Prompt Versioning and Change Management

Every earlier lesson in this unit treated prompt text as something that changes over time — instructions get refined, examples get added or removed, output requirements get tightened. In a real application, "the prompt changed" is an event with consequences: it can shift model behavior for every user of a feature, and if something goes wrong, the first question is always "what changed, and when." Without a deliberate versioning scheme, a prompt is just a mutable string that gets edited in place, with no record of what earlier behavior looked like or which version produced a given historical output. This lesson covers how to version prompts, track which version produced which output, and roll out prompt changes safely.

Why Editing a Prompt in Place Is a Problem

Treating a prompt constant as an ordinary variable that gets updated when the wording improves seems harmless:

# Before the edit
SUPPORT_INSTRUCTIONS = "You are a support assistant. Keep answers under 100 words."

# Later, edited in place
SUPPORT_INSTRUCTIONS = "You are a support assistant. Keep answers under 150 words. Always end with a follow-up question."

Git history technically preserves the diff, but that is not the same as application-level versioning. The practical problems this causes:

• No record connects a specific past output to the exact instructions that produced it. If a user complains about a response from three weeks ago, and the prompt has changed twice since, there is no easy way to know which version of the instructions was active at that time unless it was logged alongside the output.
• No way to run old and new versions side by side. Comparing "the old prompt" against "the new prompt" on the same test inputs (Lesson 9) requires having both versions available as distinct, callable objects — not one variable that has already been overwritten.
• No controlled rollout. Shipping a prompt change to 100% of traffic at once, the same way a single string edit necessarily does, is a much higher-risk deployment than gradually rolling it out the way a well-run application would roll out any other behavior change.

Pattern: Named, Immutable Prompt Versions

The fix is to treat each meaningfully different version of a prompt as a separate, named, immutable object rather than a variable that gets overwritten:

from dataclasses import dataclass

@dataclass(frozen=True)
class PromptVersion:
    name: str
    version: str
    instructions: str

SUPPORT_V1 = PromptVersion(
    name="support_assistant",
    version="v1",
    instructions="You are a support assistant. Keep answers under 100 words.",
)

SUPPORT_V2 = PromptVersion(
    name="support_assistant",
    version="v2",
    instructions=(
        "You are a support assistant. Keep answers under 150 words. "
        "Always end with a follow-up question."
    ),
)

Both SUPPORT_V1 and SUPPORT_V2 continue to exist as distinct objects even after v2 becomes the default used in production. This is the core idea behind all prompt versioning: past versions are not deleted or overwritten, they simply stop being the one that new code paths reference by default, exactly like keeping old releases of a library available even after a new one becomes the default install target.

Recording Which Version Produced an Output

Versioning is only useful if the version identifier travels with the output it produced, so that later debugging or analysis can connect a specific response back to the exact prompt that generated it:

from dataclasses import dataclass
from datetime import datetime, timezone
from openai import OpenAI

client = OpenAI()

@dataclass(frozen=True)
class PromptResult:
    output_text: str
    prompt_name: str
    prompt_version: str
    model: str
    created_at: str

def run_prompt(prompt: PromptVersion, user_input: str, model: str = "gpt-5.6-terra") -> PromptResult:
    response = client.responses.create(
        model=model,
        instructions=prompt.instructions,
        input=user_input,
    )
    return PromptResult(
        output_text=response.output_text,
        prompt_name=prompt.name,
        prompt_version=prompt.version,
        model=model,
        created_at=datetime.now(timezone.utc).isoformat(),
    )

result = run_prompt(SUPPORT_V2, "How do I reset my password?")
print(result.prompt_version)  # v2

Logging PromptResult (to a database, a structured log, or an analytics event) rather than just the bare output_text string means every historical response is traceable back to the exact prompt_name and prompt_version that produced it, along with which underlying model handled it. This is the data that makes it possible to answer, months later, "was this specific bad response produced by the prompt version we've since fixed, or is this a new problem?" — a question that is unanswerable in retrospect if only the raw text was ever kept.

Pattern: A Prompt Registry for Controlled Rollout

Beyond simply naming versions, a small registry gives application code a single place to decide which version is "current" for new requests, separate from the definitions of the versions themselves:

class PromptRegistry:
    def __init__(self):
        self._versions: dict[str, dict[str, PromptVersion]] = {}
        self._current: dict[str, str] = {}

    def register(self, prompt: PromptVersion, make_current: bool = False) -> None:
        self._versions.setdefault(prompt.name, {})[prompt.version] = prompt
        if make_current or prompt.name not in self._current:
            self._current[prompt.name] = prompt.version

    def get(self, name: str, version: str | None = None) -> PromptVersion:
        version = version or self._current[name]
        return self._versions[name][version]

    def set_current(self, name: str, version: str) -> None:
        if version not in self._versions.get(name, {}):
            raise ValueError(f"Unknown version '{version}' for prompt '{name}'")
        self._current[name] = version

registry = PromptRegistry()
registry.register(SUPPORT_V1, make_current=True)
registry.register(SUPPORT_V2)

current_prompt = registry.get("support_assistant")           # returns v1, still current
specific_prompt = registry.get("support_assistant", "v2")    # explicitly request v2

registry.set_current("support_assistant", "v2")               # roll forward

set_current is the single line that changes production behavior — everywhere in the application that calls registry.get("support_assistant") without a specific version picks up the new default the moment this line runs, with no need to hunt down and edit every call site individually. This centralization is what makes controlled rollout possible: a feature flag, a percentage-based rollout, or an instant rollback all become operations on this one registry method, rather than a redeploy of scattered code.

Pattern: Gradual Rollout Between Versions

For a change significant enough to warrant caution, route a fraction of traffic to the new version while most continues on the proven one, and compare outcomes before fully switching over:

import random

def get_prompt_for_rollout(registry: PromptRegistry, name: str, new_version: str, rollout_fraction: float) -> PromptVersion:
    if random.random() < rollout_fraction:
        return registry.get(name, new_version)
    return registry.get(name)

prompt = get_prompt_for_rollout(registry, "support_assistant", new_version="v2", rollout_fraction=0.1)
result = run_prompt(prompt, "How do I reset my password?")

Because every PromptResult records its prompt_version, outcomes from the 10% of traffic on v2 can be compared against the 90% still on the current version using ordinary analytics — response length, user follow-up rate, escalation rate to a human agent — before increasing rollout_fraction toward 1.0. This is the same gradual-rollout pattern used for any risky software change; a prompt change is a behavior change like any other and benefits from the same caution, especially because a regression in prompt quality (subtly less helpful answers, a broken output format) is often much harder to detect from logs alone than a crash or an error rate spike would be.

Rolling Back

Because old versions are never deleted from the registry, rollback is a single call, not a code revert and redeploy:

registry.set_current("support_assistant", "v1")  # instant rollback to the proven version

This is the single biggest practical payoff of not editing prompts in place: a bad prompt change can be undone as fast as it was rolled out, without needing to reconstruct the previous wording from git history or memory under incident-response time pressure.

Testing the Registry

The registry's logic — registration, lookup, current-version tracking, rollback — is ordinary Python state management and is fully testable without any model call:

def test_registry_defaults_to_first_registered_version():
    test_registry = PromptRegistry()
    test_registry.register(PromptVersion("greeter", "v1", "Say hello."))
    assert test_registry.get("greeter").version == "v1"
    print("PASS: first registered version becomes current by default")

def test_set_current_switches_active_version():
    test_registry = PromptRegistry()
    test_registry.register(PromptVersion("greeter", "v1", "Say hello."))
    test_registry.register(PromptVersion("greeter", "v2", "Say hi."))
    test_registry.set_current("greeter", "v2")
    assert test_registry.get("greeter").instructions == "Say hi."
    print("PASS: set_current switches which version is returned by default")

def test_set_current_rejects_unknown_version():
    test_registry = PromptRegistry()
    test_registry.register(PromptVersion("greeter", "v1", "Say hello."))
    try:
        test_registry.set_current("greeter", "v99")
        raise AssertionError("expected ValueError for unknown version")
    except ValueError:
        print("PASS: unknown version rejected by set_current")

test_registry_defaults_to_first_registered_version()
test_set_current_switches_active_version()
test_set_current_rejects_unknown_version()

Common Mistakes

Editing a prompt constant in place instead of creating a new named version. This destroys the ability to trace historical outputs back to the instructions that produced them, and makes side-by-side comparison of old versus new behavior impossible without manually reconstructing the previous text.

Rolling out a significant prompt change to all traffic at once. Prompt changes affect behavior the same way code changes do, but are easy to underestimate because "it's just wording." Treat a meaningful prompt change with the same rollout caution as a risky code deployment.

Logging only the model's output text, not the prompt version that produced it. Without the version recorded alongside each output, later debugging of a specific bad response cannot determine whether it came from a version that has since been fixed or represents a still-open problem.

Best Practices

Treat each meaningful prompt change as a new named, immutable version, never an in-place edit. Old versions remain available for comparison, rollback, and historical tracing even after they stop being the default.

Record the prompt name and version alongside every logged output. This is inexpensive and makes retrospective debugging and version comparison possible; omitting it cannot be fixed after the fact for outputs already produced.

Roll out significant prompt changes gradually, using a registry or feature-flag mechanism, and monitor outcome metrics before full rollout. Gate the increase in rollout fraction on the systematic evaluation approach covered in Lesson 9, not on a handful of manual spot checks.