Prompt Iteration and Versioning
Why Prompts Need Versions In an application, a prompt behaves like code: changing a single sentence can change the output for every user. Yet prompts are often edited directly in source files, in dashboards, or in chat windows, with no record of what changed, when, why, or what effect it had.
Prompt versioning means giving every meaningful state of a prompt an identifier, storing it with its metadata, and recording which version produced which results.
Problems Versioning Solves Without Versioning With Versioning "The classifier got worse last week. What changed?" Compare version 2.1.0 with 2.0.0 and their evaluation results A change causes errors, and the previous text is lost Roll back to the last good version in minutes Nobody knows which prompt produced a stored output Every logged request records the prompt version Two team members edit the same prompt differently Changes go through review, like code Evaluation results cannot be linked to prompt text Each version has its evaluation results attached What Belongs in a Prompt Version A version is more than the prompt text. Everything that affects the output should be identifiable.
Component Why It Matters Prompt text The instructions, rules, and examples Prompt identifier Which prompt this is, such as classify_ticket Version number Which state of the prompt Model The same prompt can behave differently on different models Settings Temperature, output limits, or other parameters, if used Output contract The expected output format, which other code depends on Changelog entry What changed and why Evaluation results How this version performed on the evaluation set (Lesson 66) Pin the Model Version Many providers offer both general model names that may be updated over time and dated snapshot names that stay fixed. If your application uses a general name, the model behind it can change, and your prompt's behavior may change even though you did not edit it. Where your provider offers snapshots, consider pinning a specific one in production, and re-evaluate before moving to a newer model (Lesson 43).
Version Numbers for Prompts Semantic versioning is a widely used numbering scheme with three parts: MAJOR.MINOR.PATCH. It can be adapted for prompts:
Part Increase When... Example MAJOR The output contract changes in a way that could break code using the output Labels renamed, JSON fields changed, output format changed MINOR Behavior changes intentionally, but the output contract stays compatible New rule for mixed tickets, new examples, added context PATCH Wording is clarified without any intended change in behavior Typo fixed, sentence rephrased for clarity Why This Helps The version number alone tells other developers how careful they must be:
A major change requires updating the code that parses the output. A minor change requires re-running the evaluation. A patch should be safe, but it should still be evaluated, because "no intended change" does not guarantee no actual change (Lesson 24, wording sensitivity). Storing Prompts as Versioned Files Keep prompts in files, separate from code, under version control with Git (Lesson 10, Tip 11). One simple format is a small header of metadata followed by the prompt text.
Example Prompt Files prompts/classify_ticket.v1.md
id: classify_ticket version: 1.0.0 model: gpt-5.4 changelog: First version.
Classify the support ticket in as Billing, Access, Technical, or Other. Reply with the category only. prompts/classify_ticket.v2.md
id: classify_ticket version: 2.1.0 model: gpt-5.4 changelog: Added rule for tickets that mention several issues.
Classify the support ticket in as Billing, Access, Technical, or Other. If a ticket mentions several issues, choose the category of what the customer asks us to do. Reply with the category only. Why Separate Files per Version? Keeping old versions available in the repository makes it possible to:
Run an old version and a new version side by side for comparison (Lesson 69). Roll back by changing which version the application uses, without editing text. Pin specific features to specific versions while others move to newer ones. Git already keeps the history of each file. Separate version files are an additional convenience for running several versions at the same time; for a single active version, one file plus Git history can be enough.
A Prompt Registry in Python A registry loads all prompt files and returns the requested version. The following implementation uses only the standard library.
import hashlib from dataclasses import dataclass from pathlib import Path
@dataclass(frozen=True) class PromptVersion: id: str version: str model: str changelog: str text: str
@property def content_hash(self) -> str: """Short fingerprint of the prompt text, to detect untracked edits.""" return hashlib.sha256(self.text.encode("utf-8")).hexdigest()[:12]
def parse_prompt_file(path: Path) -> PromptVersion: header, _, body = path.read_text(encoding="utf-8").partition("\n---\n") meta = dict(line.split(": ", 1) for line in header.splitlines() if ": " in line) return PromptVersion(meta["id"], meta["version"], meta["model"], meta.get("changelog", ""), body.strip())
def version_key(version: str) -> tuple[int, ...]: """Turn '2.10.0' into (2, 10, 0) so versions sort numerically.""" return tuple(int(part) for part in version.split("."))
def load_registry(directory: Path) -> dict[str, dict[str, PromptVersion]]: registry: dict[str, dict[str, PromptVersion]] = {} for path in sorted(directory.glob("*.md")): prompt = parse_prompt_file(path) registry.setdefault(prompt.id, {})[prompt.version] = prompt return registry
def get_prompt(registry, prompt_id: str, version: str | None = None) -> PromptVersion: versions = registry[prompt_id] chosen = version or max(versions, key=version_key) return versions[chosen]
registry = load_registry(Path("prompts")) latest = get_prompt(registry, "classify_ticket") pinned = get_prompt(registry, "classify_ticket", "1.0.0") print(latest.version, latest.content_hash, latest.changelog) print(pinned.version, pinned.content_hash) print(version_key("1.10.0") > version_key("1.9.0"), "1.10.0" > "1.9.0") Output (with the two example files above):
2.1.0 4433eb77d532 Added rule for tickets that mention several issues. 1.0.0 2ac80b3e618e True False Code Explanation PromptVersion A frozen (immutable) dataclass holding the metadata and text. Immutability prevents code from accidentally modifying a loaded prompt.
content_hash A SHA-256 hash of the prompt text, shortened to 12 characters. It acts as a fingerprint: if someone edits the text but forgets to change the version number, the hash changes, revealing the untracked edit.
parse_prompt_file()
partition("\n---\n") splits the file at the first separator line into header and body. Each header line of the form key: value becomes a dictionary entry. split(": ", 1) splits only at the first ": ", so values may contain colons. version_key() Converts a version string into a tuple of integers. The last line of the output shows why this matters: compared as strings, "1.10.0" > "1.9.0" is False, because "1" comes before "9" character by character. Compared as tuples, (1, 10, 0) > (1, 9, 0) is True, which is correct.
get_prompt() Returns a specific version if one is requested, or the latest version otherwise. In production, it is usually safer to request a specific version explicitly, so that adding a new file does not silently change application behavior.
Recording the Version With Every Request To know which prompt produced which output, log the prompt version alongside each request.
import json import time from datetime import datetime, timezone
def log_request(prompt: PromptVersion, input_text: str, output: str, seconds: float, log_path: str = "prompt_log.jsonl") -> None: record = { "timestamp": datetime.now(timezone.utc).isoformat(), "prompt_id": prompt.id, "prompt_version": prompt.version, "prompt_hash": prompt.content_hash, "model": prompt.model, "input_chars": len(input_text), "output": output, "seconds": round(seconds, 3), } with open(log_path, "a", encoding="utf-8") as file: file.write(json.dumps(record, ensure_ascii=False) + "\n") Code Explanation Each request is written as one JSON object per line, a format called JSON Lines (.jsonl), which is easy to append to and easy to analyze later. The record includes the prompt identifier, version, hash, and model, so any output can be traced back to the exact prompt that produced it. input_chars records the size of the input rather than the full input, which avoids storing sensitive user data in logs. Decide what to log according to your privacy requirements. datetime.now(timezone.utc) records time in UTC, which avoids confusion when servers and users are in different time zones. What This Enables Comparing error rates between prompt versions in production. Investigating a user complaint by finding the exact prompt version used. Confirming that a rollback actually took effect. The Iteration Workflow Combine versioning with the optimization process from Lesson 66:
- Create a branch and copy the current prompt to a new version file
- Make one change and update the version number and changelog
- Run the evaluation set; record results with the version
- Open a pull request containing the prompt diff and the evaluation results
- Review: does the change match the changelog? Are results better? Any regressions?
- Merge and deploy the new version (optionally to a small share of traffic first, Lesson 69)
- Monitor production logs by version
- If problems appear, roll back to the previous version Reviewing Prompt Changes A prompt pull request should show:
The text diff, so reviewers can see exactly what changed (Lesson 25, difflib). The evaluation comparison: old version versus new version on the same dataset. Examples of changed outputs, especially any that got worse (Lesson 66, paired comparison). Reviewing evaluation results alongside the text change turns prompt editing from a matter of opinion into a matter of evidence.
Rolling Back Because each version is stored and the application requests a specific version, rolling back means changing one configuration value, such as CLASSIFY_TICKET_VERSION = "2.0.0", and redeploying. No one needs to remember or reconstruct the old text.
Regression Testing Across Versions A regression is a case that worked in an earlier version and fails in a newer one. Keep a set of important test cases, especially ones that were fixed by earlier changes, and run them for every new version. If a previously fixed case breaks, the new version must not be deployed until the cause is understood.
This is covered in depth in Lesson 118 (Regression Testing for Prompt Changes).
Common Mistakes Mistake 1: Editing Prompts in Place Without Records What it is: Changing prompt text directly in code or a dashboard.
Why it happens: It is the fastest way to make a change.
What it causes: No history, no rollback, and no way to link outputs to prompt text.
How to fix it: Store prompts in versioned files under Git and change them through reviewed commits.
Mistake 2: Versioning the Text but Not the Model What it is: Recording prompt versions while the model name changes independently.
Why it happens: The model is configured elsewhere.
What it causes: Behavior changes that cannot be explained by prompt history.
How to fix it: Record the model, and settings, as part of each version, and log them with every request.
Mistake 3: Always Using "Latest" What it is: The application automatically uses the newest version of each prompt.
Why it happens: It seems convenient.
What it causes: Adding a new prompt file changes production behavior without a deliberate deployment.
How to fix it: Pin specific versions in configuration and change them deliberately.
Mistake 4: Sorting Versions as Strings What it is: Choosing the latest version with string comparison.
Why it happens: Strings are easy to compare.
What it causes: Version 1.9.0 appears newer than 1.10.0.
How to fix it: Compare versions as tuples of integers.
Mistake 5: Changes Without Evaluation Results What it is: Merging prompt changes because they "look better".
Why it happens: Evaluation takes time.
What it causes: Regressions reach production unnoticed.
How to fix it: Require evaluation results in every prompt pull request.
Best Practices Treat prompts as code. Store them in files, under version control, with review.
Version everything that affects output. Text, model, settings, and output contract.
Use semantic version numbers. Signal how risky each change is.
Record a changelog entry for every version. Explain what changed and why.
Pin versions in production. Change them deliberately, never implicitly.
Log the prompt version with every request. Every output should be traceable.
Attach evaluation results to versions. Decisions should be based on measurements.
Keep rollback simple. One configuration change should restore the previous version.
Try It Yourself Exercise 1: Set Up a Registry Create a prompts folder with three versions of a prompt you use, each with a header containing id, version, model, and changelog. Load them with load_registry() and print each version's content hash. Then edit one prompt's text without changing its version number, and observe that the hash changes.
Exercise 2: Classify Changes Decide whether each change is a MAJOR, MINOR, or PATCH version change, and explain why:
Changing the output from a single label to JSON with label and reason. Fixing the spelling of "recieve" in the instructions. Adding two few-shot examples for borderline tickets. Renaming the label "Other" to "General". Adding a rule that empty tickets should be classified as Other. Exercise 3: Trace Outputs to Versions Use log_request() while running two versions of a prompt on the same 20 inputs. Then write a short script that reads prompt_log.jsonl and reports, for each version, the number of requests and the average response time.