Testing Prompt Variants
What Is a Prompt Variant Test? A prompt variant is an alternative version of a prompt, such as a version with a new rule, different examples, a shorter instruction, or a different model. Testing prompt variants means comparing two or more versions under controlled conditions to decide which one to use.
The idea is the same as A/B testing in web development: instead of guessing which button color works better, you show each version to comparable users and measure the results.
Variant tests happen in two settings:
Setting What Is Compared Strength Limitation Offline Variants run on the same fixed evaluation dataset Controlled, repeatable, safe The dataset may not reflect every real input Online Variants serve real traffic, each to a share of users Measures real-world behavior and user outcomes Slower, riskier, needs careful monitoring A reliable process uses both: offline testing to choose a promising candidate, then online testing to confirm it with real users.
Lesson 66 explained how to measure differences with uncertainty, and Lesson 67 explained how to version prompts. This lesson focuses on how to run the comparison itself.
Offline Variant Testing Step 1: Define What "Better" Means Before comparing, decide on the primary metric and constraints (Lesson 66). For example:
Primary metric: accuracy on the labeled evaluation set Constraints: format reliability at least 99%; average input tokens at most 700 Step 2: Run Every Variant on the Same Inputs Variants must be compared on identical inputs, with the same model settings, so that differences come from the prompts alone.
Step 3: Compare Paired Results For each input, record whether each variant succeeded. The paired comparison from Lesson 66 shows how many inputs improved, how many got worse, and how many stayed the same. Always inspect the inputs that got worse: they reveal side effects of the change.
When There Is No Exact Answer For classification and extraction, outputs can be compared with expected answers. For summaries, explanations, or replies, there is often no single correct output. In these cases, two approaches are common:
Rubric scoring: score each output against defined criteria (Lesson 116). Pairwise comparison: show a judge two outputs for the same input and ask which is better. Pairwise Comparison With a Model Judge In pairwise comparison, a judge sees the input and two outputs, A and B, and decides which is better or whether they are equally good. The judge can be a person or a model with clear criteria. Using a model as a judge is often called LLM-as-a-judge.
The Position Bias Problem Model judges can be influenced by the order in which outputs are shown, for example by tending to prefer whichever output comes first. This is called position bias. If variant A is always shown first, a position-biased judge will favor A regardless of quality.
The Solution: Judge Twice With Swapped Order Ask the judge with A first and B second. Ask again with B first and A second. Accept a winner only if both judgments agree. Otherwise, count the comparison as a tie. def pairwise_verdict(judge, task: str, output_a: str, output_b: str) -> str: """Return 'A', 'B', or 'tie', judging twice with the order swapped.""" first = judge(task, output_a, output_b) # A shown first second = judge(task, output_b, output_a) # B shown first first_mapped = {"first": "A", "second": "B", "tie": "tie"}[first] second_mapped = {"first": "B", "second": "A", "tie": "tie"}[second] return first_mapped if first_mapped == second_mapped else "tie" Testing the Logic With Simulated Judges def position_biased_judge(task, x, y): return "first" # always prefers whatever is shown first
def shorter_is_better_judge(task, x, y): if len(x) == len(y): return "tie" return "first" if len(x) < len(y) else "second"
print(pairwise_verdict(position_biased_judge, "t", "short", "much longer answer")) print(pairwise_verdict(shorter_is_better_judge, "t", "short", "much longer answer")) print(pairwise_verdict(shorter_is_better_judge, "t", "much longer answer", "short")) Output:
tie A B Code Explanation pairwise_verdict()
The judge returns "first", "second", or "tie", referring to the positions in which outputs were shown. The first call shows A first, so "first" means A. The second call shows B first, so "first" means B. The two dictionaries translate positions back into variant names. A winner is returned only when both translated verdicts agree. The simulated judges
The position-biased judge always answers "first". In the first call it picks A; in the swapped call it picks B. The verdicts disagree, so the result is a tie: the bias is neutralized instead of producing a false winner. The consistent judge (here, simply preferring shorter text) gives the same verdict in both orders, so a real winner is reported: A when A is shorter, B when B is shorter. A Model Judge Requirements: pip install openai and an OPENAI_API_KEY environment variable.
from openai import OpenAI
client = OpenAI() JUDGE_MODEL = "gpt-5.4" # Replace with a model available in your account
JUDGE_INSTRUCTIONS = """You compare two responses to the same task. Criteria, in order of importance:
- Uses only facts from the task's source text
- Answers what was asked
- Clear and concise Reply with exactly one word: first, second, or tie."""
def model_judge(task: str, x: str, y: str) -> str: response = client.responses.create( model=JUDGE_MODEL, instructions=JUDGE_INSTRUCTIONS, input=f"\n{task}\n\n\n\n{x}\n\n\n\n{y}\n", ) verdict = response.output_text.strip().lower() return verdict if verdict in {"first", "second", "tie"} else "tie" Explanation The criteria are explicit and ordered, so the judge evaluates what you care about rather than its own preferences. The outputs are labeled first and second, never A and B, so the judge cannot learn which variant is which. Unclear replies are treated as ties, so they never create a false winner. Model Judges Need Checking Too A model judge is itself a prompt with possible biases, such as preferring longer or more confident answers. Before relying on it, compare its verdicts with human judgments on a sample of pairs. If they often disagree, refine the criteria or use human review. Evaluation with rubrics and judges is covered further in Lessons 116 and 117.
Online Variant Testing Offline tests use a fixed dataset. Online tests measure what happens with real users and real inputs, including outcomes such as whether customers needed to contact support again.
Assigning Users to Variants Each user must be assigned to a variant consistently: the same user should see the same variant every time, otherwise their experience is inconsistent and the results are mixed. A common method is to hash the user ID.
import hashlib from collections import Counter
def assign_variant(user_id: str, experiment: str, share_b: float = 0.5) -> str: """Deterministically assign a user to variant A or B.""" digest = hashlib.sha256(f"{experiment}:{user_id}".encode("utf-8")).hexdigest() bucket = int(digest[:8], 16) / 0xFFFFFFFF return "B" if bucket < share_b else "A"
counts = Counter(assign_variant(f"user{i}", "summary-v2-test", share_b=0.1) for i in range(10000)) print(counts) print(assign_variant("user42", "summary-v2-test", 0.1) == assign_variant("user42", "summary-v2-test", 0.1)) Output:
Counter({'A': 8986, 'B': 1014}) True Code Explanation Hashing sha256 turns the experiment name and user ID into a long hexadecimal string. The same input always produces the same hash, so the assignment is deterministic.
Bucket The first 8 hexadecimal characters are converted to an integer and divided by the largest possible 8-character value (0xFFFFFFFF), giving a number between 0 and 1 that is effectively random across users.
Share Users whose bucket is below share_b receive variant B. With share_b=0.1, about 10% of users get B: here, 1,014 of 10,000.
Including the experiment name Adding the experiment name to the hash means each experiment splits users differently. Without it, the same users would always be in the B group of every experiment, which would bias results.
Logging for Online Tests Every request should record the variant and prompt version (Lesson 67), so outcomes can be compared by variant:
Logged Field Purpose Experiment name and variant Group results Prompt id, version, and model Know exactly what was served Validation result Format reliability per variant Latency and tokens Cost and speed per variant User outcome signals Thumbs up/down, follow-up questions, escalations, task completion Guardrail Metrics A guardrail metric is a metric that must not get worse, even if the primary metric improves. Examples: error rate, complaint rate, average latency, cost per request. If a guardrail is breached, stop the experiment and return all users to variant A.
Gradual Rollout A safe way to introduce a new prompt is to increase its traffic share step by step:
Offline evaluation passes ↓ 5% of users get the new version (canary) → monitor guardrails ↓ 25% → monitor ↓ 50% → compare outcomes with enough data ↓ 100% → new version becomes the default A canary release exposes a small share of users first, so that problems affect few people and can be detected early. Because assignment is based on the hash bucket, increasing share_b from 0.05 to 0.25 keeps all original B users in group B and adds new ones, so users do not switch back and forth.
How Long Should a Test Run? Factor Effect Size of the expected difference Small differences need much more data to detect (Lesson 66) Traffic volume Low-traffic features take longer to collect enough data Variation over time Include different days of the week if user behavior varies Outcome delay Some outcomes, such as repeat support contacts, appear days later Decide the minimum sample size or duration before starting. Stopping a test as soon as one variant looks better, often called "peeking", increases the chance of choosing a variant that only looked better by chance.
Common Mistakes Mistake 1: Comparing on Different Inputs What it is: Testing variant A on last week's data and variant B on this week's.
Why it happens: Variants are evaluated at different times.
What it causes: Differences in inputs are mistaken for differences in prompts.
How to fix it: Offline, use the same dataset; online, run variants at the same time with random assignment.
Mistake 2: Ignoring Position Bias in Model Judges What it is: Always showing the new variant first or second.
Why it happens: It is simpler to call the judge once.
What it causes: The judge's order preference decides the winner.
How to fix it: Judge every pair twice with swapped order and accept only consistent verdicts.
Mistake 3: Inconsistent User Assignment What it is: Choosing a variant randomly on every request.
Why it happens: random.choice() is the obvious implementation.
What it causes: Users see inconsistent behavior, and outcomes cannot be attributed to one variant.
How to fix it: Assign deterministically by hashing the experiment name and user ID.
Mistake 4: Stopping Early What it is: Ending the test as soon as one variant appears better.
Why it happens: Early results look convincing.
What it causes: Decisions based on random fluctuations.
How to fix it: Set the sample size or duration in advance and wait for it.
Mistake 5: No Guardrails What it is: Watching only the primary metric during an online test.
Why it happens: The primary metric is the goal of the change.
What it causes: Increased errors, costs, or complaints go unnoticed.
How to fix it: Define guardrail metrics and stop the test if any is breached.
Best Practices Test offline first, then online. Use datasets to choose candidates and real traffic to confirm them.
Compare variants on identical inputs. Isolate the effect of the prompt.
Inspect regressions, not just totals. Inputs that got worse reveal side effects.
Use swapped-order pairwise judging. Neutralize position bias, and validate the judge against human ratings.
Assign users deterministically. Hash the experiment name with the user ID.
Log variant and version with every request. Results must be attributable.
Roll out gradually with guardrails. Start small, monitor, and keep rollback simple.
Decide the test size in advance. Avoid stopping on early noise.
Try It Yourself Exercise 1: Offline Pairwise Test Create two variants of a summarization prompt, for example with and without a "keep all numbers exactly" rule. Generate summaries for 15 documents with each variant and compare them with pairwise_verdict() and model_judge(). Count wins, losses, and ties, and read every pair where the judge disagreed with itself.
Exercise 2: Check the Judge Take 10 pairs from Exercise 1 and judge them yourself without looking at the model's verdicts. Compare your decisions with the model judge's. If you disagree on more than 2 of 10, revise the judge's criteria and repeat.
Exercise 3: Plan an Online Test Write a plan for testing a new prompt version in production: the experiment name, initial share for variant B, the primary metric, three guardrail metrics, the minimum duration, and the rollout steps. Then use assign_variant() to confirm that increasing the share from 5% to 25% keeps every original B user in group B.