Common Causes of Inconsistent Prompt Outputs
What Is an Inconsistent Output? A prompt produces inconsistent outputs when similar inputs, or even the same input run several times, give answers that differ in ways that matter: a different label, a different format, a different decision, or a different fact.
Some variation is normal and even desirable, for example in creative writing (Lesson 87). But in classification, extraction, decisions, and automated workflows, inconsistency is a serious problem:
Users receive different answers to the same question. Automated code breaks on unexpected formats. Evaluation results change from run to run, making it hard to know whether a prompt change helped (Lesson 118). This lesson explains how to measure inconsistency, the common causes, and the fix for each.
Measuring Consistency Before fixing inconsistency, measure it. Run each test case several times and check how often the outputs agree.
import random from collections import Counter
def consistency_report(cases: list[dict], run, repeats: int = 5, threshold: float = 0.8) -> list[dict]: """Run each case several times and report how often the most common output appears.""" report = [] for case in cases: outputs = [run(case) for _ in range(repeats)] label, count = Counter(outputs).most_common(1)[0] agreement = count / repeats report.append({"id": case["id"], "majority": label, "agreement": agreement, "unstable": agreement < threshold, "outputs": dict(Counter(outputs))}) return report
rng = random.Random(7)
def simulated_classifier(case: dict) -> str: """Stands in for a model call, with two kinds of inconsistency built in.""" if case["id"] == "mixed": return rng.choice(["Billing", "Access"]) # genuinely ambiguous input if case["id"] == "formatting": return rng.choice(["Billing", "billing", "Billing."]) # same answer, different format return {"refund": "Billing", "login": "Access"}[case["id"]]
cases = [{"id": "refund"}, {"id": "login"}, {"id": "mixed"}, {"id": "formatting"}] for row in consistency_report(cases, simulated_classifier): print(row) Output:
{'id': 'refund', 'majority': 'Billing', 'agreement': 1.0, 'unstable': False, 'outputs': {'Billing': 5}} {'id': 'login', 'majority': 'Access', 'agreement': 1.0, 'unstable': False, 'outputs': {'Access': 5}} {'id': 'mixed', 'majority': 'Billing', 'agreement': 0.6, 'unstable': True, 'outputs': {'Access': 2, 'Billing': 3}} {'id': 'formatting', 'majority': 'Billing.', 'agreement': 0.4, 'unstable': True, 'outputs': {'Billing.': 2, 'Billing': 2, 'billing': 1}} Code Explanation consistency_report()
Runs each case repeats times with the provided run function, which would normally call the model. Finds the most common output and its share, the agreement (Lesson 65 uses the same idea to find uncertain questions). Marks a case unstable if agreement is below the threshold, and shows all outputs so you can see how they differ. simulated_classifier() Simulates a model with two different problems, using a seeded random generator so the example is reproducible:
mixed alternates between two different labels, like a genuinely ambiguous message. formatting always means Billing, but writes it three different ways. Two Kinds of Inconsistency The report reveals that the two unstable cases have different causes:
Case Outputs Kind of Inconsistency Fix mixed Billing, Access Decision inconsistency A rule for the boundary case formatting Billing, billing, Billing. Format inconsistency Format instructions and normalization Normalizing Format Differences Format differences can often be removed in code:
def normalize_label(text: str) -> str: return text.strip().strip(".").capitalize()
rng = random.Random(7) print([row["agreement"] for row in consistency_report(cases, lambda c: normalize_label(simulated_classifier(c)))]) Output:
[1.0, 1.0, 0.6, 1.0] After normalization, the formatting case is fully consistent: all its outputs were the same answer. The mixed case is still unstable at 0.6, because its outputs are different decisions, which normalization cannot fix. That requires a change to the prompt.
The Common Causes and Their Fixes Cause 1: Sampling Randomness What happens: Models generate text by choosing among likely tokens. With sampling settings that allow variety, the same prompt can produce different outputs each time (Lesson 2).
Signs: Variation even for clear, unambiguous inputs.
Fixes:
For tasks that need consistency, use settings that reduce randomness, such as a low temperature, where your model and API support them. Remember that low-randomness settings reduce variation but do not always remove it completely, and some models do not expose these settings. The other fixes in this lesson remain necessary. Cause 2: Ambiguous Instructions What happens: The instructions allow more than one reasonable interpretation, and the model chooses differently on different runs (Lesson 5).
Signs: Outputs are all reasonable, but different: "summarize briefly" produces one sentence on one run and a paragraph on another.
Fix: Replace vague words with measurable requirements: "Summarize in exactly 2 sentences" (Lesson 14).
Cause 3: Missing Rules for Boundary Cases What happens: Inputs that fall between categories, like the mixed case, have no rule, so the model's decision varies (Lesson 18).
Signs: Inconsistency concentrated on specific kinds of input, while typical inputs are stable.
Fix: Add explicit rules for boundary cases, such as "If the customer paid but cannot access the course, choose Access", and test them with boundary cases (Lesson 30).
Cause 4: Underspecified Output Format What happens: The format is not fully defined, so capitalization, punctuation, extra text, or field names vary.
Signs: The same answer in different forms, like the formatting case.
Fixes:
Define the format exactly, with allowed values (Lesson 8). Use structured outputs with a schema where available (Lesson 99). Normalize and validate in code. Cause 5: Inconsistent Examples What happens: Few-shot examples use different formats, styles, or rules, so the model learns several patterns and applies them unpredictably (Lesson 29).
Signs: Outputs alternate between styles that each appear in one of the examples.
Fix: Generate examples from structured data with one formatting function, and check that examples follow the same rules as the instructions.
Cause 6: Variable Context What happens: The input to the prompt changes between runs even when the user's question is the same, for example because retrieval returns different chunks or chunks in a different order (Lesson 103).
Signs: Inconsistency in RAG answers that disappears when the context is fixed.
Fixes:
Make retrieval deterministic: stable sorting, consistent tie-breaking, fixed result counts. Log the exact context for each run, so differences can be traced (Lesson 74). Cause 7: Long or Overloaded Prompts What happens: Many competing instructions, or very long context, make it harder for the model to apply every rule every time (Lesson 15).
Signs: Different rules are ignored on different runs.
Fixes: Remove unnecessary instructions (ablation, Lesson 66), split the task into a chain (Lesson 61), and keep context focused.
Cause 8: Model or Version Changes What happens: The model behind your application changes, and outputs change with it, even though the prompt is the same (Lesson 67).
Signs: Behavior changes suddenly across many cases on a specific date.
Fixes: Pin model versions where possible, log the model with every request, and run regression tests before switching (Lesson 118).
Cause 9: Input Variation That Should Not Matter What happens: Small, meaningless differences in input, such as extra spaces, capitalization, or word order, change the output.
Signs: Paraphrased versions of the same test case get different results.
Fixes: Normalize input text where appropriate (Lesson 7), add examples with varied wording, and include paraphrased variants in your test set (Lesson 115).
A Diagnostic Process
- Measure: run each test case 5 or more times; find unstable cases
- Inspect: read the different outputs for each unstable case
- Classify: format difference, decision difference, or factual difference?
- Check the context: was the input to the model identical on every run?
- Apply the matching fix from the list above
- Measure again with the same cases and number of runs What the Outputs Show Most Likely Causes Same meaning, different form 4 (format), 5 (examples) Different decisions on specific inputs 3 (boundary rules), 2 (ambiguity) Different facts or details 6 (variable context), weak grounding (Lesson 104) Variation even on clear inputs 1 (sampling), 7 (overloaded prompt) Sudden change across many cases 8 (model change) Different results for paraphrases 9 (input variation) Consistency Is Not the Same as Correctness A prompt can be consistently wrong: it gives the same incorrect answer every time. And a prompt can be inconsistent but correct on average. Measure both:
Correct Incorrect Consistent The goal Systematic error: fix the rule or the context (Lesson 46) Inconsistent Fragile: sometimes wrong for users Unreliable and wrong Self-consistency (Lesson 45) uses majority voting over several runs to improve answers when variation is random. That can be useful, but it costs more and does not fix systematic errors. Prefer fixing the causes above where possible.
Common Mistakes Mistake 1: Testing Each Case Once What it is: Running test cases a single time.
Why it happens: It is cheaper and faster.
What it causes: Unstable cases look stable, or random variation looks like a regression.
How to fix it: Run important cases several times and measure agreement.
Mistake 2: Blaming Randomness for Everything What it is: Assuming all inconsistency comes from sampling settings.
Why it happens: Randomness is the most familiar explanation.
What it causes: Real causes, such as missing boundary rules or variable context, remain.
How to fix it: Inspect and classify the differing outputs before choosing a fix.
Mistake 3: Fixing Format Problems With Prompt Changes Only What it is: Adding more format instructions to fix capitalization and punctuation.
Why it happens: The prompt seems to be the place to fix everything.
What it causes: Small format variations continue to break code.
How to fix it: Use structured outputs and normalize in code.
Mistake 4: Ignoring Context Variation What it is: Comparing outputs without checking whether the retrieved context was identical.
Why it happens: Retrieval is treated as fixed.
What it causes: Prompt changes are made to fix a retrieval problem.
How to fix it: Log context for every run and make retrieval deterministic.
Best Practices Measure consistency with repeated runs. Find unstable cases and look at their outputs.
Classify inconsistency before fixing it. Format, decision, or factual differences have different causes.
Define formats exactly and normalize in code. Remove format variation mechanically.
Write rules for boundary cases. Most decision inconsistency is concentrated there.
Keep context stable and logged. Especially in RAG systems.
Control model versions and settings. Reduce randomness where the task needs consistency.
Measure consistency and correctness together. Consistent wrong answers are still wrong.
Try It Yourself Exercise 1: Measure Your Prompt Replace simulated_classifier() with a real call to one of your prompts. Run 15 test cases 5 times each with consistency_report(). List the unstable cases and classify each as a format, decision, or factual difference.
Exercise 2: Fix One Cause Choose the most common cause from Exercise 1. Apply the matching fix, for example a boundary rule or normalization. Re-run the same cases the same number of times and compare agreement before and after.
Exercise 3: Paraphrase Test For five test cases, write two paraphrased versions each, with the same meaning but different wording. Run all versions and check whether paraphrases receive the same output. For any that do not, decide which fix from this lesson applies.