Why Consistency Is the Hard Problem

Getting a language model to produce a good response once is not difficult — most reasonably clear prompts produce a reasonable answer most of the time. The hard problem, and the one that separates a prototype from something you can put in front of real users, is getting the model to produce a response with the same shape, the same level of detail, and the same behaviour on edge cases across hundreds or thousands of different inputs, including inputs you never anticipated while writing the prompt.

This matters because of something covered in Unit 1: models are non-deterministic by design, sampling the next token from a probability distribution rather than always picking the single most likely one. A prompt that is vague enough to admit multiple reasonable interpretations gives that non-determinism more room to produce genuinely different responses to the same or similar input. A precise prompt narrows the space of "reasonable" responses down to something much closer to one, which is what actually produces consistency — not fighting the model's inherent variability, but removing the ambiguity that lets it manifest as inconsistent output.

Ambiguity Is the Root Cause of Inconsistency

Consider a prompt for classifying customer feedback:

response = client.responses.create(
    model="gpt-5.6-luna",
    instructions="Classify the sentiment of customer feedback.",
    input="The product works, but shipping took way too long.",
)

This is a perfectly grammatical instruction, and it will produce an answer every time. But run it a hundred times across a hundred different pieces of mixed feedback like the example above, and you will see real inconsistency: sometimes "negative" (because of the shipping complaint), sometimes "mixed" or "neutral" (because the product itself is fine), sometimes a full sentence of hedged explanation instead of a single label, sometimes a label with different capitalisation or wording ("Positive" vs "positive" vs "Mixed/Neutral"). None of these responses are wrong exactly — the instruction never said what to do with mixed feedback, never specified the exact output format, and never listed the allowed category names. The model is not failing; it is correctly resolving an underspecified instruction in different ways each time, because the instruction genuinely left those decisions open.

The fix is not a different model or a "better" one-line instruction — it is removing every point of ambiguity the task actually contains:

response = client.responses.create(
    model="gpt-5.6-luna",
    instructions="""Classify the sentiment of customer feedback into exactly one of these three categories: positive, negative, mixed.

Use "mixed" when the feedback contains both a clear positive and a clear negative point, as in this example. Use "negative" only when there is no meaningful positive point. Use "positive" only when there is no meaningful negative point.

Respond with only the lowercase category word. No explanation, no punctuation, no additional text.""",
    input="The product works, but shipping took way too long.",
)

This version will reliably return mixed, every time, because every decision the model would otherwise have had to make on its own has been made for it in advance: the exact category names, the exact rule for the ambiguous "praise and complaint" case, and the exact output format. Consistency comes from closing these decision points, one by one, until the only thing left variable is the actual judgment call the task requires — in this case, which category the feedback falls into.

The Core Technique: Specify Every Decision the Model Would Otherwise Make

This is the single most useful habit for consistent prompting, and it generalises far beyond classification. Whenever you notice output varying in a way you don't want, ask: what decision did the model have to make on its own here, that I could have made for it instead? Common categories of decisions that are worth making explicit rather than leaving implicit:

Output format. Should the answer be a single word, a sentence, a paragraph, a bulleted list, JSON? "Summarise this article" leaves format entirely open; "Summarise this article in exactly three sentences" does not.

Length. "Explain how photosynthesis works" could produce two sentences or two pages depending on the model's mood. "Explain how photosynthesis works in 100-150 words" bounds it.

Edge case handling. What should happen when the input is empty, when it's ambiguous, when it doesn't fit any of your expected categories, when the requested information genuinely isn't available? If you don't specify, the model picks an approach, and it may pick a different one on different occasions.

Vocabulary and naming. If there are specific terms you want used ("customer" not "user" or "client"; "declined" not "rejected" or "denied"), state them. Otherwise the model will pick reasonable synonyms, and different runs may pick different ones.

Tone boundaries. "Be helpful" is not the same instruction as "be helpful, but never apologise more than once per response, and never use exclamation marks." The first leaves tone almost entirely up to the model's default behaviour; the second pins down two specific dimensions that otherwise vary.

Structuring Prompts for Repeatable Parsing

When a prompt's output will be parsed by code — extracted with string operations, split on a delimiter, or fed into a regular expression — consistency in structure matters even more than consistency in content, because a parser that expects a fixed shape breaks on any deviation, however reasonable that deviation might be as English prose.

instructions = """Extract the following fields from the job posting below.
Respond in exactly this format, with each field on its own line:

TITLE: <job title>
COMPANY: <company name>
LOCATION: <city, state/country, or "Remote">
SALARY: <salary range if mentioned, otherwise "Not specified">

Do not add any other text before, between, or after these four lines."""

This produces output that can be parsed with a simple line-by-line split, reliably, because the format is stated as an exact template rather than described abstractly ("extract the title, company, location, and salary"). The difference between "extract these fields" and "respond in exactly this format" is the difference between a prompt that usually works and one that works close to every time — and "usually" is not good enough for code that will run unattended against thousands of inputs.

That said, when your downstream code genuinely needs guaranteed, machine-parseable structure rather than merely well-formatted text, prompting for a format — however precise — is still fundamentally probabilistic. Unit 6 covers structured outputs with JSON schema and strict mode, which enforces the output shape mechanically rather than relying on the model choosing to follow a textual template. Use careful formatting instructions like the one above for output humans will read or for a first pass at a feature; move to schema-enforced structured outputs the moment a broken format would break your application rather than just look untidy.

Few Words, Precisely Chosen, Beat Many Words Vaguely Arranged

There is a common instinct to fight inconsistency by adding more words — more caveats, more politely-phrased requests, more repeated emphasis ("please make sure to always..."). This usually does not help, and can actively hurt, because it dilutes the signal-to-noise ratio of the instruction. Compare:

Please try to always make sure that you respond in a way that is professional
and helpful, and it would be great if you could keep things fairly brief when
possible, while still being thorough about the important points, and please
remember not to use casual language.

against:

Respond in 2-4 professional sentences. No casual language, no filler phrases.

The second is roughly a tenth of the length and produces meaningfully more consistent output, because every word in it constrains something specific ("2-4 sentences," "no casual language," "no filler"), whereas the first is mostly softening language ("please," "it would be great," "try to") that adds tokens without adding constraint. Precision, not volume, is what narrows the model's range of reasonable interpretations.

Testing for Consistency, Not Just Correctness

A single successful test run tells you a prompt can work. It does not tell you it reliably works, because a single sample says nothing about variance. The practical way to check consistency is to run the same prompt against the same input multiple times and inspect the spread:

from collections import Counter

def sample(input_text: str, n: int = 10) -> Counter:
    results = Counter()
    for _ in range(n):
        response = client.responses.create(
            model="gpt-5.6-luna",
            instructions=CLASSIFICATION_INSTRUCTIONS,
            input=input_text,
            temperature=1.0,  # deliberately high, to surface instability
        )
        results[response.output_text.strip()] += 1
    return results

print(sample("The product works, but shipping took way too long."))
# A well-specified prompt: Counter({'mixed': 10})
# A poorly-specified prompt: Counter({'negative': 6, 'mixed': 3, 'positive': 1})

Running with a deliberately high temperature here is intentional: it exaggerates whatever ambiguity remains in the prompt, making inconsistency visible with fewer samples than you'd need at a lower temperature. If ten runs at temperature=1.0 all agree, the prompt is doing real work to constrain the output rather than relying on low temperature to mask an underlying ambiguity that would resurface under different sampling settings or a different model. This is a lightweight version of the evaluation discipline Unit 13 covers in full — a handful of repeated samples per test case is enough to catch the most common consistency problems long before you need a formal eval suite.

Anchoring with a Definition Before the Task

For tasks involving any judgment call — sentiment, quality, relevance, urgency — defining the categories explicitly, before asking the model to apply them, produces far more consistent results than assuming the model shares your intuitive definition.

instructions = """You will rate customer support tickets by urgency: low, medium, or high.

high: the customer cannot use the product at all, or is threatening to cancel/churn.
medium: the customer is inconvenienced but has a working alternative or workaround.
low: a question, minor annoyance, or feature request with no urgency implied.

Read the ticket, then respond with only the urgency word."""

Without these definitions, "urgency" is left to the model's own sense of what counts as urgent, and that sense can vary — not wildly, but enough to matter at scale — from one run to the next, and certainly from one model version to another if you ever switch. With the definitions stated, you have converted a subjective judgment into something closer to a lookup against explicit criteria, which is both more consistent and, importantly, auditable: when a ticket gets classified in a way you disagree with, you can point to which line of the definition the model should have applied, rather than shrugging at an opaque disagreement over what "urgent" means.

Common Mistakes

Assuming a longer prompt is automatically a more precise one. Length and precision are different axes. A prompt can be long and still leave the actual decision points unspecified, or short and pin down everything that matters. Audit for decision points, not word count.

Testing with only "clean" examples. A prompt that works well on three tidy example inputs can fall apart on the messy, ambiguous, or unusual inputs that make up a meaningful fraction of real traffic. Deliberately include edge cases — empty input, contradictory input, input in an unexpected language — in your testing, not just the inputs that were easy to think of first.

Fixing inconsistency by raising max_output_tokens or lowering temperature alone. These can reduce the symptom — shorter, more deterministic-looking output — without addressing the cause, which is usually an underspecified instruction. Lowering temperature to 0 masks ambiguity by making the model more likely to pick the same "default" interpretation every time, but the ambiguity is still there, and it will resurface the moment you switch models, change providers, or simply hit an input the previous default interpretation doesn't handle well.

Over-constraining to the point of brittleness. The opposite failure: an instruction so rigidly specific to the examples you tested that it breaks on any input shaped slightly differently. If your instructions hardcode assumptions ("the input will always mention exactly one product"), test with inputs that violate those assumptions and see what happens — real traffic eventually will.

Not versioning prompt changes. A prompt that gets tweaked repeatedly without any record of what changed and why makes it impossible to know whether a regression in output quality came from your last edit or from something else entirely (a model update, a data drift). Treat prompt text with the same version control discipline as code, precisely because — as Lesson 1 of this unit argued — it frequently is the application logic.

Best Practices

State the format explicitly, every time output will be parsed or compared programmatically. Never rely on the model inferring a sensible format from context alone when your code depends on that format being exact.

Define subjective terms before asking the model to apply them. "Urgent," "professional," "concise," "high-quality" all mean different things to different readers; give the model your specific definition rather than assuming shared intuition.

Enumerate the exact set of valid outputs when the task is a closed classification. "Classify as positive, negative, or mixed" is more consistent than "classify the sentiment," because it removes the possibility of a fourth, unanticipated category appearing on some fraction of runs.

Test for variance, not just for one good result. Sample the same input multiple times before trusting a prompt is reliable, especially before it goes anywhere near production traffic.

Prefer removing ambiguity over adding emphasis. If a rule keeps getting missed, the fix is usually to state it more precisely and concretely, not to repeat the same vague version more forcefully or add more polite hedging around it.

Reducing Sampling Variance Directly

Everything above addresses the content of a prompt — removing ambiguity so that even under normal sampling, the range of reasonable outputs narrows. There is a separate, complementary lever: the sampling parameters themselves, which control how much randomness the generation process introduces regardless of how precise the prompt is.

temperature, covered in Unit 1, Lesson 4, is the primary lever. Setting it low (0 to 0.3) makes the model consistently favour its highest-probability tokens, which reduces run-to-run variation for any given prompt. This is the right setting for classification, extraction, and any task with a single correct or preferred answer. It is the wrong setting for brainstorming or creative writing, where you specifically want variation across runs, and where a low temperature produces flat, repetitive output.

It is important to be precise about what low temperature does and does not guarantee, because it is a common source of false confidence: temperature=0 narrows variance, it does not eliminate it. The underlying computation involves floating-point operations that can execute in different orders depending on server load and batching, which means even fully deterministic-looking settings can occasionally produce a different token at a genuine tie or near-tie in probability. For tasks where you need bit-for-bit reproducibility — rare outside of testing infrastructure — this is worth knowing so you don't build a system that assumes perfect determinism and breaks when it turns out not to hold.

seed, when supported, is a complementary parameter that, combined with an unchanged model version and unchanged parameters, aims to make repeated calls reproduce the same output more reliably than temperature alone. It is not a universal guarantee across all models and all conditions, but it is worth using in testing and debugging contexts specifically because it removes one axis of variation, letting you isolate whether an inconsistency comes from your prompt or from sampling randomness. If you observe different outputs across two calls with the same seed, the same model, and the same parameters, that is itself useful information — it tells you the inconsistency is not something your prompt engineering alone can fully resolve, and you should widen your tolerance or add a validation step downstream rather than continuing to tune prompt wording indefinitely.

Self-Consistency: Using the Model's Own Variance to Your Advantage

There is a class of task — mathematical reasoning, multi-step logic, anything where a single sampled answer can go wrong midway through a chain of steps — where the fix is not to eliminate variance but to embrace it deliberately, sample several independent answers, and take the majority result. This technique, often called self-consistency, trades extra API calls (and therefore extra cost) for higher reliability on hard problems:

from collections import Counter

def self_consistent_answer(question: str, n: int = 5) -> str:
    answers = []
    for _ in range(n):
        response = client.responses.create(
            model="gpt-5.6-terra",
            instructions="Solve the problem step by step, then give your final answer on its own line starting with 'ANSWER:'.",
            input=question,
            temperature=0.7,
        )
        text = response.output_text
        final_line = next(
            (line for line in text.splitlines() if line.startswith("ANSWER:")), ""
        )
        answers.append(final_line.replace("ANSWER:", "").strip())

    return Counter(answers).most_common(1)[0][0]

Note that this deliberately uses a higher temperature than the classification examples earlier in this lesson — the goal here is genuinely independent attempts at solving the problem, not five near-identical copies of the same reasoning path. If all five samples took an identical path and made an identical mistake, majority voting would not help at all; the technique's value comes specifically from the paths sometimes diverging and errors not being perfectly correlated across samples. This is a meaningfully more expensive approach — five calls instead of one — and it should be reserved for problems where getting the answer right matters enough to justify the multiplied cost, not applied by default to every request.

Prompt Templates: Consistency Across Many Similar Requests

Once a prompt has been tuned to produce consistent output for one input, the next challenge is keeping it consistent as it gets reused across many different inputs of the same general shape. The discipline here is separating the fixed template from the variable data cleanly, rather than hand-assembling similar-but-slightly-different strings for each call site:

from string import Template

REVIEW_SUMMARY_TEMPLATE = Template("""Summarise the following product review in exactly two sentences.
The first sentence must state the overall sentiment (positive, negative, or mixed).
The second sentence must state the single most specific point the reviewer made.

Review:
$review_text""")

def summarise_review(review_text: str) -> str:
    prompt = REVIEW_SUMMARY_TEMPLATE.substitute(review_text=review_text)
    response = client.responses.create(
        model="gpt-5.6-luna",
        input=prompt,
    )
    return response.output_text

Using string.Template (or an f-string, for simpler cases) rather than manually concatenating strings at each call site guarantees that every call goes through the identical wrapper text, with only the review content varying. This matters for consistency in a subtle way: if five different parts of a codebase each independently construct a similar-but-not-identical prompt for "summarise this review," you have effectively created five slightly different prompts, each with its own consistency profile, and a bug fix or improvement made to one has to be manually propagated to the other four. A single template, referenced everywhere the task is needed, means one improvement benefits every call site simultaneously, and the only thing that varies between calls is the data the task is genuinely about.

Consistency Across Model Versions

A prompt tuned for consistent output on one model is not guaranteed to remain equally consistent when the underlying model changes — whether because you deliberately switched models to save cost, or because a model alias you're using was quietly repointed to a newer snapshot. This is a real operational risk: Unit 1 already flagged that model names are the fastest-moving part of this whole stack, and a prompt's consistency profile is exactly the kind of thing that can shift silently when the model underneath it does.

The practical defence is the same lightweight sampling check from earlier in this lesson, run again whenever you change models or notice a model alias has moved to a new snapshot: take your existing test cases, run each one several times against the new model, and compare the distribution of outputs against what you observed before. A prompt that was reliably consistent on one model and becomes noticeably less so on another is telling you that some of the ambiguity you thought you'd removed was actually being resolved by that specific model's particular tendencies, rather than by the wording of your prompt alone — which is a sign the instruction needs to be made more explicit rather than relying on a specific model's default behaviour to fill the gap.

A Worked End-to-End Example

Pulling the techniques in this lesson together, here is the evolution of a single prompt from unreliable to consistent, for the task of extracting a delivery date from a shipping notification email.

Attempt 1 — vague:

instructions = "Find the delivery date in this email."

Problem: no format specified, so output ranges from "March 15th" to "2026-03-15" to full sentences like "The package will arrive around March 15."

Attempt 2 — format specified, but edge cases left open:

instructions = "Find the delivery date in this email and respond in YYYY-MM-DD format."

Better, but still inconsistent on emails with no delivery date at all, or a date range ("between March 14-16") — some runs guess a date, others say "not found" in varying phrasing, others return the first day of the range and some the last.

Attempt 3 — edge cases closed:

instructions = """Find the delivery date mentioned in this email.

Respond with only the date in YYYY-MM-DD format.
If a date range is given, respond with the LAST day of the range.
If no delivery date is mentioned anywhere in the email, respond with exactly: NONE"""

This final version has had every decision point identified and closed: the exact output format, the rule for the ambiguous range case, and the exact token to use when no date exists at all (a fixed, unmistakable sentinel value rather than a free-text explanation, which also makes downstream parsing trivial — checking for the literal string "NONE" is far more reliable than trying to detect the many ways a model might phrase "I couldn't find one"). Each attempt removed one specific source of variance; there was no single rewrite that fixed everything at once, and that incremental, test-driven tightening — write, sample, observe the inconsistency, close the specific gap that caused it, repeat — is the actual practice of writing prompts that get consistent results, far more than any one-time trick or template.

State What to Do, Not Only What to Avoid

A pattern worth calling out on its own: instructions phrased purely as prohibitions ("don't be verbose," "don't use jargon," "don't refuse unnecessarily") are less reliable than instructions that pair the prohibition with the positive behaviour you actually want instead. "Don't be verbose" leaves the model to infer what concise looks like in this context, and different runs can infer it differently. "Respond in 2-3 sentences" states the target directly, with nothing left to infer.

This matters more than it might seem, because a purely negative instruction defines a boundary without defining a destination — it tells the model what region of output-space to avoid without telling it which point within the remaining, still-large space to aim for, and that remaining ambiguity is exactly what produces run-to-run inconsistency. Whenever you catch yourself writing "don't X," it is worth pausing to add "instead, do Y" — not as a stylistic nicety, but because Y is the piece of information that actually removes the ambiguity, while "don't X" only narrows it.

Consider the contrast directly:

Don't give a long answer.

versus

Give a 2-3 sentence answer that states the conclusion first, then one supporting reason.

Both instructions rule out the same failure mode — a long, rambling answer — but only the second tells the model what a correct answer looks like. Run each against ten different questions and the difference in output consistency is immediately visible: the first produces answers ranging from one clipped sentence to a full paragraph the model apparently considered "not that long," while the second reliably produces answers matching its explicit shape. The lesson generalises well beyond length: any prohibition ("don't be pushy," "don't sound robotic," "don't over-explain") can be paired with a concrete positive description of the target behaviour, and doing so is one of the highest-leverage, lowest-effort edits available when a prompt's output is bouncing around more than you'd like.