The Trap of Manual Testing

Every developer who has built something with the OpenAI SDK has done this: write a prompt, run it three or four times in a script or in the Playground, read the output, nod, and ship it. The response looked reasonable. The tone was right. The JSON parsed. So the feature goes into the pull request, and the pull request gets merged.

This is called manual testing, or less kindly, vibe testing — you are judging the system by vibes, not by measurement. It is not a bad instinct. It is how most people validate any piece of software when they are moving fast. The problem is specific to language models: manual testing checks whether the model can produce a good answer. It does not check whether the model reliably produces a good answer, and reliability is the entire game once your code stops being a demo and starts being a product.

To understand why, you need to understand what is actually different about testing an LLM-powered feature compared to testing a normal function.

Traditional Code vs. Model-Powered Code

When you write a normal Python function, the same input produces the same output every time.

def add_tax(price: float, rate: float = 0.08) -> float:
    return round(price * (1 + rate), 2)

Call add_tax(100) a thousand times and you get 108.0 a thousand times. If you write a unit test that asserts add_tax(100) == 108.0, that test is permanently valid — it will pass today, tomorrow, and after any refactor that doesn't change the logic.

Now compare that to a function that calls a model:

def summarize(text: str) -> str:
    response = client.responses.create(
        model="gpt-4.1",
        input=f"Summarize this in one sentence:\n\n{text}",
    )
    return response.output_text

Call summarize() with the same paragraph ten times and you can get ten different sentences. Even at temperature=0, OpenAI does not guarantee bit-for-bit determinism, because production inference runs across many GPUs and floating-point operations are not perfectly associative across hardware and batch compositions. Your "unit test" for this function cannot be assert summarize(text) == "The exact sentence I saw once." That assertion will fail constantly, for reasons that have nothing to do with whether your code is correct.

This single fact — that the function under test is non-deterministic — is the root of almost every problem discussed in this unit. You cannot use the testing habits you built for deterministic code and expect them to catch problems in a model-powered feature.

What "It Looked Fine" Actually Verified

When you manually test a prompt three times and it looks good, here is the honest list of what you have actually confirmed:

1. The model can produce a correct answer for the three inputs you tried, under the specific sampling conditions of those three runs.
2. Your code doesn't crash on those three inputs.
3. You, personally, liked the tone and structure of three outputs.

Here is what you have not confirmed:

1. Whether the model produces a correct answer for inputs you didn't try — including the weird, messy, real-world inputs your users will actually send.
2. Whether the model's behavior stays correct next week, after OpenAI updates the underlying model snapshot, after you tweak the system prompt for an unrelated reason, or after a teammate changes a function schema.
3. Whether the rate of correct answers is 95% or 60%. Three anecdotes cannot distinguish those two very different products.
4. Whether failures are randomly distributed or concentrated in a specific, common category of input (for example, every ticket written in a language other than English, or every request that mixes two topics).

This gap between "I saw it work" and "I know how often it works, and on what" is exactly what a systematic evaluation closes. That's the subject of the rest of this unit — but before you can appreciate the solution, it helps to walk through the concrete ways manual testing fails in practice.

Failure Mode 1: Small Sample Size Hides the Real Failure Rate

Imagine you built a support ticket categorizer that assigns one of five labels: billing, technical, account, feature_request, other. You test it on five tickets you wrote yourself, all clean and unambiguous. It gets all five right. You ship it.

In production, tickets are messy. Users write in fragments, mix two issues in one message, use sarcasm, or paste error logs directly into the box. If the model's real accuracy on ticket text like that is 82%, you will not discover this from five clean examples — you will discover it three weeks later when a support lead asks why 18% of billing tickets are actually about account access.

An evaluation with even 50–100 realistic examples, scored automatically, would have surfaced that 82% figure before a single real user saw the feature. That number — a measured pass rate against a fixed dataset — is something manual testing structurally cannot produce, because no human is going to sit and manually grade 100 outputs every time they change a prompt. A computer can, cheaply and consistently, which is the entire premise behind an automated eval (covered in Lesson 2) and a grader (covered in Lesson 3).

Failure Mode 2: Silent Regressions From "Harmless" Changes

This is the failure mode that causes the most damage in real codebases, because it violates the intuition that small changes have small effects.

Say your system prompt for a customer-support assistant currently says:

You are a support agent. Be concise and professional.

A teammate decides to make the tone warmer and changes it to:

You are a friendly, empathetic support agent. Make the customer
feel heard, and be concise and professional.

This reads like a harmless, even positive change. But "empathetic" and "make the customer feel heard" can nudge the model toward longer, more apologetic responses, more hedging language, and in some cases, more agreement with the customer's framing of a problem — including agreeing to policy exceptions the business doesn't actually allow. Nobody manually tests this change against the ten edge cases that matter (refund requests, angry customers, policy violations) because nobody knows in advance which ten edge cases matter. They test two or three friendly interactions, see warmer replies, and ship it.

The same pattern applies to model upgrades. When you change model="gpt-4.1-mini" to a newer snapshot because it's cheaper or faster, you are implicitly re-running every behavior your product depends on against a different set of weights. Output formatting conventions, refusal behavior, function-calling reliability, and instruction-following precision can all shift, sometimes for the better and sometimes not. Without a fixed, repeatable set of test cases and a way to score them automatically, you have no way to know whether a "safe-looking" change quietly broke five percent of your traffic.

This is why the industry phrase for this discipline is regression testing for prompts — you are not just testing that a prompt works right now, you are building a permanent, re-runnable check that catches the next person's (or your own) unintentional damage.

Failure Mode 3: Confirmation Bias in What You Choose to Test

When you manually write test inputs, you unconsciously write inputs that you expect the model to handle well. This is not a character flaw — it's how humans generate examples. If you're testing a code-explanation assistant, you'll instinctively type a clean, well-formatted function, because that's what "a function" means to you by default. You are far less likely to paste in minified, auto-generated, or deliberately obfuscated code, even though real users absolutely will.

A rigorous evaluation dataset is deliberately built to fight this bias: it should include boundary cases, malformed input, adversarial input, and examples pulled from real production logs — not just the cases a developer would think to type by hand. Lesson 2 covers how to assemble that kind of dataset instead of relying on your own imagination for coverage.

Failure Mode 4: Silent Structural Failures

Many production LLM features don't just generate prose — they produce structured output that downstream code parses: JSON payloads, function/tool call arguments, or values that get inserted directly into a database or another API call. Manual testing tends to check whether the content looks right, but a human eyeballing an output in a terminal is bad at catching subtle structural problems, especially at 11pm before a deploy.

import json

response = client.responses.create(
    model="gpt-4.1",
    input="Extract the customer's order number and issue type as JSON.",
)

# Looks fine when you read it...
print(response.output_text)
# {"order_number": "A-88213", "issue_type": "damaged item"}

# ...but this next line is what your application actually depends on:
data = json.loads(response.output_text)  # Can raise json.JSONDecodeError

If the model occasionally wraps its answer in a sentence ("Sure! Here's the JSON: {...}") or adds a trailing comma, json.loads throws an exception. You will not catch this by reading five outputs — you catch it by running hundreds of inputs through the exact parsing code your application uses and measuring how often it throws. This is precisely the kind of thing a grader is built to check automatically (Lesson 3), and it's a category of bug that is invisible to the human eye until it's already in production logs as a stack trace.

The Four Levels Where Evaluation Matters

As your system grows in complexity, the kind of testing it needs grows with it. It's useful to think about four rough levels of complexity, because "did it work" means something different at each one:

Manual testing degrades fastest as you move down this table. For a single-turn prompt, glancing at a handful of outputs might genuinely catch obvious problems. For a single-agent system with a tool-calling loop, there are combinatorially more paths through the conversation — which tool gets called, in what order, with what arguments, how the agent recovers from a failed tool call — and a human skimming a transcript will miss most of the ways that loop can go wrong. The more autonomy you give a model, the more you need a systematic way to check its behavior, because your own attention span cannot keep up with the branching possibilities.

What a Real Evaluation Gives You Instead

A proper evaluation setup replaces "I tried it and it looked fine" with three concrete, reusable things:

1. A fixed dataset of representative inputs, including edge cases and realistic mess — not just the three clean examples you thought of.
2. A grader — code or a separate model call that scores each output automatically, consistently, and the same way every time.
3. A pass rate — a single number (or a breakdown by category) that you can compare across runs: today's prompt scored 91%, last week's scored 88%, the new model snapshot scored 79% on the technical category specifically.

None of this eliminates the need for human judgment — a human still decides what "correct" means and still spot-checks results. What it eliminates is having to re-earn your confidence from scratch, by hand, every single time you touch the prompt, the model, or a tool definition. You build the eval once; you run it in seconds, as many times as you want, forever.

Common Mistakes at This Stage

Treating a demo as proof of production readiness. A demo is optimized for a controlled, favorable path through the product. Production traffic is not curated by anyone who wants the demo to succeed.

Testing only the happy path. If every example you hand-test assumes a well-formed, polite, on-topic user message, you have no information about what happens when a user pastes in three paragraphs of unrelated text, writes in a different language, or tries to get the model to ignore its instructions.

No baseline to compare against. If you don't measure the pass rate of your current prompt, you have nothing to compare a future change to. "This looks better" is not a substitute for "this scores higher on the same 100 test cases as before."

Re-testing by memory instead of by dataset. If your "test suite" lives in your head as a mental checklist of five things you always try, it will not survive being handed to a teammate, and it will silently shrink over time as you get busy.

Do You Need an Eval for a Simple Prompt?

Not every prompt justifies the overhead of a formal, versioned evaluation dataset. A one-off script you run yourself, once, to reformat a text file, doesn't need one. The line worth watching for is: does this prompt run more than once, on input you don't fully control, and does a wrong answer cost something — a bad user experience, a wrong business decision, a broken downstream parse? If the answer to all three is yes, you're past the point where manual testing is a reasonable strategy, even if the feature feels small.

A useful rule of thumb: the moment a prompt goes from "something I run in a notebook" to "something that runs automatically every time a user does X," it has crossed into production code, and production code that can silently degrade deserves the same kind of repeatable verification any other production code gets. The size of the eval doesn't have to match the size of the feature — a 20-row dataset checked automatically on every change is still enormously more reliable than an unlimited number of manual spot checks, because the 20 rows are exactly the same every time, and a computer never gets bored or skips one to save time.

The Software Testing Analogy — and Where It Breaks

If you come from traditional software engineering, you already have a mental model for this problem: the testing pyramid. Unit tests check small pieces of logic in isolation. Integration tests check that pieces work together. End-to-end tests check the whole system from the outside. All three assume the thing under test is deterministic, or can be made deterministic for the purpose of the test (mocking a clock, mocking a random number generator, mocking a network call).

You can, and should, still write ordinary unit tests around your LLM-powered code — but notice what they can and can't cover:

def test_summarize_returns_a_string():
    result = summarize("Some long article text...")
    assert isinstance(result, str)
    assert len(result) > 0

This test is legitimate and worth having. It catches crashes, empty responses, and obvious type errors. What it cannot do is tell you whether the summary is accurate, whether it omits the most important sentence, or whether it stays under the length limit your UI needs. Those are quality questions, not correctness-of-code questions, and assert statements are the wrong tool for quality questions that have a probabilistic answer.

This is the real reason evaluations exist as their own discipline instead of just being "more unit tests." A unit test gives you a boolean: pass or fail, and it should almost always pass once the code is correct. An eval gives you a rate: this prompt passes 91% of the time on this dataset, broken down by category. You don't expect that number to hit 100%, and a single failing row usually isn't a bug in the traditional sense — it's a data point telling you where the model's weaknesses cluster. That shift in mindset, from "pass/fail" to "what's the rate, and where does it dip," is the single most important idea in this entire unit.

A Concrete Regression, Traced Step by Step

It helps to see the failure mode from Lesson 1's introduction play out with real numbers, even illustrative ones, because "regressions can happen" is abstract and "here is exactly how one happens" is not.

Suppose you run a small internal tool that classifies incoming sales leads into hot, warm, or cold based on the lead's message, using this prompt:

Classify the lead below as hot, warm, or cold based on their
buying intent. Respond with only the label.

Lead message: {{lead_text}}

You manually test it on three messages — one obviously eager buyer, one obviously not interested, one asking a generic question — and it gets all three right. You ship it. Two months later, someone on the sales team notices that leads mentioning a competitor by name are almost always being marked cold, even when the lead is clearly comparison-shopping and ready to buy — which is actually a hot signal for a sales team, not a cold one.

Nobody wrote that bug. It's a property of how the base model interprets "mentions a competitor" as a negative signal by default, absent more specific instructions. It was there from day one, invisible in three hand-picked examples, and it was silently costing the sales team real leads for two months before a human happened to notice the pattern by reading enough transcripts manually — which is exactly the slow, unscalable process an eval replaces.

If a 100-row evaluation dataset had included even five examples of competitor-mentioning leads with a human-labeled correct answer, the very first run of the eval would have shown a specific, isolated failure: "9 out of 100 rows failed, and 5 of those 9 share the tag mentions_competitor." That is an actionable, specific signal. "It looked fine when I tested it" can never produce that sentence, because it never had the coverage or the bookkeeping to notice the pattern in the first place.

Signs You Need a Real Evaluation, Not More Manual Testing

A few practical signals tend to show up right before teams realize manual testing has stopped being enough:

You've stopped remembering to re-test after changes. Once a feature has shipped, changing the prompt "just a little" starts to feel low-risk, and manual re-testing quietly stops happening at all. This is normal — humans deprioritize tedious, repetitive verification, especially under deadline pressure — and it is exactly the gap an automated eval is designed to close, because a script never gets tired of running the same 100 checks.

Two people disagree about whether an output is "good." If a code reviewer and the prompt's author look at the same model response and reach different conclusions about whether it's acceptable, you don't have a testing problem, you have an undefined-criteria problem. Writing a grader (Lesson 3) forces you to make the definition of "correct" explicit and written down, which resolves this kind of disagreement before it reaches production.

You're relying on user complaints as your test suite. If the way you find out a prompt is broken is a support ticket or a Slack message from an unhappy internal user, your feedback loop is measured in days, and every failure in that window happened to a real person. An eval that runs in CI on every prompt change catches the same failure in seconds, before anyone outside your team ever sees it.

"Should be fine" is doing a lot of work in your team's vocabulary. If you notice yourself or a teammate saying "should be fine" about a prompt or model change instead of "I ran the eval and it's fine," that phrase is a tell that the team has quietly reverted to vibes-based confidence.

Building the Habit, Not Just the Tool

It's worth being honest that an eval, by itself, doesn't fix anything — it's a tool that only pays off if you actually run it, consistently, at the moments that matter. The teams that get real value from evaluation treat it the same way experienced engineering teams treat automated test suites: as a gate, not a nice-to-have.

In practice, that means:

• Running the eval before merging any change to a prompt, a tool schema, or a model version — not after a teammate notices something looks off in staging.
• Keeping the pass rate visible somewhere the team actually looks, whether that's a dashboard, a CI check annotation, or a number posted in a deploy channel, rather than buried in a log file nobody reads.
• Growing the dataset over time by adding every real production failure you find, so the eval gets stricter and more representative the longer the product has existed — instead of staying frozen at the 20 examples someone wrote on day one.
• Treating a drop in pass rate the same way you'd treat a failing CI pipeline: something to investigate before shipping, not something to note and move past.

None of this requires exotic tooling. It requires deciding, as a team, that "I tested it and it looked fine" is no longer an acceptable justification for shipping a change to a model-powered feature — and replacing that sentence with a number you can point to. The rest of this unit shows you exactly how to produce that number.

Start Collecting Real Examples Before You Need Them

One practical habit worth adopting immediately, even before you write a single formal eval, is logging the real inputs and outputs your feature produces once it's live. This matters because the best test cases are not the ones you can imagine while writing a prompt — they are the ones your actual users generate, including the strange ones you'd never have thought to type yourself.

A minimal version of this is just appending structured records to a file or a table every time your model-powered function runs:

import json
import time

def summarize_with_logging(text: str) -> str:
    response = client.responses.create(
        model="gpt-4.1",
        input=f"Summarize this in one sentence:\n\n{text}",
    )
    output = response.output_text

    with open("summarize_log.jsonl", "a") as f:
        f.write(json.dumps({
            "timestamp": time.time(),
            "input": text,
            "output": output,
            "model": "gpt-4.1",
        }) + "\n")

    return output

This looks almost too simple to matter, but it changes what happens the first time something goes wrong in production. Instead of trying to reconstruct, from memory, what input might have caused a bad output, you have the exact input sitting in a log file, ready to become the newest row in your evaluation dataset. Over a few weeks, this log becomes a rich, realistic pool of examples — including the messy, adversarial, and boundary-case inputs that Failure Mode 3 explained you would never have thought to write by hand. Lesson 2 shows you how to turn a collection of examples like this into the structured JSONL file the Evals API expects, and how to attach ground-truth labels to a sample of them so you have something to grade against.

You don't need a sophisticated observability platform to start. A plain JSONL file, or a row in a database table, is enough to begin — the discipline of capturing real traffic matters far more than the sophistication of where you store it.

A Note on Multi-Turn Conversations

Everything above gets harder, not easier, once your feature is a conversation rather than a single request-response pair. Manual testing a multi-turn assistant usually means typing a few messages yourself, in an order you find natural, and judging the whole exchange by feel. But real conversations branch in ways a single tester rarely reproduces: a user who corrects themselves halfway through, one who asks an unrelated question and then returns to the original topic, one who repeats a request because an earlier answer didn't actually address it. Each of those is a different path through your system, and a bug that only appears on turn four of a five-turn conversation is essentially invisible to someone typing a two-turn manual test. The eval and grader concepts in the rest of this unit apply just as directly to multi-turn transcripts as they do to single prompts — a dataset row can just as easily be an entire conversation history as a single input — but it's worth flagging now that conversation-shaped features need proportionally more deliberate test coverage than single-turn ones, precisely because manual testing covers proportionally less of what can actually happen.

Where This Leads

The fix for all four failure modes above is the same: stop relying on your own eyes and memory, and build something that runs a fixed set of inputs through your model-powered code and scores the outputs automatically, every time something changes. That "something" is an eval — a first-class concept in the OpenAI API with its own dataset format, its own way of defining pass/fail criteria, and its own way of comparing runs over time.

Lesson 2 walks through building your first one from scratch, using the OpenAI Python SDK, on a realistic example. Lesson 3 goes deep on graders — the different ways you can tell the system what "correct" actually means, from exact string matching all the way up to using a second model as a judge of subjective quality.