Optimizing Prompts Systematically

What Does "Systematic" Optimization Mean? Most prompt improvement starts informally: try a change, look at a few outputs, keep it if they look better. That works for quick experiments, but it is unreliable for prompts used in real applications. Looking at a few outputs cannot tell you whether a change helped overall, whether it broke something else, or whether the improvement was just luck.

Systematic prompt optimization treats prompt improvement as a measured experiment:

Define objectives and metrics ↓ Build an evaluation dataset (with a held-out test set) ↓ Measure a baseline ↓ Form a hypothesis and create a variant ↓ Measure the variant on the same data ↓ Compare, accounting for uncertainty ↓ Keep, discard, or refine → repeat ↓ Confirm the final prompt on the test set Lesson 12 introduced iterative refinement for a single prompt. This lesson extends it into a disciplined optimization process: multiple objectives, controlled comparisons, ablation studies, cost optimization, and statistical caution.

Step 1: Define Objectives and Metrics Optimization requires knowing what you are optimizing. Most real prompts have several objectives.

Objective Example Metric How It Is Measured Quality Accuracy on labeled examples; rubric score Compare outputs to expected results (Lessons 114–117) Format reliability Percentage of outputs that pass validation Parse and validate in code (Lesson 8) Cost Average input and output tokens per request response.usage from the API Latency Average or 95th-percentile response time Time each request Robustness Accuracy on difficult inputs specifically A separate subset of edge cases Safety Rate of policy violations or unsupported claims Checks and reviews (Lessons 121–127) Primary Metric and Constraints Optimizing several numbers at once is confusing. A practical approach:

Choose one primary metric to maximize, such as accuracy. Treat the others as constraints: format reliability at least 99%, average cost below a set number of tokens, latency below 3 seconds. A variant is better only if it improves the primary metric without breaking a constraint.

Step 2: Build an Evaluation Dataset What It Contains Representative inputs from real use, not only clean examples. Expected outputs or criteria for judging each output. Edge cases: empty inputs, very long inputs, mixed languages, ambiguous cases, and manipulation attempts. Split It Set Purpose Development set Used for every comparison during optimization Test set Used once, at the end, to confirm the final prompt When you try many variants on the development set, the best one partly fits that specific data by chance. The untouched test set shows whether the improvement is real (Lesson 64).

Building evaluation datasets is covered in depth in Lessons 115 and 119.

Step 3: Measure a Baseline The baseline is the current prompt, measured on all metrics. Every variant is compared against it.

Metric Baseline Accuracy (dev set, 200 examples) 85% Format reliability 98% Average input tokens 620 Average latency 1.4 s Without a baseline, you cannot tell whether a change improved anything.

Step 4: Change One Thing at a Time Each variant should test one hypothesis, such as "adding a rule for tickets that mention several issues will fix misclassified mixed tickets". If you change several things at once and the result improves, you do not know which change was responsible, and some of the changes may have made things worse (Lesson 12).

Common Optimization Levers Lever Typical Hypothesis Lessons Clearer instructions Ambiguity causes inconsistent outputs 5, 25 Definitions and rules Boundary cases are misclassified 18, 30 Examples Format or style is inconsistent 26–31 Reasoning Multi-step errors are common 39–43 Context The model lacks necessary information 6, 19 Output format Parsing failures occur 8 Shorter prompt Unnecessary text adds cost without benefit 14 Different model A cheaper model may be good enough — Step 5: Compare Fairly and Account for Uncertainty The Problem With Small Differences Suppose a variant scores 95% and the baseline scores 85%. Is the variant better? It depends on how many examples were tested.

A confidence interval shows the range of plausible true accuracies given the observed result. The Wilson score interval is a reliable method for proportions such as accuracy.

import math

def wilson_interval(successes: int, n: int, z: float = 1.96) -> tuple[float, float]: """Approximate 95% confidence interval for a proportion (z = 1.96).""" if n == 0: return 0.0, 0.0 p = successes / n denominator = 1 + z2 / n centre = (p + z2 / (2 * n)) / denominator margin = z math.sqrt(p (1 - p) / n + z**2 / (4 n*2)) / denominator return max(0.0, centre - margin), min(1.0, centre + margin)

for successes, n in [(17, 20), (19, 20), (170, 200), (190, 200)]: low, high = wilson_interval(successes, n) print(f"{successes}/{n} = {successes / n:.0%} 95% interval: {low:.0%} to {high:.0%}") Output:

17/20 = 85% 95% interval: 64% to 95% 19/20 = 95% 95% interval: 76% to 99% 170/200 = 85% 95% interval: 79% to 89% 190/200 = 95% 95% interval: 91% to 97% Code Explanation p is the observed accuracy. The Wilson formula adjusts the center and width of the interval, which gives sensible results even for small samples and for accuracies close to 0% or 100%, where simpler formulas fail. z = 1.96 corresponds to a 95% confidence level. Reading the Result With 20 examples, 85% and 95% have overlapping intervals (64–95% and 76–99%). The difference could easily be chance. With 200 examples, the intervals (79–89% and 91–97%) do not overlap. The improvement is very likely real. Lesson: Small evaluation sets cannot reliably detect small improvements. Before trusting a difference, make sure the dataset is large enough.

Paired Comparison Because both prompts are evaluated on the same examples, you can compare them example by example. What matters is how many examples changed: correct for one prompt and wrong for the other.

def paired_comparison(results_a: list[bool], results_b: list[bool]) -> dict[str, int]: """Count agreement and disagreement between two prompts on the same examples.""" pairs = list(zip(results_a, results_b)) return { "only_A_correct": sum(a and not b for a, b in pairs), "only_B_correct": sum(b and not a for a, b in pairs), "both_correct": sum(a and b for a, b in pairs), "both_wrong": sum(not a and not b for a, b in pairs), }

baseline = [True] 16 + [False] 4 variant = [True] * 15 + [False, True, True, True, False] print(sum(baseline), sum(variant), paired_comparison(baseline, variant)) Output:

16 18 {'only_A_correct': 1, 'only_B_correct': 3, 'both_correct': 15, 'both_wrong': 1} Interpretation The variant scores 18 versus 16, but only 4 examples changed (1 got worse, 3 got better). A net gain of 2 out of 4 changed examples is weak evidence. The paired view also shows that the variant broke one example that the baseline got right, which is worth inspecting: it may reveal a side effect of the change.

Step 6: Ablation Studies An ablation study removes one component of a prompt at a time and measures the effect. It answers the question: which parts of this prompt actually matter?

Why Ablation Is Useful Prompts grow over time. Rules are added to fix specific problems, and examples are added "to be safe". Some of these components may no longer help, and they cost tokens on every request (Lesson 14).

Building Ablation Variants COMPONENTS = { "role": "You are a support specialist for an online course platform.", "definitions": "Billing: payments and refunds. Access: login and locked courses. ...", "rules": "If a ticket mentions several issues, choose what the customer asks us to do.", "format": "Reply with the category only.", }

def build_prompt(components: dict[str, str], exclude: str | None = None) -> str: return "\n\n".join(text for name, text in components.items() if name != exclude)

variants = {"full": build_prompt(COMPONENTS)} variants.update({f"without_{name}": build_prompt(COMPONENTS, exclude=name) for name in COMPONENTS}) print(list(variants)) Output:

['full', 'without_role', 'without_definitions', 'without_rules', 'without_format'] Each variant is then evaluated on the development set.

Example Ablation Results Variant Accuracy Format Reliability Input Tokens full 91% 99% 410 without_role 91% 99% 395 without_definitions 78% 99% 260 without_rules 87% 99% 385 without_format 90% 82% 400 Illustrative values.

Interpretation Definitions matter most for accuracy; removing them drops it substantially. Format matters for reliability, not accuracy. Rules add a few points of accuracy. Role makes no measurable difference here and could be removed to save tokens. Before removing it, confirm on a larger set that it truly has no effect, including on tone if that matters. Step 7: Optimize Cost Without Losing Quality Once quality meets your requirements, cost and latency often become the main objectives.

Technique Effect Risk Remove components shown by ablation to have no effect Fewer input tokens Hidden effects not covered by your metrics Shorten wording (Lesson 14) Fewer input tokens Accidentally removing meaning Limit output length Fewer output tokens, lower latency Truncated outputs (Lesson 2) Fewer or shorter examples Fewer input tokens Lower consistency (Lesson 26) Use a smaller or cheaper model Lower cost per token Lower quality on hard inputs Use code for exact steps No model cost for those steps None, if the logic is correct Put stable content first in the prompt Can enable provider-side caching of repeated prefixes Depends on the provider About Prompt Caching Some providers, including OpenAI, can reuse computation for a repeated prompt prefix, the identical beginning of a prompt, which can lower cost and latency for long prompts. To benefit, place stable content such as instructions and examples at the beginning, and variable content such as the user's input at the end. Rules and pricing differ by provider, so check the documentation for the model you use.

Evaluate Cost Changes Like Quality Changes A cheaper variant is acceptable only if it keeps the primary metric within an acceptable margin and still meets all constraints. Measure it on the same development set.

Step 8: Know When to Stop Stop optimizing when:

The primary metric meets the target and all constraints are satisfied. Further variants do not produce improvements larger than the uncertainty of your measurements. Remaining errors are caused by missing knowledge or task ambiguity, which prompt changes cannot fix (Lesson 24). Then confirm the final prompt once on the test set, record the result, and version the prompt (Lesson 67).

Keeping an Optimization Log Record every experiment so that decisions are traceable:

Version Change Hypothesis Dev Accuracy Format Tokens Decision v1 Baseline — 85% 98% 620 — v2 Added mixed-ticket rule Fix mixed tickets 88% 98% 650 Keep v3 Removed role sentence Role adds nothing 88% 98% 635 Keep v4 Added 3 examples Improve boundaries 89% 99% 900 Reject: +1% not worth +265 tokens The log prevents repeating failed experiments, explains why the prompt looks the way it does, and supports versioning.

Common Mistakes Mistake 1: Judging by a Few Outputs What it is: Deciding a change is better after looking at three or four responses.

Why it happens: It is fast and feels convincing.

What it causes: Changes that help some inputs and harm others are accepted.

How to fix it: Measure every variant on the full development set.

Mistake 2: Ignoring Uncertainty What it is: Treating a 2% difference on 50 examples as a real improvement.

Why it happens: Numbers look precise.

What it causes: Chasing noise and accepting random changes.

How to fix it: Use confidence intervals and paired comparisons; use larger datasets for small differences.

Mistake 3: Optimizing One Metric Only What it is: Improving accuracy while cost doubles and format reliability drops.

Why it happens: Only one number is tracked.

What it causes: Prompts that are "better" but unusable in production.

How to fix it: Choose a primary metric and enforce constraints on the others.

Mistake 4: Reusing the Test Set What it is: Checking the test set after every change.

Why it happens: It feels like extra validation.

What it causes: The test set becomes another development set and no longer gives an unbiased estimate.

How to fix it: Use the test set once, at the end.

Mistake 5: Keeping Components Without Evidence What it is: Never removing rules, examples, or role statements once added.

Why it happens: Removing something feels risky.

What it causes: Long, expensive prompts with components that do nothing.

How to fix it: Run ablation studies periodically.

Best Practices Define a primary metric and constraints. Optimization needs a clear target.

Use a development set for comparisons and a test set for confirmation. Never mix them.

Measure a baseline first. Every improvement is relative to it.

Test one hypothesis per variant. Make cause and effect clear.

Account for uncertainty. Use intervals and paired comparisons; prefer larger evaluation sets.

Run ablations. Find out which components earn their tokens.

Optimize cost after quality. Then verify that quality still meets the target.

Keep an optimization log. Make every decision traceable.

Try It Yourself Exercise 1: How Big Must the Dataset Be? Using wilson_interval(), find the smallest number of examples at which 85% and 90% accuracy produce non-overlapping 95% intervals. Explain what this means for detecting a 5-point improvement.

Exercise 2: Run an Ablation Take a prompt you use with at least four distinct components. Create ablation variants with build_prompt(), evaluate each on at least 50 labeled examples, and fill in a table like the one in this lesson. Decide which components to keep.

Exercise 3: Cost Optimization Starting from your best-quality prompt, create two cheaper variants: one shortened by removing components that ablation showed to be unnecessary, and one using a smaller model. Measure accuracy, tokens, and latency for each, and decide which to deploy given a constraint that accuracy may drop by at most 1 percentage point.