What Is the Automatic Prompt Engineer? Automatic Prompt Engineer (APE) is a method in which a language model generates candidate instructions for a task, each candidate is tested on examples, and the instruction with the best measured performance is selected.

In simple terms: instead of a person writing and rewriting a prompt by hand, a model proposes many prompts, and an automatic test picks the best one.

APE was introduced in the 2022 paper "Large Language Models Are Human-Level Prompt Engineers" by Zhou et al. The authors reported that automatically generated instructions matched or exceeded human-written instructions on many of the tasks they tested. One well-known result: APE found the zero-shot chain-of-thought trigger "Let's work this out in a step by step way to be sure we have the right answer.", which performed better than "Let's think step by step." on the reasoning benchmarks they evaluated (Lesson 40).

How APE Differs From Meta Prompting Lesson 55 showed how to ask a model to write a prompt, then test and revise it. APE makes this process more systematic:

Aspect Meta Prompting (Lesson 55) APE Starting point A task description Input-output examples of the task Number of prompts generated Usually one, then revisions Many candidates at once Selection Guided by test failures Score every candidate on a dataset; keep the best Search Revise the current prompt Generate variations of the best candidates The key idea is that prompt quality is measured, not judged. Every candidate is scored on the same data, and the numbers decide.

How APE Works Step 1: Instruction Generation The model sees a few input-output pairs and is asked what instruction would produce those outputs from those inputs.

I gave a colleague an instruction. They applied it to several inputs and produced these outputs:

Input: hot → Output: cold Input: fast → Output: slow Input: open → Output: closed

What was the instruction? Write only the instruction. Possible candidates (from several samples):

• Write the opposite of the given word.
• Give the antonym of the word.
• Return a word with the opposite meaning. This is called forward generation: the model infers the instruction from demonstrations. The original paper also used reverse generation, where the model fills in a missing instruction placed at the beginning of a template.

Step 2: Scoring Candidates Each candidate instruction is used, as a normal prompt, on a separate set of examples. Its outputs are compared with the expected outputs.

Candidate Score on Evaluation Set "Write the opposite of the given word." 0.92 "Give the antonym of the word." 0.95 "Return a word with the opposite meaning." 0.88 The original paper used execution accuracy (does the output match the expected output?) and, for some settings, the probability the model assigns to the expected output. Execution accuracy is the practical choice when you access models through an API.

Step 3: Selection and Iterative Search The best candidates are kept. Optionally, the model generates variations of them ("Generate a variation of the following instruction while keeping the meaning"), the variations are scored too, and the process repeats. The paper describes this as an iterative Monte Carlo search: searching the space of possible instructions near the current best ones.

Overview Training examples → generate many candidate instructions ↓ Evaluation examples → score every candidate ↓ Keep the top candidates → generate variations → score again (optional) ↓ Best instruction → confirm on held-out test examples The Most Important Rule: Separate Your Data APE chooses the prompt that scores highest on some data. If the same data is used to generate candidates, choose the winner, and report performance, the result will be overly optimistic. This is the overfitting problem from Lesson 12, applied automatically and at scale.

Split your labeled examples into three sets:

Set Used For Why Training (generation) Showing examples to the model when generating candidates Candidates are inspired by these Development (selection) Scoring candidates and choosing the best Many candidates are compared here, so the winner fits this set well Test (confirmation) Measuring the final chosen prompt once The only unbiased estimate of real performance Why the Test Set Must Be Separate When you compare 20 candidates on the development set, the winner may be lucky: it happened to suit those specific examples. Measuring it on a test set it has never influenced shows whether the improvement is real.

Implementing APE in Python Data Splitting and Scoring These functions contain no API calls, so they can be tested with a simulated model.

import random from typing import Callable

Example = tuple[str, str]

def split(examples: list[Example], seed: int = 0) -> tuple[list[Example], list[Example], list[Example]]: """Shuffle and split examples into training, development, and test sets (thirds).""" shuffled = examples[:] random.Random(seed).shuffle(shuffled) n = len(shuffled) return shuffled[: n // 3], shuffled[n // 3 : 2 n // 3], shuffled[2 n // 3 :]

def score_instruction(instruction: str, dataset: list[Example], run: Callable[[str, str], str]) -> float: """Fraction of examples where the output matches the expected output (case-insensitive).""" correct = sum(run(instruction, x).strip().lower() == y.lower() for x, y in dataset) return correct / len(dataset)

def rank_candidates(candidates: list[str], dataset: list[Example], run) -> list[tuple[float, str]]: scored = [(score_instruction(c, dataset, run), c) for c in candidates] return sorted(scored, key=lambda pair: pair[0], reverse=True)

Test with a simulated model

def fake_run(instruction: str, word: str) -> str: antonyms = {"hot": "cold", "big": "small", "fast": "slow", "up": "down", "happy": "sad", "open": "closed"} return antonyms[word] if "antonym" in instruction else word

data = [("hot", "cold"), ("big", "small"), ("fast", "slow"), ("up", "down"), ("happy", "sad"), ("open", "closed")] train, dev, test = split(data) print(len(train), len(dev), len(test)) print(rank_candidates(["Repeat the word.", "Write the antonym of the word."], dev, fake_run)) Output:

2 2 2 [(1.0, 'Write the antonym of the word.'), (0.0, 'Repeat the word.')] Code Explanation split() Shuffles a copy of the examples with a fixed seed (so the split is reproducible) and divides them into three equal parts. In practice, use far more than six examples; the small number here only demonstrates the mechanics.

score_instruction() Runs the instruction on every example and counts exact matches after normalizing case and whitespace. The run parameter is any function that executes an instruction on an input: a real model call or, for testing, a simulation.

rank_candidates() Scores every candidate on the same dataset and sorts them from best to worst.

The simulated model fake_run behaves correctly only when the instruction mentions "antonym". It lets you check that the scoring and ranking logic works before spending money on API calls.

Candidate Generation and Model Execution Requirements: pip install openai and an OPENAI_API_KEY environment variable.

from openai import OpenAI

client = OpenAI() MODEL = "gpt-5.4" # Replace with a model available in your account

GENERATE = """Here are input-output pairs produced by following one instruction:

{pairs}

Write the instruction that most likely produced these outputs. Write only the instruction, as one sentence."""

VARY = """Write a variation of the following instruction that keeps its meaning but uses different wording. Write only the new instruction.

Instruction: {instruction}"""

def call(prompt: str) -> str: response = client.responses.create(model=MODEL, input=prompt) return response.output_text.strip()

def model_run(instruction: str, text: str) -> str: response = client.responses.create( model=MODEL, instructions=instruction + "\nReply with the output only.", input=text, ) return response.output_text

def generate_candidates(train: list[Example], count: int = 8) -> list[str]: pairs = "\n".join(f"Input: {x}\nOutput: {y}" for x, y in train) prompt = GENERATE.replace("{pairs}", pairs) return list({call(prompt) for _ in range(count)})

def ape(examples: list[Example], rounds: int = 1, keep: int = 3) -> tuple[str, float]: train, dev, test = split(examples) candidates = generate_candidates(train) ranked = rank_candidates(candidates, dev, model_run)

for _ in range(rounds): best = [instruction for _, instruction in ranked[:keep]] variations = [call(VARY.replace("{instruction}", b)) for b in best] ranked = rank_candidates(best + variations, dev, model_run)

best_score, best_instruction = ranked[0] test_score = score_instruction(best_instruction, test, model_run) print(f"Best on dev: {best_score:.0%} Same prompt on test: {test_score:.0%}") return best_instruction, test_score Code Explanation GENERATE Forward instruction generation: the model sees demonstrations and writes the instruction that explains them. .replace("{pairs}", ...) is used instead of .format() so that braces elsewhere in the template cannot cause errors (Lesson 38).

generate_candidates() Calls the generation prompt several times. Because sampling varies between calls, this produces different candidates. Collecting them in a set removes exact duplicates.

model_run() Executes a candidate instruction as a normal prompt. The added line "Reply with the output only." keeps outputs comparable with the expected answers; it is applied identically to all candidates, so the comparison stays fair.

ape()

Splits the data. Generates and scores candidates on the development set. For each round, creates variations of the best candidates and re-ranks originals and variations together. Measures the final winner once on the test set. The printed comparison If the development score is much higher than the test score, the chosen instruction was overfitted to the development set, and the improvement may not be real.

Cost Scoring is the expensive part: each candidate is run on every development example. With 8 candidates, 3 variations, and 50 development examples, one round already requires hundreds of calls. Budget accordingly, and use the smallest development set that still gives stable rankings.

Practical Considerations APE Optimizes Only What You Measure The score is the only thing APE improves. If your metric is exact-match accuracy, APE will find instructions that produce exact matches, but it will ignore tone, safety, or length unless they are part of the score. Design the metric to reflect what you actually need (Lessons 114–117).

Automatic Does Not Mean Unreviewed Generated instructions can contain odd wording, hidden assumptions, or rules that happen to work on your data but are wrong in general. Always read the selected instruction before deploying it.

Instructions Interact With Everything Else APE typically optimizes the instruction part of a prompt. The full prompt may also contain context, examples, and format rules. An instruction that wins in isolation may behave differently when combined with the rest of your prompt, so evaluate the final, complete prompt.

Good Data Matters More Than Clever Search APE can only be as good as its examples. If the examples are unrepresentative, inconsistent, or wrongly labeled, APE will optimize toward the wrong target (Lesson 28).

When to Use APE Good Fit Why Tasks with clear, automatically checkable answers Scoring can be automated Tasks where small wording changes affect accuracy Search explores many wordings efficiently High-volume prompts where a few percent of accuracy matters The cost of the search is repaid over time You have enough labeled examples for three splits Fair selection and confirmation are possible Poor Fit Why Open-ended writing tasks without a reliable metric Nothing to optimize against Very few labeled examples Results will be noisy and overfitted Tasks where failures come from missing knowledge Better instructions cannot add facts (Lesson 24) Other approaches to automatic prompt optimization exist and continue to develop. They share the same essential ingredients as APE: candidate generation, measurement on data, and selection. Lesson 66 covers systematic optimization more broadly.

Common Mistakes Mistake 1: Selecting and Reporting on the Same Data What it is: Reporting the best development score as the expected performance.

Why it happens: It is the number the search produces.

What it causes: Overconfidence: the selected prompt performs worse on real inputs.

How to fix it: Confirm the final prompt once on a held-out test set.

Mistake 2: A Metric That Does Not Reflect the Goal What it is: Optimizing exact-match accuracy for a task where several phrasings are correct.

Why it happens: Exact match is easy to compute.

What it causes: Good instructions are scored as failures, and the search optimizes for a particular wording.

How to fix it: Normalize outputs, use label sets, or design a scoring function that matches the real requirement.

Mistake 3: Too Few Evaluation Examples What it is: Scoring candidates on five or ten examples.

Why it happens: Evaluation is expensive.

What it causes: Rankings are driven by chance.

How to fix it: Use enough examples for stable rankings, and check uncertainty (Lesson 66).

Mistake 4: Deploying the Winner Without Reading It What it is: Using the top-scoring instruction automatically.

Why it happens: The process is "automatic".

What it causes: Strange, unsafe, or overly narrow instructions in production.

How to fix it: Review the selected instruction and test it as part of the complete prompt.

Best Practices Start from good, representative examples. They define what "best" means.

Split data into training, development, and test sets. Generate, select, and confirm on different data.

Score with a metric that reflects the real goal. APE optimizes exactly what you measure.

Generate diverse candidates, then refine the best. Breadth first, then local search.

Test the logic with a simulated model. Verify scoring and ranking before spending on API calls.

Budget the search. Limit candidates, rounds, and evaluation size.

Review the winner as a human. Automation finds candidates; people decide what is deployed.

Try It Yourself Exercise 1: Generate Candidates Create 15 examples of a simple task, such as converting informal sentences into polite ones, or converting course names to URL slugs. Use generate_candidates() with five training examples and read all generated instructions. Note how many distinct ideas the candidates express.

Exercise 2: Run the Full Search Run ape() on a classification task with at least 45 labeled examples (15 per split). Record the development score and test score of the winner. If they differ substantially, explain why.

Exercise 3: Compare With Your Own Prompt Write your best hand-made instruction for the same task and score it on the same test set. Compare it with the APE winner. Then read both instructions and decide which you would deploy, considering clarity and maintainability as well as the score.