Active-Prompt: Selecting Useful Reasoning Examples

What Is Active-Prompt? Active-Prompt is a method for choosing which questions should receive hand-written chain-of-thought examples. Instead of picking examples randomly or by intuition, it asks the model to answer many candidate questions several times, measures where the model is most uncertain, and has people write reasoning examples for exactly those questions.

In simple terms: write worked examples for the problems the model finds hardest, because that is where examples help most.

Active-Prompt was introduced in the 2023 paper "Active Prompting with Chain-of-Thought for Large Language Models" by Diao et al., which reported improvements over standard chain-of-thought example selection on several reasoning benchmarks.

The Problem It Solves Few-shot chain-of-thought (Lesson 41) requires people to write correct reasoning examples. Writing them takes time, so you can only write a few. Which questions deserve that effort?

Selection Method Weakness Random questions May choose easy questions the model already solves The author's favorite examples Reflect what the author finds typical, not what the model finds hard Most complex questions (Lesson 50) Complexity is a proxy; it does not measure the model's actual difficulty Clustering for diversity (Lesson 42) Ensures variety, but not usefulness Most uncertain questions (Active-Prompt) Targets exactly where the model is unreliable The name comes from active learning, a machine learning approach where the system chooses which examples a human should label next, focusing on the most informative ones.

How Active-Prompt Works Step 1: Collect a Pool of Questions Gather questions representative of the task, without answers or reasoning. These can be real user questions or a training set.

Step 2: Estimate Uncertainty For each question, ask the model to answer it k times with sampling, such as 5 times (Lesson 52). Then measure how much the answers disagree.

Step 3: Select the Most Uncertain Questions Rank the questions by uncertainty and choose the top few.

Step 4: Annotate A person writes a correct, step-by-step reasoning chain and answer for each selected question.

Step 5: Use Them as Examples The annotated questions become the few-shot chain-of-thought examples for the task.

Pool of questions ↓ Sample k answers per question ↓ Measure uncertainty ↓ Select the most uncertain questions ↓ People write reasoning for them ↓ Use as few-shot CoT examples Measuring Uncertainty The original paper considered several ways to measure uncertainty. Two simple and practical ones are disagreement and entropy.

Disagreement Disagreement is the number of distinct answers divided by the number of samples.

Sampled Answers (k = 5) Distinct Answers Disagreement 1800, 1800, 1800, 1800, 1800 1 0.2 20, 20, 32, 20, 20 2 0.4 34, 40, 34, 31, 40 3 0.6 Wednesday, Tuesday, Wednesday, Thursday, Monday 4 0.8 Simple and easy to explain, but it ignores how often each answer appeared.

Entropy Entropy measures how spread out the answers are, taking frequencies into account. It is 0 when all answers are the same and increases as the answers become more evenly spread over many different values.

For example, 20, 20, 32, 20, 20 and 34, 34, 40, 40, 31 both have more than one answer, but the first is dominated by one answer, so its entropy is lower.

Other Measures The paper also discussed variance, for numeric answers, and self-confidence, where the model rates its own certainty. Self-reported confidence was found to be less reliable than measures based on actual answer variation, consistent with the caution in Lesson 18.

Computing Uncertainty in Python The following code computes disagreement and entropy for sampled answers and ranks questions by uncertainty. It runs without an API.

import math from collections import Counter

def disagreement(answers: list[str]) -> float: """Number of distinct answers divided by the number of samples.""" return len(set(answers)) / len(answers)

def entropy(answers: list[str]) -> float: """Shannon entropy of the answer distribution, in bits (0 = full agreement).""" counts = Counter(answers) total = len(answers) return sum((count / total) * math.log2(total / count) for count in counts.values())

sampled = { "Q1: 60 students, 15% leave, one-third of the rest go to evenings. Day batch?": ["34", "40", "34", "31", "40"], "Q2: A course costs 2000 with 10% off. Final price?": ["1800", "1800", "1800", "1800", "1800"], "Q3: 48 lessons, 3 per weekday, 5 on Saturday, rest Sunday, start Monday. Finish day?": ["Wednesday", "Tuesday", "Wednesday", "Thursday", "Monday"], "Q4: 80 seats, 60% booked online, 12 at the venue. Free seats?": ["20", "20", "32", "20", "20"], }

ranked = sorted(sampled, key=lambda q: (disagreement(sampled[q]), entropy(sampled[q])), reverse=True) for question in ranked: answers = sampled[question] print(f"disagreement={disagreement(answers):.1f} entropy={entropy(answers):.2f} {question[:45]}...") Output:

disagreement=0.8 entropy=1.92 Q3: 48 lessons, 3 per weekday, 5 on Saturday,... disagreement=0.6 entropy=1.52 Q1: 60 students, 15% leave, one-third of the ... disagreement=0.4 entropy=0.72 Q4: 80 seats, 60% booked online, 12 at the ve... disagreement=0.2 entropy=0.00 Q2: A course costs 2000 with 10% off. Final p... Code Explanation disagreement() set(answers) keeps only distinct values; dividing their count by the number of samples gives a value between 1/k (all identical) and 1.0 (all different).

entropy()

Counter counts how often each answer appears. For each answer with probability p = count / total, the formula adds p × log2(1/p), written here as (count / total) * math.log2(total / count). When all answers agree, there is one answer with p = 1, and log2(1) = 0, so entropy is exactly 0. Writing the formula this way avoids printing -0.00, which the equivalent form -sum(p * log2(p)) can produce. The maximum for 5 samples is log2(5) ≈ 2.32, reached when all five answers differ. Ranking Questions are sorted by disagreement first and entropy second, from most to least uncertain.

Shortened question text question[:45] keeps only the first 45 characters of each question, so each line stays short; ... marks the cut.

Reading the Result Q3 (the finishing-day problem) is the most uncertain: the model gives four different answers. A worked example of this type is likely to help. Q2 is answered consistently, so writing an example for it would add little. If you can afford three annotated examples, Active-Prompt suggests Q3, Q1, and Q4, in that order.

Collecting Samples With a Model To compute uncertainty for real questions, sample several answers per question. The sampling function from Lesson 52 can be reused:

Requirements: pip install openai and an OPENAI_API_KEY environment variable.

import re

from openai import OpenAI

client = OpenAI() MODEL = "gpt-5.4" # Replace with a model available in your account

INSTRUCTIONS = ( "Solve the problem step by step, then give only the final answer " "inside tags." )

def sample_answers(question: str, k: int = 5) -> list[str]: answers = [] for _ in range(k): response = client.responses.create(model=MODEL, instructions=INSTRUCTIONS, input=question) match = re.search(r"\s*(.?)\s", response.output_text, re.DOTALL) answers.append(match.group(1).strip().lower() if match else "") return answers

def select_for_annotation(questions: list[str], n: int = 3, k: int = 5) -> list[str]: samples = {q: sample_answers(q, k) for q in questions} ranked = sorted(questions, key=lambda q: (disagreement(samples[q]), entropy(samples[q])), reverse=True) return ranked[:n] Code Explanation Each question is answered k times; samples must vary for uncertainty to be measurable (Lesson 52). Answers are normalized to lowercase and stripped, so formatting differences do not count as disagreement. For numeric answers, use the normalization from Lessons 52 and 53. Missing answers are recorded as "", which counts as a distinct answer and therefore increases uncertainty, which is reasonable: a failure to answer is a sign of difficulty. select_for_annotation() returns the n most uncertain questions. The sampling cost is len(questions) × k requests, so start with a manageable pool. Writing the Annotations Selecting the questions is only half of the method. The value comes from high-quality reasoning written by people for those questions.

Annotation Guidelines Solve the problem completely and correctly. Verify every number, ideally with code (Lesson 58). Show every step that the model tended to get wrong. Look at the sampled answers: if wrong answers came from applying one-third to 60 instead of 51, make the "remaining students" step explicit. Use the same reasoning format in all examples. Consistent format helps the model follow the method (Lesson 41). End with a clear answer marker. It makes extraction and evaluation easy. Learning From the Wrong Samples The sampled answers are diagnostic information. They show how the model fails on each question, which tells the annotator what the example must emphasize. This is one of the most useful side effects of the method, even if you do not follow it exactly.

Active-Prompt Compared With Other Example Selection Methods Method Selects Examples By Needs Human Annotation? Main Strength Manual few-shot CoT (Lesson 41) Author's judgment Yes Full control Auto-CoT (Lesson 42) Diversity via clustering No (model-generated reasoning) No annotation effort Complexity-based (Lesson 50) Number of reasoning steps Depends Favors thorough reasoning Dynamic selection (Lesson 28) Similarity to each input Depends on the pool Relevance to the current input Active-Prompt Model uncertainty Yes, for selected questions only Focuses human effort where it matters most These methods can be combined. For example, you could cluster questions for diversity, then choose the most uncertain question within each cluster, so the examples are both diverse and difficult.

Limitations High Uncertainty Can Mean a Bad Question A question may be uncertain because it is ambiguous or badly worded, not because it is hard. An example built on an ambiguous question may confuse rather than help. Review selected questions before annotating them.

Consistently Wrong Answers Look Certain If the model gives the same wrong answer every time, uncertainty is low, and the question will not be selected, even though the model needs help with it. Uncertainty-based selection catches unstable errors, not systematic ones (Lesson 46). Where you have correct answers available, also check for questions that are consistently wrong.

Sampling Cost Estimating uncertainty requires several calls per question in the pool. For a large pool, sample a subset first.

The Model May Change Uncertainty is measured for a specific model. After switching models, the most uncertain questions may be different, and the selection should be repeated.

Common Mistakes Mistake 1: Sampling Without Diversity What it is: Sampling with settings that produce identical answers every time.

Why it happens: Deterministic settings are used throughout the application.

What it causes: Every question appears certain, so uncertainty cannot be measured.

How to fix it: Use sampling that allows variation, and confirm that answers can differ (Lesson 52).

Mistake 2: Counting Formatting Differences as Disagreement What it is: Treating "34", "34 students", and "Thirty-four" as different answers.

Why it happens: Answers are compared as raw text.

What it causes: Uncertainty is overestimated, and the wrong questions are selected.

How to fix it: Normalize answers before measuring uncertainty.

Mistake 3: Annotating Without Reviewing the Wrong Samples What it is: Writing reasoning for selected questions without looking at how the model failed.

Why it happens: The selection step seems to be the whole method.

What it causes: Examples that do not address the model's actual mistakes.

How to fix it: Read the sampled answers and make the error-prone steps explicit.

Mistake 4: Ignoring Systematic Errors What it is: Relying only on uncertainty to find difficult questions.

Why it happens: Uncertainty seems to capture difficulty.

What it causes: Consistently wrong question types are never addressed.

How to fix it: Where correct answers are known, also select questions that are consistently answered incorrectly.

Best Practices Use a representative pool of questions. The selected examples can only be as relevant as the pool.

Sample with diversity and normalize answers. Uncertainty must reflect real disagreement.

Rank by disagreement and entropy. Use both to separate close cases.

Review selected questions for ambiguity. Fix or skip unclear questions.

Write reasoning that targets the observed errors. The wrong samples show what to emphasize.

Combine with diversity when possible. Uncertain and varied examples are better than uncertain examples of one type.

Repeat selection after model changes. Uncertainty is model-specific.

Try It Yourself Exercise 1: Compute Uncertainty by Hand For each set of five sampled answers, compute disagreement and entropy with the functions above, and rank the questions:

A: 12, 12, 12, 15, 12 B: Yes, No, Yes, No, Yes C: 2027-04-22, 2027-04-21, 2027-04-22, 2027-04-23, 2027-04-20 Explain why question B has a higher entropy than question A, even though both have only two distinct answers.

Exercise 2: Select Questions for Annotation Create a pool of 15 multi-step word problems from your domain. Use select_for_annotation() with k=5 to find the three most uncertain ones. Inspect their sampled answers and describe the most common type of error for each.

Exercise 3: Measure the Benefit Write correct reasoning examples for the three selected questions. Compare a few-shot CoT prompt using these examples with one using three randomly chosen questions from the same pool (also annotated). Test both on 15 new problems and compare accuracy.