Self-Consistency Prompting

What Is Self-Consistency? Self-consistency is a technique where the model solves the same problem several times with chain-of-thought reasoning, and the final answer is chosen by majority vote among the answers.

In simple terms: instead of trusting one attempt, you ask for several independent attempts and pick the answer that most of them agree on.

┌→ Reasoning path 1 → Answer: 34 ├→ Reasoning path 2 → Answer: 34 Same problem ───┼→ Reasoning path 3 → Answer: 31 ├→ Reasoning path 4 → Answer: 34 └→ Reasoning path 5 → Answer: 34

Majority vote → 34 (4 of 5 agree) The technique was introduced in the 2022 paper "Self-Consistency Improves Chain of Thought Reasoning in Language Models" by Wang et al., which reported substantial accuracy gains over single chain-of-thought answers on arithmetic and commonsense reasoning benchmarks.

Why Self-Consistency Works Different Paths, Same Destination A complex problem can usually be solved in several valid ways. For example, to find the day batch size, you could compute the remaining students first, or compute the fraction of the original batch that ends up in the day batch. Correct reasoning paths, even when they differ, tend to arrive at the same answer.

Errors Tend to Scatter Mistakes are usually different from one path to another: one path misreads a value, another makes an arithmetic slip, a third applies a rule in the wrong order. Because errors lead to different wrong answers, they rarely outvote the correct one.

Correct answers cluster; wrong answers scatter. Majority voting exploits this pattern.

Connection to the Limitations of CoT Lesson 44 explained that a single reasoning chain can contain an error that propagates to the end, and that different runs can produce different answers. Self-consistency turns this variation into an advantage: instead of hoping a single run is correct, it uses several runs to identify the most reliable answer.

How Self-Consistency Works Step by Step Step 1: Use a Chain-of-Thought Prompt Any CoT prompt can be used: zero-shot (Lesson 40) or few-shot (Lesson 41). The prompt must end with a clearly marked final answer so answers can be compared.

Step 2: Generate Several Diverse Reasoning Paths Run the prompt multiple times. The runs must produce different reasoning paths. This requires sampling, meaning the model chooses tokens with some randomness (Lesson 2). The original method used temperature sampling to create diversity.

Step 3: Extract the Final Answer From Each Path Only the final answers are compared; the reasoning text is not.

Step 4: Choose the Most Common Answer The answer that appears most often is the final result. In the original paper's terminology, the reasoning paths are "marginalized out": what matters is which answer the most paths support.

Example Problem A course has 30 lessons. A student completes 4 lessons per weekday and 6 lessons on Saturday, resting on Sunday. Starting on Monday, on which day does the student complete the course? Five Sampled Reasoning Paths (Illustrative) Path 1: Mon–Fri = 5 × 4 = 20. Saturday = 6 → total 26 by Saturday. Sunday rest. Monday +4 = 30. Answer: Monday

Path 2: Week total = 20 + 6 = 26. Remaining after week 1 = 4. Next Monday covers 4. Answer: Monday

Path 3: Day by day: Mon 4, Tue 8, Wed 12, Thu 16, Fri 20, Sat 26, Sun 26, Mon 30. Answer: Monday

Path 4: 30 ÷ 4 = 7.5 days, so 8 weekdays needed. Counting weekdays only: Mon to the following Wednesday. Answer: Wednesday ← ignores Saturday's 6 lessons

Path 5: Mon–Fri = 20, Saturday = 6, total 26. Sunday +4 = 30. Answer: Sunday ← ignores the rest day

Vote Answer Count Monday 3 Wednesday 1 Sunday 1 Final answer: Monday. The two wrong paths made different mistakes, so neither wrong answer could outvote the correct one.

Implementing Self-Consistency in Python Requirements: pip install openai and an OPENAI_API_KEY environment variable.

import re from collections import Counter

from openai import OpenAI

client = OpenAI() MODEL = "gpt-5.4" # Replace with a model available in your account

INSTRUCTIONS = ( "Solve the problem step by step inside tags. " "Then give only the final answer, as a single word or number, inside tags." )

def sample_answer(problem: str) -> str | None: response = client.responses.create( model=MODEL, instructions=INSTRUCTIONS, input=problem, ) match = re.search(r"\s*(.?)\s", response.output_text, re.DOTALL) return match.group(1).strip().lower() if match else None

def self_consistency(problem: str, samples: int = 5) -> tuple[str | None, float, Counter]: answers = [sample_answer(problem) for _ in range(samples)] valid = [a for a in answers if a] if not valid: return None, 0.0, Counter() votes = Counter(valid) best, count = votes.most_common(1)[0] agreement = count / samples return best, agreement, votes

problem = ( "A course has 30 lessons. A student completes 4 lessons per weekday and " "6 lessons on Saturday, resting on Sunday. Starting on Monday, on which day " "does the student complete the course?" )

answer, agreement, votes = self_consistency(problem, samples=5) print("Votes:", dict(votes)) print(f"Final answer: {answer} (agreement {agreement:.0%})") Possible output:

Votes: {'monday': 5} Final answer: monday (agreement 100%) Code Explanation INSTRUCTIONS A standard CoT prompt with the answer in tags, so each run's answer can be extracted and compared.

sample_answer()

Makes one independent request. Extracts the answer and normalizes it with .strip().lower(), so that "Monday", "monday", and " Monday " count as the same vote. Returns None if the answer tag is missing, so malformed responses are excluded instead of counted as an answer. self_consistency()

Collects samples answers. Counter counts the votes; most_common(1) returns the winning answer and its count. agreement is the share of all samples that voted for the winner. Dividing by the total number of samples, not only the valid ones, means malformed responses lower the agreement score. About Diversity Self-consistency only works if the samples can differ. Many chat models sample with some randomness by default, so repeated requests produce different reasoning. If a model supports a temperature setting, a moderate value encourages diversity; a value of 0 makes runs nearly identical, which defeats the purpose. Note that some models, including some reasoning models, do not accept a temperature parameter, so check the documentation before setting it. Sampling strategies are covered in detail in Lesson 52.

Speed The loop above sends requests one after another. Because the samples are independent, they can be sent in parallel, for example with Python's concurrent.futures.ThreadPoolExecutor or the asynchronous OpenAI client, which reduces total waiting time.

Agreement as a Confidence Signal The agreement ratio is a useful by-product of self-consistency.

Agreement Interpretation Suggested Action High (for example, 5 of 5) The problem is probably clear, and the answer is likely correct Use the answer Medium (for example, 3 of 5) Some paths make errors; the majority answer is probably right Use it, possibly with extra validation Low (for example, 2 of 5 or a tie) The problem may be ambiguous, too hard, or under-specified Flag for review, add samples, or clarify the prompt Important Caution High agreement means the model is consistent, not that it is correct. If the prompt is missing a fact, or every path shares the same misunderstanding (for example, a rule that is ambiguous in the prompt), all samples can agree on the same wrong answer. Agreement measures stability, not truth.

Where Self-Consistency Applies Self-consistency requires answers that can be compared exactly. It works best when the final answer is short and has a clear correct value.

Good Fit Why Arithmetic and word problems Numeric answers are easy to compare Multiple-choice questions Answers are fixed options Classification with fixed labels Votes can be counted per label Yes/no decisions Two possible answers Logic puzzles with a single answer One correct solution Poor Fit Why Summaries and essays No two outputs are identical, so votes cannot be counted directly Creative writing There is no single correct answer Open-ended advice Different answers can all be valid For open-ended outputs, related approaches exist, such as generating several candidates and using a separate evaluation step to choose the best one (Lesson 48), but that is selection, not majority voting.

Self-Consistency for Classification Self-consistency is not only for math. It also works for borderline classification cases.

def classify_with_votes(text: str, samples: int = 5) -> tuple[str, float]: instructions = ( "Classify the student message as Doubt, Request, Complaint, or Feedback. " "Think briefly inside tags, then give only the label inside tags." ) labels = [] for _ in range(samples): response = client.responses.create(model=MODEL, instructions=instructions, input=text) match = re.search(r"\s*(.?)\s", response.output_text, re.DOTALL) if match: labels.append(match.group(1).strip()) if not labels: return "NeedsReview", 0.0 label, count = Counter(labels).most_common(1)[0] return label, count / samples A borderline message such as "Great session, but please upload the notes." might receive votes split between Request and Feedback. A low agreement score identifies exactly these cases, which can then be reviewed or used to improve the prompt with a clearer rule (Lesson 18).

Costs and Trade-Offs Benefit Cost Higher accuracy on reasoning tasks Several times the token cost: 5 samples cost about 5 times as much Built-in confidence signal Higher latency, unless requests are parallel Reduces the impact of random reasoning errors Does not fix systematic errors shared by all paths Reducing Cost Use self-consistency only for tasks where accuracy matters enough to justify it. Start with a small number of samples, such as 3, and add more only when the first ones disagree. If the first 3 agree, stop early. Combine with escalation: use a single answer first, and switch to self-consistency only when validation fails (Lesson 43). Common Mistakes Mistake 1: Sampling Without Diversity What it is: Running the prompt several times with settings that produce identical outputs.

Why it happens: Low temperature is used for consistency in other parts of the application.

What it causes: All samples repeat the same path, so voting adds cost without benefit.

How to fix it: Use sampling that allows variation, and check that the reasoning paths actually differ.

Mistake 2: Comparing Unnormalized Answers What it is: Counting "34", "34 students", and "Thirty-four" as different answers.

Why it happens: Raw output is compared directly.

What it causes: Votes are split artificially, and the wrong answer may win.

How to fix it: Require a strict answer format and normalize answers before counting.

Mistake 3: Treating Agreement as Proof What it is: Assuming a unanimous answer must be correct.

Why it happens: Agreement feels like confirmation.

What it causes: Systematic errors, such as a missing fact or ambiguous rule, are trusted.

How to fix it: Use agreement as a signal, and validate important answers independently.

Mistake 4: Using Voting for Open-Ended Outputs What it is: Trying to vote on summaries or paragraphs.

Why it happens: Applying the technique outside its intended scope.

What it causes: No meaningful majority, because every output is different.

How to fix it: Use self-consistency for short answers; use candidate evaluation for open-ended tasks.

Best Practices Combine self-consistency with a CoT prompt that has a strict answer format. Answers must be extractable and comparable.

Ensure diversity in the samples. Different reasoning paths are the source of the benefit.

Normalize answers before voting. Case, whitespace, units, and number formats.

Use the agreement ratio. Low agreement identifies difficult or ambiguous inputs.

Control cost. Use a few samples, stop early on agreement, and apply the technique only where accuracy matters.

Remember what voting cannot fix. Missing facts and shared misunderstandings require better prompts or context.

Try It Yourself Exercise 1: Observe Vote Distributions Run the self_consistency() function with samples=7 on five multi-step problems of varying difficulty. Record the vote distribution for each. Identify which problems had low agreement and inspect their reasoning paths to understand why.

Exercise 2: Early Stopping Modify self_consistency() so that it first collects 3 samples and stops if all 3 agree. Otherwise, it collects 4 more and votes over all 7. Measure how many requests it saves on 10 problems compared with always collecting 7.

Exercise 3: Find Ambiguous Classifications Use classify_with_votes() on 15 student messages, including several borderline ones. List the messages with agreement below 60%, write a rule for each pattern you find, add the rules to the instructions, and check whether agreement improves.