Comparing Chain-of-Thought and Self-Consistency

Two Techniques, One Relationship Chain-of-thought (CoT) and self-consistency are often presented as separate techniques, but they are not alternatives in the usual sense. Self-consistency is built on top of chain-of-thought.

Chain-of-thought: problem → one reasoning path → answer

Self-consistency: problem → many CoT reasoning paths → many answers → majority vote → answer Every self-consistency run uses a CoT prompt. The real decision is therefore:

Is one reasoning path enough, or do you need several and a vote?

This lesson compares the two directly: what each does well, what each costs, how to measure the difference for your task, and how to combine them efficiently.

Side-by-Side Comparison Aspect Chain-of-Thought (Single Path) Self-Consistency (Multiple Paths) Requests per problem 1 N (commonly 3 to 10, or more in research settings) Token cost 1× About N× Latency One response N responses (parallel requests reduce wall-clock time) Accuracy on multi-step problems Improved over direct answers Usually improved further over single CoT Effect of a random reasoning error Directly produces a wrong answer Often outvoted by correct paths Effect of a systematic error Wrong answer Wrong answer, often with high agreement Confidence signal None built in Agreement ratio Works for open-ended output Yes Not directly (requires comparable answers) Requires sampling diversity No Yes Implementation One prompt Prompt, loop, answer normalization, voting What Each Technique Fixes What CoT Fixes Compared With Direct Answers Missing intermediate steps: the model writes partial results it can build on (Lesson 39). Hidden errors: steps become visible and checkable. What Self-Consistency Fixes Compared With Single CoT Random errors in individual paths: an arithmetic slip or misread value in one path is likely to be outvoted (Lesson 45). Lack of a confidence measure: agreement shows how stable the answer is. What Neither Fixes Problem Why Neither Helps What Does Help Missing or wrong facts Every path reasons from the same missing fact Context, retrieval (Lessons 6, 101) Ambiguous instructions Paths may consistently follow the same wrong interpretation Clearer rules (Lesson 5) Exact computation errors at scale Votes reduce but do not eliminate arithmetic errors Compute in code (Lesson 58) Biased examples All paths share the same bias Better examples (Lesson 28) This last table is important. Before adding samples, ask whether the error is random (varies between runs) or systematic (happens every time). Self-consistency only helps with random errors.

How to Tell the Difference Run the same CoT prompt several times on a failing input:

If the answers vary, some runs are correct, and the error is at least partly random. Self-consistency is likely to help. If every run gives the same wrong answer, the error is systematic. More samples will only repeat it. Fix the prompt, the context, or the approach. Measuring the Difference on Your Task The following script compares single CoT and self-consistency on the same test set, reporting accuracy, total output tokens, and how often self-consistency changed the result.

Requirements: pip install openai and an OPENAI_API_KEY environment variable.

import re from collections import Counter

from openai import OpenAI

client = OpenAI() MODEL = "gpt-5.4" # Replace with a model available in your account

INSTRUCTIONS = ( "Solve the problem step by step inside tags. " "Then give only the final number inside tags." )

TEST_SET = [ ("A course costs 6000 rupees. Early registration gives 15% off. GST of 18% is " "added after the discount, then a 200 rupee coupon is applied. Final price?", "5818"), ("A batch starts with 60 students. 15% leave. One-third of the rest move to " "evenings. How many stay in the day batch?", "34"), ("A workshop has 80 seats. 60% are booked online and 12 at the venue. " "How many seats are free?", "20"), ("A student watches 3 lessons on Monday, twice as many on Tuesday, and 4 fewer " "on Wednesday than Tuesday. The course has 20 lessons. How many are left?", "9"), ]

def one_run(problem: str) -> tuple[str, int]: response = client.responses.create(model=MODEL, instructions=INSTRUCTIONS, input=problem) match = re.search(r"\s*(.?)\s", response.output_text, re.DOTALL) raw = match.group(1) if match else "" number = re.search(r"-?\d+(?:.\d+)?", raw.replace(",", "")) return (number.group() if number else ""), response.usage.output_tokens

def compare(samples: int = 5) -> None: cot_correct = sc_correct = changed = 0 cot_tokens = sc_tokens = 0

for problem, expected in TEST_SET: runs = [one_run(problem) for _ in range(samples)] answers = [answer for answer, _ in runs] tokens = [t for _, t in runs]

single = answers[0] # single CoT = the first run valid = [answer for answer in answers if answer] # ignore runs with no answer voted = Counter(valid).most_common(1)[0][0] if valid else ""

cot_correct += single == expected sc_correct += voted == expected changed += single != voted cot_tokens += tokens[0] sc_tokens += sum(tokens)

n = len(TEST_SET) print(f"Single CoT: {cot_correct}/{n} correct, {cot_tokens} output tokens") print(f"Self-consistency: {sc_correct}/{n} correct, {sc_tokens} output tokens") print(f"Self-consistency changed the answer on {changed} of {n} problems")

compare(samples=5) Code Explanation one_run() Makes one CoT request, extracts the number from the tag (removing thousands separators), and returns it with the number of output tokens. If no answer can be found, it returns an empty string.

Fair comparison For each problem, samples runs are collected once. The first run is treated as the single-CoT result, and all runs are used for the vote. Both methods are therefore evaluated on exactly the same model outputs, which removes luck from the comparison: self-consistency is literally "single CoT plus more samples".

Ignoring failed runs in the vote valid keeps only runs that produced an answer. Without this filter, several malformed responses could make the empty string the "majority answer". If no run produced an answer at all, the voted result is an empty string, which counts as incorrect.

Counting with booleans cot_correct += single == expected adds 1 when the comparison is True, because Python treats True as 1.

changed Counts how often voting produced a different answer from the first run. If this is almost always 0, the extra samples are rarely making a difference for this task.

Interpreting the Results Result Meaning Decision SC more accurate, and changed is often > 0 Random errors are common; voting corrects them Self-consistency is worth considering for this task Same accuracy, changed near 0 Runs nearly always agree Single CoT is enough Both inaccurate on the same problems Errors are systematic Fix the prompt, context, or use code SC only slightly better, cost N× higher Small gain for large cost Use adaptive self-consistency (below) or single CoT Note: A test set of four problems demonstrates the method. Real decisions require a larger, representative test set, especially because the differences between the two methods are usually concentrated in a minority of difficult inputs.

Adaptive Self-Consistency: Getting Most of the Benefit for Less You do not need to choose between "always one sample" and "always N samples". An adaptive approach uses more samples only when the first ones disagree.

Run 2 samples ├── Both agree → use the answer (cost: 2×) └── They disagree → run 3 more → majority vote over 5 (cost: 5×) def adaptive_answer(problem: str) -> tuple[str, int]: """Return (answer, number of samples used).""" first = [one_run(problem)[0] for _ in range(2)] if first[0] and first[0] == first[1]: return first[0], 2 more = [one_run(problem)[0] for _ in range(3)] votes = Counter(a for a in first + more if a) if not votes: return "", 5 return votes.most_common(1)[0][0], 5 Why This Works For most clear problems, the first two samples agree, so they cost only 2×. The full 5 samples are used only for difficult inputs, where disagreement shows that voting is needed. The average cost depends on how often your inputs are difficult.

Variation: Start With One Sample Plus Validation For even lower cost, start with a single CoT answer and only escalate to self-consistency when a validation check fails, such as the answer being outside a plausible range (Lesson 43, escalation).

Choosing Between Them Use Single Chain-of-Thought When The task benefits from reasoning, but runs usually agree. Latency matters, as in interactive chat or real-time features. Output is open-ended, so answers cannot be compared exactly. Volume is high and the cost of N samples is not justified. Use Self-Consistency When Answers are short and comparable: numbers, labels, choices, yes or no. Errors are at least partly random: different runs give different answers. Accuracy matters more than cost and latency for this task. A confidence signal is useful, for example to route uncertain cases to human review. Use Neither (or Something Else) When The errors are systematic: fix the prompt or provide context. The computation is exact: use code. The task does not need reasoning at all: use a direct prompt (Lesson 43). Decision Flow Does the task need step-by-step reasoning? ├── No → Direct prompt └── Yes → Use CoT ↓ Is the answer short and comparable? ├── No → Single CoT (consider candidate evaluation, Lesson 48) └── Yes → Do runs disagree on some inputs? ├── No → Single CoT └── Yes → Is the accuracy gain worth N× cost? ├── Yes → Self-consistency (fixed or adaptive) └── No → Single CoT, or adaptive self-consistency A Combined Example A platform calculates refund amounts from free-text support conversations:

CoT prompt extracts the purchase date, completion percentage, and coupon details, and checks policy conditions step by step. Self-consistency (adaptive) runs this 2 times; if the eligibility decision differs, it runs 3 more and votes. Code calculates the refund amount from the extracted values, because the calculation is exact. Agreement below 60% sends the case to a human agent. This design uses each technique for what it does best: CoT for interpretation and rule checking, self-consistency for stability on difficult cases, code for exact computation, and human review for genuinely uncertain decisions.

Common Mistakes Mistake 1: Treating Self-Consistency as a Separate Alternative What it is: Comparing self-consistency with a direct (non-CoT) prompt and attributing all gains to voting.

Why it happens: The relationship between the two techniques is not clear.

What it causes: Overestimating the benefit of extra samples.

How to fix it: Compare single CoT with self-consistency using the same CoT prompt.

Mistake 2: Adding Samples to Fix Systematic Errors What it is: Increasing the number of samples when every run gives the same wrong answer.

Why it happens: "More samples" seems like a general solution.

What it causes: Higher cost with no improvement.

How to fix it: Check whether answers vary across runs before using self-consistency.

Mistake 3: Comparing Methods on Different Outputs What it is: Running single CoT and self-consistency separately, so results differ partly by chance.

Why it happens: Each method is tested independently.

What it causes: Unreliable comparisons on small test sets.

How to fix it: Use the first sample as the single-CoT result and all samples for voting, as in the script above.

Mistake 4: Fixed N for Every Input What it is: Always generating the maximum number of samples.

Why it happens: It is the simplest implementation.

What it causes: Paying for samples on easy inputs where they never change the answer.

How to fix it: Use adaptive self-consistency.

Best Practices Think of self-consistency as an extension of CoT. Decide whether one reasoning path is enough.

Identify whether errors are random or systematic. Only random errors are fixed by voting.

Compare on identical outputs. Use the first sample as single CoT and all samples for voting.

Exclude failed runs from the vote. Missing answers must never become the majority answer.

Measure accuracy, cost, and how often voting changes the answer. If voting rarely changes anything, it is not needed.

Use adaptive sampling. Spend extra samples only on inputs where runs disagree.

Combine with code and human review. Each technique has a role; none replaces verification for important decisions.

Try It Yourself Exercise 1: Random or Systematic? Choose five problems where a single CoT run gave a wrong answer. Run each problem 7 more times. For each, decide whether the error is random (answers vary) or systematic (the same wrong answer appears consistently), and choose the appropriate fix.

Exercise 2: Run the Comparison Extend TEST_SET to 15 problems from your own domain, including at least five difficult ones. Run compare(samples=5) and record accuracy, output tokens, and how many answers voting changed. Decide which method you would deploy and justify the decision with your numbers.

Exercise 3: Measure Adaptive Savings Run adaptive_answer() on the same 15 problems. Record how many problems needed only 2 samples and how many needed 5. Calculate the total number of requests and compare it with always using 5 samples, and check whether accuracy differs from fixed self-consistency.