Self-Consistency: Sampling Multiple Reasoning Paths

Focus of This Lesson Lesson 45 introduced self-consistency: sample several chain-of-thought answers and take a majority vote. Lesson 46 compared it with single chain-of-thought. This lesson focuses on the engineering of the sampling itself, which decides whether self-consistency actually works in practice:

Question Why It Matters How do you get genuinely different reasoning paths? Identical samples give no benefit How many samples are enough? Each sample costs money and time How do you send samples efficiently? Sequential requests are slow How do you make answers comparable? Formatting differences split votes What do you do with ties and failed samples? Edge cases decide reliability How do you check that the paths really differ? Diversity should be measured, not assumed Where Diversity Comes From Self-consistency depends on the model producing different reasoning paths for the same prompt. As Lesson 2 explained, models generate text by selecting tokens from a probability distribution. Different selections early in a response lead to different paths.

Sampling Settings Setting Effect on Diversity Temperature Higher values make less likely tokens more probable, producing more varied paths. A value of 0 makes outputs nearly identical. Top P Limits choices to the most probable tokens whose combined probability reaches P. Lower values reduce diversity. Default settings Many chat models sample with some randomness by default, so repeated requests usually differ. Practical Guidance Avoid temperature 0 for self-consistency samples: near-identical runs defeat the purpose. Avoid very high temperatures: reasoning becomes erratic and more paths become wrong. A moderate temperature is the usual choice when the setting is available. The original self-consistency research used temperature sampling to obtain diverse paths. Some models, including some reasoning models, do not accept temperature or top_p. For those, rely on the model's default sampling and check that outputs actually vary. Because supported settings differ between models, the code in this lesson passes temperature only when you explicitly set it.

Diversity Through the Prompt Diversity can also be encouraged without changing settings, for example by asking different samples to approach the problem in different ways:

Solve the problem. Approach it by working forward from the given values. Solve the problem. Approach it by first deciding what the final quantity depends on, then computing each part. This prompt variation produces structurally different reasoning, which can be useful when default sampling produces very similar paths. If you use it, keep the answer format identical across variations so answers remain comparable.

Making Answers Comparable Voting only works if equivalent answers look identical. Without normalization, "5,818", "₹5818", and "5818.0" are three different votes for the same answer.

Step 1: Request a Strict Format Give only the final number inside tags, without units or currency symbols. Step 2: Normalize in Code import re

def normalize_number(text: str | None) -> str | None: """Convert a numeric answer to a canonical string, or None if no number is found.""" if text is None: return None cleaned = text.replace(",", "").replace("₹", "").strip() match = re.search(r"-?\d+(?:.\d+)?", cleaned) if not match: return None value = float(match.group()) return str(int(value)) if value.is_integer() else str(round(value, 4))

for raw in ["₹5,818", "5818.0", "34 students", "Answer: 2.50", "none"]: print(repr(raw), "→", normalize_number(raw)) Output:

'₹5,818' → 5818 '5818.0' → 5818 '34 students' → 34 'Answer: 2.50' → 2.5 'none' → None Code Explanation Thousands separators and the rupee symbol are removed before searching for the number. The regular expression finds the first number, including negative numbers and decimals. Whole numbers are converted to integer form (5818.0 becomes 5818), and decimals are rounded to 4 places, so tiny formatting differences do not split votes. Answers without a number become None, so they are excluded from voting rather than counted as a separate answer. For text answers such as labels or names, normalize case and whitespace, and map synonyms to canonical values if needed. Lesson 53 covers normalization for arithmetic and logic answers in more depth.

Voting With Ties and Failed Samples from collections import Counter

def vote(answers: list[str | None]) -> tuple[str | None, float, bool]: """Return (winning answer, agreement ratio, whether there was a tie for first place).""" valid = [a for a in answers if a is not None] if not valid: return None, 0.0, False counts = Counter(valid).most_common() best, top = counts[0] tie = len(counts) > 1 and counts[1][1] == top return best, top / len(answers), tie

print(vote(["34", "34", "40", None, "34"])) print(vote(["34", "40"])) Output:

('34', 0.6, False) ('34', 0.5, True) Code Explanation Failed samples None values, from responses without a valid answer, are excluded from the vote but still count in the agreement denominator: top / len(answers). In the first example, 3 of 5 samples voted for 34, so agreement is 0.6. Failed samples lower confidence, which is correct, because they are evidence that something went wrong.

Ties most_common() returns answers sorted by count. If the second answer has the same count as the first, there is a tie. In the second example, 34 and 40 each have one vote; Counter returns one of them first, but the tie flag shows the decision is not meaningful.

What to Do With a Tie Take more samples until the tie is broken. Treat the result as uncertain and send it for review. Never silently accept the first answer in a tie as if it were a real majority. Sending Samples Efficiently Samples are independent, so they can be requested in parallel. Python's concurrent.futures.ThreadPoolExecutor runs several API requests at the same time, which reduces total waiting time from roughly N × (one request) to about the time of the slowest request.

Requirements: pip install openai and an OPENAI_API_KEY environment variable. The code uses normalize_number() and vote() from above.

import re from concurrent.futures import ThreadPoolExecutor

from openai import OpenAI

client = OpenAI() MODEL = "gpt-5.4" # Replace with a model available in your account

INSTRUCTIONS = ( "Solve the problem step by step inside tags. Then give only the " "final number, without units or currency symbols, inside tags." )

def sample_once(problem: str, temperature: float | None = None) -> tuple[str, str | None]: """Return (reasoning, normalized answer) for one sampled reasoning path.""" options = {} if temperature is None else {"temperature": temperature} response = client.responses.create( model=MODEL, instructions=INSTRUCTIONS, input=problem, **options ) text = response.output_text reasoning = re.search(r"(.*?)", text, re.DOTALL) answer = re.search(r"(.*?)", text, re.DOTALL) return ( reasoning.group(1).strip() if reasoning else "", normalize_number(answer.group(1)) if answer else None, )

def sample_paths(problem: str, n: int = 5, temperature: float | None = None) -> list[tuple[str, str | None]]: with ThreadPoolExecutor(max_workers=n) as executor: return list(executor.map(lambda _: sample_once(problem, temperature), range(n)))

problem = ( "A batch starts with 60 students. 15% leave in the first month. Of the " "remaining students, one-third choose the evening batch. How many are in " "the day batch?" )

paths = sample_paths(problem, n=5) answer, agreement, tie = vote([a for _, a in paths]) print(f"Answer: {answer}, agreement: {agreement:.0%}, tie: {tie}") Code Explanation options temperature is added to the request only when you pass a value. This keeps the code working with models that do not accept the parameter.

options Unpacks the dictionary into keyword arguments, so create(..., {"temperature": 0.7}) is the same as create(..., temperature=0.7).

sample_once() Extracts both the reasoning and the normalized answer. Keeping the reasoning allows you to measure diversity and inspect failures.

ThreadPoolExecutor(max_workers=n) Runs up to n requests at the same time. executor.map calls sample_once once per item in range(n) and returns the results in order. The lambda _: ignores the number from range because each call uses the same problem.

Note: Parallel requests count against your API rate limits. For large batches, limit max_workers, handle rate-limit errors with retries and backoff, or use the asynchronous client (AsyncOpenAI).

Measuring Diversity You should not assume that samples differ; you can measure it. A simple measure compares the reasoning texts pairwise.

from difflib import SequenceMatcher from itertools import combinations

def diversity(reasonings: list[str]) -> float: """Return 1 minus the average pairwise similarity (0 = identical, higher = more diverse).""" pairs = list(combinations(reasonings, 2)) if not pairs: return 0.0 average = sum(SequenceMatcher(None, a, b).ratio() for a, b in pairs) / len(pairs) return 1 - average

identical = ["15% leave: 9. Remaining 51. Evening 17. Day 34."] * 3 varied = [ "15% leave: 9. Remaining 51. Evening 17. Day 34.", "Day batch is two-thirds of the 85% who stay: 60 × 0.85 × 2/3 = 34.", "Remaining = 51. Evening = 51 / 3 = 17. Day = 51 − 17 = 34.", ] print(round(diversity(identical), 2), round(diversity(varied), 2)) Output:

0.0 0.68 Code Explanation combinations(reasonings, 2) generates every pair of samples. SequenceMatcher(...).ratio() measures text similarity from 0 to 1 (Lesson 27). Diversity is 1 minus the average similarity: identical samples give 0; the three differently worded paths give 0.68. Using the Measurement If diversity is near 0, self-consistency is just repeating the same path at extra cost. Increase temperature (if supported), use prompt variations, or reconsider whether self-consistency is useful for this model and task.

Diversity of text is not the same as diversity of reasoning: two paths can use different words for the same steps. This measure is a quick check, not a precise one.

How Many Samples? Diminishing Returns The first few extra samples usually give the largest improvement. Each additional sample adds less, because the majority answer stabilizes. The original research tested larger numbers of samples, but in applications, cost usually limits the count to a small number.

Practical Starting Points Situation Typical Number Quick improvement on moderately hard problems 3 Standard use 5 High-stakes answers with budget available 7 to 10 Adaptive approach 2 first, then more only if they disagree (Lesson 46) Use Odd Numbers When Possible With two possible answers, an odd number of samples avoids an exact tie when every sample produces a valid answer.

Find Your Own Point of Diminishing Returns Run your test set with 1, 3, 5, and 7 samples, using the same samples for each count (the first 1, first 3, and so on), and plot accuracy against sample count. Choose the smallest count after which accuracy stops improving meaningfully.

Common Mistakes Mistake 1: Temperature 0 for All Samples What it is: Using deterministic settings for self-consistency samples.

Why it happens: Temperature 0 is set globally for consistency in the application.

What it causes: Identical samples and no benefit from voting.

How to fix it: Use sampling settings that allow variation for self-consistency calls, and measure diversity.

Mistake 2: Passing Unsupported Parameters What it is: Sending temperature to a model that does not accept it.

Why it happens: Code written for one model is reused with another.

What it causes: API errors.

How to fix it: Pass sampling parameters only when they are set and supported, as in sample_once().

Mistake 3: Voting on Raw Text What it is: Counting unnormalized answers.

Why it happens: Normalization is skipped.

What it causes: Equivalent answers split the vote.

How to fix it: Require a strict format and normalize answers before counting.

Mistake 4: Ignoring Ties and Failures What it is: Taking the first answer in a tie, or ignoring failed samples entirely.

Why it happens: The voting code returns a winner regardless.

What it causes: Uncertain results are presented as confident ones.

How to fix it: Detect ties, include failed samples in the agreement ratio, and handle both explicitly.

Mistake 5: Sequential Requests in User-Facing Features What it is: Sending five samples one after another while a user waits.

Why it happens: A simple loop is the easiest implementation.

What it causes: Five times the waiting time.

How to fix it: Send samples in parallel, within rate limits.

Best Practices Ensure diversity. Use appropriate sampling settings or prompt variations, and measure diversity.

Respect model capabilities. Only pass sampling parameters the model supports.

Normalize answers before voting. Strict formats in the prompt, normalization in code.

Handle ties and failures explicitly. They are signals of uncertainty.

Send samples in parallel. Manage rate limits with limited workers and retries.

Choose the number of samples by measurement. Find where extra samples stop helping.

Try It Yourself Exercise 1: Diversity and Settings If your model supports temperature, generate 5 samples for the same problem at temperatures 0, 0.5, and 1.0, and compute diversity() for each set. If it does not, compare default sampling with prompt variations. Record how many samples produced correct answers in each setting.

Exercise 2: Sample Count Curve Using 10 multi-step problems with known answers, collect 7 samples for each. Compute the voted answer using the first 1, 3, 5, and 7 samples, and record accuracy for each count. Identify where accuracy stops improving.

Exercise 3: Robust Voting Extend the code so that when vote() reports a tie, it automatically collects 2 more samples and votes again, up to a maximum of 9 samples. If the tie remains, return "Uncertain". Test it with a deliberately ambiguous problem.