Self-Consistency for Arithmetic and Logic Tasks
Why Arithmetic and Logic Need Special Handling Arithmetic and logic problems are where self-consistency is most often used, because their answers are exact and comparable (Lesson 45). They also bring specific challenges:
Challenge Example Equivalent answers look different 0.75, 3/4, and 0.750 are the same number Rounding differences 33.33 and 33.333 may both be acceptable Some answers can be checked exactly A schedule either satisfies all constraints or it does not Some problems have several correct answers Majority voting assumes there is only one Arithmetic slips are common Wrong digits in otherwise correct reasoning This lesson shows how to adapt self-consistency to these tasks: normalizing numeric answers exactly, comparing with tolerance, verifying candidates in code before voting, and handling problems with more than one valid answer.
Part 1: Arithmetic Answers Exact Normalization With Fractions Lesson 52 normalized answers as decimal strings. For arithmetic, Python's Fraction type is often better: it represents numbers exactly, so 0.75 and 3/4 become the same value.
import re from fractions import Fraction
def parse_number(text: str | None) -> Fraction | None: """Parse an integer, decimal, or simple fraction into an exact Fraction.""" if text is None: return None cleaned = text.replace(",", "").replace("₹", "").strip() match = re.search(r"-?\d+(?:.\d+)?(?:\s*/\s*\d+)?", cleaned) if not match: return None try: return Fraction(match.group().replace(" ", "")) except (ValueError, ZeroDivisionError): return None
for raw in ["3/4", "0.75", " 0.750 ", "2 / 3", "₹5,818", "1/0", "abc"]: print(repr(raw), "→", parse_number(raw)) Output:
'3/4' → 3/4 '0.75' → 3/4 ' 0.750 ' → 3/4 '2 / 3' → 2/3 '₹5,818' → 5818 '1/0' → None 'abc' → None Code Explanation Cleaning Thousands separators and the rupee symbol are removed, as in Lesson 52.
The regular expression
-?\d+ matches an optional minus sign and digits. (?:.\d+)? optionally matches a decimal part. (?:\s*/\s*\d+)? optionally matches a fraction such as / 3, with or without spaces. Fraction(...) Python's Fraction accepts strings such as "0.75" or "3/4" and stores them as exact ratios. Both become 3/4, so they count as the same vote.
Error handling "1/0" raises ZeroDivisionError, and malformed text raises ValueError. Both return None, so the sample is excluded from voting.
Voting With Exact Values from collections import Counter
raw_answers = ["0.75", "3/4", "0.8", "3/4", None] values = [parse_number(a) for a in raw_answers] votes = Counter(v for v in values if v is not None) best, count = votes.most_common(1)[0] print(best, count, float(best)) Output:
3/4 3 0.75 Three of the five samples (0.75, 3/4, 3/4) are recognized as the same answer, which wins against 0.8.
When Exact Comparison Is Too Strict Some answers are naturally approximate, such as 33.33 for one-third of 100. Two options:
Specify rounding in the prompt: Round the final answer to 2 decimal places. Then exact comparison of rounded values works. Group answers with a tolerance in code, treating values within a small difference as equal: import math
math.isclose(33.33, 33.333, abs_tol=0.01) # True Choose the tolerance based on the task: currency amounts may need exact paise, while percentages for a report may tolerate small differences.
Part 2: Verify, Then Vote The Idea For many arithmetic and logic tasks, a candidate answer can be checked, even if computing the answer from scratch is hard. When checking is possible, it is better than voting alone:
Sample several reasoning paths ↓ Extract each candidate answer ↓ Verify each candidate in code → discard invalid ones ↓ Vote among the VALID candidates only Why This Is Stronger Voting alone assumes that most samples are correct. If several samples make the same mistake, the wrong answer can win. Verification removes incorrect candidates before they can vote, so even a minority of correct samples can produce the right result.
Arithmetic Verification Examples Problem Type Verification in Code "Find x such that 3x + 7 = 31" Substitute the candidate: is 3 × x + 7 equal to 31? "Split ₹10,000 between two people in a 3:2 ratio" Do the parts sum to 10,000 and have a 3:2 ratio? "What is the final price after these steps?" Recompute the steps in code with the given values For the last row, if the steps can be computed in code, it is usually simpler to compute the answer directly than to verify the model's answer. Verification is most valuable when checking is easy but solving is hard, as with equations and constraint problems.
Part 3: Logic Problems and Constraint Checking Example Problem Three mentors (Arun, Priya, Karthik) must each take one session: Monday, Tuesday, or Wednesday. Priya cannot take Monday. Karthik must take a day after Arun. Who takes which day? Lesson 43 showed that this problem has two valid solutions:
Arun Monday, Priya Tuesday, Karthik Wednesday Arun Monday, Priya Wednesday, Karthik Tuesday This is a perfect case for verification: checking a proposed schedule against the rules is simple, even though finding all solutions requires careful reasoning.
Requesting Comparable Answers Solve the scheduling problem in step by step inside tags. Then give the schedule inside tags as a JSON object mapping each mentor's name to a day, for example {"Arun": "Monday", ...}. A JSON object is easy to parse and compare, and its structure prevents ambiguous wording.
Verification in Code from collections import Counter
DAYS = {"Monday", "Tuesday", "Wednesday"} DAY_ORDER = {"Monday": 1, "Tuesday": 2, "Wednesday": 3} MENTORS = {"Arun", "Priya", "Karthik"}
def is_valid(schedule: dict[str, str]) -> bool: """Check that a schedule satisfies every rule of the problem.""" if set(schedule) != MENTORS or set(schedule.values()) != DAYS: return False # each mentor exactly one distinct day if schedule["Priya"] == "Monday": return False # Priya cannot take Monday return DAY_ORDER[schedule["Karthik"]] > DAY_ORDER[schedule["Arun"]]
samples = [ {"Arun": "Monday", "Priya": "Tuesday", "Karthik": "Wednesday"}, {"Arun": "Monday", "Priya": "Wednesday", "Karthik": "Tuesday"}, {"Arun": "Tuesday", "Priya": "Monday", "Karthik": "Wednesday"}, # breaks Priya rule {"Arun": "Monday", "Priya": "Tuesday", "Karthik": "Wednesday"}, {"Arun": "Monday", "Priya": "Tuesday", "Karthik": "Tuesday"}, # two mentors on Tuesday ]
valid = [tuple(sorted(s.items())) for s in samples if is_valid(s)] votes = Counter(valid) print(f"{len(valid)} valid samples, {len(votes)} distinct valid solutions") for solution, count in votes.most_common(): print(count, dict(solution)) Output:
3 valid samples, 2 distinct valid solutions 2 {'Arun': 'Monday', 'Karthik': 'Wednesday', 'Priya': 'Tuesday'} 1 {'Arun': 'Monday', 'Karthik': 'Tuesday', 'Priya': 'Wednesday'} Code Explanation is_valid() Encodes every rule of the problem:
set(schedule) != MENTORS ensures all three mentors appear and no others. set(schedule.values()) != DAYS ensures all three days are used, so no two mentors share a day. The Priya rule and the ordering rule are checked directly. tuple(sorted(s.items())) Dictionaries cannot be counted by Counter because they are not hashable. Converting each schedule to a sorted tuple of (name, day) pairs creates a hashable, order-independent representation, so identical schedules are counted together.
Results
Two samples are rejected: one breaks the Priya rule, one assigns two mentors to Tuesday. Among the valid samples, there are two distinct solutions. The Multiple-Solutions Lesson Plain majority voting would report "Arun Monday, Priya Tuesday, Karthik Wednesday" as the answer, because it has the most votes. That would hide the fact that a second, equally valid solution exists. With verification, you can see that both candidates are correct.
Rule: When verification shows several distinct valid answers, do not choose by vote. Either report all valid solutions, or add a rule that makes the answer unique (for example, "If several schedules are valid, choose the one where Priya's day is earliest").
Part 4: Combining Self-Consistency With Code Execution For arithmetic-heavy problems, an even stronger approach is to ask the model to write a short program that computes the answer, run it, and vote on the program outputs. This removes arithmetic slips from the model's reasoning entirely. This approach is called Program-Aided Language Models (PAL) and is covered in Lesson 58.
Warning: Running code generated by a model is a security risk. Generated code must only be executed in an isolated sandbox with no access to your files, network, or credentials. Lesson 58 discusses safe execution.
A Complete Arithmetic Workflow
- Prompt with CoT and a strict answer format
- Sample several reasoning paths in parallel (Lesson 52)
- Parse answers exactly with Fraction (or round to an agreed precision)
- Verify each candidate if a check is possible
- Vote among valid candidates
- If no valid candidates, or a tie, or several distinct valid answers → flag for review Decision Table for Results Outcome Action One valid answer with strong agreement Accept One valid answer with weak agreement Accept with caution, or sample more Several distinct valid answers Report all, or add a rule that makes the answer unique No valid answers Flag: the problem may be misunderstood or unsolvable as stated Tie between invalid answers only Flag: do not guess Common Mistakes Mistake 1: Comparing Numbers as Strings What it is: Treating "0.75" and "3/4" as different answers.
Why it happens: Answers are counted as raw text.
What it causes: Votes for the correct answer are split.
How to fix it: Parse answers into exact numeric values before voting.
Mistake 2: Voting When Verification Is Possible What it is: Relying only on majority vote for problems whose answers can be checked.
Why it happens: Voting is the standard self-consistency approach.
What it causes: Wrong majority answers win, and invalid answers are counted.
How to fix it: Verify each candidate first and vote only among valid ones.
Mistake 3: Assuming a Single Correct Answer What it is: Picking the most common valid answer when several valid answers exist.
Why it happens: Voting implicitly assumes one correct answer.
What it causes: Hidden alternative solutions, and misleading certainty.
How to fix it: Check how many distinct valid answers exist and report them, or make the problem's requirements unique.
Mistake 4: Inconsistent Rounding What it is: Some samples round to 2 decimal places, others to 3.
Why it happens: The prompt does not specify precision.
What it causes: Equivalent answers are counted separately.
How to fix it: Specify rounding in the prompt, or compare with a tolerance.
Best Practices Parse numeric answers exactly. Use Fraction or a defined rounding rule.
Request structured answers for logic problems. JSON objects are easy to verify and compare.
Verify before voting whenever possible. Remove invalid candidates first.
Check for multiple valid solutions. Do not let a vote hide them.
Consider computing in code. For heavy arithmetic, code execution (Lesson 58) can replace error-prone reasoning.
Flag uncertain outcomes. No valid answers, ties, or multiple valid answers all need attention.
Try It Yourself Exercise 1: Exact Numeric Voting Write five problems whose answers are fractions or decimals, such as splitting amounts or computing ratios. Collect 5 samples for each, parse them with parse_number(), and vote. Count how many votes would have been split if you had compared raw strings instead.
Exercise 2: Verify an Equation Ask a model to solve "Find a whole number x such that 7x − 12 = 51" with 5 samples. Write a verification function that substitutes each candidate into the equation, and vote only among valid candidates. Compare with plain majority voting.
Exercise 3: A Logic Puzzle With Constraints Create a scheduling puzzle with four people, four time slots, and three rules. Write is_valid() for your rules, collect 7 samples from the model, and report the number of distinct valid solutions. Then add one more rule so that exactly one solution remains, and confirm it with new samples.