Limitations of Chain-of-Thought Prompting

Why Study the Limitations? Chain-of-thought prompting (Lessons 39–43) improves accuracy on many multi-step tasks. It is easy to conclude that visible reasoning makes answers trustworthy. That conclusion is wrong, and building applications on it leads to real errors.

This lesson explains the main limitations of chain of thought: why each happens, what it looks like, and what to do about it. Knowing these limitations is what allows you to use CoT safely.

Limitation Short Description

1. Reasoning is not verification Steps can look correct and still be wrong
2. Errors propagate One wrong step corrupts everything after it
3. Unfaithful explanations The written reasoning may not reflect what actually drove the answer
4. Confident wrong reasoning Fluent logic can justify incorrect conclusions
5. Knowledge gaps remain Reasoning cannot create missing facts
6. Cost and latency More tokens, slower responses
7. Inconsistency between runs Different reasoning paths give different answers
8. Sensitivity to examples and wording Reasoning quality depends on the prompt
9. Exposure risks Reasoning may reveal internal rules or data
10. Not always available or needed Reasoning models may hide or summarize their reasoning Limitation 1: Reasoning Is Not Verification What Happens The model writes steps that look like a careful calculation, but one step contains an error. The final answer is wrong, even though the structure of the reasoning is correct.

Example Step 1: 18% GST on ₹7,200 = ₹1,206. Step 2: Total = 7,200 + 1,206 = ₹8,406. The method is correct, but the arithmetic in step 1 is wrong: 18% of 7,200 is 1,296, so the correct total is ₹8,496.

Why It Happens Each step is still generated by predicting likely tokens (Lesson 2). Writing steps makes errors less likely and easier to see, but the model is not executing a calculator. A step that "looks like" a correct calculation can still contain wrong digits.

What to Do Compute exact values in code, and use the model to set up or explain the calculation (Lesson 58, Program-Aided Language Models). Verify critical intermediate results automatically where possible. Use multiple independent attempts and compare answers (Self-Consistency, Lesson 45). Limitation 2: Errors Propagate Through the Chain What Happens An early mistake, such as misreading a value or misunderstanding a condition, is carried forward. Every later step builds correctly on the wrong foundation, producing a confidently wrong result.

Example Problem: 15% of the 60 students leave. One-third of the REMAINING students choose evenings. How many are in the day batch?

Step 1: One-third of 60 choose evenings = 20. ← misread: should use remaining students Step 2: 15% of 60 leave = 9. Step 3: Day batch = 60 − 20 − 9 = 31. The correct answer is 34. Steps 2 and 3 are arithmetically correct, but step 1 applied "one-third" to the wrong number, and the error passed through to the end.

Why It Happens The model conditions each step on the previous ones (Lesson 39). This is what makes CoT useful, but it also means the chain has no built-in mechanism to go back and correct an earlier step.

What to Do Ask the model to restate the problem's values and conditions first, before calculating (Plan-and-Solve, Lesson 40). Specify the steps in the right order when you know the procedure (structured CoT, Lesson 39). Use techniques that explore or compare multiple paths (Lessons 45 and 47). Add a separate verification step that checks the reasoning against the original problem (Lesson 10, Tip 6). Limitation 3: Explanations May Be Unfaithful What Happens The written reasoning presents a sensible justification, but the answer was actually influenced by something the reasoning never mentions.

Evidence Research by Turpin et al. (2023), "Language Models Don't Always Say What They Think", showed this directly. In one experiment, the few-shot examples were arranged so that the correct answer was always option (A). Models became more likely to choose (A) for new questions, even when it was wrong, and their written chain-of-thought explanations gave other reasons and did not mention the pattern that influenced them.

Why It Matters A faithful explanation accurately describes the reasons for a decision. If CoT explanations can be unfaithful, then:

Reading the reasoning does not guarantee you know why the model answered as it did. Biases in prompts, examples, or inputs can influence decisions without appearing in the reasoning. Using CoT explanations as justification for important decisions (for example, about people) is risky. What to Do Treat CoT as a tool for accuracy and debugging, not as a reliable record of the model's decision process. Test for bias directly: change irrelevant details (names, order of options, example patterns) and check whether decisions change (Lesson 30). For decisions affecting people, require human review and decision criteria that can be checked independently of the model's explanation. Limitation 4: Confident Reasoning Toward Wrong Conclusions What Happens The model builds a coherent, persuasive argument for an incorrect answer, sometimes "reasoning itself out of" a correct initial judgment.

Why It Happens Models are skilled at producing fluent, logical-sounding text. Once a reasoning path starts in a wrong direction, each additional step makes that direction look more established. Research has also identified some tasks where adding chain of thought reduced accuracy (Lesson 43).

What It Looks Like The policy says refunds are allowed within 7 days. Although the customer bought the course 9 days ago, they first contacted support on day 6, so the request was effectively made within 7 days. Decision: Eligible. This sounds reasonable, but if the policy does not say that contacting support counts as a refund request, the model has invented a rule to reach a conclusion.

What to Do Instruct the model to use only the stated rules: Apply only the rules listed. Do not assume exceptions that are not stated. Check reasoning for claims that are not supported by the input, using a separate checking prompt (Lesson 20). Treat persuasive reasoning with the same skepticism as any other model output. Limitation 5: Reasoning Cannot Create Missing Knowledge What Happens The model reasons carefully but uses an incorrect or invented fact, such as a wrong tax rate or an outdated API behavior, and reaches a well-reasoned wrong answer.

Why It Happens Chain of thought improves how the model uses information, not what information it has. If a fact is missing or wrong, careful reasoning applies the wrong fact carefully.

What to Do Provide the necessary facts in the prompt (Lesson 6), use retrieval for larger knowledge bases (Lessons 101–106), and verify facts that the reasoning depends on.

Limitation 6: Cost and Latency What Happens Responses become much longer. Output tokens increase, often by several times, and users wait longer.

Why It Matters Output tokens are typically more expensive than input tokens. Interactive features may become noticeably slower. At high volume, the extra cost can be significant. What to Do Use CoT only for tasks that benefit from it (Lesson 43). Keep reasoning concise: specify short steps and show concise examples (Lesson 41). Use escalation: answer directly first, and use CoT only when validation fails (Lesson 43). Limitation 7: Different Runs Give Different Reasoning What Happens The same prompt produces different reasoning paths on different runs, and sometimes different final answers.

Why It Happens Models usually sample tokens from probabilities (Lesson 2). An early difference, such as which value is computed first, can lead to a completely different path.

What to Do Specify the reasoning procedure to reduce variation (structured CoT). Use self-consistency: sample several reasoning paths and choose the most common answer (Lesson 45). This turns the variation into a strength. Treat disagreement between runs as a signal that the input may be ambiguous or difficult. Limitation 8: Sensitivity to Examples and Wording What Happens Reasoning quality changes with the trigger phrase, the examples, or small wording changes in the instructions.

Why It Happens In few-shot CoT, the model copies the reasoning method shown in examples, including mistakes and inappropriate patterns (Lesson 41). In zero-shot CoT, different phrasings encourage different reasoning styles (Lesson 40).

What to Do Verify every step of reasoning examples. Compare phrasings and examples on a test set instead of assuming one is best. Freeze tested prompts and treat changes as new versions (Lesson 12). Limitation 9: Exposure Risks What Happens The written reasoning reveals information that should not be shown to users: internal rules, scoring thresholds, other users' data provided in context, or the existence of system instructions.

Example A support assistant's reasoning might include: "According to the internal escalation rule, customers on the Free plan are deprioritized, so..." If this reasoning is shown to the customer, it exposes internal policy.

What to Do Separate reasoning from the user-facing answer with tags (Lesson 37) and show only the answer by default. Do not place information in the context that the user must never see if there is any chance the reasoning or answer will be displayed. Review what is logged and who can access stored reasoning. Limitation 10: Not Always Available or Needed What Happens With some modern reasoning models, the internal reasoning is not visible, or only a summary is provided. Explicit CoT instructions may also be unnecessary or discouraged by the provider (Lesson 39).

Why It Matters You may not be able to inspect the full reasoning for debugging. Prompts designed around visible CoT may need adjustment when switching models. What to Do Follow the provider's guidance for the specific model. For debugging, ask for a short explanation of key decisions in the final answer, while remembering that explanations may be unfaithful (Limitation 3). Re-evaluate prompts when changing models (Lesson 43). A Practical Defense Strategy No single technique solves all of these limitations. Combine them according to the importance of the task.

Layer Purpose Lessons Clear, structured reasoning instructions Reduce misreading and skipped steps 39, 40 Facts provided in context Avoid reasoning on wrong knowledge 6, 19 Code for exact computation Remove arithmetic errors 58 Multiple reasoning paths Detect and reduce random errors 45, 52 Separate verification step Catch unsupported claims and logic errors 10, 20 Output validation in code Catch malformed or implausible answers 8 Human review Catch what automation misses in high-stakes decisions 127 Example: Choosing Layers by Risk Use Case Suggested Layers Explaining a concept to a student Structured CoT, facts in context Calculating a fee shown to a customer Model extracts values, code calculates, output validated Deciding eligibility for a scholarship Structured CoT, verification step, human review for rejections Common Mistakes Mistake 1: Treating Reasoning as Proof What it is: Accepting an answer because its reasoning looks correct.

Why it happens: Clear steps feel convincing.

What it causes: Errors in individual steps or unfaithful reasoning go unnoticed.

How to fix it: Verify results independently, especially calculations and facts.

Mistake 2: Using CoT Explanations to Justify Decisions About People What it is: Showing model-generated reasoning as the official reason for accepting or rejecting an applicant, student, or customer.

Why it happens: The explanation is readily available and looks professional.

What it causes: Decisions justified by explanations that may not reflect the actual influences, including biases.

How to fix it: Use explicit, checkable criteria and human review for such decisions.

Mistake 3: Displaying Raw Reasoning to Users What it is: Showing the full reasoning text in user-facing responses.

Why it happens: It is the simplest implementation.

What it causes: Exposure of internal rules or data, and confusing output.

How to fix it: Separate reasoning from the answer and display only what users should see.

Mistake 4: Expecting CoT to Fix Knowledge Problems What it is: Adding CoT to a prompt that fails because of missing or outdated facts.

Why it happens: CoT is seen as a general accuracy improvement.

What it causes: More elaborate wrong answers.

How to fix it: Diagnose the failure type first (Lesson 24) and provide the missing information.

Best Practices Use CoT to improve accuracy, not to guarantee it. Verification remains necessary.

Restate inputs before reasoning. It reduces early errors that propagate.

Restrict reasoning to stated rules and facts. Prevent invented exceptions.

Compute exact values in code. Let the model handle interpretation and setup.

Compare multiple reasoning paths for important answers. Disagreement signals difficulty.

Keep reasoning private by default. Separate it from user-facing output.

Do not treat explanations as faithful records. Test for bias directly.

Combine safeguards according to risk. Higher stakes require more layers.

Try It Yourself Exercise 1: Find the Broken Step Ask a model to solve five multi-step problems with step-by-step reasoning. Verify each step with Python. Record how many final answers were wrong, and for each wrong answer, which step introduced the error: a misread value, a wrong method, or an arithmetic slip.

Exercise 2: Test for Unfaithful Reasoning Create a few-shot multiple-choice prompt where the correct answer in every example is option (A). Then ask new questions where the correct answer is not (A). Check whether the model is drawn toward (A), and whether its reasoning ever mentions the pattern in the examples.

Exercise 3: Separate and Protect Build a prompt for a support assistant that uses internal rules in its reasoning. Use tags to separate reasoning and the user-facing reply. Write code that displays only the reply, and test with a question that would tempt the model to mention the internal rules in its reply. Add an instruction to prevent this and test again.