When Chain-of-Thought Prompting Helps
The Question This Lesson Answers Chain-of-thought prompting (Lessons 39–42) can significantly improve accuracy, but it always has costs: more output tokens, higher latency, and more text to parse. Applying it to every prompt wastes resources and can even make some results worse.
This lesson explains which tasks benefit from chain of thought, which do not, why, and how to measure the difference for your own tasks.
The core principle:
Chain of thought helps when the correct answer depends on intermediate results that the model would otherwise have to produce implicitly in a single step.
If a task needs several dependent steps, writing them down helps. If the answer can be produced directly from the input, writing steps adds cost without benefit.
Tasks Where Chain of Thought Usually Helps
- Multi-Step Arithmetic and Calculations Why it helps: Each calculation depends on the previous one. Writing intermediate results prevents mistakes such as applying a percentage to the wrong base (Lesson 39).
Example:
A course costs ₹6,000. Early registration gives 15% off. GST of 18% is added after the discount, and a ₹200 coupon is applied at the end. What is the final price? The correct answer requires four ordered steps: ₹6,000 − 15% = ₹5,100; + 18% GST = ₹6,018; − ₹200 = ₹5,818.
- Applying Several Rules or Conditions Why it helps: Each rule must be checked, and the results combined correctly. Without explicit steps, a model may check the most obvious rule and skip the others.
Example: eligibility decisions, policy checks, approval workflows (Lesson 41).
- Logic and Constraint Problems Why it helps: Scheduling, ordering, and puzzles require tracking several constraints at the same time. Writing them down keeps all constraints visible.
Example:
Three mentors (Arun, Priya, Karthik) must each take one session: Monday, Tuesday, or Wednesday. Priya cannot take Monday. Karthik must take a day after Arun. Who takes which day? Reasoning: Karthik must come after Arun, so Arun cannot take Wednesday. Priya cannot take Monday. If Arun takes Monday, then Priya and Karthik share Tuesday and Wednesday, and Karthik can take either. If Arun takes Tuesday, Karthik must take Wednesday, leaving Monday for Priya, which is not allowed. So Arun takes Monday, and two solutions remain: Priya Tuesday and Karthik Wednesday, or Karthik Tuesday and Priya Wednesday. Step-by-step reasoning reveals that the problem has two valid answers, which a direct answer would likely hide.
- Multi-Hop Questions Why it helps: The answer requires combining several facts, where each fact leads to the next.
Example:
Using the course catalog, which instructor teaches the course that is a prerequisite for "Advanced Django"? The model must first find the prerequisite, then find its instructor.
Tracing and Debugging Code Why it helps: Predicting what code does requires tracking variable values through each line. Writing the trace makes errors visible (Lesson 39).
Planning and Decomposition Why it helps: Plans have dependencies: some steps must happen before others. Reasoning first produces more coherent plans (Lesson 40, Plan-and-Solve).
Judgment Tasks With Several Factors Why it helps: Decisions such as code review severity or ticket escalation weigh several factors. Reasoning first prevents the model from committing to a conclusion before considering them (Lesson 10, Tip 4).
Tasks Where Chain of Thought Usually Does Not Help
Simple Factual Lookups What does HTTP status code 404 mean? There are no intermediate steps. Reasoning adds length without improving the answer.
Clear-Cut Classification Classify the sentiment: "Absolutely loved the course!" The answer is obvious from the text. CoT may help on borderline classification cases, but for clear cases it only adds cost.
Translation and Rewriting Translating or rewriting text is a direct transformation. Step-by-step reasoning about a translation usually adds nothing and can make the output harder to extract.
Simple Extraction Extract the email address from this message. The value is either present or not. No reasoning chain is needed.
Creative Generation Writing a tagline, a poem, or a social media post does not depend on a chain of logical steps. Planning can sometimes help with long structured writing, such as an outline before an article (Lesson 15), but step-by-step reasoning is not generally useful for short creative output.
Tasks Where Overthinking Hurts Research has also identified tasks where adding chain of thought reduced accuracy, especially tasks that rely on quick pattern recognition or intuitive judgment rather than explicit logic. Reasoning can lead the model to construct a plausible but wrong argument and talk itself out of a correct initial judgment. This is another reason to measure rather than assume.
Decision Table Task Characteristic CoT Likely Helpful? Several dependent calculations ✓ Several rules or conditions to combine ✓ Constraints that must be satisfied together ✓ Answer requires connecting multiple facts ✓ Requires tracing a process step by step ✓ Answer is directly visible in the input ✗ Single fact from general knowledge ✗ Direct transformation (translate, rewrite, format) ✗ Short creative output ✗ Clear-cut classification ✗ (consider for borderline cases only) A Quick Test Ask yourself:
"Would a careful person need scratch paper to solve this?"
If yes, chain of thought is likely to help. If a person would answer immediately, it probably will not.
The Role of the Model Model Size and Capability The original chain-of-thought research found that the benefit appeared mainly in large models; smaller models often produced reasoning that looked fluent but did not improve accuracy (Lesson 39). With today's models, CoT is generally effective for capable instruction-following models, but results still vary between models.
Reasoning Models Models designed to reason internally before answering already perform a form of chain of thought. For them:
Adding "think step by step" may provide little or no improvement. Some providers recommend against explicit step-by-step instructions for these models. Clear task definitions, constraints, and output formats remain important. The practical consequence: the same CoT instruction can help one model and be unnecessary for another. Re-evaluate when you change models.
Costs to Consider Cost Why It Matters Output tokens Reasoning can be several times longer than the answer; output tokens are often priced higher than input tokens Latency Longer outputs take longer to generate; users wait longer Parsing The answer must be extracted from a longer response Display Raw reasoning may need to be hidden from users For a background job that runs overnight, latency may not matter. For a chat interface or an autocomplete feature, a few extra seconds can matter a great deal.
Measuring Whether CoT Helps Your Task The reliable way to decide is to compare a direct prompt and a CoT prompt on the same test set, measuring accuracy, output tokens, and latency.
Requirements: pip install openai and an OPENAI_API_KEY environment variable.
import re import time
from openai import OpenAI
client = OpenAI() MODEL = "gpt-5.4" # Replace with a model available in your account
DIRECT = "Solve the problem. Reply with the final number only, with no units."
COT = ( "Solve the problem. Reason step by step inside tags, then give " "only the final number, with no units, inside tags." )
TEST_SET = [ ("A course costs 6000 rupees. Early registration gives 15% off. GST of 18% " "is added after the discount, then a 200 rupee coupon is applied. " "What is the final price?", "5818"), ("A batch starts with 60 students. 15% leave in month one. One-third of the " "rest move to evenings. How many stay in the day batch?", "34"), ("What is 25 multiplied by 4?", "100"), ("A workshop has 80 seats. 60% are booked online and 12 at the venue. " "How many seats are free?", "20"), ]
def extract_answer(text: str, use_tags: bool) -> str: if use_tags: match = re.search(r"\s*(.?)\s", text, re.DOTALL) text = match.group(1) if match else "" number = re.search(r"-?\d+(?:.\d+)?", text.replace(",", "")) return number.group() if number else ""
def evaluate(name: str, instructions: str, use_tags: bool) -> None: correct, output_tokens, seconds = 0, 0, 0.0 for problem, expected in TEST_SET: start = time.perf_counter() response = client.responses.create(model=MODEL, instructions=instructions, input=problem) seconds += time.perf_counter() - start output_tokens += response.usage.output_tokens if extract_answer(response.output_text, use_tags) == expected: correct += 1 n = len(TEST_SET) print(f"{name}: {correct}/{n} correct | " f"avg output tokens {output_tokens / n:.0f} | avg latency {seconds / n:.1f}s")
evaluate("direct", DIRECT, use_tags=False) evaluate("chain-of-thought", COT, use_tags=True) Code Explanation DIRECT and COT Two versions of the same task. The CoT version asks for reasoning in tags and the answer in separate tags (Lesson 39).
TEST_SET Problems with known answers. It mixes multi-step problems with one simple calculation (25 × 4), which helps show where CoT matters and where it only adds cost.
extract_answer()
For the CoT version, it first extracts the content of tags. It then extracts the first number, after removing thousands separators, so answers like "5,818" match "5818". time.perf_counter() A high-resolution timer used to measure how long each request takes, including network time.
response.usage.output_tokens The number of tokens the model generated. Comparing this between the two versions shows the real cost of the reasoning.
Reading the Results Outcome Interpretation CoT is more accurate on multi-step problems CoT is worth its cost for this task Both are equally accurate Use the direct prompt; CoT only adds cost CoT is less accurate The reasoning may be leading the model astray, or the answer extraction is failing; investigate before deciding CoT is more accurate but too slow Consider CoT only for hard inputs, or compute the result in code For real decisions, use a larger test set than this demonstration, with inputs from your actual application (Lesson 115).
Using CoT Selectively You do not have to choose CoT for all or none of your requests.
Strategy 1: Route by Task Type Use CoT for task types that need it (calculations, eligibility checks) and direct prompts for others (extraction, translation) within the same application.
Strategy 2: Escalate Hard Cases Answer directly first. If validation fails, for example the answer is not a valid number or a consistency check fails, retry the request with CoT. Most requests stay fast and cheap; only difficult cases pay for reasoning.
Direct answer → validate → valid? → done → invalid? → retry with chain-of-thought → validate → done or flag Strategy 3: Replace CoT With Code Where Possible If the steps are exact calculations on known values, use the model to extract the values and code to compute the result. This is often both cheaper and more accurate than any form of CoT. This idea is developed further in Program-Aided Language Models (Lesson 58).
Common Mistakes Mistake 1: Adding CoT to Every Prompt What it is: Including "think step by step" by default.
Why it happens: CoT is known to improve reasoning, so it seems safe to always use.
What it causes: Higher costs and latency for tasks that do not benefit, and occasionally lower accuracy.
How to fix it: Use the decision table and measure.
Mistake 2: Judging by How Convincing the Reasoning Looks What it is: Choosing CoT because the outputs look more thorough.
Why it happens: Detailed reasoning feels more trustworthy.
What it causes: Paying for reasoning that does not improve correctness.
How to fix it: Compare accuracy on a labeled test set, not impressions.
Mistake 3: Ignoring Latency in User-Facing Features What it is: Using long reasoning in interactive features.
Why it happens: Testing focuses on accuracy only.
What it causes: Slow responses and a poor user experience.
How to fix it: Measure latency and use selective strategies such as escalation.
Mistake 4: Not Re-Evaluating After Model Changes What it is: Keeping CoT instructions after switching to a model that reasons internally.
Why it happens: Prompts are carried over unchanged.
What it causes: Extra cost without benefit, or reduced performance if the instructions conflict with the model's own reasoning approach.
How to fix it: Re-run the comparison whenever you change models.
Best Practices Use CoT for tasks with dependent steps. Calculations, rule combinations, constraints, multi-hop questions, tracing, and planning.
Skip CoT for direct tasks. Lookups, clear classification, translation, extraction, and short creative writing.
Apply the scratch-paper test. Would a careful person need to write things down?
Measure accuracy, tokens, and latency together. Decide based on data from your own task.
Use CoT selectively. Route by task type or escalate only hard cases.
Prefer code for exact computation. Let the model extract values and code calculate.
Re-evaluate when models change. Reasoning models may not need explicit CoT.
Try It Yourself Exercise 1: Classify Tasks For each task, decide whether CoT is likely to help and explain why using the scratch-paper test:
Converting a date from "15th Nov 2026" to "2026-11-15" Deciding whether a student qualifies for a scholarship under four rules Translating a notification into Hindi Calculating the total cost of a course with installments and late fees Classifying a clearly positive review Exercise 2: Run the Measurement Extend TEST_SET with 10 problems from your own domain: five multi-step problems and five simple ones. Run the comparison and record accuracy, average output tokens, and latency separately for the simple and multi-step groups.
Exercise 3: Implement Escalation Write a function that first calls the direct prompt, validates that the result is a number within a plausible range, and only if validation fails retries with the CoT prompt. Run it on your test set and record how many requests needed escalation and how the total token usage compares with always using CoT.