Automatic Chain-of-Thought (Auto-CoT)

What Is Auto-CoT? Automatic Chain-of-Thought (Auto-CoT) is a method for building few-shot chain-of-thought prompts without writing the reasoning examples by hand. Instead, the model generates the reasoning examples itself, and an automatic selection process makes sure those examples are diverse.

It was introduced in the 2022 paper "Automatic Chain of Thought Prompting in Large Language Models" by Zhang et al.

The Problem It Solves Few-shot CoT (Lesson 41) works well, but it requires someone to write correct, varied reasoning examples for each type of task. This is slow and requires expertise. Zero-shot CoT (Lesson 40) needs no examples, but gives less control and sometimes produces flawed reasoning.

Auto-CoT tries to get the benefits of few-shot CoT while removing the manual work:

Approach Who Writes the Reasoning Examples? Effort Zero-shot CoT Nobody (no examples) Very low Few-shot CoT A human expert High Auto-CoT The model, using zero-shot CoT, with automatic selection Low after setup The Key Insight: Diversity Matters The obvious way to automate example creation would be:

Take the new question. Find the most similar questions from a collection. Generate reasoning for them with zero-shot CoT. Use them as examples. The Auto-CoT research found a problem with this approach. Zero-shot CoT sometimes produces wrong reasoning. When examples are chosen only for similarity, several examples may share the same type of mistake, and the model then repeats that mistake on the new question. The authors described this as being misled by similarity.

The Solution Choose examples that are diverse: questions from different groups, so that if the model makes a mistake on one type of question, that mistake affects only one example rather than all of them.

Diversity limits the damage of any single flawed example.

How Auto-CoT Works Auto-CoT has two stages.

Stage 1: Question Clustering Collect a set of questions for the task (without answers). Convert each question into a numerical representation of its meaning. The original paper used Sentence-BERT embeddings. Group the questions into k clusters with the k-means algorithm, where k is the number of examples you want. Each cluster contains questions of a similar type, such as percentage problems, time calculations, or rate problems.

Stage 2: Demonstration Sampling For each cluster:

Sort the questions by their distance to the cluster center, so the most representative questions come first. Take the first question that passes simple quality filters. Generate its reasoning with zero-shot CoT ("Let's think step by step"). Use the question, reasoning, and answer as one example. The Quality Filters The paper used simple heuristics to prefer examples that are more likely to be correct:

Short questions (the paper used a limit of 60 tokens). Short reasoning (the paper used a limit of 5 reasoning steps). Shorter, simpler examples are less likely to contain reasoning errors, and they keep the final prompt compact.

Overview Collection of questions ↓ Embed and cluster into k groups ↓ For each group: pick the most central question that passes the filters ↓ Generate reasoning for each picked question with zero-shot CoT ↓ Filter out reasoning that is too long ↓ k diverse reasoning examples ↓ Use them as few-shot CoT examples for new questions A Simplified Implementation in Python The following implementation demonstrates the Auto-CoT process. It uses TF-IDF vectors from scikit-learn instead of Sentence-BERT embeddings, so it runs with a common library and no extra model downloads. TF-IDF represents questions by their important words rather than their meaning, so it groups questions less accurately than embeddings, but the overall process is the same.

Requirements:

pip install openai scikit-learn numpy You also need an OPENAI_API_KEY environment variable.

Step 1: Cluster and Select Representative Questions import numpy as np from sklearn.cluster import KMeans from sklearn.feature_extraction.text import TfidfVectorizer

QUESTIONS = [ "A batch has 60 students and 15% leave. How many remain?", "A course costs 4000 rupees with a 10% discount. What is the final price?", "A student watches 3 lessons a day for 5 days. How many lessons in total?", "A workshop has 80 seats and 60% are booked. How many seats are free?", "A laptop costs 50000 rupees and GST is 18%. What is the total price?", "Priya studies 2 hours on weekdays and 4 hours on Saturday. How many hours per week?", "A course has 12 modules and a student finished 7. What percentage is complete?", "A train leaves at 9:40 and the trip takes 2 hours 35 minutes. When does it arrive?", "A class starts at 6:15 PM and lasts 90 minutes. When does it end?", "A coupon gives 500 rupees off a 2500 rupee course. What percent discount is that?", ]

def select_representatives(questions: list[str], k: int = 4, max_words: int = 25) -> list[str]: vectors = TfidfVectorizer().fit_transform(questions) kmeans = KMeans(n_clusters=k, n_init=10, random_state=0).fit(vectors) distances = kmeans.transform(vectors)

selected = [] for cluster in range(k): members = np.where(kmeans.labels_ == cluster)[0] ordered = sorted(members, key=lambda i: distances[i, cluster]) for i in ordered: if len(questions[i].split()) <= max_words: selected.append(questions[i]) break return selected

representatives = select_representatives(QUESTIONS) for question in representatives: print("-", question) Possible output:

• A batch has 60 students and 15% leave. How many remain?
• A course costs 4000 rupees with a 10% discount. What is the final price?
• A class starts at 6:15 PM and lasts 90 minutes. When does it end?
• A student watches 3 lessons a day for 5 days. How many lessons in total? The exact selection can differ between library versions, but the selected questions should come from different types of problems: percentages, prices, times, and totals.

Code Explanation TfidfVectorizer().fit_transform(questions) Converts each question into a vector. TF-IDF (term frequency–inverse document frequency) gives high weight to words that are frequent in one question but rare across the collection, so distinctive words like "GST", "arrive", or "seats" matter more than common words like "a" or "the".

KMeans(n_clusters=k, n_init=10, random_state=0)

n_clusters=k creates one cluster per example you want. n_init=10 runs the algorithm 10 times with different starting points and keeps the best result. random_state=0 makes the result reproducible. kmeans.transform(vectors) Returns, for each question, its distance to every cluster center. distances[i, cluster] is the distance of question i to the center of its own cluster.

Selection loop

np.where(kmeans.labels_ == cluster)[0] gets the indices of the questions in the cluster. Sorting by distance puts the most central, most representative question first. The first question that passes the length filter (max_words) is selected. The word limit is a simple stand-in for the token limit used in the paper. Step 2: Generate Reasoning With Zero-Shot CoT from openai import OpenAI

client = OpenAI() MODEL = "gpt-5.4" # Replace with a model available in your account

GENERATION_INSTRUCTIONS = ( "Solve the problem. Let's think step by step. " "Write each step on its own line, then end with a line in the form " "'The answer is X.'" )

def generate_demo(question: str, max_steps: int = 5) -> dict | None: response = client.responses.create( model=MODEL, instructions=GENERATION_INSTRUCTIONS, input=question, ) text = response.output_text.strip() lines = [line for line in text.splitlines() if line.strip()] steps = [line for line in lines if not line.startswith("The answer is")] has_answer = any(line.startswith("The answer is") for line in lines)

if not has_answer or len(steps) > max_steps: return None # Reject demonstrations that are incomplete or too long return {"question": question, "solution": "\n".join(lines)}

demos = [demo for q in representatives if (demo := generate_demo(q)) is not None] Code Explanation GENERATION_INSTRUCTIONS Uses the zero-shot CoT trigger and asks for one step per line, ending with a fixed answer sentence. The fixed format makes the steps countable and the answer extractable.

generate_demo()

Splits the response into non-empty lines. Counts the reasoning steps (lines that are not the answer line). Rejects the demonstration if it has no answer line or more than max_steps steps, mirroring the reasoning-length filter in the original method. demos = [...] Uses the "walrus" operator := (Python 3.8+) to call generate_demo() once per question and keep only accepted results.

Step 3: Build the Final Few-Shot CoT Prompt def build_auto_cot_instructions(demos: list[dict]) -> str: examples = "\n\n".join(f"Q: {d['question']}\nA: {d['solution']}" for d in demos) return ( "Solve the problem step by step, following the style of the examples. " "End with 'The answer is X.'\n\n" + examples )

def solve(question: str, instructions: str) -> str: response = client.responses.create( model=MODEL, instructions=instructions, input=f"Q: {question}\nA:", ) return response.output_text

AUTO_COT_INSTRUCTIONS = build_auto_cot_instructions(demos) print(solve("A batch has 45 students and one-third join the weekend batch. " "How many are in the weekday batch?", AUTO_COT_INSTRUCTIONS)) The generated examples are now used exactly like hand-written few-shot CoT examples (Lesson 41).

Improving on the Basic Method The original Auto-CoT method does not check whether generated reasoning is correct; it relies on diversity and simple filters. In practical applications, you can add stronger safeguards.

Verify Answers When Ground Truth Exists If you have correct answers for some questions (for example, from past data), select only demonstrations whose generated answer matches the known answer.

def answer_matches(solution: str, correct: str) -> bool: return f"The answer is {correct}" in solution Review Generated Demonstrations Once Auto-CoT generates examples once, during setup, not for every request. A human can review a handful of generated examples in minutes, which is much faster than writing them from scratch. This combines automation with human verification.

Use Better Similarity Measures For real use, replace TF-IDF with an embedding model, which groups questions by meaning. For example, "What is 15% of 60?" and "How many students remain if 9 of 60 leave?" use different words but are both percentage problems.

Regenerate When the Task Changes If the type of incoming questions changes, rerun the clustering and generation on a new collection so that the examples continue to represent the questions you actually receive.

Auto-CoT Compared With Other CoT Methods Aspect Zero-Shot CoT Manual Few-Shot CoT Auto-CoT Human effort Minimal High Low (setup and optional review) Needs a question collection No No Yes Reasoning quality of examples No examples High (if experts write them carefully) Variable; reduced risk through diversity and filters Diversity of examples Not applicable Depends on the author Systematic through clustering Setup complexity None Writing examples Clustering, generation, filtering Prompt cost per request Low Higher Higher (similar to manual few-shot) When Auto-CoT Is a Good Fit You have a collection of questions representative of the task, but no worked solutions. Many task types or domains need CoT examples, and writing them all manually is impractical. You want examples that cover the variety of real inputs systematically. When It Is Not Needed Zero-shot CoT already meets your accuracy requirements. The task is narrow enough that a few hand-written examples are easy to produce. The model you use is a reasoning model that performs well without CoT examples (Lesson 39). Common Mistakes Mistake 1: Selecting Examples Only by Similarity What it is: Choosing the generated examples most similar to each new question.

Why it happens: Similar examples seem most relevant.

What it causes: Several examples may share the same reasoning error, which the model then repeats. This is the problem Auto-CoT was designed to avoid.

How to fix it: Select examples from different clusters to ensure diversity.

Mistake 2: No Quality Filters What it is: Using every generated reasoning chain as an example.

Why it happens: Filtering seems unnecessary when the model generated the reasoning.

What it causes: Long, complex, or incomplete reasoning enters the prompt and increases errors and cost.

How to fix it: Filter by question length, number of steps, and presence of a clear answer.

Mistake 3: Unrepresentative Question Collection What it is: Clustering a small or unusual set of questions.

Why it happens: Using whatever sample data is easily available.

What it causes: Examples that do not reflect the questions the system will actually receive.

How to fix it: Use a collection drawn from real inputs, large enough to contain the main question types.

Mistake 4: Assuming Generated Examples Are Correct What it is: Treating model-generated reasoning as verified.

Why it happens: The process is automatic, so its output feels authoritative.

What it causes: Incorrect examples that teach wrong methods.

How to fix it: Verify answers against known results where possible, and review the final examples.

Best Practices Prioritize diversity when selecting examples. Clustering ensures that different question types are represented and limits the impact of any single error.

Use simple filters for length and completeness. Short, complete reasoning chains make better examples.

Generate examples once, then reuse them. Auto-CoT is a setup process, not a per-request step.

Verify and review where possible. Check answers against ground truth and review the small set of final examples.

Use embeddings for real systems. TF-IDF is fine for learning; embeddings group by meaning.

Compare with zero-shot CoT and manual examples. Use Auto-CoT only if it improves results on your test set (Lesson 43).

Try It Yourself Exercise 1: Run the Clustering Create a collection of 20 questions covering at least four types of problems from your domain, such as fee calculations, time calculations, percentages, and schedules. Run select_representatives() with k=4 and check whether each selected question comes from a different type. Then try k=6 and observe what changes.

Exercise 2: Generate and Filter Demonstrations Run the full pipeline. Print each generated demonstration, check its correctness manually, and record how many were rejected by the filters. Adjust max_steps and observe how it affects which demonstrations are kept.

Exercise 3: Compare Three Approaches On a test set of 15 new questions with known answers, compare zero-shot CoT, Auto-CoT, and a version where you replace the Auto-CoT examples with the most similar generated examples for each question (no clustering). Compare accuracy and note whether the similarity-based version repeats any specific error across several questions.