Exploring Multiple Solutions with Tree of Thoughts

From Puzzles to Real Decisions Lesson 47 introduced Tree of Thoughts with the Game of 24, a puzzle with one exact goal and a perfect code-based evaluator. Most real problems are different:

There is no single correct answer, only better and worse solutions. Solutions must balance several criteria: cost, effort, risk, user experience. No code can calculate which solution is "correct". Examples include choosing an architecture, planning a course launch, designing a database schema, or deciding how to reduce student drop-off. For these problems, the Tree of Thoughts idea is still valuable in a practical form:

Generate several genuinely different approaches ↓ Evaluate each against explicit criteria ↓ Discard weak approaches, keep the strongest ↓ Develop the survivors in more detail ↓ Evaluate again and select (or combine) This lesson shows how to implement this explore → evaluate → prune → expand workflow with prompts and code.

Why Explore Instead of Asking for One Answer? The First Idea Is Rarely the Best One When asked for a single solution, a model usually produces the most common, most "typical" answer. That answer may be reasonable, but it is chosen because it is likely, not because it was compared with alternatives.

Exploration vs Self-Consistency Both techniques generate several outputs, but with opposite goals:

Aspect Self-Consistency (Lesson 45) Exploring Multiple Solutions Goal Find the answer most paths agree on Find the best of several different options Desired diversity Different reasoning paths, same answer Different solutions Selection method Majority vote Evaluation against criteria Suitable for Problems with one correct answer Open-ended problems with trade-offs Benefits Better solutions: the chosen option has been compared with alternatives. Visible trade-offs: the evaluation shows what each option gains and loses. Better decisions by people: the output is a comparison you can review, not a single recommendation you must accept or reject. Step 1: Define the Problem and the Criteria Exploration only works if you know how solutions will be judged. Write the criteria before generating options.

Example Problem A small learning platform built with Django needs to send class reminders to about 5,000 students, 30 minutes before each live class. Most students are in India and use mobile phones. The team has one backend developer and a monthly budget of about ₹5,000 for this feature. Example Criteria Criterion Weight Meaning Reach 3 How likely students are to actually see the reminder in time Cost 3 Monthly running cost within the ₹5,000 budget Implementation effort 2 Feasible for one developer Reliability 2 Reminders are sent on time, every time Why Weights? Weights state the priorities explicitly. Without them, the evaluation treats all criteria as equally important, which rarely reflects the real situation. Here, reach and cost matter most.

Step 2: Generate Genuinely Different Approaches The Diversity Problem If you simply ask for "three options", the model often returns three variations of the same idea, such as three email providers. Exploration only helps if the options differ in their core approach.

Prompt for Diverse Approaches Propose 4 approaches to the problem in .

Requirements:

• Each approach must use a different core mechanism (for example, a different delivery channel or a different system design), not a variation of another approach.
• For each approach, give a short name, a 2-sentence description, and the main assumption it depends on.

Return only a JSON array of objects with keys: name, description, assumption. Why Each Requirement Matters "Different core mechanism" prevents near-duplicate options. "Main assumption" surfaces what must be true for the approach to work. For example, "students have installed our mobile app". Assumptions are often the deciding factor in evaluation. Possible Approaches Name Core Mechanism Main Assumption Email reminders Email via a transactional email service Students check email shortly before class SMS reminders SMS via a provider SMS cost per message fits the budget WhatsApp messages WhatsApp Business messaging Template approval and per-message pricing are acceptable Push notifications Web or mobile push Students have granted notification permission Step 3: Evaluate Each Approach Against the Criteria Use a Separate Evaluation Prompt Generating and evaluating in the same prompt tends to favor the options the model just created with enthusiasm. A separate evaluation step, with explicit criteria, produces more balanced judgments (Lesson 10, Tip 6).

Evaluate the approach in for the problem in .

Score each criterion from 1 (poor) to 5 (excellent):

• reach: how likely students are to see the reminder within 30 minutes
• cost: how well the monthly cost fits a budget of about ₹5,000 for 5,000 students
• effort: how feasible it is for one backend developer
• reliability: how consistently reminders will be delivered on time

For each score, give a one-sentence reason. State any assumption you are unsure about instead of treating it as fact.

Return only JSON: {"scores": {"reach": n, "cost": n, "effort": n, "reliability": n}, "reasons": {...}, "uncertain_assumptions": [...]} Why These Details Matter Defined scale and criteria make scores comparable between approaches. Reasons make each score reviewable. Uncertain assumptions are important for real decisions. For example, the model does not know current SMS or WhatsApp prices; it should flag them for verification rather than presenting estimates as facts. Step 4: Combine Scores in Code and Prune Weighted totals and ranking are exact calculations, so they belong in code.

WEIGHTS = {"reach": 3, "cost": 3, "effort": 2, "reliability": 2}

def weighted_score(scores: dict[str, int]) -> float: total_weight = sum(WEIGHTS.values()) return sum(scores[c] * w for c, w in WEIGHTS.items()) / total_weight

evaluations = { "Email reminders": {"reach": 2, "cost": 5, "effort": 5, "reliability": 4}, "SMS reminders": {"reach": 5, "cost": 3, "effort": 4, "reliability": 4}, "WhatsApp messages": {"reach": 5, "cost": 3, "effort": 3, "reliability": 4}, "Push notifications": {"reach": 3, "cost": 5, "effort": 3, "reliability": 3}, }

ranked = sorted(evaluations, key=lambda name: weighted_score(evaluations[name]), reverse=True) for name in ranked: print(f"{weighted_score(evaluations[name]):.2f} {name}")

KEEP = 2 survivors = ranked[:KEEP] print("Expand:", survivors) Output:

4.00 SMS reminders 3.90 Email reminders 3.80 WhatsApp messages 3.60 Push notifications Expand: ['SMS reminders', 'Email reminders'] The scores in this example are illustrative, chosen to demonstrate the calculation.

Code Explanation WEIGHTS The priorities from Step 1, stored in one place.

weighted_score() Multiplies each criterion's score by its weight, sums the results, and divides by the total weight. This gives a weighted average on the same 1–5 scale, which is easier to interpret than a raw sum. For SMS: (5×3 + 3×3 + 4×2 + 4×2) ÷ 10 = 40 ÷ 10 = 4.00.

Ranking and pruning Approaches are sorted by weighted score, and only the top KEEP survive. This is the pruning step of Tree of Thoughts: weak branches are not developed further, which saves effort.

Look at Close Scores Carefully Scores of 4.00, 3.90, and 3.80 are close. Model-generated scores are approximate judgments, not measurements. When scores are close, treat the options as roughly equivalent and let the reasons and uncertain assumptions decide. A sensible rule is to keep any option within a small margin of the top score, not only a fixed number.

Watch for Critical Weaknesses Email survives here with a reach score of only 2, because its low cost and low effort compensate in the weighted average. But a reminder that students do not see in time fails the main purpose of the feature. Weighted averages can hide a critical weakness like this. A common solution is a minimum threshold: reject any option that scores below a set value on an essential criterion, before ranking.

MINIMUMS = {"reach": 3}

eligible = [ name for name, scores in evaluations.items() if all(scores[c] >= minimum for c, minimum in MINIMUMS.items()) ] With this rule, Email is excluded, and the top two eligible options become SMS (4.00) and WhatsApp (3.80).

Step 5: Expand the Survivors The surviving approaches are now developed in more detail. This is the next level of the tree.

Develop the approach in into an implementation plan for the problem in .

Include:

1. The main components and how they connect to the existing Django application
2. How reminders are scheduled 30 minutes before each class
3. The main risks and how to reduce each
4. What must be verified before committing (prices, approvals, limits)

Keep it under 250 words. Why Expand Only the Survivors Detailed plans are expensive to produce and to read. Developing only the most promising approaches is the efficiency gain of tree search: depth is spent only on branches worth exploring.

Combining Branches Expansion sometimes reveals that the best solution combines approaches. For example, a detailed plan might show that sending push notifications to students who have the app installed, and SMS only to the rest, keeps reach high while reducing cost. Exploration is valuable precisely because it makes such combinations visible.

Step 6: Final Comparison and Decision The final step compares the expanded plans, ideally in a format suitable for a human decision:

Compare the implementation plans in and . Create a table with rows: monthly cost estimate, implementation time, reach, main risk, items to verify. Then recommend one plan, or a combination, in 3 sentences, stating the assumption the recommendation depends on most. The output is a decision document, not just an answer: it shows options, trade-offs, and the key assumption to verify, such as current SMS pricing.

The Complete Workflow in Python Requirements: pip install openai and an OPENAI_API_KEY environment variable.

import json

from openai import OpenAI

client = OpenAI() MODEL = "gpt-5.4" # Replace with a model available in your account

WEIGHTS = {"reach": 3, "cost": 3, "effort": 2, "reliability": 2} MINIMUMS = {"reach": 3}

GENERATE = """Propose 4 approaches to the problem in . Each approach must use a different core mechanism, not a variation of another approach. Return only a JSON array of objects with keys: name, description, assumption."""

EVALUATE = """Evaluate the approach in for the problem in . Score each criterion from 1 (poor) to 5 (excellent): reach, cost, effort, reliability. Return only JSON: {"scores": {"reach": n, "cost": n, "effort": n, "reliability": n}, "uncertain_assumptions": [list of strings]}"""

EXPAND = """Develop the approach in into an implementation plan for the problem in . Include components, scheduling, main risks, and what must be verified before committing. Keep it under 250 words."""

def call(instructions: str, input_text: str) -> str: response = client.responses.create(model=MODEL, instructions=instructions, input=input_text) return response.output_text

def weighted_score(scores: dict) -> float: return sum(scores[c] * w for c, w in WEIGHTS.items()) / sum(WEIGHTS.values())

def meets_minimums(scores: dict) -> bool: return all(scores[c] >= minimum for c, minimum in MINIMUMS.items())

def explore(problem: str, keep: int = 2) -> list[dict]: problem_block = f"\n{problem}\n" approaches = json.loads(call(GENERATE, problem_block))

for approach in approaches: approach_block = f"\n{json.dumps(approach)}\n" evaluation = json.loads(call(EVALUATE, f"{problem_block}\n\n{approach_block}")) approach["scores"] = evaluation["scores"] approach["uncertain"] = evaluation.get("uncertain_assumptions", []) approach["score"] = weighted_score(evaluation["scores"])

eligible = [a for a in approaches if meets_minimums(a["scores"])] eligible.sort(key=lambda a: a["score"], reverse=True) survivors = eligible[:keep]

for approach in survivors: approach_block = f"\n{json.dumps(approach)}\n" approach["plan"] = call(EXPAND, f"{problem_block}\n\n{approach_block}") return survivors

problem = ( "A small Django learning platform must send class reminders to about 5,000 " "students, 30 minutes before each live class. Most students are in India and " "use mobile phones. One backend developer; budget about ₹5,000 per month." )

for option in explore(problem): print(f"\n=== {option['name']} (score {option['score']:.2f}) ===") print("Verify:", option["uncertain"]) print(option["plan"]) Code Explanation Three prompts, three roles GENERATE explores, EVALUATE judges, and EXPAND develops. Each prompt has one job (Lesson 10, Tip 9).

explore()

Generates diverse approaches. Evaluates each one separately, with the problem and the approach clearly delimited. Removes approaches that fail a minimum threshold, then computes weighted scores in code and sorts the rest. Expands only the top keep approaches into plans. uncertain The evaluator's list of assumptions to verify is kept with each option and printed with the plan, so a person reviewing the result knows what must be checked before deciding.

Cost For 4 approaches and 2 survivors, the workflow makes 1 + 4 + 2 = 7 requests. This is far more than a single answer, which is why the technique is suited to important decisions rather than routine requests.

Note: For production use, add JSON extraction and validation (Lessons 8 and 22), and check that scores are integers from 1 to 5 before computing weighted totals.

Making Evaluations More Reliable Model-based evaluation is the weakest part of this workflow, because there is no exact evaluator as in the Game of 24. Several techniques improve it:

Technique How It Helps Explicit criteria with definitions Reduces vague, inconsistent scoring Separate evaluation prompts Avoids bias toward options just generated Evaluate each option independently Prevents scores from being influenced by the order of options Multiple evaluation samples, averaged Reduces random variation in scores (similar to Lesson 45) Minimum thresholds for essential criteria Prevents averages from hiding critical weaknesses Ask for reasons and uncertain assumptions Makes judgments reviewable and highlights what to verify Provide real data in context Replaces guesses with facts, such as actual prices or user statistics Comparing outputs with rubrics is covered in depth in Lesson 116.

Common Mistakes Mistake 1: Near-Duplicate Options What it is: Generated approaches are variations of one idea.

Why it happens: The prompt does not require different core mechanisms.

What it causes: Exploration provides no real comparison.

How to fix it: Require different mechanisms and check the options before evaluating them.

Mistake 2: Criteria Defined After Seeing the Options What it is: Choosing evaluation criteria that favor an option you already like.

Why it happens: Criteria are written last.

What it causes: A biased process that only confirms a preference.

How to fix it: Define criteria and weights before generating options.

Mistake 3: Treating Scores as Measurements What it is: Choosing an option because it scored 4.00 instead of 3.90.

Why it happens: Numbers look precise.

What it causes: Decisions based on noise in model judgments.

How to fix it: Treat close scores as ties and decide using reasons, assumptions, and verified facts.

Mistake 4: Relying Only on Weighted Averages What it is: Ranking options only by their weighted score.

Why it happens: A single number is easy to sort.

What it causes: An option with a fatal weakness, such as very low reach, can still rank highly.

How to fix it: Apply minimum thresholds to essential criteria before ranking.

Mistake 5: Accepting Model Estimates of Prices and Limits What it is: Using the model's estimates of service prices, rate limits, or approval requirements as facts.

Why it happens: The evaluation includes them confidently.

What it causes: Decisions based on outdated or invented information.

How to fix it: Ask the evaluator to flag uncertain assumptions, and verify them against official provider documentation.

Best Practices Define criteria, weights, and minimums first. They make evaluation consistent and honest.

Require genuinely different approaches. Different mechanisms, not variations.

Evaluate in a separate step. Generation and judgment are different jobs.

Calculate weighted scores in code. Use the model for judgments, code for arithmetic.

Prune, then expand. Spend detailed work only on promising options.

Look for combinations. The best solution may merge strengths of several branches.

Keep humans in the decision. Produce a comparison with trade-offs and assumptions to verify, not just a single answer.

Try It Yourself Exercise 1: Explore a Real Decision Choose a decision from your own work, such as how to host a new web application, how to structure a course, or how to collect student feedback. Define three to four weighted criteria and at least one minimum threshold, then run the explore() workflow. Review whether the generated approaches were genuinely different.

Exercise 2: Stability of Scores Run the evaluation step three times for the same approach and compare the scores. Modify the workflow to average the scores from three evaluations, and check whether the ranking of approaches becomes more stable.

Exercise 3: Find a Combination After expanding the top two approaches, write a prompt that asks whether combining them would score better on the criteria than either alone, and why. Evaluate the combined approach with the same EVALUATE prompt and compare its weighted score.