Maieutic Prompting
What Is Maieutic Prompting? Maieutic prompting is a technique for answering true/false questions more reliably by asking the model to explain both possible answers, then checking which explanations the model consistently believes, and choosing the answer best supported by consistent explanations.
The name comes from the Socratic method, called maieutics in Greek: a way of reaching the truth by asking questions that test whether a person's beliefs are consistent with each other. Instead of trusting the model's first answer, maieutic prompting questions the model's own explanations.
The technique was introduced in the 2022 paper "Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations" by Jung et al. It was designed for commonsense and factual true/false questions, where models often give inconsistent answers.
The Problem It Addresses Models Can Explain Anything Ask a model to explain why a statement is true, and it will usually produce a fluent explanation. Ask it to explain why the same statement is false, and it will often produce a fluent explanation for that too.
Statement: "Python's list.sort() returns the sorted list."
Explanation for TRUE: "list.sort() returns the sorted list, so you can write sorted_items = items.sort()."
Explanation for FALSE: "list.sort() sorts the list in place and returns None." Both explanations sound confident. Only one is correct: list.sort() sorts in place and returns None; the separate function sorted() returns a new sorted list.
The Insight A model's ability to produce an explanation does not mean it believes that explanation consistently. Maieutic prompting tests each explanation:
Does the model agree the explanation is true? Does the model agree the opposite of the explanation is false? An explanation that passes both checks is logically consistent for the model. An explanation where the model agrees with both the statement and its opposite is unreliable and should be discarded.
How Maieutic Prompting Works The original method has three main parts.
- Abductive Explanation Generation Abductive reasoning means finding the explanation that would make a conclusion true. For a question Q, the model generates:
Q is true, because ... Q is false, because ... Each generated explanation is itself a statement that can be questioned.
Recursive Explanation (the Maieutic Tree) For explanations that are not yet reliable, the method asks for explanations of the explanations, building a tree of statements. The recursion continues until the statements are ones the model treats consistently.
Logical Integrity and Consistency Solving A statement is logically integral when the model's judgments about it are consistent: it judges the statement true and its negation false, or the statement false and its negation true.
The original method then:
Assigns weights to statements based on the model's confidence (belief) and the consistency of relationships between statements. Uses a MAX-SAT solver, a program that finds the assignment of true/false values to all statements that satisfies as many weighted constraints as possible. Reads the answer to the original question from that assignment. Overview Question Q ├── "Q is true, because E1" │ ├── Check E1 and NOT E1 → consistent? │ └── (if not, explain E1 further...) └── "Q is false, because E2" ├── Check E2 and NOT E2 → consistent? └── (if not, explain E2 further...)
Keep only consistent statements → weigh them → find the most consistent overall answer A Simplified Version You Can Implement The full method, with recursion and a MAX-SAT solver, is complex. A simplified version captures its central idea and is practical to build:
Generate one explanation for "true" and one for "false". For each explanation, generate its negation. Ask the model, separately, whether the explanation is true and whether the negation is true. Keep only explanations with consistent judgments. Choose the answer supported by consistent explanations the model believes. If none qualify, return "Uncertain". Note: This simplified version omits the recursive tree and the MAX-SAT solver. It demonstrates the core principle of testing explanations for consistency, not the full algorithm from the paper.
Walking Through the Example Question: Does Python's list.sort() return the sorted list?
Supports Explanation (E) Model judges E Model judges NOT E Consistent? Believed? True list.sort() returns the sorted list, so you can assign its result. False True Yes No False list.sort() sorts the list in place and returns None. True False Yes Yes The "true" explanation is consistent but not believed: the model judges it false and its negation true. It provides no support. The "false" explanation is consistent and believed. It supports the answer False. Result: False. list.sort() does not return the sorted list.
Implementing the Simplified Version in Python Requirements: pip install openai and an OPENAI_API_KEY environment variable.
from openai import OpenAI
client = OpenAI() MODEL = "gpt-5.4" # Replace with a model available in your account
def ask(instructions: str, text: str) -> str: response = client.responses.create(model=MODEL, instructions=instructions, input=text) return response.output_text.strip()
def explain(question: str, side: str) -> str: return ask( f"Write one sentence explaining why the answer to the question is {side}. " "Write only the explanation as a factual statement.", question, )
def negate(statement: str) -> str: return ask( "Write the logical negation of the statement as one clear sentence. " "Write only the negated statement.", statement, )
def judge(statement: str) -> bool | None: answer = ask("Is the statement true? Reply with exactly one word: True or False.", statement) if answer.lower().startswith("true"): return True if answer.lower().startswith("false"): return False return None
def belief(statement: str) -> bool | None: """Return True/False if the model judges the statement consistently, else None.""" judged = judge(statement) judged_negation = judge(negate(statement)) if judged is None or judged_negation is None or judged == judged_negation: return None # inconsistent: model agrees (or disagrees) with both return judged
def decide(supporting: list[tuple[bool, bool | None]]) -> str: """supporting: list of (answer the explanation supports, model's belief in it).""" support = {True: 0, False: 0} for supports, believed in supporting: if believed is True: support[supports] += 1 if support[True] == support[False]: return "Uncertain" return "True" if support[True] > support[False] else "False"
def maieutic_answer(question: str) -> str: results = [] for side, value in (("TRUE", True), ("FALSE", False)): explanation = explain(question, side) results.append((value, belief(explanation))) print(f"[{side}] {explanation} → belief: {results[-1][1]}") return decide(results)
print(maieutic_answer("Does Python's list.sort() return the sorted list?")) Possible output:
[TRUE] list.sort() returns the sorted list so it can be assigned to a variable. → belief: False [FALSE] list.sort() sorts the list in place and returns None. → belief: True False Code Explanation explain() Generates an abductive explanation for one side of the question. Asking for a "factual statement" makes the explanation something that can be judged as true or false.
negate() Asks the model to write the logical opposite of a statement. Writing the negation as a full sentence, rather than adding "not", avoids awkward or ambiguous negations.
judge() Asks for a single-word true/false judgment in a separate request, so the judgment is not influenced by the reasoning that produced the statement. Unclear replies return None.
belief() The logical integrity check:
If the model judges the statement and its negation differently, its belief is consistent, and the function returns the judgment of the statement. If both judgments are the same, for example both "True", the model's belief is contradictory, and the function returns None so the explanation is discarded. decide() Counts support only from explanations that are consistently believed. If neither side has more support, it returns "Uncertain" instead of guessing.
Testing the Decision Logic Without the API The decision step is pure logic, so it can be checked directly:
print(decide([(True, False), (False, True)])) # False: only the FALSE explanation is believed print(decide([(True, True), (False, None)])) # True: FALSE explanation was inconsistent print(decide([(True, None), (False, None)])) # Uncertain: no consistent explanation Output:
False True Uncertain Cost Each side requires one explanation, one negation, and two judgments: 4 requests per side, 8 in total. This is expensive compared with a single answer, so the technique is suited to questions where reliability matters and a wrong confident answer is costly.
When Maieutic Prompting Helps Good Fit Why True/false factual questions The method is designed for binary answers Common misconceptions The model may produce a plausible wrong explanation that fails consistency checks Questions where the model's first answer is unstable Consistency checking reveals whether beliefs are solid Verifying claims in generated content Each claim can be checked for consistent belief Poor Fit Why Open-ended questions No clear true/false structure Questions requiring missing knowledge Consistency cannot create facts the model lacks Calculations Code verification is more reliable High-volume, low-stakes tasks The number of requests is too high An Important Limit Maieutic prompting measures consistency, not truth. A model can consistently believe something false: it may judge a false statement true and its negation false. Consistency checks remove contradictory beliefs, but they cannot correct a belief that is consistently wrong. For important facts, verify against authoritative sources.
Using the Idea Without the Full Method Even without implementing the algorithm, the principle is useful in everyday prompting:
Ask for Both Sides Statement: "Indexes always make database queries faster."
- Give the strongest argument that the statement is true.
- Give the strongest argument that the statement is false.
- Decide which argument is better supported, and state the conclusion with any conditions under which it holds. This prevents the model from simply defending whichever answer it chose first, and often reveals that the correct answer is conditional. Here, indexes speed up many reads but add cost to writes and can be ignored by the query planner, so "always" is false.
Check Key Claims in Generated Content When a model produces an explanation with several factual claims, extract the key claims and ask about each one and its negation separately. Claims judged inconsistently are good candidates for manual verification.
Common Mistakes Mistake 1: Trusting Fluent Explanations What it is: Accepting an answer because its explanation sounds convincing.
Why it happens: Fluent explanations feel like evidence.
What it causes: Plausible but wrong answers, especially on common misconceptions.
How to fix it: Generate and test explanations for both answers.
Mistake 2: Judging Statements in the Same Context That Created Them What it is: Asking the model to evaluate an explanation in the same response or conversation where it wrote it.
Why it happens: It requires fewer requests.
What it causes: The model tends to agree with what it just wrote.
How to fix it: Judge each statement and negation in separate, independent requests.
Mistake 3: Forcing an Answer When Evidence Is Inconsistent What it is: Always returning True or False, even when no explanation is consistently believed.
Why it happens: The application expects a binary answer.
What it causes: Confident answers in exactly the cases where the model is least reliable.
How to fix it: Return "Uncertain" and route such cases to review or further verification.
Mistake 4: Equating Consistency With Correctness What it is: Treating a consistently believed explanation as verified fact.
Why it happens: Consistency checks feel like proof.
What it causes: Consistently held misconceptions are accepted.
How to fix it: Use consistency as a reliability signal and verify important facts independently.
Best Practices Generate explanations for both answers. Do not let the first answer frame the reasoning.
Test each explanation and its negation separately. Consistent judgments indicate a stable belief.
Discard inconsistent explanations. Agreement with both a statement and its opposite is not evidence.
Allow "Uncertain" as an outcome. It identifies questions that need verification.
Use the both-sides principle in everyday prompts. It reduces one-sided, overconfident answers.
Reserve the full method for important true/false questions. It requires many requests.
Try It Yourself Exercise 1: Test Common Misconceptions Run maieutic_answer() on five programming statements that are commonly misunderstood, for example: "In JavaScript, const makes an object immutable" or "Django's QuerySet.count() always loads all rows into memory". Check each result against official documentation, and record cases where the result was "Uncertain".
Exercise 2: Find Inconsistent Beliefs For ten factual statements from your field, call judge() on each statement and on its negation (using negate()). List the statements where both judgments were the same. Explain why these are the statements most worth verifying manually.
Exercise 3: Both-Sides Prompting Write a "both sides" prompt (as in the section above) for three statements that contain words such as "always", "never", or "best". Compare the conclusions with direct answers to the same statements, and note how often the both-sides version produced a more conditional, accurate answer.