Generate Knowledge Prompting

What Is Generate Knowledge Prompting? Generate knowledge prompting is a two-step technique:

Knowledge generation: ask the model to write relevant facts about the question before answering it. Knowledge integration: ask the question again, this time with the generated facts included in the prompt, and let the model answer using them. In simple terms: before answering, the model "recalls what it knows" about the topic, and then uses that recalled knowledge to answer.

The technique was introduced in the 2022 paper "Generated Knowledge Prompting for Commonsense Reasoning" by Liu et al., which reported improved accuracy on several commonsense reasoning benchmarks.

Example Direct question:

Is a website that uses HTTPS always safe for entering your card details? Answer Yes or No. A model may answer Yes, because HTTPS is strongly associated with "secure" in everyday language.

Step 1 — Generate knowledge:

Write 3 short factual statements about what HTTPS does and does not guarantee. Possible output:

1. HTTPS encrypts the data sent between the browser and the website.
2. HTTPS confirms that the connection is to the domain shown in the address bar, but not that the website's owner is trustworthy.
3. Phishing and scam websites can also obtain certificates and use HTTPS. Step 2 — Answer with the knowledge:

Knowledge:

1. HTTPS encrypts the data sent between the browser and the website.
2. HTTPS confirms that the connection is to the domain shown in the address bar, but not that the website's owner is trustworthy.
3. Phishing and scam websites can also obtain certificates and use HTTPS.

Using the knowledge above, answer: Is a website that uses HTTPS always safe for entering your card details? Answer Yes or No, then explain in one sentence. Possible output:

No. HTTPS only protects the connection; a scam website can use HTTPS too, so you must also check that the site itself is legitimate. Example Explanation The direct question invites a quick association ("HTTPS = safe"). Generating knowledge first brings the relevant distinctions into the context: encryption is not the same as trustworthiness. With those facts written down, the correct answer becomes the most natural continuation.

Why It Works Recalled Facts Become Part of the Context As explained in Lesson 2, a model's answer depends on everything in its context. When a question is asked directly, the model must connect the question to relevant knowledge internally, in a single step. Generating the knowledge first makes that knowledge explicit text that the answer can build on, in the same way that chain-of-thought makes intermediate reasoning explicit (Lesson 39).

It Counters Surface Associations Many wrong answers come from strong word associations: "HTTPS" and "safe", "index" and "fast", "free" and "no cost". Writing out factual statements first encourages the model to consider what is actually true, rather than what the words usually suggest.

Knowledge Generation Is a Simpler Task "List facts about HTTPS" is easier and more open-ended than "answer this tricky yes/no question". The model performs the easier task first, and its output then supports the harder one.

The Two Steps in Detail Step 1: Knowledge Generation The original research used few-shot prompts for this step: a few examples showing a question followed by a useful knowledge statement. This taught the model what kind of knowledge to produce.

Generate one factual statement that is useful for answering the question.

Question: Does a bigger file always take longer to download? Knowledge: Download time depends on file size and on the speed of the connection.

Question: Can a Python list contain items of different types? Knowledge: Python lists can hold any objects, including a mix of strings, numbers, and other lists.

Question: Is a website that uses HTTPS always safe for entering your card details? Knowledge: Design Guidance for Step 1 Ask for facts, not answers. If the knowledge statements already contain the conclusion, step 2 just copies it, including any error. Ask for several statements. Different statements cover different aspects of the question. Keep statements short and specific. Long, general text dilutes the useful facts. Focus on distinctions and conditions. Statements such as "X does A but not B" are especially useful for tricky questions. Step 2: Knowledge Integration The original method integrated knowledge in a specific way: the question was answered separately with each knowledge statement, and the final answer was the one with the strongest support, based on the model's confidence. In practice, two simpler approaches are common:

Approach How It Works Trade-Off All knowledge together Include all statements in one prompt and answer once One extra call; simple One statement at a time, then vote Answer once per statement and take the majority answer More calls; limits the influence of a single bad statement The second approach is similar in spirit to self-consistency (Lesson 45): each knowledge statement produces one "vote".

Implementing Generate Knowledge Prompting in Python Requirements: pip install openai and an OPENAI_API_KEY environment variable.

import json from collections import Counter

from openai import OpenAI

client = OpenAI() MODEL = "gpt-5.4" # Replace with a model available in your account

GENERATE_KNOWLEDGE = """Write {count} short, factual statements that are useful for answering the question in . Focus on what is true, including important conditions and exceptions. Do not answer the question itself. Return only a JSON array of strings."""

ANSWER_WITH_KNOWLEDGE = """Use the knowledge in to answer the question in . Reply with exactly one word: Yes or No."""

def call(instructions: str, text: str) -> str: response = client.responses.create(model=MODEL, instructions=instructions, input=text) return response.output_text.strip()

def generate_knowledge(question: str, count: int = 3) -> list[str]: instructions = GENERATE_KNOWLEDGE.replace("{count}", str(count)) return json.loads(call(instructions, f"\n{question}\n"))

def answer_with(question: str, knowledge: list[str]) -> str: knowledge_text = "\n".join(f"- {fact}" for fact in knowledge) prompt = f"\n{knowledge_text}\n\n\n\n{question}\n" return call(ANSWER_WITH_KNOWLEDGE, prompt).rstrip(".").capitalize()

def generated_knowledge_answer(question: str) -> dict: facts = generate_knowledge(question) combined = answer_with(question, facts) # all knowledge together votes = Counter(answer_with(question, [fact]) for fact in facts) # one statement at a time return {"facts": facts, "combined": combined, "votes": dict(votes)}

result = generated_knowledge_answer( "Is a website that uses HTTPS always safe for entering your card details?" ) print(json.dumps(result, indent=2)) Possible output:

{ "facts": [ "HTTPS encrypts data between the browser and the server.", "A valid certificate shows the connection is to the named domain, not that the business is honest.", "Phishing websites can use HTTPS certificates." ], "combined": "No", "votes": { "No": 2, "Yes": 1 } } Code Explanation GENERATE_KNOWLEDGE Asks for facts, explicitly including conditions and exceptions, and forbids answering the question. The {count} placeholder is replaced with .replace() rather than .format(), because the instructions mention JSON, which uses braces (Lesson 38).

answer_with() Inserts the knowledge as a bullet list inside tags and asks for a one-word answer. .rstrip(".").capitalize() normalizes answers such as "no." to "No" so they can be counted.

Two integration methods

combined answers once with all statements. votes answers once per statement. In the sample output, one statement alone ("HTTPS encrypts data...") led to "Yes", because encryption by itself sounds like safety. This shows why individual statements can mislead, and why voting or combining statements is more robust. json.dumps(result, indent=2) Prints the result as indented JSON, with each list item and dictionary entry on its own line, which makes the facts and votes easy to read.

The Main Risk: Generated Knowledge Can Be Wrong Generated knowledge is produced by the same model that might answer incorrectly. If the model has a misconception, it may generate that misconception as a "fact", and step 2 will then use it confidently.

Risk Example Mitigation Incorrect generated fact "HTTPS guarantees the website is verified by the government." Generate several facts; vote; review Outdated fact A deprecated library behavior stated as current Provide current documentation as context (RAG) Fact contains the conclusion "HTTPS websites are always safe." Instruct: facts only, do not answer the question Irrelevant facts Facts about HTTP history Ask for facts useful for this question Generated Knowledge vs Retrieved Knowledge Aspect Generated Knowledge Retrieved Knowledge (RAG) Source The model's own training Your documents or a trusted source Setup None Requires a document store and retrieval Accuracy Limited by the model's knowledge and its errors As accurate as the source documents Up to date No, limited to training data Yes, if documents are updated Good for General and commonsense knowledge Company-specific, recent, or critical facts Use generated knowledge to help the model use what it already knows more carefully. Use retrieval (Lessons 101–106) when the facts must be correct, specific, or current.

When to Use Generate Knowledge Prompting Good Fit Why Commonsense or general-knowledge questions with tempting wrong answers Explicit facts counter surface associations Questions involving distinctions ("does X imply Y?") Facts make the distinction visible Explanatory answers for learners The generated facts double as a teaching summary Poor Fit Why Company-specific or recent information The model does not have the knowledge to generate Exact calculations Facts do not replace computation Simple questions the model already answers reliably The extra call adds cost without benefit Variations Single-Prompt Version Both steps can be combined in one prompt, similar to chain of thought:

Question: Is a website that uses HTTPS always safe for entering your card details?

First, list 3 facts about what HTTPS does and does not guarantee. Then answer the question using those facts. Format: Facts:

1. ... Answer: Yes or No, with one sentence of explanation. This is cheaper, but the facts are generated with the question's likely answer already in mind, which reduces their independence.

Knowledge for Explanations Generated knowledge is also useful for teaching. For a student question, generating the key facts first and then writing an explanation produces more structured, fact-focused explanations than answering directly.

Common Mistakes Mistake 1: Knowledge That Contains the Answer What it is: The generation step writes "The answer is No because...".

Why it happens: The prompt asks for "information about the question" without forbidding an answer.

What it causes: Step 2 simply repeats the conclusion, adding no independent check.

How to fix it: Ask for factual statements and explicitly forbid answering the question.

Mistake 2: Trusting Generated Facts as Verified What it is: Treating generated knowledge as reliable reference material.

Why it happens: The facts are written in a confident, encyclopedic style.

What it causes: Misconceptions are presented as facts and used to justify answers.

How to fix it: Generate several facts, use voting, and verify facts that matter.

Mistake 3: Using It for Missing Knowledge What it is: Generating "knowledge" about your own products, policies, or recent changes.

Why it happens: The technique seems like a way to give the model information.

What it causes: Invented details.

How to fix it: Provide the actual information as context or use retrieval.

Mistake 4: Too Much Generic Knowledge What it is: Long paragraphs of general background.

Why it happens: The prompt asks for "everything about the topic".

What it causes: The useful facts are buried, and the answer becomes less focused.

How to fix it: Ask for a few short facts that are useful for this specific question.

Best Practices Generate facts, not answers. Keep the two steps independent.

Ask for conditions and exceptions. They are the facts most likely to change the answer.

Use several statements and combine or vote. Limit the impact of a single wrong fact.

Separate the steps when independence matters. Use the single-prompt version only when cost matters more.

Use retrieval for facts that must be correct or current. Generated knowledge is only as good as the model's training.

Measure the benefit. Compare with direct answers on a test set of tricky questions.

Try It Yourself Exercise 1: Find Tempting Wrong Answers Write five yes/no questions from your field where a common association leads to the wrong answer, such as "Does adding more RAM always make a website faster?" or "Is code with 100% test coverage free of bugs?" Ask each directly, then with generated knowledge. Compare the answers.

Exercise 2: Compare Integration Methods Run generated_knowledge_answer() on your five questions. For each, compare the combined answer with the voting result. Identify any individual fact that led to a wrong vote, and explain why that fact was misleading on its own.

Exercise 3: Detect Wrong Knowledge For one question in your field where models commonly hold a misconception, generate 5 knowledge statements three times. Check each statement against official documentation, and count how many were incorrect. Decide whether this question needs retrieval instead of generated knowledge.