Few-Shot Prompting

Common Problems and Fixes How Few-Shot Prompts Fail Few-shot prompts fail in recognizable ways. Because the examples are the strongest signal in the prompt (Lesson 26), most failures trace back to something the examples teach that you did not intend, or something they fail to teach.

This lesson is a troubleshooting guide. Each problem is described by its symptom (what you see in the output), its cause (what in the prompt produces it), and its fix. A diagnostic table and an automated example checker follow at the end.

Problem 1: The Model Copies the Examples Symptom Outputs reuse phrases, sentence structures, or specific details from the examples. Every product description starts with "Discover", or every output has exactly three bullet points because the examples did.

Cause The model cannot tell which features of the examples are essential and which are incidental (Lesson 27). Features shared by all examples look like rules.

Fix Vary the examples in wording, structure, and length. State the purpose of the examples: The examples show the tone and format only. Do not reuse their wording. Measure similarity between outputs and examples to confirm the fix (Lesson 27). Example Before (all examples start the same way):

Course: Python Basics Tagline: Master Python one small program at a time.

Course: SQL Essentials Tagline: Master SQL one query at a time. After (varied openings and structures):

Course: Python Basics Tagline: Master Python one small program at a time.

Course: SQL Essentials Tagline: Turn messy tables into clear answers.

Course: Git and GitHub Tagline: Never lose a line of code again. Problem 2: Values Leak From Examples Into Outputs Symptom A phone number, name, date, or price from an example appears in the output for an input that does not contain it.

Cause When a field is missing from the input, the model fills it with a value it has seen in a similar context, which is the example (Lesson 31).

Fix Include examples where optional fields are null. Add the rule Never copy values from the examples. Use distinctive example values and detect them in outputs with code (Lesson 31). Problem 3: The Output Includes the Label Prefix Symptom You expect Billing, but the output is Category: Billing. Your validation code rejects it.

Cause The examples show Category: Billing as a complete line. In chat-style APIs, the model writes a full reply, and it may include the label because that is how outputs appeared in the examples. This is especially common when examples are placed in the instructions and the real input is sent without an output cue.

Fix Choose one consistent approach:

End the input with the cue (Category:) so the model continues after it. Use conversation-turn examples (Lesson 26, Method 2) where the assistant messages contain only the label. Normalize in code as a safety net: def clean_label(raw: str, prefix: str = "Category:") -> str: value = raw.strip() if value.lower().startswith(prefix.lower()): value = value[len(prefix):].strip() return value.strip(".") Explanation: The function removes surrounding whitespace, removes the prefix if present (case-insensitively), and removes a trailing period. Validation against the allowed labels should still follow.

Problem 4: The Model Continues Generating Examples Symptom After answering, the model writes a new invented input and its output, then another, continuing the pattern.

Billing

Ticket: My invoice is wrong. Category: Billing

Ticket: ... Cause In completion-style prompts, the pattern "input, output, input, output" has no natural end. The model continues it.

Fix Add Reply with the category only. or Return only the output for the final input. Use conversation turns, where each reply naturally ends after one answer. If supported by the API, use a stop sequence such as \nTicket:. In code, keep only the first line when a single-line output is expected. Problem 5: Examples Are Treated as Part of the Input Symptom A summary includes content from the example texts. A classifier answers about the first example instead of the real input. An extractor returns items from the examples.

Cause The prompt does not clearly separate demonstrations from the actual task, so the model treats everything as material to process.

Fix Wrap examples in an container and the real input in a separate, clearly labeled section (Lesson 29). State: The examples are demonstrations only. Process only the text in . Place the real input after the examples, immediately before the output cue. Problem 6: Examples Contradict the Instructions Symptom The instructions say "one sentence", but outputs are three sentences long. The instructions say "use null", but outputs contain empty strings.

Cause At least one example does not follow the instructions. When instructions and examples conflict, models often follow the examples.

Fix Check every example against every instruction. This is easy to automate for measurable rules, as shown in the checker at the end of this lesson. Whenever an instruction changes, review all examples.

Problem 7: Bias Toward One Label Symptom One label is predicted far more often than it appears in your real data.

Cause That label has more examples than others (majority label bias). The last examples in the prompt share that label (recency bias). That label's examples are the most general, so it acts as a catch-all. Fix Balance the number of examples per label. Shuffle examples with a fixed seed so labels are mixed (Lesson 30). Compare the distribution of predictions with the expected distribution on a test set. Problem 8: Good on Examples, Poor on Real Inputs Symptom The prompt performs perfectly in early tests but poorly on real data.

Cause The test inputs were the examples themselves, or very similar to them. The examples are clean and formal, while real inputs are short, informal, misspelled, or in mixed languages (Lesson 28, representativeness). Fix Test on a separate set of real inputs that are never used as examples. Replace idealized examples with anonymized real ones. Add examples for input types that fail. Problem 9: Results Change When Example Order Changes Symptom The same input receives different labels when the examples are rearranged.

Cause The input is a borderline case, and the model's decision depends on small influences such as which example appeared last.

Fix Add a boundary example or rule that decides the case explicitly. Fix the order with a seed so results are reproducible. Test several orders on borderline inputs to find unstable cases (Lesson 30). Problem 10: The Prompt Becomes Too Long Symptom Response times and costs increase; the prompt is difficult to maintain; new examples no longer improve results.

Cause Examples were added for every failure without removing redundant ones.

Fix Remove examples that teach nothing new, such as several typical cases for the same label. Keep boundary examples, which give the most value per token. Consider dynamic example selection, which includes only the most relevant examples for each input (Lesson 28). Reconsider whether a clearer rule could replace several examples. Problem 11: Few-Shot Performs Worse Than Zero-Shot Symptom After adding examples, accuracy drops compared to the zero-shot version.

Cause Examples are incorrect, unrepresentative, or biased. Examples narrow the model's behavior to patterns that do not fit many real inputs. Some modern reasoning-focused models perform well with clear zero-shot instructions, and examples can constrain their approach unnecessarily (Lesson 23). Fix Always compare few-shot and zero-shot versions on the same test set (Lesson 33). If zero-shot is better, investigate the examples; if they are correct and representative, keep zero-shot.

Problem 12: Examples Become Outdated Symptom After a business change, such as a new category, a renamed product, or a new output field, outputs are partly wrong or use old values.

Cause The instructions were updated, but the examples still show the old labels, fields, or rules.

Fix Treat examples as part of the prompt's specification. Store them as data, version them with the prompt (Lesson 67), and validate them automatically against the current label set and schema.

Diagnostic Table Symptom Likely Problem First Fix to Try Repeated phrases or structure across outputs 1. Copying Vary examples; state their purpose Values from examples in outputs 2. Leakage Null examples; leak check in code Category: Billing instead of Billing 3. Label prefix Output cue or normalization Extra invented examples after the answer 4. Continuation "Return only..." and conversation turns Example content appears in results 5. Examples treated as input Container tags and explicit statement Instructions ignored 6. Contradiction Check examples against instructions One label predicted too often 7. Label bias Balance and shuffle Good tests, poor production 8. Unrepresentative examples Test on real, separate data Labels change when order changes 9. Order sensitivity Boundary rule or example Slow, expensive, hard to maintain 10. Too many examples Remove redundant examples Accuracy lower than zero-shot 11. Harmful examples Compare and investigate Old labels or fields in outputs 12. Outdated examples Validate examples automatically Automatically Checking an Example Set Many of these problems can be caught before the prompt is used, by checking the examples themselves. The following script checks a classification example set for common issues.

from collections import Counter

LABELS = {"Billing", "Access", "Technical", "Other"} MAX_WORDS_PER_OUTPUT = 1

EXAMPLES = [ {"input": "charged twice for the same course", "output": "Billing"}, {"input": "paid but course still locked", "output": "Access"}, {"input": "video stuck at 2:15 in chrome", "output": "Technical"}, {"input": "do you have Tamil courses?", "output": "Other"}, {"input": "coupon says invalid", "output": "Billing"}, {"input": "charged twice for the same course", "output": "Billing"}, {"input": "certificate download fails", "output": "Technical issue"}, ]

TEST_INPUTS = [ "paid but course still locked", "OTP not received", ]

def check_examples(examples, labels, test_inputs): problems = [] outputs = [ex["output"] for ex in examples]

# Problem 6 and 12: outputs must use current labels and expected format for i, ex in enumerate(examples, start=1): if ex["output"] not in labels: problems.append(f"Example {i}: unknown label {ex['output']!r}") if len(ex["output"].split()) > MAX_WORDS_PER_OUTPUT: problems.append(f"Example {i}: output longer than {MAX_WORDS_PER_OUTPUT} word(s)")

# Coverage: every label demonstrated at least once missing = labels - set(outputs) if missing: problems.append(f"Labels with no example: {sorted(missing)}")

# Problem 7: balance counts = Counter(o for o in outputs if o in labels) if counts and max(counts.values()) > 2 * min(counts.values()): problems.append(f"Unbalanced label counts: {dict(counts)}")

# Duplicates teach nothing new (Problem 10) duplicate_inputs = [text for text, n in Counter(ex["input"] for ex in examples).items() if n > 1] if duplicate_inputs: problems.append(f"Duplicate example inputs: {duplicate_inputs}")

# Problem 8: examples must not appear in the test set overlap = {ex["input"] for ex in examples} & set(test_inputs) if overlap: problems.append(f"Examples also used as test inputs: {sorted(overlap)}")

# Problem 7 (recency): last two examples share a label if len(outputs) >= 2 and outputs[-1] == outputs[-2]: problems.append(f"Last two examples share the label {outputs[-1]!r}")

return problems

for problem in check_examples(EXAMPLES, LABELS, TEST_INPUTS): print("-", problem) Output:

• Example 7: unknown label 'Technical issue'
• Example 7: output longer than 1 word(s)
• Unbalanced label counts: {'Billing': 3, 'Access': 1, 'Technical': 1, 'Other': 1}
• Duplicate example inputs: ['charged twice for the same course']
• Examples also used as test inputs: ['paid but course still locked'] Code Explanation Label and format check Each example output must be one of the current LABELS and no longer than the expected length. Example 7 uses an old or invented label, "Technical issue", which would teach the model a value your code rejects.

Coverage check labels - set(outputs) finds labels that are never demonstrated. Here all four labels appear, so no coverage problem is reported.

Balance check Counter counts examples per valid label. The check flags the set when the most common label has more than twice as many examples as the least common one. Billing has 3 examples while others have 1, so it is flagged. The factor of 2 is a simple starting threshold; adjust it for your situation.

Duplicate check Counts identical example inputs. Duplicates add tokens and overweight one label without teaching anything new.

Overlap check The set intersection & finds example inputs that are also test inputs. Testing on examples produces misleadingly good results.

Recency check Flags when the last two examples share a label, which can encourage recency bias. Here, the last two outputs are "Billing" and "Technical issue", so this check does not trigger.

Using the Checker Run it whenever examples or labels change, for example as part of your test suite. It catches structural problems instantly, without calling the model. Problems that depend on model behavior, such as copying and order sensitivity, still require running the prompt on test data.

Common Mistakes When Fixing Few-Shot Problems Adding More Examples for Every Failure Not every failure needs a new example. Sometimes a rule, a better example, or removing a misleading example is the right fix. Adding examples by default leads to Problem 10.

Fixing One Symptom Without Retesting Everything Changing examples affects the whole prompt. A new example that fixes one input can shift decisions on others. Re-run the full test set after each change (Lesson 12).

Treating Few-Shot as Always Better Few-shot is a tool for specific problems: format, boundaries, and style. If zero-shot meets your criteria, examples add cost and maintenance without benefit.

Try It Yourself Exercise 1: Diagnose From Outputs For each observation, name the problem from this lesson and the fix:

Every generated email ends with "Looking forward to hearing from you soon!", exactly like the only example. The model returns Sentiment: Positive. and your code expects Positive. After the new Refund label was added, no output ever uses it. A summary of a new article includes a sentence from an example summary. Exercise 2: Extend the Checker Add two checks to check_examples():

Flag examples whose input is shorter than 3 words, which may be unrepresentative. Flag labels whose examples all contain the same keyword (for example, every Billing example contains "charged"), which may teach a keyword shortcut. Exercise 3: Repair a Few-Shot Prompt Build a few-shot classifier with deliberately flawed examples: one wrong label, three examples of one label and one of another, and an example also used in testing. Measure accuracy on 15 test inputs. Then fix each problem using the checker and the diagnostic table, and measure again.