Choosing Good Examples for Few-Shot Prompts

Why Example Choice Matters In a few-shot prompt, the examples are the strongest signal the model receives. When examples and instructions disagree, models often follow the examples (Lesson 26). This means:

The quality of a few-shot prompt is mostly the quality of its examples.

Well-chosen examples can make a short prompt highly reliable. Poorly chosen examples can make a prompt worse than its zero-shot version, by teaching wrong labels, narrow patterns, or details that leak into outputs.

This lesson explains the qualities of good examples, how to build an example set, and how to choose examples automatically for each input.

Quality 1: Correctness What It Means Every example must show exactly the output you want: correct labels, correct values, correct format, and correct handling of special cases.

Why It Matters The model treats examples as the definition of correct behavior. A single mislabeled example can teach the wrong boundary between two categories, and the error appears on every similar input.

Example of an Incorrect Example Ticket: I paid but the course is still locked. Category: Billing If your team's rule is that locked courses after payment belong to Access, this example teaches the opposite. Every similar ticket will be misrouted.

A Note on Research Research by Min et al. (2022), "Rethinking the Role of Demonstrations", found that for some classification tasks, models gained a surprising amount of benefit from examples even when the example labels were randomized, because the examples still showed the label set, the kind of inputs, and the format. This does not mean correctness is unimportant. It means examples teach several things at once, including format and label space. For your applications, incorrect labels still risk teaching wrong boundaries, especially on borderline cases, and there is no reason to accept that risk.

Quality 2: Representativeness What It Means Examples should look like the real inputs the prompt will receive: similar length, language, writing style, and topics.

Why It Matters If examples are clean, formal sentences but real inputs are short, informal messages with spelling mistakes and mixed languages, the model sees a pattern that does not match its actual task.

Unrepresentative Example Representative Example I would like to request a refund for the course I purchased. refund pls, course not useful The video player does not function correctly. video stuck at 2:15 every time in chrome I am unable to access my account. login aagala, OTP varala (Tamil in English letters: "can't log in, OTP not coming") How to Achieve It Use real inputs from your application as the basis for examples, with personal information removed or replaced.

Quality 3: Diversity What It Means Examples should differ from each other in the ways real inputs differ: length, phrasing, structure, and difficulty.

Why It Matters As explained in Lesson 27, when all examples share a feature, the model assumes the feature is part of the pattern. If all examples are short, the model may handle long inputs poorly. If all examples start with "I", the model may treat that as significant.

Dimensions to Vary Dimension Variation Length One short, one medium, one long input Phrasing Question, complaint, statement Language English, Tamil or Hindi in Latin script, mixed Formality Formal and casual Output Different outputs within the allowed range Avoid Accidental Patterns Check your example set for patterns you did not intend:

Are all positive examples long and all negative examples short? Does every example mention a course name? Do all examples for one label use the same keyword? If the label Billing is always shown with the word "payment", the model may learn to classify by that keyword instead of by meaning, and fail on billing tickets that use words like "charged" or "invoice".

Quality 4: Coverage What It Means The example set should cover all important cases: every label in classification, and every important type of input or output in other tasks.

Coverage Checklist Task What to Cover Classification At least one example per label Extraction A complete input, an input with missing fields, an input with multiple items Summarization A short input, a long input, an input with numbers and dates Generation Different topics or subjects within the expected range Any task At least one edge case, such as empty or irrelevant input, if it can occur Example: Extraction Coverage Example 1: All fields present Example 2: Phone number missing → null Example 3: Two courses mentioned → array with two items Three examples, each teaching something different. A fourth example with all fields present would teach little new.

Quality 5: Boundary Cases What It Means Include examples that sit near the boundary between two outputs, where the decision is not obvious.

Why It Matters Clear cases are usually handled correctly even in zero-shot prompts. The inconsistencies come from borderline cases (Lesson 24). Examples placed exactly at the boundary show the model where you draw the line, which is often the main reason to use few-shot prompting at all.

Example For sentiment classification, typical examples are:

"Amazing course, loved it!" → Positive "Waste of money." → Negative These teach little. Boundary examples teach much more:

"It was fine, I learned the basics." → Neutral "Good content, but the audio was terrible." → Mixed "Not bad at all, actually quite useful." → Positive The third example is especially useful: it contains the words "not bad", which a keyword-based approach might classify as negative.

Quality 6: Consistency With Instructions What It Means Examples must follow every rule stated in the instructions.

Common Inconsistencies Instruction Inconsistent Example Reply with the label only. Category: Billing (because it mentions a refund) Maximum 20 words. A 35-word example output Use null for missing values. An example with "phone": "" Dates as YYYY-MM-DD. An example with "15 Nov 2026" When this happens, the model receives two conflicting signals, and the examples often win. Review examples against instructions every time either one changes.

Quality 7: Safe Content What It Means Examples should not contain real personal data, confidential information, or details that could leak into outputs.

Why It Matters Examples are sent with every request, so sensitive data in examples is repeatedly exposed to the model provider and logs. Specific details in examples, such as a phone number or a price, can be copied into outputs for unrelated inputs (Lesson 27, detail copying; Lesson 31). How to Achieve It Replace real names, phone numbers, and emails with realistic but fictional values. Use values that are obviously example-specific, such as an example phone number that is clearly not used in real data. How Many Examples and in What Order? Number Start with the minimum that covers the important cases, then add examples only to fix specific failures found in testing. Typical starting points are given in Lesson 26.

Order The order of examples can influence results:

Recency effect: Examples near the end of the prompt, closest to the real input, may have more influence. Research on few-shot classification, such as Zhao et al. (2021), "Calibrate Before Use", reported effects where the label of the final examples and the most frequent label in the examples could bias predictions. Grouping by label (all Billing examples, then all Access examples) can create a pattern where the model expects the next label to follow the sequence. Practical guidance:

Mix labels rather than grouping them. Avoid ending the example list with several examples of the same label. For important classifiers, test a few different orders and check whether results change. Building an Example Set Step by Step Step 1: Collect Candidates Gather real inputs from your application. If you have no real data yet, run a zero-shot prompt on realistic inputs and review its outputs (Lesson 23).

Step 2: Label Them Carefully Write the correct output for each candidate. For classification, have a second person check difficult cases; if two people disagree, the boundary needs a clearer rule.

Step 3: Select for Coverage and Diversity Choose a small set that covers every label or case type, includes boundary cases, and varies in length and phrasing.

Step 4: Remove Sensitive Data Replace personal details with fictional ones.

Step 5: Test and Refine Evaluate the few-shot prompt on a separate test set, not the examples themselves. Testing on the examples would show perfect results that mean nothing.

Step 6: Replace Weak Examples If a type of input keeps failing, add or replace an example to cover it. If an example seems to cause copying or bias, replace it with a more varied one.

Dynamic Example Selection What It Is Instead of using the same fixed examples for every input, dynamic example selection chooses the examples most similar to the current input from a larger pool.

Why It Helps If a pool contains 50 labeled tickets, the 3 most similar to the current ticket are usually more informative than 3 fixed generic ones. For a ticket about a coupon problem, examples about coupons and payments are more useful than an example about video playback.

Simple Implementation Production systems usually measure similarity with embeddings (numerical representations of meaning). The following example uses a simpler word-overlap measure so that it runs without extra dependencies and shows the idea clearly.

import re

EXAMPLE_POOL = [ {"ticket": "Coupon GOCOURSE10 shows invalid at checkout", "label": "Billing"}, {"ticket": "charged twice for the same course", "label": "Billing"}, {"ticket": "paid but course still locked", "label": "Access"}, {"ticket": "OTP not coming, cannot login", "label": "Access"}, {"ticket": "video stuck at 2:15 every time in chrome", "label": "Technical"}, {"ticket": "app crashes when I open the quiz", "label": "Technical"}, {"ticket": "do you have courses in Tamil?", "label": "Other"}, ]

def words(text: str) -> set[str]: return set(re.findall(r"[a-z0-9]+", text.lower()))

def similarity(a: str, b: str) -> float: """Jaccard similarity: shared words divided by all distinct words.""" wa, wb = words(a), words(b) if not wa or not wb: return 0.0 return len(wa & wb) / len(wa | wb)

def select_examples(ticket: str, k: int = 3) -> list[dict]: ranked = sorted(EXAMPLE_POOL, key=lambda ex: similarity(ticket, ex["ticket"]), reverse=True) return ranked[:k]

def format_examples(examples: list[dict]) -> str: return "\n\n".join(f"Ticket: {ex['ticket']}\nCategory: {ex['label']}" for ex in examples)

new_ticket = "my coupon code is not working at checkout" print(format_examples(select_examples(new_ticket))) Output:

Ticket: Coupon GOCOURSE10 shows invalid at checkout Category: Billing

Ticket: OTP not coming, cannot login Category: Access

Ticket: video stuck at 2:15 every time in chrome Category: Technical Code Explanation words() Converts text to lowercase and extracts words and numbers with a regular expression. Returning a set removes duplicates.

similarity() Calculates Jaccard similarity: the number of shared words divided by the total number of distinct words in both texts. Identical texts score 1.0; texts with no shared words score 0.0.

select_examples() Sorts the pool by similarity to the new ticket, highest first, and returns the top k.

format_examples() Builds the example block in a consistent format, ready to insert into a prompt.

Interpreting the Output The coupon example is correctly ranked first because it shares "coupon" and "checkout". The second and third results share almost nothing with the new ticket: they are ranked by very small or zero overlap. This shows two important points:

Word overlap is a weak measure of meaning. "Not working" and "invalid" mean similar things but share no words. Embedding-based similarity handles this much better. Pure similarity can destroy coverage. The selected examples happen to include three labels here, but for other inputs the top results might all have the same label, which could bias the classifier toward it. A common production approach combines both ideas: select the most similar examples, but ensure that each label is represented at least once.

Common Mistakes Mistake 1: Choosing Only Easy Examples What it is: Examples that are obviously in one category.

Why it happens: Clear examples are easy to think of and easy to label.

What it causes: The prompt does not improve on the difficult cases that caused problems in the first place.

How to fix it: Include boundary cases deliberately.

Mistake 2: Testing on the Examples What it is: Measuring accuracy using inputs that are also in the prompt.

Why it happens: It is convenient to reuse the same data.

What it causes: Overly optimistic results that do not reflect real performance.

How to fix it: Keep a separate test set that is never used as examples.

Mistake 3: Accidental Shared Features What it is: All examples of one label share a keyword, length, or phrasing.

Why it happens: Examples are written or selected without checking for patterns.

What it causes: The model learns the shortcut instead of the meaning.

How to fix it: Vary wording within each label and test with inputs that use different vocabulary.

Mistake 4: Similarity Without Coverage What it is: Dynamic selection that picks examples only by similarity.

Why it happens: Similarity seems like the only relevant criterion.

What it causes: Example sets dominated by one label, biasing predictions.

How to fix it: Combine similarity with a rule that ensures label coverage.

Best Practices Treat examples as the specification. Review them with the same care as instructions and code.

Use real, representative inputs. Anonymize them, but keep their natural style and mistakes.

Cover every label and important case. Each example should teach something the others do not.

Include boundary cases. They provide the most value per token.

Vary examples to avoid accidental patterns. Different lengths, wording, and structure.

Keep examples consistent with instructions. Re-check whenever either changes.

Mix example order and test for order effects. Avoid grouping labels and ending with a run of one label.

Evaluate on a separate test set. Never measure quality using the examples themselves.

Consider dynamic selection for large pools. Combine similarity with coverage.

Try It Yourself Exercise 1: Audit an Example Set Review this example set for a sentiment classifier and list every problem you find (correctness, diversity, coverage, boundaries, accidental patterns):

"I loved the course, it was amazing and very helpful!" → Positive "I really loved the projects, they were amazing!" → Positive "I loved the instructor, amazing teaching!" → Positive "Bad." → Negative Then rewrite it as a balanced set of five examples.

Exercise 2: Build a Boundary Set For a ticket classifier with the labels Billing, Access, Technical, and Other, write three boundary examples that could reasonably be confused, and the rule that decides each one. Compare zero-shot results with and without these examples on 10 new borderline tickets.

Exercise 3: Improve Dynamic Selection Modify select_examples() so it always includes at least one example of each label before filling the remaining slots with the most similar examples. Test it with a ticket that is very similar to several Billing examples and confirm that other labels still appear.