Few-Shot Classification with Examples

What Is Few-Shot Classification? Few-shot classification is text classification (Lesson 18) where the prompt includes labeled examples that show how each category should be applied.

In zero-shot classification, labels are defined by their names and descriptions. In few-shot classification, labels are also defined by demonstration: the model sees real texts placed in each category and uses them to judge new texts.

Example Classify each student message as Doubt, Request, Complaint, or Feedback.

Message: Why does my for loop print the last item twice? Category: Doubt

Message: Can you extend my assignment deadline by two days? Category: Request

Message: The mentor didn't show up for the live session again. Category: Complaint

Message: The new practice questions in Module 3 really helped. Category: Feedback

Message: Is it possible to get the session recording? I missed it. Category: Expected output:

Request Example Explanation The final message contains a question mark and starts with "Is it possible", which looks like a Doubt if you rely on surface features. But the examples show that Doubt means a question about course content, while Request means asking for something to be done or provided. Asking for a recording is a request. The examples communicate this distinction more precisely than the label names alone.

When to Move From Zero-Shot to Few-Shot Classification Start with zero-shot classification and clear definitions (Lesson 18). Move to few-shot when testing shows one of these patterns:

Signal Why Examples Help The same two labels are repeatedly confused Boundary examples show where you draw the line Labels are specific to your organization Examples show meaning that names cannot Results change between runs on borderline cases Examples make one interpretation clearly preferred The model classifies by keywords instead of meaning Varied examples show that meaning matters Definitions keep growing with special cases A few examples can replace many rules The confusion analysis from Lesson 18 is the best guide: it shows exactly which pairs of labels need boundary examples.

Designing the Example Set for Classification Principle 1: At Least One Example per Label Every label should appear at least once. A label that never appears in the examples has to be inferred from its name alone, which puts it at a disadvantage compared to labels that were demonstrated.

Principle 2: Extra Examples for Confused Labels If confusion analysis shows that Doubt and Request are often mixed up, add examples specifically on that boundary:

Message: Can you explain how Django signals work? Category: Doubt

Message: Can you share the slides from today's session? Category: Request Both start with "Can you", which makes the distinction clear: explaining a concept is a Doubt; providing something is a Request.

Principle 3: Balance Research on few-shot classification, including Zhao et al. (2021), "Calibrate Before Use", observed that models can be biased toward labels that appear more often in the examples (majority label bias) and toward the labels of the last examples (recency bias).

Practical guidance:

Keep roughly equal numbers of examples per label, unless you have tested a reason to do otherwise. Mix labels in the example order instead of grouping them. Do not end with several examples of the same label. Principle 4: Varied Wording Within Each Label If every Complaint example contains the word "again", the model may learn "again" as a signal for Complaint. Use examples that express the same category in different ways (Lesson 28).

Principle 5: Include the Hard Cases Hard Case Example Looks like one label, is another "Is it possible to get the recording?" → Request (not Doubt) Contains two labels' features "Great session, but please upload the notes." → decide by a rule Short or vague "not working" → decide whether this is Complaint or needs Other Mixed language "Session super, notes upload pannunga" → Request (Tamil in English letters: "please upload the notes") Combining Rules and Examples Examples are powerful, but they work best together with written rules. Rules state the principle; examples show it in action.

Classify each student message as Doubt, Request, Complaint, or Feedback.

Definitions:

• Doubt: a question about course content or code
• Request: asking for something to be done or provided (extension, recording, notes)
• Complaint: dissatisfaction with a service, person, or experience
• Feedback: an opinion or suggestion without a request or complaint

Rule: If a message contains both feedback and a request, choose Request, because it needs action.

Examples: ... Why Both Are Needed Rules without examples leave boundaries open to interpretation. Examples without rules can be generalized in unintended ways, because the model must guess the principle behind them. Rules plus examples give the principle and its application, which is the most reliable combination. Few-Shot Classification in Code The following example builds a few-shot classifier from labeled data, mixes example order, validates the output, and reports the distribution of predictions to detect bias.

Requirements: pip install openai and an OPENAI_API_KEY environment variable.

import random from collections import Counter

from openai import OpenAI

client = OpenAI() MODEL = "gpt-5.4" # Replace with a model available in your account

LABELS = ["Doubt", "Request", "Complaint", "Feedback"]

DEFINITIONS = """- Doubt: a question about course content or code

• Request: asking for something to be done or provided (extension, recording, notes)
• Complaint: dissatisfaction with a service, person, or experience
• Feedback: an opinion or suggestion without a request or complaint"""

EXAMPLES = [ ("Why does my for loop print the last item twice?", "Doubt"), ("Can you explain how Django signals work?", "Doubt"), ("Can you extend my assignment deadline by two days?", "Request"), ("Can you share the slides from today's session?", "Request"), ("The mentor didn't show up for the live session again.", "Complaint"), ("I've been waiting 4 days for a reply from support.", "Complaint"), ("The new practice questions in Module 3 really helped.", "Feedback"), ("Maybe add more SQL exercises in the next batch.", "Feedback"), ]

def build_instructions(seed: int = 42) -> str: shuffled = EXAMPLES[:] random.Random(seed).shuffle(shuffled) example_text = "\n\n".join( f"Message: {message}\nCategory: {label}" for message, label in shuffled ) return ( f"Classify the student message as one of: {', '.join(LABELS)}.\n\n" f"Definitions:\n{DEFINITIONS}\n\n" "Rule: If a message contains both feedback and a request, choose Request.\n" "The examples below are demonstrations only, not part of the input.\n\n" f"Examples:\n\n{example_text}\n\n" "Reply with the category only." )

INSTRUCTIONS = build_instructions()

def classify(message: str) -> str: response = client.responses.create( model=MODEL, instructions=INSTRUCTIONS, input=f"Message: {message}\nCategory:", ) label = response.output_text.strip() if label not in LABELS: raise ValueError(f"Unexpected label: {label!r}") return label

test_messages = [ "Is it possible to get the session recording? I missed it.", "What's the difference between a list and a tuple?", "Great session, but please upload the notes.", "Nobody has fixed the broken quiz for a week.", "I liked the pace of today's class.", ]

predictions = [classify(message) for message in test_messages] for message, label in zip(test_messages, predictions): print(f"{label:<10} {message}")

print("\nPrediction distribution:", Counter(predictions)) Possible output:

Request Is it possible to get the session recording? I missed it. Doubt What's the difference between a list and a tuple? Request Great session, but please upload the notes. Complaint Nobody has fixed the broken quiz for a week. Feedback I liked the pace of today's class.

Prediction distribution: Counter({'Request': 2, 'Doubt': 1, 'Complaint': 1, 'Feedback': 1}) Code Explanation LABELS and DEFINITIONS The labels are defined once. The same list is used in the prompt and in validation, so they cannot drift apart (Lesson 18).

EXAMPLES Two examples per label, balanced. The pairs are chosen to cover boundaries: both Doubt and Request include "Can you..." examples, which teaches that the deciding factor is meaning, not the opening words.

build_instructions(seed)

EXAMPLES[:] makes a copy so the original order is not changed. random.Random(seed).shuffle(...) shuffles the examples so labels are mixed rather than grouped. Using a fixed seed keeps the order identical every time the program runs. A changing order would make results harder to reproduce and test. The examples are formatted with identical Message: and Category: labels (Lesson 29). INSTRUCTIONS = build_instructions() The instructions are built once, not for every request, because they do not change.

classify()

The input uses the same Message: label as the examples and ends with Category: as the output cue. The result is validated against LABELS. Counter(predictions) Shows how often each label was predicted. On a larger, representative test set, compare this distribution with the expected distribution. If one label is predicted much more often than it should be, the examples may be biasing the model toward it.

Evaluating Zero-Shot vs Few-Shot Before keeping a few-shot classifier, confirm it is actually better than the zero-shot version on the same test set:

Step Action 1 Build a labeled test set that does not include any of the examples 2 Run the zero-shot version and record predictions 3 Run the few-shot version and record predictions 4 Compare accuracy and confusion pairs (Lesson 18) 5 Compare token cost per request (Lesson 23) Keep the few-shot version only if the improvement justifies the extra tokens. Often, few-shot gives the largest gains on exactly the confused label pairs that boundary examples were chosen for.

Few-Shot Multi-Label Classification For multi-label classification (Lesson 18), examples must show texts with one, several, and zero labels, so the model learns all three cases.

Identify every topic mentioned in the feedback: Content, Instructor, Platform, Pricing. Return a JSON array.

Feedback: The instructor explains clearly. Topics: ["Instructor"]

Feedback: Great lessons, but the app crashes and it's overpriced. Topics: ["Content", "Platform", "Pricing"]

Feedback: Thanks! Topics: []

Feedback: Videos buffer a lot, but the projects are excellent. Topics: Expected output:

["Platform", "Content"] Explanation The second example shows that several labels can apply at once. The third example shows that an empty array is valid, which reduces over-labeling. The order of labels in the output may vary; your code should compare them as sets, as in Lesson 18. Common Mistakes Mistake 1: Unbalanced Examples What it is: Five examples of one label and one of another.

Why it happens: Some labels are more common in real data, so more examples are available.

What it causes: A tendency to predict the over-represented label.

How to fix it: Balance examples across labels, then check the prediction distribution.

Mistake 2: Grouped Example Order What it is: All Doubt examples first, then all Request examples, and so on.

Why it happens: Grouping seems organized.

What it causes: Possible recency effects, where the last group's label is favored.

How to fix it: Shuffle examples with a fixed seed.

Mistake 3: No Boundary Examples What it is: Only clear, typical examples for each label.

Why it happens: Clear examples are easy to write.

What it causes: Few-shot gives little improvement over zero-shot, because the difficult cases are still undefined.

How to fix it: Use confusion analysis to identify boundaries and add examples on them.

Mistake 4: Evaluating on the Examples What it is: Testing with messages that also appear as examples.

Why it happens: Reusing data is convenient.

What it causes: Artificially perfect results.

How to fix it: Keep examples and test data strictly separate.

Best Practices Start from zero-shot and confusion analysis. Add examples where the data shows they are needed.

Show every label, and extra examples on confused boundaries. Boundary examples give the largest improvements.

Balance and mix the examples. Equal counts per label, shuffled with a fixed seed.

Combine definitions, rules, and examples. Principles plus demonstrations are most reliable.

Use identical formatting and an output cue. The real input must look like the examples.

Validate labels and monitor prediction distribution. Detect invalid outputs and bias early.

Compare zero-shot and few-shot on the same test set. Keep examples only if they improve results enough to justify their cost.

Try It Yourself Exercise 1: Build and Compare Take the student message classifier from this lesson. Create a test set of 20 messages not used as examples, including at least 6 borderline cases. Run a zero-shot version (definitions only) and the few-shot version. Compare accuracy and identify which messages the examples fixed.

Exercise 2: Test for Bias Change EXAMPLES so that it contains four Complaint examples and only one of each other label. Run the same test set and compare the prediction distribution with the balanced version.

Exercise 3: Test Order Effects Run the balanced classifier with three different seeds in build_instructions(seed). Check whether any test messages change label depending on example order. If they do, decide whether a boundary example or rule would make the decision stable.