Pre-Decision Status Note

A Status Note Before the Decision Framework

Before working through when fine-tuning makes sense, you need to know where things actually stand. OpenAI is winding down its self-serve fine-tuning platform. As of the most recent guidance, it is no longer open to new users — only organizations with an existing fine-tuning history can create new jobs, and even for those existing users, that window is explicitly framed as temporary ("the coming months," not indefinitely). Separately, OpenAI's deprecation timeline lists staged restrictions on new job creation through mid-2026, tightening further into 2027. What doesn't change, and matters most if you already have a fine-tuned model in production: inference on existing fine-tuned models keeps working until the underlying base model itself is deprecated — a fine-tuned model isn't switched off the moment the training platform is.

This lesson still teaches the decision framework in full, for two reasons. First, if your organization already has fine-tuning access, this is exactly the reasoning you need to decide whether to use it. Second, and more importantly, the underlying question — prompting vs. retrieval vs. fine-tuning — is not specific to OpenAI's platform. It's the same question you face with any model provider, including ones actively investing in fine-tuning today, and the same tradeoffs (cost, latency, maintenance, flexibility) apply regardless of whose infrastructure runs the training job. Treat the specific API calls in Lesson 5 as tied to OpenAI's current offering; treat the reasoning in this lesson as durable.

The Real Question: What Are You Actually Trying to Fix?

"Should I fine-tune?" is usually the wrong first question, because it skips past a more useful one: what specifically is wrong with the current approach, and which layer of the system actually owns that problem? A model-powered feature has several distinct layers you can intervene on, and fine-tuning is only the right lever for a specific subset of failures.

Broadly, in order of how cheap and reversible they are to try:

1. Prompt engineering — better instructions, clearer formatting requirements, few-shot examples in the prompt itself.
2. Structured outputs and function calling — constraining the model's output shape at the API level rather than hoping the prompt is followed.
3. Retrieval-augmented generation (RAG) — giving the model access to specific facts or documents it doesn't have memorized, at request time.
4. Fine-tuning — actually changing the model's weights so a behavior becomes the default, without needing to re-explain it in every prompt.

The temptation, especially once a team has heard "fine-tuning" as the sophisticated, grown-up solution, is to reach for it early. In practice this is almost always the wrong order. Fine-tuning is the most expensive, slowest, and least flexible of the four options, and it solves a narrower set of problems than people expect — it changes how a model responds, not what it knows.

What Fine-Tuning Actually Changes (and What It Doesn't)

This distinction is the single most common source of wasted fine-tuning effort, so it's worth being precise about it.

Fine-tuning is good at: teaching a model a consistent style, format, or behavior pattern that would otherwise require a long, repeated set of instructions or few-shot examples in every single prompt. Classification into a fixed taxonomy. Correcting a specific, persistent instruction-following failure — for example, a model that keeps adding a disclaimer paragraph you've explicitly told it not to include, across hundreds of prompt-engineering attempts. Generating content in a rigid, specific format your business requires, where "close enough" formatting isn't acceptable. Producing shorter, more consistent outputs than a base model tends to produce by default, which can meaningfully reduce token costs at scale.

Fine-tuning is bad at: teaching a model new facts it doesn't already know, or facts that change over time. If you fine-tune a model on your product's current pricing, and pricing changes next quarter, the fine-tuned model keeps confidently stating the old numbers — you've baked stale knowledge into the weights, and updating it means retraining, not editing a document. This exact scenario is what RAG exists to solve instead: keep the facts in a retrievable knowledge base, and let the model reference them fresh at request time, so an update to the source document is instantly reflected without touching the model at all.

A genuinely common and costly mistake is fine-tuning a model to "know about" a company's product catalog or internal documentation. It appears to work at first, because the training data captured a snapshot of the truth — and then it quietly drifts wrong as the real information changes underneath a model that can't update itself. If your problem is "the model doesn't know X," the answer is almost always retrieval, tool calling against a live data source, or both — not fine-tuning.

Decision Framework: Work Through the Cheap Options First

Step 1: Can better prompting fix this? This sounds obvious, but it's skipped more often than you'd expect. Before concluding a model "can't" do something reliably, make sure you've actually tried: clearer instructions, an explicit output format specification, a handful of well-chosen few-shot examples embedded in the prompt, and breaking a complex instruction into smaller, more explicit steps. A surprising fraction of "we need to fine-tune" situations resolve completely once someone rewrites the prompt with real care, because the failure was ambiguity in the instructions, not a limitation of the model itself.

Step 2: Is this actually a structure problem, not a content problem? If the issue is that the model's output is inconsistently formatted — sometimes valid JSON, sometimes prose wrapped around JSON — that's frequently solved by using the API's structured output features (a JSON schema the model is constrained to follow) rather than fine-tuning a model to "learn" to always produce JSON. Structured outputs solve format reliability at the API level, deterministically, which is both cheaper and more reliable than hoping a fine-tuned model never deviates.

Step 3: Is the model missing information, not missing a skill? If the model gives wrong answers because it doesn't have access to the specific facts it needs — your company's current refund policy, a customer's order history, today's inventory levels — that's a retrieval problem. Feed the relevant facts into the prompt at request time (via a vector search over your documents, a direct database lookup, or a tool call), and re-test. This resolves an enormous share of "the model doesn't know our stuff" complaints without ever touching a fine-tuning job.

Step 4: Only now, ask whether fine-tuning is the right tool. If you've genuinely worked through the first three and you still have a persistent behavioral gap — the model knows the right facts (you've confirmed this by giving it directly in the prompt) but still doesn't consistently apply your desired tone, format, or decision pattern, even with careful prompting and examples — that's the specific situation fine-tuning is built for.

Concrete Signals That Point Toward Fine-Tuning

A few real, specific situations tend to justify moving to fine-tuning rather than continuing to iterate on prompts:

Your few-shot prompt is enormous and still not reliable enough. If you've found that including 15 or 20 examples directly in every prompt gets you close to the behavior you want, but you can't fit that many examples alongside the actual user input without hitting context or cost limits, fine-tuning effectively lets you "bake in" far more examples than could ever fit in a single prompt, at no per-request token cost for those examples.

You need much shorter, cheaper responses at scale, with consistent formatting. A fine-tuned model that has learned your exact desired output pattern often needs a shorter prompt to reproduce it than a base model coached with lengthy instructions and examples every single call — and at high request volume, that per-call token savings compounds into a real cost difference.

You're doing distillation from a larger, more capable model. A common and well-supported pattern: use a larger, more expensive model with careful prompting to generate high-quality labeled examples for a hard task, then fine-tune a smaller, cheaper model on those examples. The result is often a small model that performs close to the large model's quality on that narrow task, at a fraction of the inference cost — because the smaller model no longer needs an elaborate prompt to reach the same behavior; it's already baked into the weights.

A specific instruction-following failure survives every prompting attempt. If you've tried rewording, reordering, emphasizing, and giving explicit counter-examples, and the model still does the wrong thing at a stubborn, consistent rate, that persistence — not occasional failure, but a pattern that resists prompt-level fixes — is a strong signal that the behavior needs to be trained in rather than instructed in.

The Real Costs of Fine-Tuning That Prompting Doesn't Have

It's worth being honest about what you're signing up for, because fine-tuning trades a cheap, reversible cost (editing a prompt) for a set of more expensive, less reversible ones.

Data curation is real work, and quality matters more than quantity. OpenAI's own fine-tuning guidance is blunt about this: a smaller amount of high-quality, carefully checked training data is generally more effective than a larger amount of low-quality data. Every training example needs to be internally consistent, free of contradictions with other examples, correctly formatted, and — critically — contain all the information the model needs to produce the desired output, since a fine-tuned model can't reason its way to information that simply isn't present anywhere in its training examples or the prompt. Auditing a dataset of hundreds of examples for grammar, logic, and stylistic consistency, and checking inter-annotator agreement if multiple people wrote the examples, is genuine, ongoing labor — not a one-time setup cost.

You now own a retraining lifecycle. A prompt can be edited and redeployed in minutes. A fine-tuned model, once your underlying task or desired behavior shifts even slightly, generally needs new training data and a new training job to catch up — you can't just patch a fine-tuned model's behavior the way you'd patch a prompt's wording. If your product's requirements change frequently, this retraining cadence becomes a real, recurring engineering cost that a pure-prompting approach doesn't have.

Evaluation becomes mandatory, not optional. Everything from Lessons 1 through 3 in this unit is a prerequisite for fine-tuning responsibly, not an optional nice-to-have. You need a held-out evaluation set, separate from your training data, to actually confirm a fine-tuned model performs better than the base model it's replacing — without that, you're trusting that training "probably helped," which is exactly the kind of unverified confidence Lesson 1 warned against, just applied to a more expensive and slower-to-iterate process.

It reduces flexibility, not just cost. A fine-tuned model has learned a specific pattern. If a future business requirement needs the model to deviate from that pattern in an edge case, you're now fighting the training, not just adjusting an instruction — prompts layered on top of a fine-tuned model can conflict with what training baked in, in ways that are harder to predict and debug than adjusting a prompt-only system.

A Cost Comparison Worth Actually Running

Before deciding fine-tuning is worth the investment, it's worth estimating the numbers rather than assuming. Consider a classification task currently handled with a long, carefully-crafted few-shot prompt:

# Approach A: prompting with embedded examples
# ~1,800 tokens of instructions + 20 few-shot examples per request
# ~150 tokens of actual user input
# Total input tokens per request: ~1,950

# Approach B: fine-tuned model, examples baked into training instead
# ~200 tokens of instructions (much shorter, no few-shot examples needed)
# ~150 tokens of actual user input
# Total input tokens per request: ~350

At meaningful request volume, that difference — 1,950 input tokens versus 350 — is the actual, measurable saving fine-tuning can offer, separate from any quality improvement. Whether that saving justifies the upfront cost of curating training data, running training jobs, and maintaining an evaluation pipeline depends entirely on your request volume: at a thousand requests a month, the token savings are unlikely to offset the engineering time invested; at a million requests a month, they very well might. Do this arithmetic with your own real prompt length and real request volume before committing — "fine-tuning will save tokens" is true in general and can still be the wrong call for your specific scale.

A Common Mistake: Shortening Instructions to Save Money Before Fine-Tuning

OpenAI's fine-tuning guidance specifically flags a mistake worth calling out here because it's counterintuitive: teams sometimes try to reduce cost by shortening their prompt's instructions before fine-tuning, assuming a fine-tuned model needs less guidance overall. In practice this backfires — a shorter, vaguer instruction set requires more training examples to reach the same level of reliability, because the model has less explicit guidance to generalize from and has to infer more of the pattern purely from examples. If anything, keep your instructions precise and complete going into a fine-tuning job, and let the token savings come from no longer needing extensive few-shot examples in every request, not from stripping down the core instructions themselves.

Real-World Example: A Support Ticket Priority Classifier

To make the decision framework concrete, consider a team building a feature that assigns incoming support tickets a priority level (urgent, high, normal, low) based on message content and account tier.

They start with prompting (Step 1): a clear instruction set plus five examples per priority level. This gets them to roughly 85% agreement with human-assigned priorities on their evaluation set (Lesson 2's workflow) — good, but not good enough for a feature that auto-routes urgent tickets to on-call staff without human review.

They check whether it's a structure or knowledge problem (Steps 2–3): the model already has every fact it needs in the prompt (message text, account tier), and output format isn't the issue — priority assignment is the issue. This rules out RAG and structured outputs as the fix.

They try harder prompting first: tightening the definition of each priority level, adding more contrastive examples showing near-miss cases (a frustrated but non-urgent message versus a genuinely urgent one). This gets them to 90%, but plateaus there even after substantial iteration — the remaining 10% of misclassifications cluster around a nuanced judgment call (account tier interacting with message tone) that resists further prompt tuning.

At this point, Step 4 applies: they have a persistent, prompt-resistant behavioral gap on a well-defined classification task with plenty of historical, labeled examples already sitting in their support system. This is close to the ideal fine-tuning case — a fixed taxonomy, abundant real labeled data, and a ceiling that prompting alone couldn't break through. Lesson 5 picks this exact scenario back up and walks through the full process of turning it into a fine-tuning job.

Side-by-Side: Prompting, RAG, and Fine-Tuning

It helps to see the three approaches compared directly against the same criteria, since the right choice usually comes down to which row of this comparison matters most for your specific feature.

Note that these aren't mutually exclusive — most mature production systems use all three at once: RAG to supply current facts, careful prompting to structure the request and handle edge cases, and, only where a genuine behavioral gap remains after the first two, a fine-tuned model to close it. Treating this as an either/or choice is itself a common misconception; the realistic target for most teams is "prompting plus RAG, with fine-tuning reserved for the specific residual problem neither one solves."

When Fine-Tuning Looks Right but Isn't

A few situations look like good fine-tuning candidates on the surface but usually aren't, and it's worth naming them explicitly because they're easy traps.

"We want the model to always be more concise." This is almost always a prompting problem, not a fine-tuning problem — explicit length constraints, a maximum word count, or a structured output schema with a bounded field size solve this directly and instantly, with none of fine-tuning's overhead. Reach for fine-tuning here only if you've already tried explicit length instructions carefully and the model still, consistently, ignores them.

"We want the model to stop hallucinating about our product." As covered above, this is a knowledge-grounding problem, solved by retrieval, not a behavioral problem solved by training. Fine-tuning a model on a snapshot of correct answers doesn't teach it to say "I don't know" when it's actually uncertain about something outside that snapshot — it just teaches it to confidently reproduce whatever pattern was in the training data, which can make hallucination on new questions look even more confident and convincing, not less.

"Our task is unusual, so we assume the base model can't do it well without training." In practice, modern general-purpose models handle a very wide range of tasks well with good prompting alone, including tasks that feel specialized. It's worth actually measuring baseline performance with a careful prompt and a real evaluation dataset before assuming the task needs fine-tuning — many teams skip this measurement step and pay for training a model that turns out to perform about the same as a well-prompted base model would have.

"We fine-tuned once, so now every new requirement should be added to the fine-tune." This slowly turns fine-tuning into a substitute for basic product iteration. If a new requirement can be handled by adjusting the prompt layered on top of your fine-tuned model, do that first — retraining should be reserved for changes substantial enough to actually need the model's baseline behavior to shift, not every minor tweak.

Frequently Asked Questions

Can I fine-tune and use RAG at the same time? Yes, and this is a common, effective combination. The fine-tuned model handles style, tone, and task-specific behavior; retrieval supplies the current, specific facts the model needs for a given request. Neither approach replaces the other — they address different failure modes entirely.

Does fine-tuning make a model "smarter" at reasoning tasks it currently struggles with? Generally no. Supervised fine-tuning, as covered in this unit, is best understood as teaching a model to reliably reproduce a demonstrated pattern of input-to-output behavior — it is not a general capability upgrade, and it won't reliably improve a model's ability to solve problems that require reasoning genuinely outside the patterns shown in your training examples. If your problem is "the model isn't capable enough," a more capable base model, or a fundamentally different prompting strategy such as giving the model more room to reason step by step, is a more direct fix than fine-tuning.

How do I know if my prompting attempts have genuinely "plateaued," versus me just not having tried hard enough? This is exactly what the evaluation habits from Lessons 1 through 3 are for — track your pass rate across successive prompt iterations on a fixed dataset. If several genuinely different prompting strategies (not just minor rewordings of the same approach) all land within a percentage point or two of each other, that's a real plateau, not an under-tried prompt. If you haven't tried at least a few structurally different approaches — different instruction framing, different example selection, breaking the task into smaller steps — you likely haven't exhausted prompting yet.

Is fine-tuning still worth learning if the platform is being restricted to existing users? Yes, for two separate reasons. If your organization already has access, the decision framework and mechanics in this unit are directly usable today, for as long as OpenAI continues to support existing users' training jobs. And even if you never touch OpenAI's specific fine-tuning API, the same tradeoffs — training-time cost versus prompt-time flexibility, style-and-format learning versus knowledge-grounding — apply identically to fine-tuning on any other provider or open-source model, so the reasoning itself remains directly transferable.

Organizational Readiness, Not Just Technical Fit

A task can be a perfect technical fit for fine-tuning and still be a bad idea for a specific team right now, because fine-tuning done responsibly requires ongoing ownership, not a one-time project. Before committing, it's worth asking a few questions that have nothing to do with the model itself.

Who owns the training dataset going forward? Datasets drift out of date the same way prompts do, except the fix isn't a quick edit — it's regenerating training data and running a new job. If nobody is clearly responsible for noticing when the fine-tuned model's behavior stops matching current requirements and refreshing the data, the model will quietly become stale in the exact way Lesson 1 warned about for prompts, just slower and harder to detect.

Do you already have an evaluation pipeline, or would you be building one for the first time just for this? If Lessons 1 through 3 describe a discipline your team hasn't adopted yet, fine-tuning is a genuinely risky place to adopt it for the first time, because the cost of an unmeasured mistake is much higher — a bad prompt change is a five-minute revert, a bad fine-tuning job is wasted training cost plus the time spent curating the data that produced it, and without an evaluation pipeline you may not even find out it was bad until it's already affecting users.

Is the underlying task stable enough to be worth training into weights? A task whose definition is still actively changing week to week — a new feature the product team is iterating on rapidly — is a poor fit for fine-tuning almost by definition, since you'd be committing to a snapshot of requirements that may be outdated before training even finishes. Fine-tuning rewards stability; save it for behaviors you're confident will still be correct in three months, and keep actively-evolving requirements in prompts, where they're cheap to adjust.

Best Practices for the Decision Itself

Always establish your evaluation framework — dataset and graders, from Lessons 2 and 3 — before you fine-tune anything, so you have an honest before-and-after comparison rather than an impression. Try the cheaper interventions in order and actually measure each one's effect, rather than skipping straight to fine-tuning because it feels like the "real" solution. Be specific about which failure mode you're trying to fix — a knowledge gap, a formatting inconsistency, or a genuine behavioral pattern — because only the last of those three is actually fixed by fine-tuning. And do the token-cost arithmetic for your actual request volume before assuming fine-tuning pays for itself; it very often does at scale, and just as often doesn't at low volume, where the ongoing maintenance burden outweighs the savings.

One Last Sanity Check Before Committing

Before starting the process in Lesson 5, it's worth writing down, in one or two sentences, the specific answer to a single question: "prompting plateaued at what exact metric, on what exact evaluation dataset, and fine-tuning needs to beat that number to be worth the investment." If you can't fill in that sentence concretely — a real percentage, from a real eval run, not an impression — that's a sign the decision isn't actually ready to be made yet, and it's worth going back to Lessons 1 through 3 and building that measurement first. Committing to a multi-day data curation and training effort without a clear, numeric target for what "success" means is one of the most common ways fine-tuning projects quietly become a sunk cost nobody can confidently evaluate afterward — you want to walk away from Lesson 5 able to say either "yes, this measurably beat the baseline" or "no, it didn't, and here's the number that shows it," not "it feels like it's probably better."

Lesson 5 assumes you've worked through this framework and landed on fine-tuning as the right tool for a specific, well-scoped task, and walks through the entire supervised fine-tuning process end to end — from preparing a training dataset to using the finished model in production.