Current Availability Note
Before You Start: Current Availability
This walkthrough uses OpenAI's supervised fine-tuning API as it exists today. It's important to be direct about its status: OpenAI has been winding down the self-serve fine-tuning platform, and as of the most recent documentation, new fine-tuning job creation is restricted to organizations with an existing fine-tuning history — it is not currently open to brand-new users. If your organization has used fine-tuning before, you can continue creating jobs for the near term. If you're trying this for the first time and don't already have that history, the API calls below may not be available to your account, and you should check your organization's current access before investing time building a pipeline around them.
Even so, walking through the full process — from data preparation to a working custom model — is worth doing. Every step here (structuring training examples, choosing hyperparameters, evaluating a trained model against a holdout set) is a skill that transfers directly to fine-tuning on any other platform, and if you're building on the assumption OpenAI's platform stays available to your organization, this is exactly how the process works today.
We'll continue the example from Lesson 4: a support ticket priority classifier that plateaued around 90% accuracy with prompting alone, with a persistent, prompt-resistant gap on nuanced cases involving account tier and message tone.
Step 1: Prepare Your Training Data
Supervised fine-tuning data uses the same chat-format structure as a normal Chat Completions or Responses API call — a JSONL file where each line is one complete training example, expressed as a list of messages.
Format and Minimums
Each line must be a JSON object with a messages array following the standard chat structure:
{"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}The platform's stated minimum is 10 examples, but that floor exists to let the API accept a file — it is not a realistic number to actually improve a model's behavior. OpenAI's own guidance recommends starting around 50 well-crafted demonstrations as a practical baseline, with 50 to 100 examples as the range where teams typically start seeing solid, measurable improvement. Beyond that, more high-quality data tends to help further, though gains generally follow a pattern of diminishing but real returns: expect a similar-sized improvement each time you roughly double your dataset, rather than linear scaling.
priority_training.jsonl
{"messages": [{"role": "system", "content": "You are a support ticket priority classifier. Respond with exactly one label: urgent, high, normal, or low."}, {"role": "user", "content": "Account tier: Enterprise. Message: Our production integration just stopped authenticating and every API call is failing right now."}, {"role": "assistant", "content": "urgent"}]}
{"messages": [{"role": "system", "content": "You are a support ticket priority classifier. Respond with exactly one label: urgent, high, normal, or low."}, {"role": "user", "content": "Account tier: Free. Message: Just wondering if you plan to add dark mode at some point."}, {"role": "assistant", "content": "low"}]}
{"messages": [{"role": "system", "content": "You are a support ticket priority classifier. Respond with exactly one label: urgent, high, normal, or low."}, {"role": "user", "content": "Account tier: Pro. Message: I'm getting billed twice this month and I need this fixed before my card gets charged again tomorrow."}, {"role": "assistant", "content": "high"}]}
{"messages": [{"role": "system", "content": "You are a support ticket priority classifier. Respond with exactly one label: urgent, high, normal, or low."}, {"role": "user", "content": "Account tier: Enterprise. Message: Small UI glitch on the settings page, the save button is slightly misaligned."}, {"role": "assistant", "content": "normal"}]}A real training file would have 50–100+ lines like these, and — this is the part that matters most — it should be drawn heavily from real, historical support tickets your team already labeled, not invented from scratch. This is precisely the scenario Lesson 4 described as ideal for fine-tuning: a task with abundant real, labeled historical data sitting in an existing system, rather than a task you'd have to synthesize examples for from imagination.
Data Quality Checklist Before You Upload
Apply these checks before training, because OpenAI's own best-practices guidance is explicit that a fine-tuned model can only be as consistent as the data it's trained on:
- Every example follows the exact same system message. If some examples have a slightly different system prompt than others, the model receives mixed signals about what context it should expect at inference time.
- Labels are internally consistent. If two nearly identical tickets have different assigned priorities in your training set because two different people labeled them, the model has no way to learn a consistent rule — it will average toward inconsistency. Resolve disagreements before training, not after.
- Every example contains all the information needed to produce the label. If a human labeler used context that isn't in the
usermessage itself (like knowledge of a specific customer's history that isn't written into the ticket text), the model can't learn that pattern — it will look inconsistent, because from the model's perspective the same input sometimes gets different answers. - No example exceeds a reasonable token length. Training examples that run long are truncated from the end, which can silently cut off the assistant's actual answer. Check your longest examples specifically, rather than assuming your typical example length is representative.
Step 2: Upload the Training File
from openai import OpenAI
client = OpenAI()
training_file = client.files.create(
file=open("priority_training.jsonl", "rb"),
purpose="fine-tune",
)
print(training_file.id)Note the purpose value here is "fine-tune", distinct from the "evals" purpose used in Lesson 2 — the API uses this field to validate the file against the correct expected schema for its intended use. If you want to evaluate your fine-tuned model afterward with a held-out set (strongly recommended, covered in Step 6), prepare that as a separate file now too, structured the same way, but never included in the training file itself.
Step 3: Create the Fine-Tuning Job
job = client.fine_tuning.jobs.create(
training_file=training_file.id,
model="gpt-4.1-nano-2025-04-14",
method="supervised",
)
print(job.id, job.status)Walking through the parameters: training_file is the file ID from Step 2. model is the base model you're customizing — the currently supported base models for supervised fine-tuning are the gpt-4.1 family: gpt-4.1-2025-04-14, gpt-4.1-mini-2025-04-14, and gpt-4.1-nano-2025-04-14. gpt-4.1-nano is a reasonable starting point for a well-defined classification task like this one — it's the cheapest and fastest of the three to both train and run at inference time, and classification tasks with a small, fixed label set generally don't need the largest base model's extra capacity. method defaults to "supervised", which is the technique this entire lesson covers; OpenAI's platform also supports other fine-tuning methods for different use cases, but supervised fine-tuning on labeled input/output pairs is the standard starting point and the right fit for the classifier example here.
Step 4: Tune Hyperparameters (Optional, But Know What They Do)
If you don't specify hyperparameters, the API uses defaults that work reasonably well for most tasks. It's still worth understanding what they control, because if your first training run underperforms, adjusting these — rather than just throwing more data at the problem — is often the actual fix.
job = client.fine_tuning.jobs.create(
training_file=training_file.id,
model="gpt-4.1-nano-2025-04-14",
method="supervised",
hyperparameters={
"n_epochs": 3,
"batch_size": 4,
"learning_rate_multiplier": 1.0,
},
)n_epochs (default: 10) controls how many full passes the training process makes over your dataset. More epochs mean the model sees your examples more times, which helps it learn the pattern more thoroughly — but past a certain point, more epochs on the same fixed dataset causes the model to overfit, memorizing the specific training examples rather than generalizing the underlying rule. OpenAI's guidance suggests increasing epochs by one or two if the model seems to be underfitting (still not reliably following the pattern by the end of training) and decreasing epochs if your dataset lacks diversity, since a less diverse dataset overfits faster.
batch_size (default: 1) controls how many training examples are processed together in each training step before the model's weights are updated. A larger batch size tends to make training more stable (each update is based on an average across more examples, reducing noise) but also slows down training and can require more epochs to reach the same level of learning, since the model gets fewer weight updates per epoch at a larger batch size.
learning_rate_multiplier (default: 1) scales how large each weight update is. If training loss plateaus early and doesn't seem to be improving further, increasing this can help the model converge past a stall. Too high a value can cause training to become unstable rather than better.
There's no universally correct combination — the practical approach is to start with defaults, evaluate the result against your holdout set (Step 6), and adjust one hyperparameter at a time based on a specific, observed problem, rather than guessing at a full configuration upfront.
Step 5: Monitor the Job
Fine-tuning jobs run asynchronously and can take anywhere from minutes to hours depending on dataset size and base model. Poll the job status, and inspect the events endpoint for detailed progress:
import time
while True:
job = client.fine_tuning.jobs.retrieve(job.id)
if job.status in ("succeeded", "failed", "cancelled"):
break
time.sleep(30)
print(job.status)
events = client.fine_tuning.jobs.list_events(job.id, limit=20)
for event in events.data:
print(event.created_at, event.message)The events stream reports meaningful milestones — file validation, the start of each training step, and, importantly, checkpoint creation at the end of each epoch. Checkpoints from the last three epochs remain available after training completes, which means you can compare the model's behavior at, say, epoch 8, 9, and 10, and choose to actually deploy an earlier checkpoint if it turns out to generalize better than the final one — a real, useful option if you suspect the final epochs caused some overfitting.
Safety Evaluation
Once training itself finishes, OpenAI runs an automated safety evaluation against the resulting model across a defined set of categories — including harassment, hate speech, self-harm, sexual content, and violence, 13 categories in total. A model that fails these thresholds is blocked from being used for inference, regardless of how well it performed on your own task-specific measure of quality. You can review this step's outcome by checking events with type moderation_checks. For a narrow, business-context task like ticket priority classification trained on clean internal support data, this check is rarely the source of a problem, but it's worth knowing it exists so a blocked deployment doesn't come as a surprise — and so you don't mistake a safety block for a bug in your own code when a job otherwise "succeeds" but the model won't serve requests.
Step 6: Evaluate Before You Trust It
This is the step it's easiest to skip under time pressure, and the one this entire unit has been building toward not skipping. Once your job's status is succeeded, the response includes a fine_tuned_model field — an ID in the format ft:gpt-4.1-nano-2025-04-14:openai::BTz2REMH — that you use exactly like a base model name in any Responses or Chat Completions call.
Before routing real traffic to it, run it through the exact eval workflow from Lessons 2 and 3, against a held-out set of tickets that were never part of the training file:
run = client.evals.runs.create(
eval_config.id, # the same eval object from Lesson 2, reused
name="fine-tuned nano vs base prompt",
data_source={
"type": "responses",
"model": job.fine_tuned_model,
"input_messages": {
"type": "template",
"template": [
{
"role": "system",
"content": "You are a support ticket priority classifier. Respond with exactly one label: urgent, high, normal, or low.",
},
{"role": "user", "content": "{{ item.ticket_text }}"},
],
},
"source": {"type": "file_id", "id": holdout_file.id},
},
)Because this reuses the same eval definition and grading criteria as your earlier prompting-only runs, result_counts gives you a direct, apples-to-apples comparison: did the fine-tuned model actually beat the 90% ceiling the prompting-only approach plateaued at in Lesson 4, on data it never saw during training? If it didn't meaningfully improve on your holdout set, that's a real, important result — it means either the training data needs more or better examples, a hyperparameter needs adjusting, or the underlying gap wasn't actually the kind of problem fine-tuning fixes after all, and it's worth revisiting the Lesson 4 framework rather than assuming more training will eventually help.
Step 7: Use the Fine-Tuned Model in Production
Once you've confirmed a real improvement on your holdout evaluation, using the model is identical to using any base model:
response = client.responses.create(
model=job.fine_tuned_model,
input=[
{
"role": "system",
"content": "You are a support ticket priority classifier. Respond with exactly one label: urgent, high, normal, or low.",
},
{
"role": "user",
"content": "Account tier: Pro. Message: The export feature has been broken for three days and my team can't finish our report.",
},
],
)
print(response.output_text) # e.g. "high"Notice the prompt here is dramatically shorter than the heavily few-shot-laden prompt from Lesson 4's prompting-only approach — the model has learned the classification pattern from training rather than needing it re-demonstrated in every call, which is exactly the token-cost benefit discussed in that lesson's cost comparison.
Suggested Project Structure
As with the eval setup in Lesson 2, it's worth separating the one-time data-preparation and job-creation steps from the parts you'll want to re-run or reference repeatedly:
priority-classifier-finetune/
├── .env
├── requirements.txt
├── data/
│ ├── priority_training.jsonl # training examples, never touched by evals
│ └── priority_holdout.jsonl # held-out set, used only for evaluation
├── prepare_data.py # builds/validates the JSONL files from raw ticket exports
├── create_job.py # uploads the training file and starts the job
├── monitor_job.py # polls status and prints events until done
└── evaluate_model.py # runs the Lesson 2/3 eval against the finished modelKeeping data/priority_training.jsonl and data/priority_holdout.jsonl in clearly separate files, rather than one large file you split programmatically each time, makes it much harder to accidentally leak holdout examples into training — a mistake that's easy to make silently and hard to detect after the fact, since a model trained on leaked holdout data will look deceptively good on that same holdout set.
A Distillation Variant Worth Knowing
One especially useful pattern, directly supported by this same workflow, is training a small, cheap model to imitate a larger, more expensive one — a technique generally called distillation. Instead of hand-labeling your training examples yourself, you generate them by running a larger, more capable model (with careful prompting, possibly including the kind of extensive few-shot examples Lesson 4 discussed) over a batch of real inputs, and use its outputs as the assistant messages in your training file.
# Step A: generate labels using a larger, more careful-prompted model
large_model_response = client.responses.create(
model="gpt-4.1",
input=[
{"role": "system", "content": long_careful_instructions_with_examples},
{"role": "user", "content": ticket_text},
],
)
label = large_model_response.output_text
# Step B: write that as a training example for the smaller model
training_example = {
"messages": [
{"role": "system", "content": short_instructions},
{"role": "user", "content": ticket_text},
{"role": "assistant", "content": label},
]
}Repeated across a large batch of real ticket inputs, this produces a training file where the labels come from a strong model's careful judgment rather than manual human labeling — useful when you don't have a large volume of pre-existing human-labeled data, but do have budget to spend a larger model's inference cost once, upfront, to generate it. The resulting fine-tuned small model (gpt-4.1-nano, in this lesson's example) often ends up performing close to the larger model's quality on that specific narrow task, while costing far less to run at inference time going forward — you've effectively paid the larger model's cost once, during data generation, instead of on every single production request. Always validate a sample of the generated labels by hand before training on them; a distillation pipeline inherits any systematic errors the larger model makes just as faithfully as it inherits its correct judgments.
What Happens to This Model Over Time
Given the platform status covered at the top of this lesson, it's worth being clear about what does and doesn't change once your model is trained and deployed. The fine-tuning job creation API — the part restricted to existing users — is what's being wound down; it governs whether you can start a new training run. A model you've already successfully trained keeps serving inference requests normally, billed and rate-limited the same way as any other model, for as long as its underlying base model remains supported. OpenAI's guidance states fine-tuned models remain available for inference until their base model itself is deprecated — a separate, later event than the fine-tuning platform's shutdown.
Practically, this means: if you train the priority classifier from this lesson today, it keeps working in production on its normal schedule. What becomes harder is the next retraining pass — if your organization loses fine-tuning access before you need to retrain on fresher data, you'd need to fall back to the prompting-plus-RAG approaches from Lesson 4, or evaluate whether a different provider's fine-tuning offering can pick up where OpenAI's leaves off using the same training data format concepts covered in this lesson. This is a real, practical reason to treat the evaluation habits from Step 6 as non-negotiable now — you want hard evidence of how much fine-tuning is actually buying you, in case a future version of this exact decision has to be made without OpenAI's fine-tuning API as an option.
Frequently Asked Questions
How much does a fine-tuning job cost compared to just calling the base model? Training itself is billed separately from inference, based on the size of your training file and the number of epochs — more data and more epochs mean a larger one-time training cost. After training, inference on the fine-tuned model is billed per token, generally at a different (often somewhat higher) per-token rate than the equivalent base model, offset in practice by the shorter prompts a fine-tuned model typically needs, as discussed in Lesson 4's cost comparison. Check current pricing directly before committing, since these rates are set independently of the mechanics covered in this lesson and can change.
Can I cancel a running job? Yes — client.fine_tuning.jobs.cancel(job.id) stops a job that's still in progress, which is useful if you spot a data problem in the events log shortly after starting and don't want to wait out a job you already know is compromised.
What if I want to keep improving the same fine-tuned model rather than starting over? Some fine-tuning workflows support continuing training from an existing fine-tuned model as the new starting point rather than the original base model, which can be a faster way to incorporate a batch of new examples without re-learning everything from scratch. Confirm current support for this in your organization's available API surface before relying on it, since fine-tuning platform capabilities are exactly the area currently in flux.
Do I need to remove the training and holdout examples from my regular eval dataset used in Lessons 2 and 3? Keep them conceptually separate even if they cover the same underlying task. The eval dataset from Lesson 2 is meant to track a deployed system's quality over time and should reflect the real, current distribution of production traffic; your fine-tuning holdout set is a fixed snapshot used specifically to validate one training run against its own training data. It's fine, and often useful, for your eval dataset to include some of the same holdout examples once you're comfortable the model generalizes well, but don't let the two purposes blur into a single file you're not sure how to interpret.
Should I retrain from scratch or just add new examples and rerun? Given that a full run isn't prohibitively expensive at these dataset sizes, the safer default is retraining from the original base model with your full, updated dataset (old examples plus new ones), rather than repeatedly incrementally training on top of prior fine-tunes — this avoids compounding drift where small quirks from each successive round accumulate in ways that are hard to trace back to a specific batch of training data.
Common Mistakes
Training on a dataset with inconsistent labels. If your historical support data has cases where similar tickets were prioritized differently by different agents, the model learns that inconsistency as noise, capping how well it can ever perform. Clean and reconcile your labels before training, not after a disappointing result.
Evaluating against training data instead of a genuine holdout set. A model can score suspiciously well against examples it was directly trained on without that meaning anything about how it handles new tickets. Always keep a separate holdout set the model has never seen, and be disciplined about not letting the two files mix.
Assuming more epochs always means a better model. Past the point where the model has learned your pattern, additional epochs on the same fixed dataset tend to cause overfitting — strong performance on training examples, worse generalization to new ones. Watch your holdout evaluation score across different epoch checkpoints, not just the training loss curve.
Shortening the system instructions to save training cost. As covered in Lesson 4, this backfires — vaguer instructions require more training examples to reach the same reliability, not fewer. Keep your instructions precise going into training.
Troubleshooting
Job status moves to failed. Check the events list for the specific failure reason — the most common causes are a malformed JSONL file (validate every line parses as JSON before uploading) or a file that doesn't match the expected chat-format schema for the given method.
The resulting model performs worse than the prompting-only baseline on your holdout eval. This usually traces back to data quality rather than hyperparameters — inconsistent labels, too few examples, or examples that don't actually contain the information needed to justify their labels. Revisit the data quality checklist in Step 1 before touching n_epochs or learning_rate_multiplier.
A completed job's model can't be used for inference. Check the moderation_checks events for that job — this is the signature of the automated safety evaluation blocking the model. If your training data is legitimate business content and this still triggers, review the specific flagged category in the event details rather than assuming it's a false positive.
Training examples appear to have been cut off. This is token-limit truncation — examples are truncated from the end when they exceed the per-example token limit. Check your longest examples specifically and either shorten them or split the content so the important part (particularly the assistant's answer) isn't at risk of being cut.
The model performs well on common cases but poorly on rare ones. This is almost always a data distribution problem rather than a hyperparameter problem — if 90% of your training examples are normal priority tickets and only a handful cover urgent, the model has seen far too few examples of the rare, high-stakes category to learn it reliably. Deliberately oversample the categories that are rare in your raw historical data but important to get right, rather than assuming a purely random sample of past tickets will produce a balanced training set on its own.
Best Practices
Keep the exact holdout evaluation results from every training run you produce, even ones you discard — a simple log entry with the date, the hyperparameters used, the training file's contents (or a hash of it, at minimum), and the resulting pass rate on your holdout set. When a second or third training attempt improves on the first, that log is what lets you say with confidence which specific change caused the improvement, instead of just knowing the final model is better than where you started without being able to explain why. This kind of lightweight experiment tracking costs almost nothing to maintain and pays for itself the first time a training run underperforms and you need to work out which variable to adjust next.
Always hold out a genuine evaluation set before you start, using the exact same eval-and-grader workflow from Lessons 2 and 3, so "did fine-tuning actually help" has a real, comparable number attached to it instead of an impression. Start with a base model and hyperparameter defaults appropriate to your task's difficulty and label-set size — don't reach for the largest base model or heavily customized hyperparameters before you've established a baseline result with the defaults. Treat your training dataset as a living asset: as new real-world tickets come in and get correctly labeled by your team, they're candidates for a future retraining pass, the same way Lesson 2 argued eval datasets should keep growing from real production traffic. And remember Lesson 4's core lesson going into all of this: fine-tuning is the right tool for a narrow, well-defined behavioral gap that prompting couldn't close — it's not a general-purpose upgrade, and its value only shows up when you measure it honestly against the alternative you already had.