The 2023 version of this question was about knowledge: “should we fine-tune so the model knows our domain?” That framing is dead in 2025. Current models — Claude Sonnet 4, GPT-4o/4.1, Gemini 2.5, Llama 4 for the self-hosted crowd — arrive with enormous context windows, reliable structured output, and prompt caching that makes long system prompts nearly free on cache hits. “Teaching” a model facts is now done by retrieval, not weights. The question that matters today is narrower and sharper: is fine-tuning buying you something that a better prompt, a retrieval layer, or a different model can’t? Usually the answer is no. Sometimes it’s emphatically yes, and the mistake is refusing to see it.

What the money actually buys you

Fine-tuning earns its keep in three places, and none of them are “the model knows our domain”:

1. Format and behavior consistency. You need structured output that won’t drift — a 500-field extraction, a strict JSON schema, a house style of prose. Prompting gets you 90% there; fine-tuning closes the long tail of formatting regressions that an eval harness can measure.
2. Latency and cost at volume. Every token in your system prompt costs TTFT and per-token price. Distilling a 2,000-token instruction set into a fine-tuned 100-token prefix can cut TTFT by 30-50% on long-context calls, and dropping to a smaller fine-tuned model (GPT-4o mini, or an 8B Llama via LoRA) slashes per-call cost at high volume.
3. Distillation of a bigger model. You don’t need the frontier model per-request; you need its judgment. Generate 10,000 labeled examples with the big model, fine-tune a small one, and keep the small one in production.

Prompt engineering’s advantages are the mirror image: iteration is measured in minutes and cents instead of a training run; retrieval means the knowledge changes without a retrain; and vendor model churn (we’ve swapped Claude models twice this year alone) doesn’t invalidate a prompt the way it can a checkpoint.

The failure-mode heuristic

Don’t fine-tune on vibes — fine-tune on evidence. We gate it behind a checklist:

• There is a measurable eval gap. A held-out set scores >=5-10 points better on the fine-tuned candidate than on the prompted baseline. If the gap is under 5 points, the prompt is the problem.
• The failures are consistent, not random. Logs show the same malformed output, the same hallucination on the same class of input, repeatedly. Random noise won’t be fixed by gradient descent.
• You have high-quality paired data. 500-2,000 curated prompt-output pairs beat 100,000 rows of scraped garbage. Every fine-tuning war story is a data-cleaning story.
• The volume justifies the fixed cost. Training, evaluation, and checkpoint maintenance are ongoing — the traffic has to pay for them.

And the reverse test, which we run first: does retrieval-plus-prompt-caching close the gap? For knowledge-heavy tasks it usually does, and it’s a weekend of work.

An eval harness before you spend anything

Whatever path you take, you need the harness first. Ours is boring on purpose — a held-out set of 200 real inputs, a scoring function, and a report:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
require "openai" # or any provider SDK

def score_extraction(provider, model, system, inputs)
  correct = 0
  inputs.each do |input, expected|
    out = provider.chat(model, system: system, messages: [{ role: "user", content: input }])
    parsed = JSON.parse(out)
    correct += 1 if parsed["status"] == expected["status"] && parsed["total_cents"].to_i == expected["total_cents"]
  end
  correct.to_f / inputs.size
end

baseline = score_extraction(OpenAI, "gpt-4o", SYSTEM_PROMPT, EVAL_SET)
# run again against your fine-tuned checkpoint, then compare

Run this on every candidate: prompted frontier model, prompted small model, fine-tuned small model. The decision becomes a number instead of an argument.

What the landscape looks like now

The 2025 fine-tuning menu is wider than the 2023 one: OpenAI offers fine-tuning on GPT-4o and the GPT-4.1 family; open models fine-tune cheaply with LoRA/QLoRA on a single GPU, and the platform quality bar is high enough that a well-fine-tuned 8B model beats a sloppy prompted 70B on a narrow task. What has not changed is the cost structure of being wrong: a fine-tuned model you can’t explain, can’t retrain quickly, and can’t audit is a liability wrapped in an accuracy bump.

The playbook

Start with the cheapest lever that produces a measurable improvement — prompt design, then retrieval with prompt caching, then structured output constraints. Keep monitoring eval scores and failure rates as the traffic drifts. Fine-tune only when a prompted baseline demonstrably cannot reach the target, you have the data to do it well, and the volume pays for the maintenance. In 2025 that’s maybe one in ten AI features — and that’s the right ratio.