Fine-tuning a model for internal tooling usually means one thing is missing: the data. Our first attempt to adapt a model for internal ticket routing needed 10,000 labeled examples, and human annotation at our quality bar cost roughly $2.40 per example across two labelers and a reviewer — $24k and six weeks for a dataset we would outgrow in a quarter. Frontier models were cheaper than our labelers and faster than our review queue. But “generate variations with GPT-4” produces a dataset that looks great and collapses your model within a week. The difference between a dataset and a pipeline is the machinery around the generator: seeds, constraints, verification, and a golden set you never let the LLM touch.

The pipeline shape

Our pipeline has five stages and runs as a batch job, not inline:

1. Seed set. 150–300 hand-curated, edge-case-heavy examples. These encode the structure of the domain — the failure modes a generator cannot invent because it does not know your internal systems.
2. Expansion. A generator (GPT-4-class at the time) receives a seed, a schema, and a distribution spec, and emits variations with controlled perturbations: entity swaps, slot-filling, length and formality ranges, injected ambiguity.
3. Verification. Every generated sample must pass execution checks, not just schema checks. For code we run it in a sandbox; for routing we re-run the deterministic router and assert the label is stable; for SQL we execute against a scratch Postgres.
4. Deduplication. MinHash over normalized text, plus exact-match on the programmatic key fields.
5. Budgeting. We oversample deliberately from the tails of the difficulty distribution, because an LLM’s default output is the modal case, and a fine-tune on modal cases is a fine-tune that never sees the long tail.

A generator that holds the distribution

The failure mode of naive self-instruct is that the generator ignores your constraints and emits the same three sentence templates. You do not fix that with a longer prompt; you fix it by making the constraints a schema you validate against and a distribution you can measure. We emit structured records and assert on them:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
import json
import random
from pydantic import BaseModel, Field, ValidationError

class RoutingSample(BaseModel):
    prompt: str = Field(..., min_length=10)
    intent: str
    difficulty: int = Field(..., ge=1, le=5)
    needs_escalation: bool

def generate_batch(client, seeds, n=200):
    """Generate a batch, validate schema, and enforce a difficulty target."""
    target = {1: 0.05, 2: 0.25, 3: 0.45, 4: 0.20, 5: 0.05}
    samples, counts = [], {}
    while len(samples) < n:
        seed = random.choice(seeds)
        raw = client.chat.completions.create(
            model="gpt-4o",
            messages=[{
                "role": "user",
                "content": (
                    f"Rewrite this routing case: {seed.prompt}\n"
                    "Vary entities, phrasing, length, and formality. "
                    "Emit one JSON object with keys: prompt, intent, "
                    "difficulty (1-5), needs_escalation (bool)."
                ),
            }],
            response_format={"type": "json_object"},
            temperature=0.9,
        ).choices[0].message.content
        try:
            s = RoutingSample(**json.loads(raw))
        except ValidationError:
            continue                      # schema reject, do not retry silently
        if counts.get(s.difficulty, 0) / n >= target[s.difficulty]:
            continue                      # distribution budget met, skip
        samples.append(s)
        counts[s.difficulty] = counts.get(s.difficulty, 0) + 1
    return samples

The budget check is the part most people skip, and it is the whole game: without it the generator floods difficulty 2 and your model learns “moderate effort is normal.” With it, you control the curriculum from the data file, which is far easier to tune than a prompt.

Verification is where the value is

A rejected sample costs a token round-trip; an unverified sample costs a degraded model you will not notice for weeks. Our hard rule: if the task has a ground truth you can compute, compute it. We re-ran the deterministic router against generated prompts and kept only samples where the router and the generator agreed on the label. That single check removed ~30% of generated samples, and the model trained on the surviving 70% beat the model trained on all of them on every eval we ran.

Production lessons

• Audit for bias clusters. LLM generators overuse signature phrases — “delve”, “testament”, “it is important to note”. We grep generated sets for a stoplist of these and reweight the generator prompt. Left in place, they become the model’s dialect after one epoch of fine-tuning.
• Keep 10% golden data. We hold out 1,000 human-curated examples and mix them into every fine-tune. They anchor the model to real distribution and measurably slow representation collapse when the generator drifts between model upgrades.
• Pin the generator version. When we upgraded the generator, the generated distribution shifted enough to move eval scores by 2-3 points. We regenerate from frozen seeds and diff the distributions before retraining.
• Budget, then dedupe, then verify — in that order. Verification is the expensive stage; do it last so you never execute samples that would have been discarded for duplication or over-budget.
• Sample selection beats prompt engineering. We spent more time on the rejection criteria than on the generator prompt, and the criteria were what actually moved quality.

Synthetic data did not replace human annotation; it replaced 80% of it. The seeds, the golden set, and the verification checks are all human labor — concentrated where it counts instead of spread over 10,000 repetitive examples. That is the sustainable shape: use the model to do the tedious work, and spend your humans on the work that decides whether the dataset is good.