We rewrote a production pipeline from LangChain to DSPy last spring, and the reason was not a benchmark — it was a two-day incident. A security team bumped our model from gpt-3.5-turbo to gpt-4-turbo, and the extractor started emitting its JSON wrapped in markdown fences. Fifty prompt chains in LangChain each had their own parse-and-retry glue, so fifty small fixes had to be found by paging through logs. Prompts in LangChain are data glued to code, and nothing recompiles when the world changes. The model changed; the whole system was wrong.

The failure mode LangChain codified

LangChain solved the problem of “call a model, maybe use a tool” and in doing so made the chain structure the artifact you maintain. Your system prompt is a hardcoded string, your few-shot examples are hardcoded lists, your output format is enforced by custom parsers, and none of it reacts to a metric. It is a framework for writing the same brittle code with a nicer API. We kept it for exactly one thing — agents with tool routing — where the framework’s integration surface (tool schemas, memory, vendor SDKs) is the value.

What DSPy actually is

DSPy’s claim is that prompts are code, so let a compiler write them. You declare input/output signatures, assemble modules, and define a metric. An optimizer then searches the prompt space — instruction text, few-shot examples bootstrapped from the training set, and (with MIPROv2) candidate instructions — to maximize your metric on a dev set. The workflow moves from “hand-tune a string until it works” to “declare the contract and let the optimizer earn its keep.”

The signature is the contract:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
import dspy
from dspy.evaluate import Evaluate
from dspy.teleprompt import MIPROv2

class Rewriter(dspy.Signature):
    """Rewrite a support ticket as a terse, entity-preserving search query."""
    ticket: str = dspy.InputField()
    query: str = dspy.OutputField(desc="3-8 words, keep product names and IDs")

class QueryAgent(dspy.Module):
    def __init__(self):
        super().__init__()
        self.rewrite = dspy.ChainOfThought(Rewriter)
        self.classify = dspy.Predict("query -> category")

    def forward(self, ticket):
        q = self.rewrite(ticket=ticket).query
        return self.classify(query=q)

def metric(example, pred, trace=None):
    return (example.category == pred.category and
            _token_overlap(example.query, pred.query) > 0.5)

teleprompter = MIPROv2(prompt_model=dspy.OpenAI(model="gpt-4o"),
                       task_model=dspy.OpenAI(model="gpt-4o", temperature=0))
compiled = teleprompter.compile(QueryAgent(), trainset=dev, metric=metric,
                                max_bootstrapped_demos=8, num_candidate_programs=10)
compiled.save("query_agent_v3.json")

Two details matter at compile time. task_model at temperature=0 makes the bootstrapped demonstrations deterministic, which is what lets the optimizer reason about which prompt changes actually helped. And you need a real dev set — 100-200 samples is our floor; below that the optimizer is fitting noise and the “compiled” prompt can be worse than the hand-written one.

The honest numbers

On our ticket-routing eval (400 held-out samples, 14 categories), the hand-tuned LangChain prompt scored 0.83 exact-match accuracy. DSPy with BootstrapFewShotWithRandomSearch — ~10 minutes of compile and about $2 of API spend — hit 0.88. MIPROv2 pushed to 0.90 but cost ~30 minutes and ~$12. The runtime profile barely changed: a compiled module is still one or two model calls; the overhead is compile time, not inference time. That is the trade-off to internalize: DSPy moves your engineering hours into a batch job that re-runs when the model changes, which is exactly where you want them.

Production lessons

• Never run the optimizer in the request path. Compile offline, compiled.save() to disk, load the artifact at boot. We saw teams call compile() inside a server handler; that is a $/request and a minutes-of-latency anti-pattern rolled into one.
• Keep a holdout eval set the optimizer never sees. The optimizers are greedy maximizers; without a holdout you will watch them memorize your dev set. We freeze a 100-sample holdout and gate deploys on it.
• Pin the compiler and the model together. Upgrading the model without recompiling is the exact mistake that started this post, inverted. We recompile in CI whenever either changes and diff the eval report.
• Log the compiled prompt artifacts. You cannot debug a prompt the optimizer wrote unless you can retrieve it. Store the compiled JSON next to the eval report.
• LangChain still has a lane. Tool-calling agents, vendor integrations, memory orchestration — LangChain and LangGraph remain reasonable there. The mistake is using them for prompt-bound extraction pipelines where a signature + optimizer gives you a regression-testable contract instead of a string.

DSPy is not “LangChain but newer.” It is a different division of labor: you write the contract and the metric, the optimizer writes the prose. That division is what makes the pipeline reproducible — the same signature, metric, and train set compile to the same prompt, no matter which model is under the hood. After the rewrite, the model bump that caused our incident became a 20-minute CI job instead of a two-day incident.