The naive fix for agents that forget is to buy a bigger context window. We ran that playbook and it is a trap: the failure mode of a long conversation isn’t a hard token limit, it’s signal decay. A 100-token summary of a 4-hour support session keeps the plot but loses the error code, the user’s exact phrasing, and the third attempt at a fix. Truncation keeps the facts but kills the plot. MemGPT’s insight — treat the model like an OS with virtual memory instead of a transcript with a cap — is the first architecture that actually manages the trade-off, and it shipped just in time for us (arXiv 2310.08560, October 2023).

The OS analogy, made literal

MemGPT splits the model’s working set into three tiers, exactly like a paging hierarchy:

• Core memory: a small fixed buffer with the system prompt, user profile, and persistent facts. Always in-context, always billed.
• Recall memory: append-only transcript of the conversation, kept outside the context and searched on demand.
• Archival memory: the durable store — documents, prior sessions, long-term facts — retrievable via a separate search tool.

The agent itself decides when to page: it emits tool calls like core_memory_append, recall_memory_search, and archival_memory_insert to evict and reload. Context management stops being a prompt-engineering hack and becomes a first-class function the model can reason about. On the paper’s deep-memory-retrieval benchmark, MemGPT on a plain 8K-window GPT-4 hit 100% task completion where the strongest baseline (fixed summarization over the same model) managed under 8%.

The loop in code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
MEMORY_BUDGET = 8000  # tokens; paper's GPT-4 default

def agent_turn(agent, user_message):
    agent.context.append({"role": "user", "content": user_message})

    if agent.tokens_used() > MEMORY_BUDGET:
        # 1. page out the oldest evictable events
        evicted = agent.evict_to_archival()
        agent.archival_memory_insert(evicted)
        # 2. decide what matters for the current turn and page it in
        query = agent.extract_search_query()
        hits = agent.recall_memory_search(query, k=10)
        agent.inject(hits)

    return agent.step()  # loop until the model calls "pause"

Two details in that loop matter more than they look. First, extract_search_query() is itself a model call — the agent generates the query that will retrieve its own history, so retrieval quality tracks model capability. Second, the budget check happens before the next model call, so the context never silently overflows; either the model pages or we return to the user. Deterministic budget accounting is the whole game in production.

Why this still matters with 128K models

By late 2023 GPT-4 Turbo shipped 128K context and it still did not retire this pattern, for three practical reasons:

• Cost. At $0.01/1K input tokens, a 100K-token context costs $1.00 per query before a single output token. Agents that loop ten times per user request are throwing dollars at the window instead of managing memory.
• Latency. Prefill time scales roughly with context length; a 128K prefill is materially slower than an 8K one, and agents already feel slow.
• Retrieval quality. Long contexts degrade into “lost in the middle”; putting the right 4K tokens in front of the model beats giving it 100K and hoping it looks.

We now treat context window size as a capacity budget the agent manages, not as permission to skip the memory layer.

Production lessons

• Cap memory tool calls per turn. Two is our ceiling; more and the agent loops on paging churn, burning tokens and latency on bookkeeping instead of answering.
• Deduplicate before you insert. Agents emit redundant self-reflection; we filter near-duplicate memory records at write time, and our archival store stopped growing at 3x the rate of genuinely new facts.
• Index recall with time and vectors. Semantic search on transcripts is great for “what did we say about X”; timestamps resolve “what did we say first”. Conflict resolution without a temporal index is guesswork.
• Sandbox the memory tools. An agent that can write arbitrary archival rows can poison its own future context. We restrict memory writes to schema-validated records and audit them out-of-band.

The uncomfortable truth: building an agent that remembers is a storage-engineering problem wearing an LLM costume. MemGPT gave us the vocabulary for it — and the memory hierarchy we built around it now outlives any single model we plug in.