Vector RAG answered “what does the payment service do?” beautifully and went blind on “what did the author build for the payment service while reporting to project manager X?” The first is a single fact; the second is a relationship spanning three documents. On our internal Q&A over 1.4M docs, dense retrieval scored 0.87 recall@20 on single-entity questions and 0.41 on multi-entity ones — because vector search returns isolated chunks, and the relationship lives in the space between them. We rebuilt around a knowledge graph: extract entities and relations into Neo4j, use vectors as entry points, and let bounded Cypher traversal assemble the cross-document context a plain chunk never contains.

The indexing pipeline

We chunk documents, run an LLM extractor over each chunk with a strict JSON schema for (subject, predicate, object) triples, and write the results to the graph. Two rules keep the graph from turning into garbage:

• Schema-validate every triple. The extractor must emit {"s": {"type": "person", "name": ...}, "p": "REPORTED_TO", "o": {...}} against a fixed enum of predicates. A free-form triple is a fact you cannot query; malformed JSON is a graph corruption.
• Resolve entities asynchronously. “Rails”, “Ruby on Rails”, and “the framework” must collapse to one node before insertion. We run a normalized-name + embedding-based resolution pass in the pipeline; skip it and every answer that needs a “sibling” relation fails silently.

Query time: vectors find the door, Cypher walks the hall

The agent does not generate a Cypher query from scratch — that is how you get RETURN 1000 nodes and a 4-second query. Instead the flow is: embed the question, hit the node vector index for the top-k entry nodes, then run a fixed, bounded traversal from those entries along a small allowed set of relation types, capped at 2-3 hops. The agent picks the entry nodes and the traversal frontier; it does not invent syntax:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import os
from neo4j import GraphDatabase
from openai import OpenAI

driver = GraphDatabase.driver("bolt://neo4j:7687", auth=("neo4j", os.environ["NEO4J_PASSWORD"]))
client = OpenAI()

TRAVERSAL = """
CALL db.index.vector.queryNodes('entity_embeddings', 8, $query_vector)
YIELD node, score
WITH node, score ORDER BY score DESC
MATCH (node)-[r:DEVELOPED_FOR|REPORTED_TO|PART_OF|MENTIONS]-(n2)
WITH node, score, type(r) AS rel, n2, startNode(r) AS from, endNode(r) AS to
RETURN from.name AS subject, rel, to.name AS object, score
ORDER BY score DESC LIMIT 40
"""

def agentic_retrieve(question):
    emb = client.embeddings.create(model="text-embedding-3-small",
                                   input=question).data[0].embedding
    with driver.session() as s:
        rows = s.run(TRAVERSAL, query_vector=emb).data()
    ctx = [f"{r['subject']} --{r['rel']}--> {r['object']}" for r in rows]
    return "\n".join(ctx)  # assembled context goes into the answer prompt

db.index.vector.queryNodes (Neo4j 5.11+) lets the vector index return graph nodes directly, so the entry-point hop is a single call. The bounded, relation-whitelisted traversal is what keeps p95 at ~35 ms on a 3-node Neo4j cluster holding 6M nodes — versus the 900 ms+ we measured for the “generate any Cypher and hope” approach.

The agentic part is the context assembly

The retrieval result is a set of triples, not an answer. The agent — a GPT-4o class model with a tool for graph retrieval — decides which triples are on-topic, walks a second hop on promising subjects, and writes the final answer, citing the source documents of the triples it used. The graph gives the agent a map; that is what lets it answer “did X and Y ever work together?” without any single chunk containing the answer. We keep the graph strictly for evidence assembly — every triple in context exists in the database; the model never invents edges.

Production lessons

• Cap the hops and the node fan-out. A 2-hop traversal over a hub node can explode to tens of thousands of matches. We hard-cap depth at 3, limit results per hop, and precompute a “hub score” that penalizes high-degree nodes as traversal targets. There is no correct infinite traversal; treat this as a cost model.
• Timebox every traversal. We set the Neo4j query timeout to 500 ms and the retrieval deadline to 150 ms; on timeout the agent gets an empty graph, not a half-answered question.
• Watch the extraction tax. Extraction at ~2,000 docs/hour with a 4x model costs about $18 per 100k docs in tokens and ~2x wall-clock because JSON schema retries dominate. Extraction quality, not retrieval code, is your recall ceiling.
• Prompt-injected Cypher is an attack surface. Since we never execute model-generated Cypher, a user who convinces the model to emit MATCH (n) DETACH DELETE n gets a triple in context, not a dropped database.
• Hybrid graph + vector, not either/or. We keep a parallel vector store over chunks for “describe this topic” questions, where the graph’s structured answer is wrong-footed. The routing agent picks graph vs chunk retrieval per query; forcing every query through the graph cost 6 points on single-fact evals.

Alternatives worth knowing

If you cannot run a graph database, Microsoft’s GraphRAG (2024) gets relational answers via hierarchical summarization — but that local-to-global pass is expensive to build and refresh. If most of your queries are single-entity, skip the graph; extraction cost only pays off when relationships are the query currency, and for us that was ~46 points of recall@20 on multi-entity questions. Start with bounded vector entrypoints and a fixed traversal; it is the only design we shipped that answers cross-document questions both fast and genuinely sourced.