Let me correct the framing up front: Llama 3.1 is text-only. The 8B, 70B, and 405B models released in July 2024 consume tokens, not pixels. Any pipeline that claims to feed page images into Llama 3.1 is either hallucinating an API or wrapping a vision model that isn’t Llama 3.1. The correct design — and the one that actually worked for us — treats images as their own embedding modality, indexes them in a separate vector space, and lets Llama 3.1 read the text side plus captions.

This matters because our enterprise docs are full of exactly what text chunking destroys: architecture diagrams, revenue charts, and layout-heavy pages where the position of a label carries meaning. Naive chunkers produced retrieval that confidently answered “where is the revenue chart?” with prose paragraphs about revenue. Recall was embarrassing.

The pipeline

1. Extract text and layout with pdfplumber, and render each page to PNG for the visual modality (page-level rendering, not figure detection — simpler and robust).
2. Embed the visual side with CLIP ViT-B/32 (512-d), which is cheap and surprisingly good at layout-and-chart-level retrieval. Optionally run a VLM like Qwen-VL to produce a text caption per page — that caption becomes retrievable text and a compact stand-in for the image.
3. Index both spaces as named vectors in Qdrant — text (384-d, bge-small) and image (512-d) — on the same point, so each page is one document with two embedding axes.
4. Retrieve with hybrid fusion (RRF) across both spaces.
5. Answer with Llama 3.1 on the retrieved text plus captions. It never sees pixels; it doesn’t need to.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct, HybridFusion

client = QdrantClient("http://qdrant.internal:6333")

client.upsert(
    collection_name="enterprise_docs",
    points=[
        PointStruct(
            id=page_id,
            vector={"text": text_vec, "image": image_vec},
            payload={
                "doc_title": "architecture_v2.pdf",
                "page": 4,
                "source": "platform-arch",
                "caption": "Deployment topology: api-gateway -> workers -> postgres",
            },
        )
    ],
)

hits = client.query_points(
    collection_name="enterprise_docs",
    prefetch=[
        {"query": "where is the revenue chart", "using": "text", "limit": 30},
        {"query": image_query_vec, "using": "image", "limit": 30},
    ],
    query=HybridFusion.RRF,
    limit=15,
)

Why named vectors instead of one concatenated blob

A single fused vector loses the axis distinction. A page’s text says “FY2023 results” while the actual numbers live in the chart; text-only retrieval finds the prose but not the figure, and image-only retrieval finds the figure but can’t confirm it’s the FY2023 one. Named vectors let the retriever keep the modalities separate and recombine them per query type: questions like “show me the revenue chart” naturally weight the image axis, “what does the doc say about capacity” weights text. RRF over the two spaces gives you both without hand-tuning weights.

Numbers from our deployment

We indexed ~40K pages of architecture and ops documentation (2.4M text chunks plus 40K page images) in about an hour on a single indexing box. On a chart-heavy evaluation set of 120 questions, recall@10 went from 0.52 (text-only) to 0.78 with the hybrid setup. Query latency added ~14ms p99 from the second prefetch — negligible. The page render, not the embedding, is the slow stage: pdfplumber is single-threaded, so we parallelize page rendering across a worker pool, and that stage dominates at ~120ms/page.

Cost and latency trade-offs

• Never send full-res images to the generator. Each high-resolution page costs real tokens. Cache page-level image embeddings once and only fetch the actual image on explicit user request — or, better, send the caption instead. The caption is 99% of the value at 1% of the cost.
• Keep the payload index. Qdrant’s payload indexes on source/doc_title turn a multi-tenant query from a full scan into a filtered search. We filter by tenant before doing any vector math.
• Hierarchy matters. Model one parent document that links to its page vectors, text summaries, and chart vectors. When a chart’s caption changes, you refresh one subtree, not the whole corpus.

Alternatives worth knowing

ColPali (late 2024) approaches this at a different level: a vision-language model that retrieves at the page level directly from rendered pages, handling layout natively without a separate text pipeline. We prototyped it. It indexed about 3x slower and queried about 2x slower, and the CLIP + bge hybrid was already at 0.78 recall@10 — so we stayed with the cheaper stack. If your docs are diagram-heavy and captions can’t carry the semantics, ColPali-style page retrieval is the right escalation. Otherwise, a text-space plus an image-space plus RRF is the boring, reliable answer.