Catalog search lives and dies on the visual queries text search never sees: “floral dress, but not the one with the collar”, “refrigerator that fits under a 32-inch counter”, “product photo that looks like a loft interior”. Metadata cannot answer these because the attributes are not tagged — nobody catalogs “vibe”. Training a classifier per attribute does not scale past a handful of concepts. So in 2023 we indexed our catalog with CLIP and relied on zero-shot cross-modal search. This year we migrated the same pipeline to SigLIP. The migration was not a model swap; it changed three things silently, and two of them broke thresholds until we understood them.

What the loss function actually changes

CLIP (Radford et al., 2021) uses a softmax contrastive loss over the batch: each image-text pair competes against every other pair in the batch as negatives. That makes the gradient quality a function of batch size — 32,768 was the paper’s sweet spot, and a 2,048 batch produces noticeably worse embeddings. SigLIP (Zhai et al., October 2023) replaces the softmax with a per-pair sigmoid loss, treating each negative independently. The result is that batch size stops being a quality dial: the paper shows sigmoid trained at batch 4,096 matching or beating softmax at 32,768, while using 8x less memory.

That property matters more in practice than the headline accuracy. We fine-tune these models on product data, and fine-tuning at 32,768 batch is a multi-node job. SigLIP let us fine-tune on a single A100 node with a 4,096 batch and still gain on zero-shot recall. On our internal 8,000-image catalog test set, SigLIP-base-patch16-224 lifted zero-shot retrieval recall@10 from 0.81 (CLIP ViT-B/32) to 0.87, which for a search team is the difference between “sort of works” and “canonical.”

The two silent changes

Normalization and thresholds. Both models L2-normalize embeddings before computing logits, so within an index, cosine and dot-product rankings are identical — the “dot product skews rankings” warning you see floating around applies to unnormalized vectors, not to these. What does change is the temperature scale (logit_scale) and the projection: SigLIP logits are t·cos + b with both learnable, and HF’s SiglipModel applies a projection plus normalization per modality. The practical effect is that a 0.28 cosine threshold tuned for CLIP keeps ~12% more junk in SigLIP. We re-tuned on a 500-pair labeled slice rather than trusting the old cutoff.

Preprocessing. CLIP wants resize-to-224-then-center-crop; SigLIP just resizes with bicubic interpolation and normalizes with mean/std 0.5. Feeding SigLIP the CLIP-cropped images silently degrades retrieval on edge-important queries (patterns, logos). Keep the processor paired to the model — this is where AutoProcessor earns its keep.

The embedding path

1
2
3
4
5
6
7
8
9
10
11
12
13
14
from transformers import AutoProcessor, AutoModel
import torch

processor = AutoProcessor.from_pretrained("google/siglip-base-patch16-224")
model = AutoModel.from_pretrained("google/siglip-base-patch16-224").eval()

def embed(image, text=None):
    inputs = (
        processor(text=[text], images=image, return_tensors="pt")
        if text else processor(images=image, return_tensors="pt")
    )
    with torch.no_grad():
        out = model(**inputs)
    return out.image_embeds.squeeze(0), out.text_embeds.squeeze(0)

Batch these, not per-item: a single 224×224 forward is ~4 ms on an A10, and batching 32 images brings it to ~1.3 ms/image. We embed images offline at write time into a pgvector / Qdrant index, and text only at query time — per-query encoder latency is ~5 ms, which disappears next to the ANN search and rerank stages.

Production lessons

• Calibrate thresholds per model, per metric. We keep a labeled golden set of pairs and recompute precision@k cutoffs after any model bump. The temperature change is exactly the kind of thing that shifts a p95 “good enough” threshold into a silent recall regression.
• Embed once, at the edge. Resize and normalize on the client before upload. We cut processor latency out of the ingest path and halved the bytes we ship to the embedding service by downscaling to 512 px before transmit; the model resizes to 224 anyway.
• Keep image and text on the same normalization. We once computed text embeddings with a fine-tuned checkpoint and image embeddings with base — silently different projection heads, and every cross-modal score was garbage. Pin the checkpoint hash in the embedding service config.
• Watch patch size vs. catalog domain. patch16 sees more detail than patch32 at 224; for pattern-heavy catalogs the extra tokens cost ~1.5x inference but paid for itself in recall@10 on logos.

Alternatives worth knowing

SigLIP is our default because it is a drop-in CLIP with better small-batch fine-tuning. If your queries are more language-heavy than visual, a text-strong pair (e.g., a SigLIP image tower with a stronger text encoder) helps; if you need product-level accuracy and can label, fine-tune CLIP-family embeddings on your own triplets — zero-shot is a floor, not a ceiling. And for very large corpora, remember the vector index, not the encoder, becomes your real cost once you pass a few million rows.

The short version: SigLIP is CLIP with the batch-size tax removed, a smarter loss, and quietly different output calibration. Treat the model swap as a migration with its own threshold tuning, and it is the single cheapest recall win we have shipped in a search system.