All Stories
Practical Guide to Model Distillation: From DeepSeek-R1 to edge-ready SLMs
When DeepSeek-R1 landed in January 2025, it proved that chain-of-thought plus reinforcement learning could push open models to near-frontier reasoning on math and code. It also proved something less convenient:...
Structured Logging Performance: Custom zap and slog Configurations in Go
Logging is the last thing anyone profiles and the first thing that sinks a latency budget. A service doing 10k req/s with two or three log lines per request is...
Implementing Llama Guard: Multi-stage Safety Filtering for LLM Gateways
We run a customer-facing LLM gateway in front of a hosted model, and the threat model is unglamorous but real: prompt injection trying to exfiltrate another tenant’s data, jailbreaks attempting...
KV-Cache Eviction and PagedAttention in High-Throughput LLM Serving
The first time we load-tested vLLM for a multi-tenant embedding-and-completion service, we watched nvidia-smi and assumed something was leaking. Four Mistral-7B instances on two A100s, steady 512-token requests, and the...
Self-hosting Llama 3 70B: Inference Optimization with vLLM and Triton
A client in EU pharma needs inference that never leaves the data center: weights, prompts, and outputs all stay on-prem, and the model has to be legally auditable. Llama 3...
Debugging Solid Queue Deadlocks: Transaction Isolation Levels and Row Locks
Rails 8 shipped on November 7, 2024 with Solid Queue as the default background-job adapter — a Postgres-backed queue that fits the “one less database” story. We migrated a roughly...