RandomCommits
A personal blog of Sijin where he likes to randomly scribble his thoughts
Tag go
The memory math nobody does up front
Why serialization keeps topping your pprof
The simplest multi-tenant design in Go is also the most dangerous: one *sql.DB per tenant, looked up from a map at request time. It works beautifully for ten tenants and...
Go’s concurrency story rests on one runtime component most engineers never look at: the scheduler. If you have ever seen a service with 300,000 goroutines and 12 cores stay responsive,...
When Go 1.18 shipped generics, the design had to thread a needle: be fast like C++ templates without their binary bloat, and be ergonomic without Java’s boxing tax. The answer...
A goroutine leak is the sneakiest way to lose memory in Go, because the memory is not garbage: it is a live stack, referenced by a blocked goroutine, unreachable to...
The advice “share memory by communicating” has done more damage than good. It was written for clarity, not throughput, and teams have shipped channel-based counters and caches that run 5-10x...
Every few months someone on the team proposes “we should just use a faster JSON library” and benchmarks encoding/json against a dropped-in replacement, sees a 2x, and ships it. That’s...
The difference between a handler that does 40 allocations per request and one that does four is usually not your data structures. It is whether the compiler managed to keep...
An API gateway sits in front of every upstream call, so its router is the hottest code path in the fleet. A linear scan over registered patterns is O(N) in...
A 2GB Kubernetes pod running a Go service with default GOGC=100 has a GC that will happily let the heap double before it starts collecting. That is by design: the...
Logging is the last thing anyone profiles and the first thing that sinks a latency budget. A service doing 10k req/s with two or three log lines per request is...
Our Redis client was pushing a sustained 8% CPU on every API node, and pprof could not tell us why. The CPU profile blamed runtime.futex, which is not an answer;...
We spent a week hunting a checkout outage the tracing dashboards could not explain. The gateway started a span, then the trace went dark: payment logged an empty trace ID,...
What the handshake actually costs
Why the write path hates B-trees
Go already has a netpoll – know why you would reimplement it
The real cost of an interface call
sync.WaitGroup answers one question: “is everyone done?” It cannot tell you someone failed, and it certainly cannot stop the others when one does. For fan-out over shards, partitions, or upstreams...
gRPC is built on HTTP/2, and HTTP/2 wants a few long-lived connections, not many short ones. Put a classic L4 or L7 load balancer in front and you have re-introduced...
A crawler or an ingress resolving thousands of distinct hostnames per second exposes a gap in the standard library. The folklore is that net.LookupHost blocks OS threads via cgo —...
Guessing where CPU goes is how services die slowly. Go ships a world-class profiler; the real skill is using it correctly, because each profile type answers a different question and...
Goroutines are cheap — a couple of KiB of stack, sub-microsecond spawn — so teams over-rotated from “thread per request” to “goroutine per request” and hit a wall. Unbounded concurrency...
A read syscall costs a few hundred nanoseconds to a few microseconds depending on kernel and storage. That sounds cheap until you multiply by bytes. A 1 GiB log file...
Kubernetes terminates pods the same way every time: it sends SIGTERM, waits terminationGracePeriodSeconds (30 by default), then sends SIGKILL. If your service treats SIGTERM as “start dying right now”, every...
Every object you allocate passes through the runtime allocator, and in a latency-sensitive Go service it is quietly responsible for more of your p99 than almost anything else. It decides...
A shared index-update queue on one of our workers was the wall in every profile: sixteen goroutines contending on a single mutex, and the flame graph showing sync.(*Mutex).Lock as a...
Our ingest path parses text records off a network buffer and indexes them. At 1.2M events/s, the parser was the hottest function in the service – not because parsing is...
Every gateway node in our fleet needs the same piece of truth at boot: the shard-to-node mapping, the config revision, the list of revoked keys. Before Raft, that truth lived...
Our request marshaling path allocated a handful of small buffers per request – fine at 1k req/s, a garbage problem at 20k. sync.Pool looked like the obvious fix, and it...
Our gateway terminates traffic for public APIs, and the first thing every unauthenticated request hits is a per-API-key limiter. With 50k active keys and a sustained 100k req/s, the limiting...
Eight bytes per struct. That is what a careless field order cost us in a 1M-record in-memory index we cache in a service that answers every lookup request. Eight bytes...
We ran a checkout pipeline that fronted a legacy payments gateway. The gateway had a 3-second timeout and, once every few weeks, it would wedge for ten minutes. The client...
Every packet that crosses our gateway gets decoded into a struct. The naive path – encoding/binary reading seven fields one at a time, on every packet – is slow in...
The append loop that grows a slice one element at a time is the most common allocation smell we find in Go production code. It is not wrong, exactly –...
Tag slice
The append loop that grows a slice one element at a time is the most common allocation smell we find in Go production code. It is not wrong, exactly –...
Tag preallocation
The append loop that grows a slice one element at a time is the most common allocation smell we find in Go production code. It is not wrong, exactly –...
Decode is memory-bandwidth-bound, and that single fact is why speculative decoding is the cheapest throughput win an inference team can spend a week on. Every autoregressive step loads the entire...
The memory math nobody does up front
The last place I look when an API endpoint is slow is the serializer, and that’s a mistake. On a typical list endpoint the serializer is where most of the...
The simplest multi-tenant design in Go is also the most dangerous: one *sql.DB per tenant, looked up from a map at request time. It works beautifully for ten tenants and...
When Go 1.18 shipped generics, the design had to thread a needle: be fast like C++ templates without their binary bloat, and be ergonomic without Java’s boxing tax. The answer...
Sidekiq’s contract is “push work, forget it,” but the moment you have an event bus that emits per-entity changes, you discover how literally Sidekiq honors that contract. Ten indexing jobs...
A 2GB Kubernetes pod running a Go service with default GOGC=100 has a GC that will happily let the heap double before it starts collecting. That is by design: the...
In Ruby 3.3, YJIT is production-ready. Shopify’s numbers, the Optcarrot benchmark, and the Rails core team all point the same direction: roughly 15–25% latency improvement on real Rails workloads, for...
includes is the most convenient lie in ActiveRecord. You write one method call, and ActiveRecord decides for you whether to emit two queries joined by an IN clause (preload) or...
Quantization is how a 7B model becomes a weekend desktop project instead of a dedicated server: fp16 weights are 14 GB, int8 is 7 GB, 4-bit is about 3.5 GB....
What the handshake actually costs
WebSockets look like the same stack as HTTP and are, in every way that matters, the opposite. An HTTP request occupies a thread for a hundred milliseconds and leaves. A...
The classic slow controller is a dashboard that aggregates three independent services. Balance from billing, usage from the analytics service, unread count from notifications. Called sequentially, a request that should...
Every packet that crosses our gateway gets decoded into a struct. The naive path – encoding/binary reading seven fields one at a time, on every packet – is slow in...
The append loop that grows a slice one element at a time is the most common allocation smell we find in Go production code. It is not wrong, exactly –...
Tag activesupport
Rails instruments almost every meaningful point in the request lifecycle already: SQL execution, view and partial rendering, cache reads, and full controller actions. It publishes all of it on a...
Tag notifications
Rails instruments almost every meaningful point in the request lifecycle already: SQL execution, view and partial rendering, cache reads, and full controller actions. It publishes all of it on a...
Tag telemetry
If you serve LLM inference at any real scale, your standard monitoring stack has a blind spot exactly where your tokens actually spend time: the kernel. Prometheus exporters and Datadog...
Our Redis client was pushing a sustained 8% CPU on every API node, and pprof could not tell us why. The CPU profile blamed runtime.futex, which is not an answer;...
We spent a week hunting a checkout outage the tracing dashboards could not explain. The gateway started a span, then the trace went dark: payment logged an empty trace ID,...
Guessing where CPU goes is how services die slowly. Go ships a world-class profiler; the real skill is using it correctly, because each profile type answers a different question and...
Rails instruments almost every meaningful point in the request lifecycle already: SQL execution, view and partial rendering, cache reads, and full controller actions. It publishes all of it on a...
Tag opentelemetry
Rails logs measure the controller. They don’t measure the queue, the auth middleware, the rate limiter, or the request as the client actually experienced it. That gap is exactly where...
Rails instruments almost every meaningful point in the request lifecycle already: SQL execution, view and partial rendering, cache reads, and full controller actions. It publishes all of it on a...
Tag unsafe
Every packet that crosses our gateway gets decoded into a struct. The naive path – encoding/binary reading seven fields one at a time, on every packet – is slow in...
Tag pointer
Every packet that crosses our gateway gets decoded into a struct. The naive path – encoding/binary reading seven fields one at a time, on every packet – is slow in...
Tag circuit_breaker
We ran a checkout pipeline that fronted a legacy payments gateway. The gateway had a 3-second timeout and, once every few weeks, it would wedge for ten minutes. The client...
Tag resiliency
We ran a checkout pipeline that fronted a legacy payments gateway. The gateway had a 3-second timeout and, once every few weeks, it would wedge for ten minutes. The client...
Tag microservices
Kubernetes terminates pods the same way every time: it sends SIGTERM, waits terminationGracePeriodSeconds (30 by default), then sends SIGKILL. If your service treats SIGTERM as “start dying right now”, every...
We ran a checkout pipeline that fronted a legacy payments gateway. The gateway had a 3-second timeout and, once every few weeks, it would wedge for ten minutes. The client...
Tag struct
Eight bytes per struct. That is what a careless field order cost us in a 1M-record in-memory index we cache in a service that answers every lookup request. Eight bytes...
Tag padding
Eight bytes per struct. That is what a careless field order cost us in a 1M-record in-memory index we cache in a service that answers every lookup request. Eight bytes...
Tag memory
Since Rails 7.2 made YJIT the default JIT in August 2024, most Rails apps picked up a 15-25% throughput bump for free. The bill shows up on Kubernetes in a...
The naive fix for agents that forget is to buy a bigger context window. We ran that playbook and it is a trap: the failure mode of a long conversation...
Every object you allocate passes through the runtime allocator, and in a latency-sensitive Go service it is quietly responsible for more of your p99 than almost anything else. It decides...
Our ingest path parses text records off a network buffer and indexes them. At 1.2M events/s, the parser was the hottest function in the service – not because parsing is...
Our request marshaling path allocated a handful of small buffers per request – fine at 1k req/s, a garbage problem at 20k. sync.Pool looked like the obvious fix, and it...
Eight bytes per struct. That is what a careless field order cost us in a 1M-record in-memory index we cache in a service that answers every lookup request. Eight bytes...
Tag rate_limiter
Our gateway terminates traffic for public APIs, and the first thing every unauthenticated request hits is a per-API-key limiter. With 50k active keys and a sustained 100k req/s, the limiting...
Tag token_bucket
Our gateway terminates traffic for public APIs, and the first thing every unauthenticated request hits is a per-API-key limiter. With 50k active keys and a sustained 100k req/s, the limiting...
Tag concurrency
“Real-time dashboard” is the feature that silently becomes an architecture decision. The naive version polls a JSON endpoint every second: one client is nothing, but 500 dashboards at 1 req/s...
Go’s concurrency story rests on one runtime component most engineers never look at: the scheduler. If you have ever seen a service with 300,000 goroutines and 12 cores stay responsive,...
The advice “share memory by communicating” has done more damage than good. It was written for clarity, not throughput, and teams have shipped channel-based counters and caches that run 5-10x...
Sidekiq’s contract is “push work, forget it,” but the moment you have an event bus that emits per-entity changes, you discover how literally Sidekiq honors that contract. Ten indexing jobs...
The incident is boring and predictable: someone bumps Sidekiq concurrency from 10 to 25, and within an hour pager duty is looking at ActiveRecord::ConnectionTimeoutError across every web and worker process....
The trap: a “concurrent” test that never races
sync.WaitGroup answers one question: “is everyone done?” It cannot tell you someone failed, and it certainly cannot stop the others when one does. For fan-out over shards, partitions, or upstreams...
Goroutines are cheap — a couple of KiB of stack, sub-microsecond spawn — so teams over-rotated from “thread per request” to “goroutine per request” and hit a wall. Unbounded concurrency...
A shared index-update queue on one of our workers was the wall in every profile: sixteen goroutines contending on a single mutex, and the flame graph showing sync.(*Mutex).Lock as a...
Our gateway terminates traffic for public APIs, and the first thing every unauthenticated request hits is a per-API-key limiter. With 50k active keys and a sustained 100k req/s, the limiting...
Tag sync_pool
Our request marshaling path allocated a handful of small buffers per request – fine at 1k req/s, a garbage problem at 20k. sync.Pool looked like the obvious fix, and it...
Tag gc
A 2GB Kubernetes pod running a Go service with default GOGC=100 has a GC that will happily let the heap double before it starts collecting. That is by design: the...
Our request marshaling path allocated a handful of small buffers per request – fine at 1k req/s, a garbage problem at 20k. sync.Pool looked like the obvious fix, and it...
Tag raft
Every gateway node in our fleet needs the same piece of truth at boot: the shard-to-node mapping, the config revision, the list of revoked keys. Before Raft, that truth lived...
Tag distributed_systems
Every gateway node in our fleet needs the same piece of truth at boot: the shard-to-node mapping, the config revision, the list of revoked keys. Before Raft, that truth lived...
Tag consensus
Every gateway node in our fleet needs the same piece of truth at boot: the shard-to-node mapping, the config revision, the list of revoked keys. Before Raft, that truth lived...
Tag activerecord
We run a 12-server Puma fleet against a Postgres primary and two read replicas. For a long time our read/write splitting was manual: a sidecar script rewrote database.yml on deploy...
includes is the most convenient lie in ActiveRecord. You write one method call, and ActiveRecord decides for you whether to emit two queries joined by an IN clause (preload) or...
The pattern starts innocently:
There’s a point in every growing service where the primary database becomes the bottleneck, and it’s not about reads. Read replicas solved reads. The pain is writes: one node, one...
validates_uniqueness_of is not a data-integrity mechanism. It is a user-experience mechanism, and treating it as anything more is how you end up with duplicate invoices. The failure mode is embarrassingly...
Tag validations
validates_uniqueness_of is not a data-integrity mechanism. It is a user-experience mechanism, and treating it as anything more is how you end up with duplicate invoices. The failure mode is embarrassingly...
Tag constraints
validates_uniqueness_of is not a data-integrity mechanism. It is a user-experience mechanism, and treating it as anything more is how you end up with duplicate invoices. The failure mode is embarrassingly...
Tag database
The simplest multi-tenant design in Go is also the most dangerous: one *sql.DB per tenant, looked up from a map at request time. It works beautifully for ten tenants and...
We run a 12-server Puma fleet against a Postgres primary and two read replicas. For a long time our read/write splitting was manual: a sidecar script rewrote database.yml on deploy...
Why the write path hates B-trees
There’s a point in every growing service where the primary database becomes the bottleneck, and it’s not about reads. Read replicas solved reads. The pain is writes: one node, one...
The migration that nearly took us down wasn’t a clever one. It was add_index :orders, :customer_id, run at 2pm against a table with 40 million rows. Postgres holds a SHARE...
validates_uniqueness_of is not a data-integrity mechanism. It is a user-experience mechanism, and treating it as anything more is how you end up with duplicate invoices. The failure mode is embarrassingly...
Tag migrations
The migration that nearly took us down wasn’t a clever one. It was add_index :orders, :customer_id, run at 2pm against a table with 40 million rows. Postgres holds a SHARE...
Tag strong_migrations
The migration that nearly took us down wasn’t a clever one. It was add_index :orders, :customer_id, run at 2pm against a table with 40 million rows. Postgres holds a SHARE...
Tag rails
In Ruby 3.3, YJIT is production-ready. Shopify’s numbers, the Optcarrot benchmark, and the Rails core team all point the same direction: roughly 15–25% latency improvement on real Rails workloads, for...
The migration that nearly took us down wasn’t a clever one. It was add_index :orders, :customer_id, run at 2pm against a table with 40 million rows. Postgres holds a SHARE...
Tag zero_allocation
Our ingest path parses text records off a network buffer and indexes them. At 1.2M events/s, the parser was the hottest function in the service – not because parsing is...
Tag parsing
Our ingest path parses text records off a network buffer and indexes them. At 1.2M events/s, the parser was the hottest function in the service – not because parsing is...
Tag rack
Rails logs measure the controller. They don’t measure the queue, the auth middleware, the rate limiter, or the request as the client actually experienced it. That gap is exactly where...
Tag middleware
Rails logs measure the controller. They don’t measure the queue, the auth middleware, the rate limiter, or the request as the client actually experienced it. That gap is exactly where...
Tag prometheus
Rails logs measure the controller. They don’t measure the queue, the auth middleware, the rate limiter, or the request as the client actually experienced it. That gap is exactly where...
Tag atomic
A shared index-update queue on one of our workers was the wall in every profile: sixteen goroutines contending on a single mutex, and the flame graph showing sync.(*Mutex).Lock as a...
Tag lock_free
A shared index-update queue on one of our workers was the wall in every profile: sixteen goroutines contending on a single mutex, and the flame graph showing sync.(*Mutex).Lock as a...
Tag allocator
Every object you allocate passes through the runtime allocator, and in a latency-sensitive Go service it is quietly responsible for more of your p99 than almost anything else. It decides...
Tag tcmalloc
Every object you allocate passes through the runtime allocator, and in a latency-sensitive Go service it is quietly responsible for more of your p99 than almost anything else. It decides...
Tag sharding
There’s a point in every growing service where the primary database becomes the bottleneck, and it’s not about reads. Read replicas solved reads. The pain is writes: one node, one...
Tag rails6
There’s a point in every growing service where the primary database becomes the bottleneck, and it’s not about reads. Read replicas solved reads. The pain is writes: one node, one...
Tag concurrent_ruby
The classic slow controller is a dashboard that aggregates three independent services. Balance from billing, usage from the analytics service, unread count from notifications. Called sequentially, a request that should...
Tag multithreading
The classic slow controller is a dashboard that aggregates three independent services. Balance from billing, usage from the analytics service, unread count from notifications. Called sequentially, a request that should...
Tag controllers
The classic slow controller is a dashboard that aggregates three independent services. Balance from billing, usage from the analytics service, unread count from notifications. Called sequentially, a request that should...
Tag graceful_shutdown
Kubernetes terminates pods the same way every time: it sends SIGTERM, waits terminationGracePeriodSeconds (30 by default), then sends SIGKILL. If your service treats SIGTERM as “start dying right now”, every...
Tag systems
Our Redis client was pushing a sustained 8% CPU on every API node, and pprof could not tell us why. The CPU profile blamed runtime.futex, which is not an answer;...
Kubernetes terminates pods the same way every time: it sends SIGTERM, waits terminationGracePeriodSeconds (30 by default), then sends SIGKILL. If your service treats SIGTERM as “start dying right now”, every...
Tag io
A read syscall costs a few hundred nanoseconds to a few microseconds depending on kernel and storage. That sounds cheap until you multiply by bytes. A 1 GiB log file...
Tag reader
A read syscall costs a few hundred nanoseconds to a few microseconds depending on kernel and storage. That sounds cheap until you multiply by bytes. A 1 GiB log file...
Tag writer
A read syscall costs a few hundred nanoseconds to a few microseconds depending on kernel and storage. That sounds cheap until you multiply by bytes. A 1 GiB log file...
Tag security
We run a customer-facing LLM gateway in front of a hosted model, and the threat model is unglamorous but real: prompt injection trying to exfiltrate another tenant’s data, jailbreaks attempting...
A client POSTs a payment, the connection drops mid-response, and the client retries. If your endpoint is not idempotent, the second request charges the card twice. At roughly 3M mutating...
Rate limiting is a liveness problem
Two ways to isolate tenants in one database
Browser XSS filters were always a poor backstop, and they’re not even that anymore. The real defense is telling the browser exactly what it’s allowed to execute, and Content Security...
Tag csp
Browser XSS filters were always a poor backstop, and they’re not even that anymore. The real defense is telling the browser exactly what it’s allowed to execute, and Content Security...
Tag sri
Browser XSS filters were always a poor backstop, and they’re not even that anymore. The real defense is telling the browser exactly what it’s allowed to execute, and Content Security...
Browser XSS filters were always a poor backstop, and they’re not even that anymore. The real defense is telling the browser exactly what it’s allowed to execute, and Content Security...
Tag worker_pool
Goroutines are cheap — a couple of KiB of stack, sub-microsecond spawn — so teams over-rotated from “thread per request” to “goroutine per request” and hit a wall. Unbounded concurrency...
Tag channels
The advice “share memory by communicating” has done more damage than good. It was written for clarity, not throughput, and teams have shipped channel-based counters and caches that run 5-10x...
Goroutines are cheap — a couple of KiB of stack, sub-microsecond spawn — so teams over-rotated from “thread per request” to “goroutine per request” and hit a wall. Unbounded concurrency...
Tag pprof
Guessing where CPU goes is how services die slowly. Go ships a world-class profiler; the real skill is using it correctly, because each profile type answers a different question and...
Tag profiling
Guessing where CPU goes is how services die slowly. Go ships a world-class profiler; the real skill is using it correctly, because each profile type answers a different question and...
Tag dns
A crawler or an ingress resolving thousands of distinct hostnames per second exposes a gap in the standard library. The folklore is that net.LookupHost blocks OS threads via cgo —...
Tag resolver
A crawler or an ingress resolving thousands of distinct hostnames per second exposes a gap in the standard library. The folklore is that net.LookupHost blocks OS threads via cgo —...
Tag networking
Go already has a netpoll – know why you would reimplement it
A crawler or an ingress resolving thousands of distinct hostnames per second exposes a gap in the standard library. The folklore is that net.LookupHost blocks OS threads via cgo —...
Tag grpc
gRPC is built on HTTP/2, and HTTP/2 wants a few long-lived connections, not many short ones. Put a classic L4 or L7 load balancer in front and you have re-introduced...
Tag load_balancing
gRPC is built on HTTP/2, and HTTP/2 wants a few long-lived connections, not many short ones. Put a classic L4 or L7 load balancer in front and you have re-introduced...
Tag resolvers
gRPC is built on HTTP/2, and HTTP/2 wants a few long-lived connections, not many short ones. Put a classic L4 or L7 load balancer in front and you have re-introduced...
Tag errgroup
sync.WaitGroup answers one question: “is everyone done?” It cannot tell you someone failed, and it certainly cannot stop the others when one does. For fan-out over shards, partitions, or upstreams...
Tag error_handling
sync.WaitGroup answers one question: “is everyone done?” It cannot tell you someone failed, and it certainly cannot stop the others when one does. For fan-out over shards, partitions, or upstreams...
Tag interfaces
The real cost of an interface call
Tag devirtualization
The real cost of an interface call
Tag compiler
The difference between a handler that does 40 allocations per request and one that does four is usually not your data structures. It is whether the compiler managed to keep...
The real cost of an interface call
Tag actioncable
ActionCable’s Redis dependency always annoyed me, and not for the reasons people usually cite. The latency argument against database pub/sub is mostly wrong at small scale — the real cost...
A canvas is a bad reason to go SPA
WebSockets look like the same stack as HTTP and are, in every way that matters, the opposite. An HTTP request occupies a thread for a hundred milliseconds and leaves. A...
Tag websockets
The memory math nobody does up front
ActionCable’s Redis dependency always annoyed me, and not for the reasons people usually cite. The latency argument against database pub/sub is mostly wrong at small scale — the real cost...
A canvas is a bad reason to go SPA
WebSockets look like the same stack as HTTP and are, in every way that matters, the opposite. An HTTP request occupies a thread for a hundred milliseconds and leaves. A...
Tag redis
The Redis cache bill is a weird tax. You’re paying premium RAM prices for data that is, by definition, disposable — regenerate it and nothing breaks. When our cache footprint...
Sidekiq’s contract is “push work, forget it,” but the moment you have an event bus that emits per-entity changes, you discover how literally Sidekiq honors that contract. Ten indexing jobs...
A client POSTs a payment, the connection drops mid-response, and the client retries. If your endpoint is not idempotent, the second request charges the card twice. At roughly 3M mutating...
Rate limiting is a liveness problem
WebSockets look like the same stack as HTTP and are, in every way that matters, the opposite. An HTTP request occupies a thread for a hundred milliseconds and leaves. A...
Tag netpoll
Go already has a netpoll – know why you would reimplement it
Tag epoll
The memory math nobody does up front
Go already has a netpoll – know why you would reimplement it
Tag multitenancy
The simplest multi-tenant design in Go is also the most dangerous: one *sql.DB per tenant, looked up from a map at request time. It works beautifully for ten tenants and...
Two ways to isolate tenants in one database
Tag postgres
Rails 8 shipped on November 7, 2024 with Solid Queue as the default background-job adapter — a Postgres-backed queue that fits the “one less database” story. We migrated a roughly...
The incident is boring and predictable: someone bumps Sidekiq concurrency from 10 to 25, and within an hour pager duty is looking at ActiveRecord::ConnectionTimeoutError across every web and worker process....
Two ways to isolate tenants in one database
Tag saas
Two ways to isolate tenants in one database
Tag lsm_tree
Why the write path hates B-trees
Tag storage_engine
Why the write path hates B-trees
Tag rack_attack
Rate limiting is a liveness problem
Tag rate_limiting
Rate limiting is a liveness problem
Tag rspec
The trap: a “concurrent” test that never races
Tag testing
BLEU and ROUGE did not catch a single regression for us. The rewrite that scored +3 ROUGE against the old transcripts was, by any human reading, worse — it hedged,...
The trap: a “concurrent” test that never races
Tag race_conditions
The trap: a “concurrent” test that never races
Tag tls
What the handshake actually costs
Tag http2
What the handshake actually costs
Tag rag
Text-to-SQL looks solved in demos and falls apart in production, always in the same four ways: the model invents column names the schema never had, it emits syntax that parses...
Let me correct the framing up front: Llama 3.1 is text-only. The 8B, 70B, and 405B models released in July 2024 consume tokens, not pixels. Any pipeline that claims to...
Off-the-shelf embedders understand generic language and nothing else. BGE-small and OpenAI’s Ada-002 encode “what is the refund policy” beautifully; ask them about an internal part number like OCT-4472-REV3 or a...
Vector RAG answered “what does the payment service do?” beautifully and went blind on “what did the author build for the payment service while reporting to project manager X?” The...
We shipped our first RAG pipeline with a single dense retriever and got paged within a week: users searching for a firmware error code like ERR_0x84F2 or a part number...
Tag bm25
We shipped our first RAG pipeline with a single dense retriever and got paged within a week: users searching for a firmware error code like ERR_0x84F2 or a part number...
Tag rerank
We shipped our first RAG pipeline with a single dense retriever and got paged within a week: users searching for a firmware error code like ERR_0x84F2 or a part number...
Tag cross_encoder
We shipped our first RAG pipeline with a single dense retriever and got paged within a week: users searching for a firmware error code like ERR_0x84F2 or a part number...
Tag quantization
When DeepSeek-R1 landed in January 2025, it proved that chain-of-thought plus reinforcement learning could push open models to near-frontier reasoning on math and code. It also proved something less convenient:...
Quantization is how a 7B model becomes a weekend desktop project instead of a dedicated server: fp16 weights are 14 GB, int8 is 7 GB, 4-bit is about 3.5 GB....
Tag gguf
Quantization is how a 7B model becomes a weekend desktop project instead of a dedicated server: fp16 weights are 14 GB, int8 is 7 GB, 4-bit is about 3.5 GB....
Tag awq
Quantization is how a 7B model becomes a weekend desktop project instead of a dedicated server: fp16 weights are 14 GB, int8 is 7 GB, 4-bit is about 3.5 GB....
Tag vector_db
The naive fix for agents that forget is to buy a bigger context window. We ran that playbook and it is a trap: the failure mode of a long conversation...
Every vector index is a bet on a triangle: recall, latency, and memory. Flat scan sits at one corner — perfect recall, but at 1M 1536-d vectors (about 6 GB...
Tag hnsw
Every vector index is a bet on a triangle: recall, latency, and memory. Flat scan sits at one corner — perfect recall, but at 1M 1536-d vectors (about 6 GB...
Tag pgvector
Every vector index is a bet on a triangle: recall, latency, and memory. Flat scan sits at one corner — perfect recall, but at 1M 1536-d vectors (about 6 GB...
Tag qdrant
Let me correct the framing up front: Llama 3.1 is text-only. The 8B, 70B, and 405B models released in July 2024 consume tokens, not pixels. Any pipeline that claims to...
Every vector index is a bet on a triangle: recall, latency, and memory. Flat scan sits at one corner — perfect recall, but at 1M 1536-d vectors (about 6 GB...
Tag hotwire
A canvas is a bad reason to go SPA
Tag turbo_streams
A canvas is a bad reason to go SPA
Tag memgpt
The naive fix for agents that forget is to buy a bigger context window. We ran that playbook and it is a trap: the failure mode of a long conversation...
Tag agents
Almost every agent starts as a naive while True: call_model, run_tool, call_model loop, and almost every production agent ends up as something else. The reason is that the three canonical...
The model got smarter, but the reason agents work in production in 2025 is the harness around them. We run them in customer-support triage, internal ops, and incident summarization, and...
Vector RAG answered “what does the payment service do?” beautifully and went blind on “what did the author build for the payment service while reporting to project manager X?” The...
The naive fix for agents that forget is to buy a bigger context window. We ran that playbook and it is a trap: the failure mode of a long conversation...
Tag ruby
Since Rails 7.2 made YJIT the default JIT in August 2024, most Rails apps picked up a 15-25% throughput bump for free. The bill shows up on Kubernetes in a...
Rails 7.2 (August 2024) did not ship a magic filter that “skips patterns” during eager loading — that idea circulates in a lot of upgrade blog posts, but it is...
If you read the Ruby 3.3 changelog looking for “right-hand patterns”, stop. Pattern matching arrived in Ruby 2.7, and the => rightward assignment shipped in the same release; the syntax...
Tag pattern_matching
If you read the Ruby 3.3 changelog looking for “right-hand patterns”, stop. Pattern matching arrived in Ruby 2.7, and the => rightward assignment shipped in the same release; the syntax...
Tag ruby3
In Ruby 3.3, YJIT is production-ready. Shopify’s numbers, the Optcarrot benchmark, and the Rails core team all point the same direction: roughly 15–25% latency improvement on real Rails workloads, for...
If you read the Ruby 3.3 changelog looking for “right-hand patterns”, stop. Pattern matching arrived in Ruby 2.7, and the => rightward assignment shipped in the same release; the syntax...
Tag refinements
If you read the Ruby 3.3 changelog looking for “right-hand patterns”, stop. Pattern matching arrived in Ruby 2.7, and the => rightward assignment shipped in the same release; the syntax...
Tag elasticsearch
The pattern starts innocently:
Tag sidekiq
Sidekiq’s contract is “push work, forget it,” but the moment you have an event bus that emits per-entity changes, you discover how literally Sidekiq honors that contract. Ten indexing jobs...
The incident is boring and predictable: someone bumps Sidekiq concurrency from 10 to 25, and within an hour pager duty is looking at ActiveRecord::ConnectionTimeoutError across every web and worker process....
The pattern starts innocently:
Tag callbacks
The pattern starts innocently:
Tag dpo
Classic RLHF is a four-model circus: a frozen reference policy, a trainable actor, a reward model, and a critic network, plus a KL term to keep the actor honest. It...
Tag rlhf
The standard recipe for a “reasoning” model is crushing amounts of supervised data: millions of human-curated chain-of-thought traces, distilled from a bigger model, cleaned by contractors, and fine-tuned into the...
Classic RLHF is a four-model circus: a frozen reference policy, a trainable actor, a reward model, and a critic network, plus a KL term to keep the actor honest. It...
Tag alignment
Fine-tuning a model for internal tooling usually means one thing is missing: the data. Our first attempt to adapt a model for internal ticket routing needed 10,000 labeled examples, and...
Classic RLHF is a four-model circus: a frozen reference policy, a trainable actor, a reward model, and a critic network, plus a KL term to keep the actor honest. It...
Tag dpo_vs_ppo
Classic RLHF is a four-model circus: a frozen reference policy, a trainable actor, a reward model, and a critic network, plus a KL term to keep the actor honest. It...
Tag idempotency
A client POSTs a payment, the connection drops mid-response, and the client retries. If your endpoint is not idempotent, the second request charges the card twice. At roughly 3M mutating...
Tag api
A client POSTs a payment, the connection drops mid-response, and the client retries. If your endpoint is not idempotent, the second request charges the card twice. At roughly 3M mutating...
Tag activejob
Rails 8 shipped on November 7, 2024 with Solid Queue as the default background-job adapter — a Postgres-backed queue that fits the “one less database” story. We migrated a roughly...
The incident is boring and predictable: someone bumps Sidekiq concurrency from 10 to 25, and within an hour pager duty is looking at ActiveRecord::ConnectionTimeoutError across every web and worker process....
Tag eager_loading
Rails 7.2 (August 2024) did not ship a magic filter that “skips patterns” during eager loading — that idea circulates in a lot of upgrade blog posts, but it is...
includes is the most convenient lie in ActiveRecord. You write one method call, and ActiveRecord decides for you whether to emit two queries joined by an IN clause (preload) or...
Tag sql
Text-to-SQL looks solved in demos and falls apart in production, always in the same four ways: the model invents column names the schema never had, it emits syntax that parses...
includes is the most convenient lie in ActiveRecord. You write one method call, and ActiveRecord decides for you whether to emit two queries joined by an IN clause (preload) or...
Tag moe
Dense scaling hits a hard wall: every parameter you add is a parameter you pay for on every token, in both VRAM and bandwidth. Mixture-of-Experts breaks that coupling by making...
“128K context” sounds like a feature but it is two separate engineering problems stapled together, and they fail independently. The first is arithmetic: attention is O(n²) in time and memory,...
Dense scaling hits a hard wall: every parameter you add is a parameter you pay for on every token, in both VRAM and bandwidth. Mixture-of-Experts breaks that coupling by making...
Tag mixtral
Dense scaling hits a hard wall: every parameter you add is a parameter you pay for on every token, in both VRAM and bandwidth. Mixture-of-Experts breaks that coupling by making...
Tag routing
An API gateway sits in front of every upstream call, so its router is the hottest code path in the fleet. A linear scan over registered patterns is O(N) in...
Dense scaling hits a hard wall: every parameter you add is a parameter you pay for on every token, in both VRAM and bandwidth. Mixture-of-Experts breaks that coupling by making...
Tag context
We spent a week hunting a checkout outage the tracing dashboards could not explain. The gateway started a span, then the trace went dark: payment logged an empty trace ID,...
Tag tracing
We spent a week hunting a checkout outage the tracing dashboards could not explain. The gateway started a span, then the trace went dark: payment logged an empty trace ID,...
Tag peft
Full-parameter fine-tuning of a 7B model needs about 112 GB of VRAM: fp16 weights (14 GB) plus gradients, fp32 Adam moments, and fp32 master weights add up to roughly 16...
Tag lora
Full-parameter fine-tuning of a 7B model needs about 112 GB of VRAM: fp16 weights (14 GB) plus gradients, fp32 Adam moments, and fp32 master weights add up to roughly 16...
Tag qlora
Full-parameter fine-tuning of a 7B model needs about 112 GB of VRAM: fp16 weights (14 GB) plus gradients, fp32 Adam moments, and fp32 master weights add up to roughly 16...
Tag finetuning
Off-the-shelf embedders understand generic language and nothing else. BGE-small and OpenAI’s Ada-002 encode “what is the refund policy” beautifully; ask them about an internal part number like OCT-4472-REV3 or a...
Fine-tuning a model for internal tooling usually means one thing is missing: the data. Our first attempt to adapt a model for internal ticket routing needed 10,000 labeled examples, and...
Full-parameter fine-tuning of a 7B model needs about 112 GB of VRAM: fp16 weights (14 GB) plus gradients, fp32 Adam moments, and fp32 master weights add up to roughly 16...
Tag yjit
Since Rails 7.2 made YJIT the default JIT in August 2024, most Rails apps picked up a 15-25% throughput bump for free. The bill shows up on Kubernetes in a...
In Ruby 3.3, YJIT is production-ready. Shopify’s numbers, the Optcarrot benchmark, and the Rails core team all point the same direction: roughly 15–25% latency improvement on real Rails workloads, for...
Tag llm
The 2023 version of this question was about knowledge: “should we fine-tune so the model knows our domain?” That framing is dead in 2025. Current models — Claude Sonnet 4,...
The model got smarter, but the reason agents work in production in 2025 is the harness around them. We run them in customer-support triage, internal ops, and incident summarization, and...
“128K context” sounds like a feature but it is two separate engineering problems stapled together, and they fail independently. The first is arithmetic: attention is O(n²) in time and memory,...
Tag attention
“128K context” sounds like a feature but it is two separate engineering problems stapled together, and they fail independently. The first is arithmetic: attention is O(n²) in time and memory,...
Tag rope
“128K context” sounds like a feature but it is two separate engineering problems stapled together, and they fail independently. The first is arithmetic: attention is O(n²) in time and memory,...
Tag kamal
The platform bill grows linearly with a problem you did not choose: a few years of real traffic later, the Heroku/Render invoices looked like a third engineering salary, and Kubernetes...
Tag docker
The platform bill grows linearly with a problem you did not choose: a few years of real traffic later, the Heroku/Render invoices looked like a third engineering salary, and Kubernetes...
Tag tailscale
The platform bill grows linearly with a problem you did not choose: a few years of real traffic later, the Heroku/Render invoices looked like a third engineering salary, and Kubernetes...
Tag deployment
The platform bill grows linearly with a problem you did not choose: a few years of real traffic later, the Heroku/Render invoices looked like a third engineering salary, and Kubernetes...
Tag clip
Catalog search lives and dies on the visual queries text search never sees: “floral dress, but not the one with the collar”, “refrigerator that fits under a 32-inch counter”, “product...
Tag siglip
Catalog search lives and dies on the visual queries text search never sees: “floral dress, but not the one with the collar”, “refrigerator that fits under a 32-inch counter”, “product...
Tag multimodal
Let me correct the framing up front: Llama 3.1 is text-only. The 8B, 70B, and 405B models released in July 2024 consume tokens, not pixels. Any pipeline that claims to...
Catalog search lives and dies on the visual queries text search never sees: “floral dress, but not the one with the collar”, “refrigerator that fits under a 32-inch counter”, “product...
Tag embeddings
Off-the-shelf embedders understand generic language and nothing else. BGE-small and OpenAI’s Ada-002 encode “what is the refund policy” beautifully; ask them about an internal part number like OCT-4472-REV3 or a...
Catalog search lives and dies on the visual queries text search never sees: “floral dress, but not the one with the collar”, “refrigerator that fits under a 32-inch counter”, “product...
Tag synthetic_data
Fine-tuning a model for internal tooling usually means one thing is missing: the data. Our first attempt to adapt a model for internal ticket routing needed 10,000 labeled examples, and...
Tag data_engineering
Fine-tuning a model for internal tooling usually means one thing is missing: the data. Our first attempt to adapt a model for internal ticket routing needed 10,000 labeled examples, and...
Tag ebpf
If you serve LLM inference at any real scale, your standard monitoring stack has a blind spot exactly where your tokens actually spend time: the kernel. Prometheus exporters and Datadog...
Our Redis client was pushing a sustained 8% CPU on every API node, and pprof could not tell us why. The CPU profile blamed runtime.futex, which is not an answer;...
Tag rails-7
Rails 7.2 (August 2024) did not ship a magic filter that “skips patterns” during eager loading — that idea circulates in a lot of upgrade blog posts, but it is...
Tag zeitwerk
Rails 7.2 (August 2024) did not ship a magic filter that “skips patterns” during eager loading — that idea circulates in a lot of upgrade blog posts, but it is...
Tag dspy
We rewrote a production pipeline from LangChain to DSPy last spring, and the reason was not a benchmark — it was a two-day incident. A security team bumped our model...
Tag langchain
We rewrote a production pipeline from LangChain to DSPy last spring, and the reason was not a benchmark — it was a two-day incident. A security team bumped our model...
Tag programming_paradigms
We rewrote a production pipeline from LangChain to DSPy last spring, and the reason was not a benchmark — it was a two-day incident. A security team bumped our model...
Tag evals
BLEU and ROUGE did not catch a single regression for us. The rewrite that scored +3 ROUGE against the old transcripts was, by any human reading, worse — it hedged,...
Tag metrics
BLEU and ROUGE did not catch a single regression for us. The rewrite that scored +3 ROUGE against the old transcripts was, by any human reading, worse — it hedged,...
Tag llm_judge
BLEU and ROUGE did not catch a single regression for us. The rewrite that scored +3 ROUGE against the old transcripts was, by any human reading, worse — it hedged,...
Tag kubernetes
Since Rails 7.2 made YJIT the default JIT in August 2024, most Rails apps picked up a 15-25% throughput bump for free. The bill shows up on Kubernetes in a...
Tag graph_db
Vector RAG answered “what does the payment service do?” beautifully and went blind on “what did the author build for the payment service while reporting to project manager X?” The...
Tag neo4j
Vector RAG answered “what does the payment service do?” beautifully and went blind on “what did the author build for the payment service while reporting to project manager X?” The...
Tag gpt4o
GPT-4o function calling still managed to call charge_payment before create_transaction produced an ID. The model emitted both calls in one turn — valid tool calls, invalid order — and the...
Tag function_calling
GPT-4o function calling still managed to call charge_payment before create_transaction produced an ID. The model emitted both calls in one turn — valid tool calls, invalid order — and the...
Tag state_machine
GPT-4o function calling still managed to call charge_payment before create_transaction produced an ID. The model emitted both calls in one turn — valid tool calls, invalid order — and the...
Tag verification
GPT-4o function calling still managed to call charge_payment before create_transaction produced an ID. The model emitted both calls in one turn — valid tool calls, invalid order — and the...
Tag solid_queue
Rails 8 (November 2024) did something bold: it made the database the default home for everything that used to need Redis. Solid Queue for background jobs, Solid Cache for the...
Rails 8 shipped on November 7, 2024 with Solid Queue as the default background-job adapter — a Postgres-backed queue that fits the “one less database” story. We migrated a roughly...
Tag deadlocks
Rails 8 shipped on November 7, 2024 with Solid Queue as the default background-job adapter — a Postgres-backed queue that fits the “one less database” story. We migrated a roughly...
Tag self_hosting
A client in EU pharma needs inference that never leaves the data center: weights, prompts, and outputs all stay on-prem, and the model has to be legally auditable. Llama 3...
Tag llama3
A client in EU pharma needs inference that never leaves the data center: weights, prompts, and outputs all stay on-prem, and the model has to be legally auditable. Llama 3...
Tag vllm
Decode is memory-bandwidth-bound, and that single fact is why speculative decoding is the cheapest throughput win an inference team can spend a week on. Every autoregressive step loads the entire...
If you serve LLM inference at any real scale, your standard monitoring stack has a blind spot exactly where your tokens actually spend time: the kernel. Prometheus exporters and Datadog...
The first time we load-tested vLLM for a multi-tenant embedding-and-completion service, we watched nvidia-smi and assumed something was leaking. Four Mistral-7B instances on two A100s, steady 512-token requests, and the...
A client in EU pharma needs inference that never leaves the data center: weights, prompts, and outputs all stay on-prem, and the model has to be legally auditable. Llama 3...
Tag triton
A client in EU pharma needs inference that never leaves the data center: weights, prompts, and outputs all stay on-prem, and the model has to be legally auditable. Llama 3...
Tag inference
Decode is memory-bandwidth-bound, and that single fact is why speculative decoding is the cheapest throughput win an inference team can spend a week on. Every autoregressive step loads the entire...
The first time we load-tested vLLM for a multi-tenant embedding-and-completion service, we watched nvidia-smi and assumed something was leaking. Four Mistral-7B instances on two A100s, steady 512-token requests, and the...
Tag kv_cache
The first time we load-tested vLLM for a multi-tenant embedding-and-completion service, we watched nvidia-smi and assumed something was leaking. Four Mistral-7B instances on two A100s, steady 512-token requests, and the...
Tag paged_attention
The first time we load-tested vLLM for a multi-tenant embedding-and-completion service, we watched nvidia-smi and assumed something was leaking. Four Mistral-7B instances on two A100s, steady 512-token requests, and the...
Tag llama_guard
We run a customer-facing LLM gateway in front of a hosted model, and the threat model is unglamorous but real: prompt injection trying to exfiltrate another tenant’s data, jailbreaks attempting...
Tag safety
We run a customer-facing LLM gateway in front of a hosted model, and the threat model is unglamorous but real: prompt injection trying to exfiltrate another tenant’s data, jailbreaks attempting...
Tag guardrails
We run a customer-facing LLM gateway in front of a hosted model, and the threat model is unglamorous but real: prompt injection trying to exfiltrate another tenant’s data, jailbreaks attempting...
Tag slog
Logging is the last thing anyone profiles and the first thing that sinks a latency budget. A service doing 10k req/s with two or three log lines per request is...
Tag zap
Logging is the last thing anyone profiles and the first thing that sinks a latency budget. A service doing 10k req/s with two or three log lines per request is...
Tag logging
Logging is the last thing anyone profiles and the first thing that sinks a latency budget. A service doing 10k req/s with two or three log lines per request is...
Tag distillation
When DeepSeek-R1 landed in January 2025, it proved that chain-of-thought plus reinforcement learning could push open models to near-frontier reasoning on math and code. It also proved something less convenient:...
Tag deepseek
The standard recipe for a “reasoning” model is crushing amounts of supervised data: millions of human-curated chain-of-thought traces, distilled from a bigger model, cleaned by contractors, and fine-tuned into the...
When DeepSeek-R1 landed in January 2025, it proved that chain-of-thought plus reinforcement learning could push open models to near-frontier reasoning on math and code. It also proved something less convenient:...
Tag slm
When DeepSeek-R1 landed in January 2025, it proved that chain-of-thought plus reinforcement learning could push open models to near-frontier reasoning on math and code. It also proved something less convenient:...
Tag gomemlimit
A 2GB Kubernetes pod running a Go service with default GOGC=100 has a GC that will happily let the heap double before it starts collecting. That is by design: the...
Tag rails7_2
The Redis cache bill is a weird tax. You’re paying premium RAM prices for data that is, by definition, disposable — regenerate it and nothing breaks. When our cache footprint...
We run a 12-server Puma fleet against a Postgres primary and two read replicas. For a long time our read/write splitting was manual: a sidecar script rewrote database.yml on deploy...
Tag connection_pool
We run a 12-server Puma fleet against a Postgres primary and two read replicas. For a long time our read/write splitting was manual: a sidecar script rewrote database.yml on deploy...
Off-the-shelf embedders understand generic language and nothing else. BGE-small and OpenAI’s Ada-002 encode “what is the refund policy” beautifully; ask them about an internal part number like OCT-4472-REV3 or a...
Tag api_gateway
An API gateway sits in front of every upstream call, so its router is the hottest code path in the fleet. A linear scan over registered patterns is O(N) in...
Tag radix_tree
An API gateway sits in front of every upstream call, so its router is the hottest code path in the fleet. A linear scan over registered patterns is O(N) in...
Tag propshaft
Sprockets was designed for the Rails era where the framework compiled CoffeeScript, SCSS, and ERB-templated JavaScript on the fly. That era is over. Modern browsers ship ES modules and CSS...
Tag sprockets
Sprockets was designed for the Rails era where the framework compiled CoffeeScript, SCSS, and ERB-templated JavaScript on the fly. That era is over. Modern browsers ship ES modules and CSS...
Tag rails7
Sprockets was designed for the Rails era where the framework compiled CoffeeScript, SCSS, and ERB-templated JavaScript on the fly. That era is over. Modern browsers ship ES modules and CSS...
Tag assets
Sprockets was designed for the Rails era where the framework compiled CoffeeScript, SCSS, and ERB-templated JavaScript on the fly. That era is over. Modern browsers ship ES modules and CSS...
Tag llama3_1
Let me correct the framing up front: Llama 3.1 is text-only. The 8B, 70B, and 405B models released in July 2024 consume tokens, not pixels. Any pipeline that claims to...
Tag escape_analysis
The difference between a handler that does 40 allocations per request and one that does four is usually not your data structures. It is whether the compiler managed to keep...
Tag inlining
The difference between a handler that does 40 allocations per request and one that does four is usually not your data structures. It is whether the compiler managed to keep...
Tag solid_cache
Rails 8 (November 2024) did something bold: it made the database the default home for everything that used to need Redis. Solid Queue for background jobs, Solid Cache for the...
The Redis cache bill is a weird tax. You’re paying premium RAM prices for data that is, by definition, disposable — regenerate it and nothing breaks. When our cache footprint...
Tag caching
The Redis cache bill is a weird tax. You’re paying premium RAM prices for data that is, by definition, disposable — regenerate it and nothing breaks. When our cache footprint...
Tag ai
The 2023 version of this question was about knowledge: “should we fine-tune so the model knows our domain?” That framing is dead in 2025. Current models — Claude Sonnet 4,...
The model got smarter, but the reason agents work in production in 2025 is the harness around them. We run them in customer-support triage, internal ops, and incident summarization, and...
Tag production
The model got smarter, but the reason agents work in production in 2025 is the harness around them. We run them in customer-support triage, internal ops, and incident summarization, and...
Tag monitoring
If you serve LLM inference at any real scale, your standard monitoring stack has a blind spot exactly where your tokens actually spend time: the kernel. Prometheus exporters and Datadog...
Tag json
The last place I look when an API endpoint is slow is the serializer, and that’s a mistake. On a typical list endpoint the serializer is where most of the...
Prompting a model to “return valid JSON” fails at a rate that is invisible until it isn’t: at low volume a 2% malformed-output rate is a retry. At 5M calls/day,...
Every few months someone on the team proposes “we should just use a faster JSON library” and benchmarks encoding/json against a dropped-in replacement, sees a 2x, and ships it. That’s...
Tag code_generation
Every few months someone on the team proposes “we should just use a faster JSON library” and benchmarks encoding/json against a dropped-in replacement, sees a 2x, and ships it. That’s...
Tag easyjson
Every few months someone on the team proposes “we should just use a faster JSON library” and benchmarks encoding/json against a dropped-in replacement, sees a 2x, and ships it. That’s...
Tag rails8
Rails 8 (November 2024) did something bold: it made the database the default home for everything that used to need Redis. Solid Queue for background jobs, Solid Cache for the...
ActionCable’s Redis dependency always annoyed me, and not for the reasons people usually cite. The latency argument against database pub/sub is mostly wrong at small scale — the real cost...
Tag solid_cable
Rails 8 (November 2024) did something bold: it made the database the default home for everything that used to need Redis. Solid Queue for background jobs, Solid Cache for the...
ActionCable’s Redis dependency always annoyed me, and not for the reasons people usually cite. The latency argument against database pub/sub is mostly wrong at small scale — the real cost...
Tag mutex
The advice “share memory by communicating” has done more damage than good. It was written for clarity, not throughput, and teams have shipped channel-based counters and caches that run 5-10x...
Tag fine_tuning
The 2023 version of this question was about knowledge: “should we fine-tune so the model knows our domain?” That framing is dead in 2025. Current models — Claude Sonnet 4,...
Tag prompt_engineering
The 2023 version of this question was about knowledge: “should we fine-tune so the model knows our domain?” That framing is dead in 2025. Current models — Claude Sonnet 4,...
Tag goroutines
A goroutine leak is the sneakiest way to lose memory in Go, because the memory is not garbage: it is a live stack, referenced by a blocked goroutine, unreachable to...
Tag memory_leaks
A goroutine leak is the sneakiest way to lose memory in Go, because the memory is not garbage: it is a live stack, referenced by a blocked goroutine, unreachable to...
Tag debugging
A goroutine leak is the sneakiest way to lose memory in Go, because the memory is not garbage: it is a live stack, referenced by a blocked goroutine, unreachable to...
Tag generics
When Go 1.18 shipped generics, the design had to thread a needle: be fast like C++ templates without their binary bloat, and be ergonomic without Java’s boxing tax. The answer...
Tag monomorphization
When Go 1.18 shipped generics, the design had to thread a needle: be fast like C++ templates without their binary bloat, and be ergonomic without Java’s boxing tax. The answer...
Tag r1
The standard recipe for a “reasoning” model is crushing amounts of supervised data: millions of human-curated chain-of-thought traces, distilled from a bigger model, cleaned by contractors, and fine-tuned into the...
Tag reasoning
The standard recipe for a “reasoning” model is crushing amounts of supervised data: millions of human-curated chain-of-thought traces, distilled from a bigger model, cleaned by contractors, and fine-tuned into the...
Tag scheduler
Go’s concurrency story rests on one runtime component most engineers never look at: the scheduler. If you have ever seen a service with 300,000 goroutines and 12 cores stay responsive,...
Tag runtime
Go’s concurrency story rests on one runtime component most engineers never look at: the scheduler. If you have ever seen a service with 300,000 goroutines and 12 cores stay responsive,...
Tag prompt_caching
Every RAG or agent application I’ve reviewed re-sends the same few thousand tokens on every request: the system prompt, tool schemas, a hundred pages of policy docs, the conversation so...
Tag latency
Every RAG or agent application I’ve reviewed re-sends the same few thousand tokens on every request: the system prompt, tool schemas, a hundred pages of policy docs, the conversation so...
Tag anthropic
Every RAG or agent application I’ve reviewed re-sends the same few thousand tokens on every request: the system prompt, tool schemas, a hundred pages of policy docs, the conversation so...
Tag gemini
Every RAG or agent application I’ve reviewed re-sends the same few thousand tokens on every request: the system prompt, tool schemas, a hundred pages of policy docs, the conversation so...
Tag decoding
Prompting a model to “return valid JSON” fails at a rate that is invisible until it isn’t: at low volume a 2% malformed-output rate is a retry. At 5M calls/day,...
Tag grammars
Prompting a model to “return valid JSON” fails at a rate that is invisible until it isn’t: at low volume a 2% malformed-output rate is a retry. At 5M calls/day,...
Tag outlines
Prompting a model to “return valid JSON” fails at a rate that is invisible until it isn’t: at low volume a 2% malformed-output rate is a retry. At 5M calls/day,...
Tag serialization
Why serialization keeps topping your pprof
The last place I look when an API endpoint is slow is the serializer, and that’s a mistake. On a typical list endpoint the serializer is where most of the...
Tag oj
The last place I look when an API endpoint is slow is the serializer, and that’s a mistake. On a typical list endpoint the serializer is where most of the...
Tag text_to_sql
Text-to-SQL looks solved in demos and falls apart in production, always in the same four ways: the model invents column names the schema never had, it emits syntax that parses...
Tag agent
Text-to-SQL looks solved in demos and falls apart in production, always in the same four ways: the model invents column names the schema never had, it emits syntax that parses...
Tag protobuf
Why serialization keeps topping your pprof
Tag flatbuffers
Why serialization keeps topping your pprof
Tag react
Almost every agent starts as a naive while True: call_model, run_tool, call_model loop, and almost every production agent ends up as something else. The reason is that the three canonical...
Tag planning
Almost every agent starts as a naive while True: call_model, run_tool, call_model loop, and almost every production agent ends up as something else. The reason is that the three canonical...
Tag reflection
Almost every agent starts as a naive while True: call_model, run_tool, call_model loop, and almost every production agent ends up as something else. The reason is that the three canonical...
Tag speculative_decoding
Decode is memory-bandwidth-bound, and that single fact is why speculative decoding is the cheapest throughput win an inference team can spend a week on. Every autoregressive step loads the entire...
Tag actioncontroller_live
“Real-time dashboard” is the feature that silently becomes an architecture decision. The naive version polls a JSON endpoint every second: one client is nothing, but 500 dashboards at 1 req/s...
Tag sse
“Real-time dashboard” is the feature that silently becomes an architecture decision. The naive version polls a JSON endpoint every second: one client is nothing, but 500 dashboards at 1 req/s...
Tag streaming
“Real-time dashboard” is the feature that silently becomes an architecture decision. The naive version polls a JSON endpoint every second: one client is nothing, but 500 dashboards at 1 req/s...
