The incident is boring and predictable: someone bumps Sidekiq concurrency from 10 to 25, and within an hour pager duty is looking at ActiveRecord::ConnectionTimeoutError across every web and worker process. Then the web tier’s p99 climbs because Postgres has run out of connections that the workers are squatting on.

I have danced this dance on three systems. The tuning is not a single flag — it is a budgeting problem with two scarce resources: database connections and CPU (specifically the GVL).

The connection budget

Every process — Puma or Sidekiq — checks connections out of its pool, and the pool is sized per process in database.yml. The rule that has never steered us wrong:

1
pool_size >= max_threads + headroom

The headroom covers nested calls: a job that spawns a transaction and touches two models still uses one connection, but callbacks and with_lock can transiently add one, and connection-holding during IO stacks up. We run RAILS_MAX_THREADS=8 on web and SIDEKIQ_CONCURRENCY=15 on workers, and size the pool from the environment so the two never drift:

1
2
3
4
5
# config/database.yml
production:
  adapter: postgresql
  database: app_production
  pool: <%= ENV.fetch("DB_POOL", [ENV.fetch("RAILS_MAX_THREADS", 5).to_i, ENV.fetch("SIDEKIQ_CONCURRENCY", 5).to_i].max + 4) %>

The +4 is not laziness: it absorbs after_commit callbacks, the checkouts that happen during slow queries, and the occasional job that legitimately nests work. Postgres’ max_connections then becomes the global constraint:

1
total = (web_processes x web_pool) + (worker_processes x worker_pool)

We run max_connections=400 and budget three-quarters of it; the rest is for migrations, admin consoles, and the pooling layer. When we exceeded that, the fix was not a bigger Postgres — it was fewer, bigger workers.

What concurrency actually buys you

This is where people get the GVL wrong. Threads in one Ruby process share the GVL, so a 25-thread Sidekiq running CPU-bound jobs gets maybe 1.2 cores’ worth of Ruby. Our measured curve on a 4-core worker box:

• 10 threads: ~240 jobs/min
• 20 threads: ~390 jobs/min (+62%)
• 30 threads: ~430 jobs/min (+10%)
• 40 threads: ~410 jobs/min (negative — GVL contention and context switching)

The plateau is real. Once jobs are IO-bound (HTTP calls, DB queries), more threads help — but you cannot know which you have without measuring. We instrument jobs/min per process and hold concurrency at the knee of the curve. When we need more throughput, we add worker processes (SIDEKIQ_CONCURRENCY=15 x N processes) rather than threads: process-level parallelism sidesteps the GVL entirely, at the cost of more RSS.

Connection starvation, in practice

The failure mode that bit us hardest: a worker wrapping an external API call in a transaction. Slow third-party response -> the transaction holds the DB connection for the whole call -> every other thread in the process blocks on checkout -> the pool drains -> the web tier shares the database and dies next.

1
2
3
4
5
6
7
8
9
10
11
12
13
class ImportWorker
  include Sidekiq::Worker

  def perform(ids)
    ids.each do |id|
      # BAD: the transaction spans a network call
      Account.transaction do
        data = http_get("/vendors/#{id}")   # holds the connection for 300ms+
        account.update!(data)
      end
    end
  end
end

The rule is short transactions: fetch, then transact.

1
2
3
4
5
6
def perform(ids)
  ids.each do |id|
    data = http_get("/vendors/#{id}")
    Account.transaction { account(id).update!(data) }
  end
end

That one change — never holding a connection across network IO — eliminated our starvation class and let us trim pool headroom back down.

Middleware, not sleeps

Rate limiting belongs in Sidekiq middleware, not in sleep calls that burn a thread and its connection for the duration:

1
2
3
4
5
6
7
8
9
10
11
12
13
class RateLimitMiddleware
  def call(_worker, _job, queue)
    limiter = RateLimiter.new("vendor-api", 10, 1.minute)
    limiter.consume!(queue)   # Redis-backed token bucket
    yield
  end
end

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add RateLimitMiddleware
  end
end

A sleeping thread holds its pool slot for the whole sleep; a middleware rejection returns instantly and Sidekiq retries the job later without occupying a connection.

The checklist we deploy with

• Size the pool from the environment, tied to thread counts — never a hardcoded constant.
• Measure jobs/min per process and find the concurrency knee before raising it.
• Treat Postgres max_connections as a global budget that includes every service that connects.
• Never hold a connection across external IO. Short transactions, fetch-then-write.
• Rate-limit in middleware, not with sleep in the job body.
• After any concurrency change, watch both ActiveRecord::ConnectionTimeoutError and worker RSS. A thread-unsafe gem that leaks thread-local state surfaces first as slow RSS creep.

Background jobs are embarrassingly parallel until they are not. Budget the two shared resources explicitly, and the “just bump the threads” instinct stops being a footgun.