Rails 8 shipped on November 7, 2024 with Solid Queue as the default background-job adapter — a Postgres-backed queue that fits the “one less database” story. We migrated a roughly 2M-jobs-per-week Sidekiq workload onto it within days of launch. By week two we had a deadlock incident: around 0.3% of job claims raised ActiveRecord::Deadlocked under normal load, and on deploy-heavy days that number tripled to 3%, with some jobs retried so many times they landed in the failed table for no real reason.

Why Solid Queue deadlocks

Solid Queue claims work with a SELECT ... FOR UPDATE SKIP LOCKED over solid_queue_ready_executions, then inserts into solid_queue_claimed_executions and updates the parent solid_queue_jobs row to mark it claimed. Enqueues take the same parent row to flip its state. You now have two code paths locking the parent and child rows in opposite orders, and under enough concurrency that is a textbook lock-order inversion. Postgres reports it as error 40P01 / DeadlockDetected; ActiveRecord surfaces it as ActiveRecord::Deadlocked.

The signature in pg_locks was unmistakable: two sessions, each holding a KeyShareLock on a solid_queue_jobs row and each blocked on the other session’s execution row. The default Read Committed isolation level has nothing to do with this — deadlocks are a locking problem, not an isolation problem, so bumping default_transaction_isolation to SERIALIZABLE only makes it worse. The fixes are ordering, shorter lock hold times, and headroom.

The retry belt comes first

Retries are not a fix, they are a seatbelt — you put them on because the crash will happen again before you finish the real repair.

1
2
3
4
5
6
7
class ApplicationJob < ActiveJob::Base
  retry_on ActiveRecord::Deadlocked, attempts: 5, wait: :exponentially_longer

  def perform
    # ... job body
  end
end

Solid Queue already re-enqueues jobs that fail to claim cleanly, but retry_on gives a deterministic policy with backoff. Five attempts with exponential wait swallowed our 3% storms without leaking into the failed queue. Job bodies that are themselves deadlock-prone (updating two records that get touched in different orders by other workers) get their own retry_on closer to the offending code, with attempts: 2 so a genuinely broken job dies fast.

What actually reduced the deadlock rate

1. Consistent claim ordering. We saw the fetch query’s ordering (priority, then job_id) interact badly with enqueue paths that inserted rows in a different order. Making the enqueue path insert with the same ordering keys removed most of the inversion window. If you can’t change the gem’s SQL, at least keep your own priority values coarse — one big priority class per queue instead of per-job values.

2. Short transactions. A worker that runs a multi-minute transaction while holding claim locks is the real amplifier: every other worker in the same queue parks on SKIP LOCKED and the contention graph turns into a knot. Keep claim-time work minimal, commit before slow I/O, and move anything heavy into the job body after the claim transaction closes.

3. Connection pool sizing. The pool must exceed the sum of worker threads across all supervisors plus headroom. With pool: 5 and a supervisor running 5 worker threads, the claim query itself can block waiting on a connection while holding its transaction open — a great way to turn a rare deadlock into a routine one. We run pool at roughly max_threads + 4.

4. Database isolation. We moved the queue into its own Postgres database on the same RDS cluster using Solid Queue’s connects_to support. The reporting queries that held long table locks on the same instance were the original trigger for our incident; separating the queue from the analytics workload removed the whole class of problem. You don’t need a second server — one database per concern is enough.

What the numbers looked like after

Post-fix, deadlock events went to zero over a month of production traffic. Median claim latency was unchanged; the retry belt added nothing measurable because it only fires on the (now rare) collision. The real cost saving was operational: no more 3AM pages about a saturated failed-queue table full of jobs that were never going to fail twice.

Production lessons that stuck with us:

• Monitor pg_locks filtered on solid_queue_* relations and pg_stat_activity for transactions older than a few seconds; that’s your canary before the first deadlock log.
• retry_on ActiveRecord::Deadlocked belongs in the base job class on day one — nobody has ever regretted the seatbelt.
• Default Read Committed is correct. If you feel the urge to change isolation levels for a queue, you’re solving the wrong problem.
• Size the pool from thread count, not from the Rails app’s pool: habit. A queue adapter consumes connections differently than a web request does.
• A dedicated queue database costs you a connection string and buys you a clean separation of lock domains.