Sidekiq’s contract is “push work, forget it,” but the moment you have an event bus that emits per-entity changes, you discover how literally Sidekiq honors that contract. Ten indexing jobs for the same record in two seconds is not ten jobs’ worth of work — it’s one job that hasn’t run yet, enqueued nine extra times. The other failure mode is social: one noisy tenant can enqueue so much that their 2 AM data-import queue dwarfs everyone else’s and the Redis LPOP that should be feeding your payment retries is feeding their retry storm. Both problems have the same shape: you need a gate in front of the queue, not after it.

Client-side deduplication

Dedupe before enqueue, not before perform. If you dedupe server-side, the jobs still occupy queue memory, still get counted in metrics, and still burn a Redis round-trip to pop. Client middleware runs in the process that calls perform_async, which is exactly where the redundant decision is being made:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
class DeduplicationMiddleware
  LOCK_TTL = 900 # seconds; must exceed worst-case job runtime

  def call(worker_class, job, queue, redis_pool)
    key = dedup_key(job["class"], job["args"])
    if redis_pool.with { |r| r.set(key, "1", ex: LOCK_TTL, nx: true) }
      yield
    end
  end

  private

  def dedup_key(klass, args)
    digest = Digest::SHA256.hexdigest(Marshal.dump(args))
    "dedup:#{klass}:#{digest}"
  end
end

SET NX EX is one atomic round-trip, so two processes enqueuing concurrently can’t both win. Two details are load-bearing. First, hash the args — keys with raw JSON blobs inside blow up memory and slow Redis. Second, the TTL is a correctness parameter: if a job legitimately runs longer than the lock, a re-enqueue gets silently dropped and work is lost. We set TTL to roughly twice the p99 runtime of the slowest job in the class, and that choice has its own failure mode — a genuinely duplicated job is now deduped against anything in a 15-minute window. That’s the trade-off, and it’s the right one for idempotent indexing work.

Before you hand-roll, know that sidekiq-unique-jobs exists and its until_expired / until_executed lock types cover most of this. We hand-rolled anyway because we needed per-tenant and global dedup keys in one middleware and we wanted the logic testable without the gem’s Redis fixtures. Both are legitimate; the gem wins if you want battle-tested re-locking on retries.

Per-tenant rate limiting

For rate limiting, client-side alone isn’t enough — a tenant’s already-enqueued jobs still run after they burn their budget. We gate twice: client middleware refuses new enqueues when a tenant is over budget, and server middleware is a backstop that drops jobs that slipped through, using a Lua token bucket so the check-and-decrement is atomic:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
class TenantRateLimitMiddleware
  SCRIPT = <<~LUA
    local current = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
    local tokens = tonumber(current[1]) or tonumber(ARGV[2])
    local ts = tonumber(current[2]) or tonumber(ARGV[3])
    local now = tonumber(ARGV[3])
    local elapsed = now - ts
    tokens = math.min(tokens + elapsed * tonumber(ARGV[1]), tonumber(ARGV[2]))
    if tokens >= 1 then
      redis.call('HSET', KEYS[1], 'tokens', tokens - 1, 'ts', now)
      return 1
    else
      return 0
    end
  LUA

  def call(worker_class, job, queue, redis_pool)
    tenant = job["args"].first["tenant_id"]
    bucket = "limit:#{tenant}:#{worker_class}"
    ok = redis_pool.with do |r|
      r.eval(SCRIPT, keys: [bucket], argv: [rate_per_s, burst, Time.now.to_f])
    end
    ok == 1 ? yield : false
  end
end

Lua scripts execute without interleaving on Redis’s single-threaded event loop, so the refill-and-decrement is race-free. The refill math means a throttled tenant gradually regains capacity instead of hard-stalling at a reset boundary — which matters, because a hard reset at midnight turns every off-peak batch job into a midnight thundering herd.

A note for teams that can pay for it: Sidekiq Enterprise ships both unique jobs and a Redis rate limiter with sorted-set history. If you’re on the OSS version, the ~60 lines above are the standard replacement.

What to measure

Queue length lies. A queue with 50,000 jobs and a fast worker is healthier than one with 500 jobs behind a blocked dependency. We alert on Sidekiq::Queue.new.latency (age of the oldest job) and on per-tenant enqueue rate rather than absolute depth. After shipping dedup, our enqueue rate for the entity-index topic dropped about 55% and queue latency on the affected queues went from minutes to single-digit seconds — the dedup had been silently doing nothing useful before we added it, because the dedup key contained a timestamp arg. Normalize keys or dedup doesn’t dedup.

Production lessons

• Client-side for dedup, both sides for rate limits. Client checks save queue and worker time; server checks catch the overshoot.
• TTL is a grace period, not a convenience. Too short drops work; too long blocks legitimate re-runs. Anchor it to measured job runtime, not intuition.
• Never let args with timestamps, request IDs, or backtraces into the key. Hash a canonicalized subset, or you’re paying for a lock that never fires.
• Test the race. Two threads enqueueing the same payload must yield exactly one job; add it to CI or the middleware rots.

Redis as a coordination point is cheap and correct here precisely because these operations are single-key and atomic. Keep them that way — the moment your dedup needs to read-modify-write across keys, you’ve outgrown the middleware and need a different design.