A client POSTs a payment, the connection drops mid-response, and the client retries. If your endpoint is not idempotent, the second request charges the card twice. At roughly 3M mutating requests a day, even a 0.05% retry rate produces thousands of duplicate records a day — and for payments those duplicates are chargebacks and support tickets.

The fix is not to make retries impossible; it is to make them harmless. An idempotency layer gives each logical operation one stable outcome no matter how many times it is submitted.

The contract

Clients send an Idempotency-Key header — a UUID the client generates per operation. The server guarantees: the first request executes and stores the response; every replay returns the stored response with Idempotent-Replay: true; concurrent duplicates never both execute.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
module IdempotentRequest
  extend ActiveSupport::Concern

  IDEMPOTENCY_TTL = 30.minutes

  included do
    before_action :require_idempotency_key, only: :create
  end

  private

  def require_idempotency_key
    return if request.headers["Idempotency-Key"].present?

    render json: { error: "missing Idempotency-Key" }, status: :400
  end

  def run_idempotent
    fingerprint = sha256(
      request.method,
      request.path,
      canonical_body(request.raw_post)
    )
    redis_key = "idem:#{fingerprint}"

    claimed = IdempotencyStore.claim(redis_key, IDEMPOTENCY_TTL) do
      response = yield
      store_response(redis_key, response, IDEMPOTENCY_TTL)
    end

    if claimed.replayed?
      headers["Idempotent-Replay"] = "true"
      render json: claimed.body, status: claimed.status
    end
  end

  def sha256(*parts)
    Digest::SHA256.hexdigest(parts.join("\x1f"))
  end

  def canonical_body(raw)
    JSON.generate(JSON.parse(raw).deep_transform_keys(&:to_s).sort_by(&:first))
  end
end

Two details matter more than they look. First, we fingerprint the request (method + path + SHA-256 of canonicalized JSON) rather than trusting the key alone. A correct client sends a fresh UUID per operation anyway, but the fingerprint catches clients that lazily reuse one key across different payloads — we return 409 Conflict for that instead of replaying the wrong cached body.

Second, canonicalization. Raw JSON has whitespace and key-ordering variance, so {"a":1,"b":2} and {"b":2,"a":1} must hash identically. Sorting keys and regenerating JSON gets us that; for typical payment bodies it costs about 20µs.

Claiming and replaying

IdempotencyStore.claim uses Redis SET ... NX EX as both a lock and the first cache write:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
class IdempotencyStore
  PROCESSING = "processing"

  def self.claim(key, ttl)
    return Replay.new(processing: true) if Redis.current.get(key) == PROCESSING

    if Redis.current.set(key, PROCESSING, ex: ttl, nx: true)
      begin
        result = yield
        Redis.current.set(key, encode(result), ex: ttl)
        result
      rescue => e
        Redis.current.del(key)
        raise
      end
    else
      Replay.new(read_response(key))
    end
  end
end

The nx: true guarantee is what makes this correct: two retries arriving in the same millisecond both perform the SET, and exactly one wins. The loser sees PROCESSING and returns 202 Accepted with a Retry-After — it polls, or the client retries once the winner has stored the body. Only terminal outcomes are cached, so if the first attempt raises, we delete the key and a retry genuinely re-runs instead of being handed a half-finished response.

The numbers

• Duplicate charge rate dropped from ~0.02% to a handful of incidents per quarter, all traceable to clients generating keys non-deterministically per retry — their bug, now visible in our logs.
• Redis cost: 3M ops/day at a 2–4 KB stored body peaks around 8 GB over a 30-minute window, comfortably inside a 12 GB primary; every replay is one GET.
• A 30-minute TTL covers virtually all retry storms while bounding memory. Clients that need a longer window re-issue with a fresh key.

Trade-offs and edges

• Only mutating endpoints need this. GET/HEAD skip the layer entirely — cache and rate-limit those normally.
• We store the full response body so replays return identical payloads, but keep stored bodies bounded (truncate at ~64 KB) and never cache streaming or async responses.
• The lock serializes the first request with a duplicate; if the operation is slow (a webhook fan-out), raise the TTL or duplicates will hit the PROCESSING path — correct, just slower.
• This is not a substitute for transactional logic. The yield still runs inside an ActiveRecord transaction; idempotency makes the write repeat-safe, not the business rule.

What we would do differently

We rolled this into the controller layer. Starting over, I would extract it into a Rack middleware with the store behind an interface, because the second team that needs idempotency — there is always a second team — should not copy a concern. But the semantics above — claim, execute, cache terminal state, replay — are the load-bearing part, and they are not specific to Rails.