The pattern starts innocently:

1
2
3
class Article < ApplicationRecord
  after_save :index_in_elasticsearch
end

By the time it sits on your biggest table, one shard has gone yellow, a save takes 180ms p95, and an Elasticsearch restart is a paging event. This post is the story of pulling search indexing out of the request path entirely.

Why synchronous indexing is a trap

Every save serializes on the Elasticsearch round trip. The write path inherits the search cluster’s tail latency, and Elasticsearch, unlike Postgres, does not fail fast under memory pressure — it thrashes, and your web threads wait. Worse, after_save runs inside the transaction, so a failed index request rolls back your database write. We shipped exactly this and watched a single slow shard drag API p95 from 90ms to 400ms on an otherwise healthy service.

Decoupling on commit

The rule is simple: index after the transaction commits, on a queue, and never hold an ActiveRecord object in job arguments.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
class Article < ApplicationRecord
  after_commit :enqueue_indexing, on: [:create, :update]
  after_commit :enqueue_deletion, on: :destroy

  private

  def enqueue_indexing
    ArticleIndexingWorker.perform_async(id, updated_at.to_f)
  end

  def enqueue_deletion
    ArticleIndexingWorker.perform_async(id, nil)
  end
end

after_commit matters because after_save still fires before the COMMIT is visible to other readers. A worker that picks a record up mid-transaction either reads a stale row or — the common failure — races ahead of the commit and raises RecordNotFound. The retry then runs once the row is finally visible, which is exactly how you end up with duplicate documents when the upsert is not idempotent.

The worker owns serialization, queue priority, and dedup:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
class ArticleIndexingWorker
  include Sidekiq::Worker

  sidekiq_options queue: :search, retry: 8

  def perform(article_id, updated_at)
    return delete_document(article_id) if updated_at.nil?

    key = "search:lock:#{article_id}"
    return if Sidekiq.redis { |r| r.set(key, 1, ex: 10, nx: true) }.nil?

    article = Article.find(article_id)
    index(article)
  rescue ActiveRecord::RecordNotFound
    delete_document(article_id)
  end
end

The nx lock collapses bursts of rapid saves: if an article is touched 12 times in a minute, the first job indexes the latest row (the fetch is the source of truth) and the other 11 skip. On a CMS where documents get updated in cascades, that cut our indexing volume by roughly 80%.

Ordering and staleness

The lock has one failure mode: an old job wins the lock while a newer save is queued, indexing stale content. Because we serialize updated_at and the worker refetches, the fix is to re-enqueue when you lose the race, or — if you can afford it — dirty-check: compare the fetched updated_at to the job’s timestamp and bail when a newer job is pending. For a search index that tolerates a few seconds of lag, we skip the extra round trip.

Reconciliation, because retries are not a backup plan

Retries cover transient failures, not lost jobs — a box dying between commit and enqueue, or a deploy draining the queue. We run a nightly reconciliation job that reindexes anything newer than the last successful sweep:

1
2
3
4
5
def reconcile(from: 24.hours.ago)
  Article.where("updated_at > ?", from).find_in_batches(batch_size: 5_000) do |batch|
    SearchIndexBulkWorker.perform_bulk(batch.map { |a| ["index", a.id, a.updated_at.to_f] })
  end
end

Bulk requests (_bulk, 5k docs per request) turn a 2M-document full reindex from a 40-minute grind into roughly 12 minutes at 25 concurrent workers. We build the fresh index under a new name and swap the alias at the end, so search never sees a half-built index.

Lessons that stick

• Keep indexing out of the request path entirely; the queue is the API contract, not the database row.
• Serialize primitives (id, timestamp), never ActiveRecord instances — job args are JSON.
• Idempotent upserts (fixed _id) make retries and reconciliation safe by construction.
• During bulk reindexes, set refresh_interval to -1 and refresh once before switching the alias.
• Watch queue depth and search:lock hit rates. When depth grows, fan out with perform_bulk; do not buy more retries.

We now run about 1.2M indexing operations a day, and the save path is a pure database transaction: p95 ~38ms, flat, regardless of what Elasticsearch is doing. That is the whole point — search availability should never be a write-path dependency.