The migration that nearly took us down wasn’t a clever one. It was add_index :orders, :customer_id, run at 2pm against a table with 40 million rows. Postgres holds a SHARE lock for the duration of a CREATE INDEX, so every write to orders queued behind it. Read replicas lagged, the primary sat at 100% IO, and requests that touched orders timed out for eleven minutes. The index build itself wasn’t even slow; the lock made it fatal.

That is the entire argument for a zero-downtime migration discipline: the dangerous operations aren’t exotic, they’re the ones you run without thinking. In this post I’ll cover what strong_migrations blocks, the Postgres-safe patterns that replace each dangerous operation, and the deploy ordering that makes it all stick.

Correcting the common wisdom first

Half the “zero-downtime Rails migration” articles you’ll find describe a three-step dance for adding a column with a default: add the column bare, add the default, backfill in batches. That advice is a fossil. Since Postgres 11 (2018), add_column with a constant default does not rewrite the table or lock it beyond a millisecond — the default is metadata, not a backfill. So on modern Postgres this is genuinely safe in one step:

1
2
3
4
5
class AddActiveToUsers < ActiveRecord::Migration[6.0]
  def change
    add_column :users, :active, :boolean, default: true
  end
end

The three-step dance still matters for MySQL (pre-8.0.12) and for computed defaults, where you’re backfilling derived data, not a constant. Get the version of your database into your brain before you write a migration, because the same migration can be a no-op on one host and an outage on another.

Let strong_migrations do the version check

Install strong_migrations early and let it encode the rules we all learn the hard way. It inspects every migration and raises StrongMigrations::UnsafeMigration for the classics: adding an index without concurrently, adding a column with a default on old Postgres, changing a column type, renaming a column or table, removing a column, and the execute-based backfill helpers. It also wraps migrations in an advisory lock so two deploys can’t run schema changes at once.

1
2
3
# config/initializers/strong_migrations.rb
StrongMigrations.lock_timeout = 10
StrongMigrations.start_after = 20200101000000

lock_timeout makes every migration run with SET lock_timeout = '10s': if a migration has to wait on a long-running query, it fails fast instead of queueing behind it. start_after tells it to ignore migrations that predate the discipline, so adopting the gem doesn’t mean rewriting history. The pay-off is that “unsafe” becomes a CI failure with a suggestion, not an incident.

The three patterns that actually matter

Adding an index. Always concurrently, and never inside a transaction:

1
2
3
4
5
6
class AddIndexOnOrdersCustomer < ActiveRecord::Migration[6.0]
  disable_ddl_transaction!
  def change
    add_index :orders, [:customer_id, :created_at], algorithm: :concurrently
  end
end

CREATE INDEX CONCURRENTLY trades the long lock for extra CPU and IO while it builds, which is the right trade. Our 40M-row build took about 40 minutes concurrently; the old blocking version took three minutes but locked the table for all of them. Monitor pg_stat_progress_create_index if you’re nervous. One caveat: it can’t run inside a transaction, hence disable_ddl_transaction!.

Backfilling derived data. Adding the column is cheap; filling it is the job that deserves a job. Keep batches small and don’t hold a connection across them:

1
2
3
4
5
6
7
def up
  add_column :users, :slug, :string
  User.where(slug: nil).find_each(batch_size: 2000) do |u|
    u.update_columns(slug: Sluggifier.call(u.name))
    sleep 0.05 if (u.id % 20_000).zero?
  end
end

update_columns skips validations and callbacks, so the loop only pays for the actual UPDATE. The sleep is a poor man’s throttle; if you’re on replicas, the real signal is replication lag, and the batch size should shrink until lag flattens. Above a few million rows, run this as a background job between deploys rather than inside the migration.

Renaming anything. A rename_column is atomic and that’s the problem: the instant it commits, the old code still running from your previous deploy starts failing with no column named: status. The expand/contract pattern is a multi-deploy sequence:

1
2
3
4
5
6
class AddStateToUsers < ActiveRecord::Migration[6.0]
  def up
    add_column :users, :state, :string
    User.where(state: nil).find_each(batch_size: 2000) { |u| u.update_columns(state: u.status) }
  end
end

Deploy one: add and backfill state, write both columns from new code. Deploy two: cut reads over to state. Deploy three: drop status. Budget for at least one deploy being mid-rollout while you’re doing this, because that’s exactly when both columns must exist.

Production lessons

• Migrate before you ship the code that depends on the schema. Write-new-first is the rule; a release that deploys code and migration together is already violating it.
• Run migrations as a separate step in the deploy, before instances roll, so a lock-waiting migration fails the deploy instead of the site.
• Set lock_timeout on every migration. A migration that waits 25 minutes behind a long report query is not a migration, it’s a scheduled outage.
• Check pg_stat_activity for lock waiters before and during any schema change, and know what your longest-running query is before you start.
• Treat the rename and type-change patterns as rehearsed procedures, not on-the-fly improvisation. Practice them on a staging copy with production-shaped data.

Zero-downtime migrations aren’t a gem feature; they’re a habit. strong_migrations makes the unsafe path a loud error instead of a quiet lock, and the patterns above make every migration boring enough to run at 2pm again.