Procedure
How we run a scheduled cutover
Every step below is written so it can be handed to someone who was not in the planning meeting. Each one names the moment it happens, the failure that stops it, and what we do about it.
The run sheet
- 01
Freeze schema migrations
T-7 days
Engineers migrate against the old primary for a week, so the replication target is never mid-migration.
- 02
Replicate the primary
T-24 hours
Continuous replication runs for a full day so lag stays observable instead of surprising.
- 03
Verify row counts and checksums
T-4 hours
Compare table row counts and a per-table checksum between source and target.
- 04
Warm the application cache
T-60 minutes
Prime the cache so the first requests do not compile cold against the new target.
- 05
Lower the DNS TTL
T-30 minutes
Drop the record TTL to 60 seconds and wait two TTL periods before anyone is moved.
- 06
Open the read-only window
T-10 minutes
Writes stop on the old primary; readers keep working while the last delta is applied.
- 07
Switch the traffic
T-0, 5 minutes
Move the record to the new target and watch the error rate continuously.
- 08
Smoke test the product
T+15 minutes
Place one real order, log in, and trigger the background jobs that usually break.
- 09
Restore the TTL and unfreeze
T+60 minutes
Raise the TTL to 3600 seconds and allow migrations against the new primary.
Where the clock runs out
The schedule is dominated by waiting, not by work. Replication needs a full day so that lag stays visible, checksum verification needs four hours, and the cache needs an hour to warm. The traffic switch itself is five minutes.
Because the expensive steps sit before the switch, a failed attempt is cheap. Where the failure happens after the switch, the rollback window is what protects the moving day.