A schema migration is safe during a rolling release only when every application version that may still receive traffic can read the transitional schema. Design for overlap, not for the instant when a migration command completes.
Expand first
Add a new nullable field or table without removing the old one. Deploy code that can tolerate the old and new shape. If new writes must populate both representations, define the writer and monitor drift. Do not add a non-null constraint to an unpopulated large table as a surprise in the release window.
ALTER TABLE orders ADD COLUMN customer_ref_new text;
-- Deploy dual-compatible readers and writers; backfill in bounded batches.
-- Only later enforce constraints and retire the old column.Backfill as an operation, not a migration side effect
Process rows in bounded batches with a cursor or stable key, make the operation resumable, and record progress. Compare counts and sample records; do not assume that a completed job implies semantic equality. Plan what happens to rows written while the backfill runs.
Database engines differ in lock behavior, online index support and DDL transaction semantics. Inspect the exact operation against the deployed engine version and rehearse it on representative data. A small staging table does not expose the lock and replication effects of a large production table.
Contract after the fleet moves
Only remove the old field after all application instances, workers and external consumers have stopped reading it. Keep a clear point at which rollback shifts from a route/deploy reversal to a data restoration exercise. That point deserves explicit approval in the release plan.
For a tiny table and a maintenance window the business accepts, a simple migration may be less risky than an elaborate compatibility sequence. Use the expand-and-contract pattern where concurrent versions or table size make it necessary.
Example: renaming a customer reference
A direct rename breaks old code still deployed on one server. Instead, add a new column, teach the writer to populate it and backfill historical rows. Teach readers to prefer the new field with a temporary fallback. Once every server and worker uses the new field, enforce its constraint, stop the fallback and remove the old column in a later release.
Watch concurrent writes during the backfill. A batch copying yesterday's value must not overwrite a fresh update. Use a condition tied to source version or update time, and reconcile remaining nulls or mismatches before enforcing the new invariant.
Lock and replication budget
For every DDL statement, estimate table size, expected lock, index build behavior and replica lag on the actual database version. Set a bounded lock timeout so the migration fails instead of silently queuing application writes behind it. Inspect active transactions that might hold locks longer than expected.
Do not bundle destructive cleanup with the first release of new code. A separated contract phase buys time to observe forgotten consumers and makes application rollback possible. Document when the window closes, including backup and recovery expectations.
A rehearsal should include a worker from the old release, not just two web servers. Background jobs often outlive the web rollout and are the last writer of an old schema.
Sources and further reading
Services This Relates To
Written by KYCONNECTS Engineering.