In short
A safe migration is backward-compatible, deployed separately from the code that depends on it, and reversible. The pattern that avoids most incidents is expand-then-contract: add the new structure, deploy code that writes to both, backfill, switch reads, and only then remove the old structure. Renames and non-null columns with defaults are the changes most likely to lock a table.
The rule that prevents most incidents
Deploy the schema change and the code that needs it separately, in that order.
The reason is rollback. If a migration and its code ship together and you have to roll back, the code reverts but the schema does not — and the old code now runs against a schema it does not understand. You have turned one problem into two, during an incident.
So: make the schema change backward-compatible, deploy it, confirm it is fine, then deploy the code. Each step is independently reversible.
Which changes are actually risky
Risk depends on your database and version, but the shape is consistent.
Usually safe: adding a nullable column, adding a table, adding an index concurrently, dropping an index.
Usually dangerous on a large table: adding a column with a non-null constraint and a default — historically this rewrote the entire table while holding a lock, and on older versions still can. Changing a column type, which rewrites. Adding a foreign key, which scans the referenced table to validate. Adding an index without the concurrent option, which blocks writes for the duration.
Always dangerous: renaming or dropping a column that running code still references.
The deciding factor is usually how long a lock is held, not what the change is called. On a table with a thousand rows nothing matters. On one with fifty million, a lock held for thirty seconds is an outage.
Expand and contract
The pattern that makes breaking changes safe. Renaming user_name to full_name:
Expand. Add full_name as a nullable column. Deploy. Nothing reads it yet.
Dual write. Deploy code writing to both columns. Now new data is correct in both places.
Backfill. Copy existing values in batches, not one statement — a single UPDATE across millions of rows holds locks and bloats the transaction log. Small batches with a pause between them.
Switch reads. Deploy code reading full_name. Both are still written, so rollback remains trivial.
Contract. Once you are confident, stop writing the old column, then drop it in a later deploy.
Five deploys instead of one. Each is reversible, and none takes the site down. The instinct to compress this into one step is exactly what produces the incident.
Backfills need care
Backfilling is where migrations most often go wrong at scale.
Batch it. Update a few thousand rows at a time, committing between batches. One enormous transaction locks rows, grows the log, and cannot be interrupted safely.
Make it resumable. Track progress so an interrupted backfill continues rather than restarting.
Make it idempotent. Running it twice should be harmless, because you will run it twice — see idempotency.
Run it outside the migration. A backfill taking hours does not belong in a deployment step. Ship the schema change, then run the backfill as a separate job you can monitor and pause.
Throttle it. A backfill competing with production traffic degrades the site while succeeding.
Making migrations reversible
Write the down path, and try it — in development at least once, against realistic data.
Some changes cannot be truly reversed. Dropping a column destroys data; the down migration recreates the column but not its contents. Know which of your migrations are one-way and treat those with more caution: take a backup first, and schedule them deliberately rather than as part of a routine deploy.
In production, prefer forward-only recovery. If a migration causes a problem, the safer response is usually a new migration fixing it rather than reversing the original, because reversal can lose data written since.
Practical safeguards
Set a lock timeout on migrations, so a statement that cannot acquire a lock fails fast instead of queueing behind a long transaction and blocking everything behind it. A failed migration is recoverable; a five-minute pile-up is an outage.
Set a statement timeout so a runaway migration aborts rather than running for an hour.
Test against realistic volumes. A migration that runs in 50ms on a development database with 200 rows tells you nothing about 20 million.
Keep migrations in version control, applied in order, tracked in a table. Every framework does this; the failure is people applying manual changes outside it, after which no environment matches.
Never edit a migration that has run. Write a new one. Editing history means environments diverge silently.
Reviewing a migration
Before approving one, ask: what does this lock and for how long on the largest affected table; is it backward-compatible with the code currently running; can it be rolled back, and has anyone tried; does it need a backfill, and is that batched and separate; and has it been tested against realistic data volumes?
Most migrations pass all five in seconds. The ones that do not are exactly the ones worth slowing down for.
If you have a migration you are nervous about running against production, book a call.
Common questions
Why should migrations deploy separately from code?
Because of rollback. If they ship together and you roll back, the code reverts but the schema does not, leaving old code running against a schema it does not understand. Deploying the backward-compatible schema change first, confirming it, then deploying the code keeps each step independently reversible.
Which database migrations are dangerous?
Anything holding a long lock on a large table: adding a non-null column with a default, changing a column type, adding a foreign key that scans the referenced table, and adding an index without the concurrent option. Renaming or dropping a column that running code still references is always dangerous.
How do you rename a column safely?
Expand and contract, across five deploys. Add the new nullable column, deploy code that writes to both, backfill existing rows in batches, switch reads to the new column, then stop writing and drop the old one. Each step is reversible, which compressing it into one migration is not.
How should a large backfill be run?
In batches of a few thousand rows with commits between them, outside the migration itself, as a separate job you can monitor, pause and resume. One enormous UPDATE locks rows, bloats the transaction log, and cannot be interrupted safely. Throttle it so it does not degrade production traffic.
Should you roll back a bad migration in production?
Usually not — prefer forward-only recovery with a new migration that fixes the problem. Reversing can lose data written since the original ran, and some changes such as dropping a column cannot be truly reversed at all, because the down migration recreates the column but not its contents.
