Infrastructure gets replaced because something reached end of support, a lease expired, or a system finally became too slow to defend. The programme is scoped, quoted and approved as a purchase.
Then it has to actually happen, against a business that is still operating, and that is where these projects go wrong. The failures are rarely about the new equipment. They are about sequence, dependencies nobody documented, and a rollback plan that turned out not to be one.
The pattern that causes most of the damage
The common shape of a bad migration is a big-bang cutover at a weekend, with everything moved at once, no way back after a certain hour, and discovery of a dependency at 2am on Sunday.
It happens for an understandable reason: incremental migration means running two things in parallel, which is more work and looks more expensive. The comparison being made is against a smooth big-bang cutover, which is not the alternative on offer.
| Big bang | Incremental | |
|---|---|---|
| Total effort | Lower if it works | Higher |
| Blast radius of a mistake | Everything | One workload |
| Rollback | Under pressure, on the night | Per workload, calmly |
| Where problems are found | During the cutover | During the parallel run |
| What a failure costs | The whole business, for a day | One team, for an hour |
There is one case where big bang is correct: where the systems genuinely cannot coexist. That case is rarer than it is claimed, and it is worth testing the claim before accepting it.
Sequencing
- 1
Map the dependencies before anything moves
What talks to what, including the things nobody documented — a scheduled task on a workstation, a hard-coded address in an application, a report that reads from a share. This is where migrations actually fail.
- 2
Move the least critical thing first
Not the easiest and not the most valuable. The least critical, because the first move is where you discover what your process gets wrong.
- 3
Move dependencies before dependants
A system moved before the thing it depends on will work in testing and fail when the dependency moves later.
- 4
Keep the old path available
Until the new one has run through a full business cycle, including whatever happens at month end. Month-end processes are the classic thing nobody tested.
- 5
Move the most critical thing last
By then the process is rehearsed and the surprises have already been found on things that mattered less.
What a rollback plan actually requires
Most rollback plans are a sentence saying the old system remains available. That is not a plan, and its weakness is only discovered at the moment it is needed.
- A decision point with a time on it. If it is not working by this hour, we roll back. Without that, the decision gets made by exhaustion at 4am.
- A named person who makes that call, present and awake.
- Steps written down, in order, tested at least once — not described in general terms.
- An honest answer about data. If the new system has accepted writes, rolling back loses them or requires reconciliation. That is the constraint that makes rollback impossible after a point, and the point should be known in advance.
- A time estimate for the rollback itself. A four-hour rollback started at 3am does not fit in the window.
The data point is the one that turns a rollback plan into an illusion. Once real transactions exist on the new system, going back is a reconciliation exercise rather than a switch. Knowing exactly when that threshold is crossed is part of the plan.
The practical details that decide the day
- Reduce DNS time-to-live several days before a cutover, so a name change takes effect in minutes rather than hours. Restore it afterwards. This single step converts a class of slow, confusing failure into a fast one.
- Check certificate names cover whatever the new system will be called, before the day rather than during it.
- Find the hard-coded addresses. Search configuration and application code for literal addresses before assuming names are used everywhere.
- Identify service accounts and confirm their credentials are known. A service that will not restart because nobody has its password is a common and avoidable stoppage.
- Verify licensing follows the workload. Some licences are bound to hardware characteristics and will refuse to start on new hardware.
- Confirm the backup of the old system is current and restorable immediately before the change, not the night before.
None of these is sophisticated. All of them are things found at 2am by teams that did not check them at 2pm.
Parallel running, done properly
Running old and new together is the technique that removes most of the risk, and it has two failure modes of its own worth naming.
The first is running in parallel indefinitely. Two systems doing the same job will diverge, and the business ends up maintaining both while trusting neither. Parallel running needs an end date decided before it starts.
The second is parallel running without comparison. If both systems are producing output and nobody checks that the outputs agree, the parallel period has proved nothing except that the new system runs. Comparison is the point.
After the cutover
- 1
Watch for a full business cycle
A week is not enough if anything happens monthly. Month-end, payroll and reporting cycles are where the untested paths are.
- 2
Keep the old system recoverable, then decommission it deliberately
On a date, with a check that nothing still references it. Systems left running indefinitely become unpatched machines nobody owns.
- 3
Update the documentation while it is fresh
Addressing, dependencies, credentials, and what the new arrangement actually is. This is the point at which it gets written or never.
- 4
Record what surprised you
The undocumented dependency you found is almost certainly not the only one of its kind, and the next migration should look for it.
Should infrastructure be migrated all at once or incrementally?
Incrementally, in almost all cases. Big-bang cutovers require less total effort only if nothing goes wrong, and the comparison people make is against a smooth big bang rather than the realistic one. Incremental migration limits the blast radius of a mistake to one workload, allows a calm rollback per workload rather than a pressured one on the night, and surfaces problems during a parallel run instead of during the cutover. The exception is systems that genuinely cannot coexist, which is claimed more often than it is true.
What makes migrations fail?
Undocumented dependencies, almost always, and usually from a small set: a hard-coded IP address, a scheduled task on someone's desktop, a service account whose password nobody knows, a firewall rule referencing an old address, or a certificate issued for a name that is about to change. Mapping what talks to what before anything moves is the single most valuable preparation step.
What does a real rollback plan need?
A decision point with a time on it, so the choice is not made by exhaustion at 4am; a named person present to make it; written, ordered steps that have been tested at least once; a time estimate for the rollback itself, since a four-hour rollback started at 3am does not fit the window; and an honest answer about data. Once the new system has accepted real writes, rolling back loses them or requires reconciliation, and the point at which that threshold is crossed should be known in advance.
Why reduce DNS time-to-live before a cutover?
So that a name pointing at a new address takes effect in minutes rather than hours. Without it, some clients continue reaching the old system long after the change, producing a slow and confusing partial failure that is difficult to diagnose while it is happening. Reduce it several days ahead so the shorter value has propagated, and restore it afterwards.
How long should old and new systems run in parallel?
Long enough to cover a full business cycle including month end, and no longer than a date decided before the parallel run starts. Two systems doing the same job diverge, and a business running both indefinitely maintains two things while trusting neither. Parallel running also has to include comparing the outputs — if nobody checks that both systems agree, the period has proved only that the new one runs.
Sources and further reading
- An IT roadmap for a growing business— the programme this executes
- Windows Server: the practices that actually matter— the lifecycle dates that trigger these migrations
- Linux servers for business— the same lifecycle pressure on the other platform
- CRM data migration and reconciliation— data migration specifically, and verification beyond row counts
- High availability for business systems— RTO, which decides how much cutover risk is acceptable
- Hybrid infrastructure: deciding what goes where— placement, decided before the migration is planned
- VPN deployment: a practical guide— a worked example of phased rollout and addressing collisions
- NIST SP 800-34 Rev. 1: Contingency Planning Guide— recovery planning structure, applicable to the rollback question
Services This Relates To
Written by KYCONNECTS Engineering. Client names are withheld under confidentiality.