Why Fast Deployments Still Feel Dangerous

Most engineering teams know their deployment time by heart. Four minutes, maybe six. It is on a dashboard somewhere and it got celebrated when it improved.

Far fewer know their recovery time — how long from "this release is bad" to "this release is no longer in production." In a lot of organisations that number is forty minutes, or ninety, or genuinely unknown because it depends who happens to be awake.

Put those two numbers side by side and you have the explanation for something that puzzles a lot of engineering leaders: why, after two years of pipeline investment, nobody wants to release on a Friday afternoon.

Speed improves. Safety does not.

This asymmetry is built into what gets measured.

Deployment frequency and lead time improve visibly when you invest in a pipeline. They go up and to the right. People outside engineering understand them.

Change failure rate and time to restore behave differently. They are harder to instrument, they only look good when nothing dramatic happens, and — this is the important part — they do not improve as a side effect of a faster pipeline. You have to work on them deliberately.

So effort goes where the visible improvement is. Two years later there is an excellent deployment system and roughly the original recovery capability.

What fear turns into

Watch a team that does not trust its recovery path.

It adds approval gates. Not deliberately — nobody writes a policy saying let us go slower. But a release manager who cannot quickly undo a mistake will find other ways to prevent one. A second reviewer. A pre-release checklist. A change board that meets weekly.

Every one of those is a sensible local response to a real risk. Together they cancel out the automation investment entirely.

This is usually misread in retrospectives as cultural resistance. It is not. It is evidence that the automation never addressed the thing people were actually afraid of.

Making mistakes cheap

Three changes do most of the work.

Separate deploying from releasing. With feature flags, shipping code and turning behaviour on become different events. Deployment stops being a decision and becomes plumbing. Rollback becomes a configuration change measured in seconds rather than a redeploy measured in pipeline duration.

Let the machine decide the rollback. Send a small share of traffic to the new version, watch the health signals, promote or revert automatically. The key word is automatically — a canary that waits for a person to read a dashboard is just a slower manual deployment. All the value is in the decision happening in seconds, at 3am, without waking anyone.

Actually test the rollback. A rollback path that has never been used is not a capability, it is a guess. Exercise it on a schedule the way you would test a backup restore.

The part that stalls everything

You can roll back code. You cannot roll back code across a schema change that dropped a column — reverting gives you an application expecting something that no longer exists.

So automated recovery is not really an infrastructure feature. It is a rule about how every migration gets written: add the new structure, migrate while both work, remove the old one only after the previous release is fully retired.

It costs more per change. It requires discipline from people who are not thinking about rollback when they write the migration. And it never lands on a roadmap, because it looks like a code review standard rather than a project.

An organisation that has not solved this does not have automated recovery, whatever the deployment tooling says.

Two questions worth asking

When did you last roll back, and how long did it take? If nobody remembers, that is not stability. That is an untested path.

How many people touch a change between merge and production? Count the real path, including the person who has to be asked and the deploy only one team member is comfortable running. That number is the most honest measure of automation coverage there is, and it is usually higher than the people who built the pipeline think.

Longer version with the full argument, costs, and a 90-day plan: https://techcirkle.com/blog/devops-automation

Frequently Asked Questions

Why does recovery time matter more than deployment speed?

Because it determines how risky each release feels, and perceived risk drives behaviour. A team that cannot undo a change quickly compensates with approval gates and checklists, which reintroduces the delay automation was supposed to remove. Fast deployment with slow recovery produces an organisation technically able to ship continuously and unwilling to.

What is a good time to restore service?

With progressive delivery and automated rollback, reverting should take under five minutes and usually under one, because it is a traffic or flag change rather than a redeploy. If recovery requires rebuilding and redeploying, your floor is pipeline duration — which is the strongest practical argument for decoupling deployment from release.

Why can't you roll back a database migration?

Destructive changes like dropping a column break the previous version of the application, so reverting the code does not restore a working system. The standard mitigation is expand-migrate-contract: add new structure, migrate while both work, and remove the old structure only after the previous release is retired.

How do you test a rollback without causing an incident?

Do it deliberately in a low-traffic window, the way you would test restoring a backup. Deploy something harmless, roll it back, measure how long it took. Some teams make rollback part of the normal release routine for a while so it becomes ordinary. The goal is that nobody's first rollback is during a real emergency.

Are feature flags worth the added complexity?

Yes, provided they are removed. Flags that live forever accumulate into combinations of state nobody can reason about. Give every flag an owner and an expiry date, with a scheduled audit that deletes expired ones. The debt is real but manageable, and the alternative — deployment and release being the same event — is worse.