Releasing and measuring
Shipping is not the end of delivery; it is the moment the team starts finding out whether it was right. Two practices make that moment safer and more informative: releasing gradually, and deciding in advance what you will measure.
Separate deploying from releasing
Deploying puts new code on the servers. Releasing makes a change visible to people. Treating these as the same event is what makes launches stressful: the moment the code goes live, everyone sees it, and any problem affects everyone.
Feature flags pull the two apart. New behaviour is deployed switched off and turned on separately, for chosen groups, at a time of the team’s choosing. If something goes wrong, the flag is switched off again in seconds, without a new deployment.
A staged rollout
A typical sequence:
- The team itself. Use the feature in real conditions for a day or two.
- Internal users or a few friendly customers. People who will tell you when something is odd.
- A small percentage of everyone, perhaps 5%. Watch error rates, performance and support tickets.
- Half, then everyone, once the numbers look normal.
Decide in advance what would make you stop and roll back: an error rate above a threshold, a drop in a key metric, a particular kind of complaint. A rollback decided in calm conditions is much easier to carry out than one argued about during an incident.
Decide what success looks like before launch
The same discipline as in assumption testing applies here. Before release, write down:
- the primary metric — usually the outcome the work was meant to move;
- its current baseline, measured over a period long enough to smooth out weekly noise;
- the change you expect, and the change that would count as clearly not working;
- one or two guardrail metrics that must not get worse: support contacts, page load time, cancellations;
- when you will look, and who will look.
Without this, results get interpreted after the fact, and almost any result can be made to look like a success.
Comparisons
Where there is enough traffic, the most reliable comparison is a controlled experiment: some people get the change, a similar group does not, and the difference is measured. Where there is not enough traffic — which is common in smaller products and in business software — a careful before-and-after comparison is still useful, as long as you are honest about what else changed in the same period: seasons, marketing campaigns, pricing.
Give it time. New things attract curiosity clicks for the first few days, and some benefits only appear after people have changed their habits. Two to four weeks is a reasonable default for a first look.
The post-launch review
A short meeting, a few weeks after release, with three questions:
- What did we expect to happen?
- What actually happened?
- What do we do now — keep, improve, or remove?
Removal deserves to be a real option. Features that did not work still cost something every month: in maintenance, in complexity, in confusing new users. Taking one out is a perfectly respectable result.
Also remove the feature flags once a change is fully released. Old flags are a quiet source of bugs and confusion.
Closing the loop
Whatever the review finds goes back into discovery: the opportunity map is updated, new questions join the interview guide, and the next round starts better informed than the last. That loop — from discovery through delivery and back — is the whole cycle. The teams that do it well are rarely the ones with the best ideas; they are the ones that find out fastest which of their ideas are good.