A product experiment should be allowed to lose
Let’s explore this page together
Why a product test is only meaningful if a worse buyer outcome can lead to revision or rollback instead of being reframed as success.
This article reflects the named expert’s practical perspective. See NovAsia’s editorial policy for how material is prepared and reviewed.
A product experiment is easy to celebrate when one number rises. Imagine a hypothetical enquiry form that becomes shorter and is submitted more often. The result looks clean: the new version produced more enquiries.
That may be good. It may also be incomplete. A property service does not exist merely to produce form submissions. The form is part of a longer sequence in which the buyer needs to express a real question and the team needs enough context to respond usefully.
Define the downside before seeing the result
If a team decides what success means only after the test, almost any outcome can be defended. More enquiries prove the form is better. Fewer enquiries prove that low-intent users were filtered out. More follow-up questions prove that conversations are deeper. Fewer follow-up questions prove that the process is more efficient.
A genuine experiment needs a possibility of disappointment.
Take a purely hypothetical change. The old form asks whether the buyer is looking for a home to live in, a rental property, or is still exploring. The new form removes that question to reduce friction. Submission volume rises.
I would then look beyond the first event. Do staff now spend more time re-establishing the buyer's basic purpose? Are there more conversations in which the buyer expected a simple answer but the team interpreted the message as a request for a full property search? Does the first useful response take longer? Are people being asked to repeat context they thought they had already supplied elsewhere?
None of these observations automatically defeats the new form. They reveal the trade-off the headline metric cannot show.
There should also be a threshold at which harm does not need a long statistical debate. If a change creates ambiguity about consent, hides a material condition or causes the service to treat a question as an authorised action, the team should not keep exposing users merely to obtain a prettier sample. Some problems are product boundaries before they are optimisation questions.
Rolling back is a result, not an embarrassment
The difficult moment comes after the evidence turns against a change. Work has already been spent on design and implementation. People have explained internally why the idea should succeed. The new version may look more modern. Reverting can therefore feel like wasting effort.
That is exactly why rollback has to be acceptable before the test begins.
A useful question is: if both versions appeared today with no history attached, which would we choose from what we now know? That helps separate sunk effort from the next user's experience.
Rollback does not always mean restoring the old design permanently. The test may reveal that shortening the form was valuable while one removed question carried essential context. A third version can preserve the lower effort and restore the missing meaning in another way. The important thing is that the data are allowed to change the product rather than merely decorate the original decision.
Negative results belong in the team's memory
Teams often remember successful launches better than failed ones. That creates a distorted internal history: every change appears to have been an improvement, and future decisions are built on a catalogue of wins.
A useful failed experiment records what the team expected, what actually happened, what limit became visible and what changed afterwards. No villain is required. A hypothesis can be reasonable and still be wrong.
Experimentation becomes politically difficult inside a team when every failed hypothesis is treated as somebody's personal failure. People then design tests that cannot really lose or avoid testing uncomfortable assumptions at all.
I would rather have a culture in which the embarrassment lies in ignoring a clear downside, not in discovering one.
The final output is a decision, not a slide
A mature experiment ends with a product choice: keep the new version, revise it, return to the previous one, or investigate a specific unresolved issue before expanding exposure. The outcome should correspond to what happened to users after the change.
More enquiries can be valuable. So can fewer but clearer enquiries. Neither is inherently superior without the purpose of the product and the downstream effects.
If one outcome is forbidden in advance—if the new feature must remain because too much work went into it—then the organisation is no longer running an experiment. It is searching for a metric that can justify a decision already made.
The most useful experiment is not the one that produces the nicest percentage. It is the one that gives the team permission to learn something inconvenient and then act on it.