The ROI of Product Experimentation

Most product leaders have a visceral understanding of why experimentation is important for their product development efforts. Yet ask them why this is and most struggle to articulate that experimenting is the activity with highest ROI development teams can be doing.

If you hanged around product development for long enough there are a few things you know for sure – these are almost the defining characteristics of modern product development environments:

• There is a, quite often large, gap between what users say they do and what they actually do
• Product analytics are great at telling “what happened” but “why it happened” is left as an opinion
• Whatever you build will quite often be less impactful that you imagine

And so, the best product people armed with this knowledge know to approach product development iteratively, shipping features in thin slices, prioritising development of higher impact features (in the best scenario informed by hard data, in the worst scenario based on some opinion / heuristics).

Taking this approach is sure to dramatically improve value output: in every iteration you are shipping something of “some” value. This article shows why that is: https://www.linkedin.com/pulse/minimum-viable-product-vs-lubomir-malo/. And thus product teams obsess about MVP definition, building in stages, measuring as they go, etc.

Yet teams are leaving money on the table when they don’t take experimentation seriously. Let’s see why this is with a simplified toy example.

Imagine you are a product manager faced with 4 potential features to build, for simplicity sake let’s assume they all take the same effort to build. So their ROI simply is their “impact/value”. You have looked at the analytics data, done some survey’s with users, etc and you’ve estimated their value to be:

1. Feature 1: 4
2. Feature 2: 3
3. Feature 3: 2
4. Feature 4: 1

As a rational product team everyone in the team commits to build feature 1.

Now, the 3rd principle of product development kicks in: “whatever you build will quite often be less impactful than what you imagine”.

So imagine your success rate is 50% – this is way higher than most reports. According to some, less than 25% of features are successful. Then, the expected return on investment of your decision to build feature 1 is then 2.

Now imagine that you can devise an experiment to know with high degree of certainty if each of your features are to be successful. Imagine to run these experiments and build the feature that turns out to be successful doubles the investment needed. The ROI of this approach is 4*0.5+3*0.5+2*0.5+1*0.5 / 2 = 2.5.

That’s a 25% increase on ROI from your product development efforts.

Of course you can play with the numbers and show lower ROI – for example if testing all 4 features costs 3 times as much as building one, you might as well just build and skip experimentation (do the math :)). If experimentation only marginally increase your success rate (say from 50% to 80%) then your experimentation ROI is the same as building feature 1: 2.5*0.8 = 2.

But the point of this article is not to advocate for or against experimentation. But to give a mental framework that product managers can use to think about experimentation in terms of ROI. As imperfect as this toy model is it highlights two points: just build it is not always the best approach. Nor is “experiment with everything”. It depends. Depends on how well you can craft your experiment to maximise your success and what is the cost of that experiment. Above all the point of this article to make the case for the ROI of experimentation – too often “experimentation” is seen as a dirty word by senior executives, let’s start making the case in terms that any executive can understand: return on investment.