The ROI of Product Experimentation
Most product leaders have a visceral understanding of...
Read More
I’ve been talking with Product people around Asia about their discovery habits. A common theme that I have come across is a difficulty in recruiting users to participate in interviews or surveys – this is specially true in the B2B space. I can relate, getting people to give you their time is time consuming and you hear more No’s than Yes’s (that is if you hear anything at all…) – so a huge thank you for those that have been talking to me!
Of course you can leverage platforms and recruitment companies and pay for their time – which makes it expensive. No wonder that there are people now looking to Generative AI and promising that they can “synthesise users”. I confess, to put it mildly, I’m skeptical of such solutions.
So what can we do? How can we learn from users effectively? What other tools can we leverage to maximise learnings? Approaches vary depending if you are exploring a new problem area and a new market or if you are looking to optimise market fit in a known problem area. Let me cover a few ways that go beyond user interviews and surveys.
If you have a product in market your options are plentiful: user analytics, session recordings, support interactions, in-product surveys, sales interactions, etc. One of the reasons why launching early and being in market with a MVP is so important – it kick starts the product market fit flywheel. The difficulty here is choosing the right source and method for the right question and making sense of all the different sources.
In product surveys, are best used when embedded in the user journey and when are hyper relevant to the user action – a typical example, a cancelation survey. In product surveys where general feedback is sought or a NPS question is asked are best viewed as a pulse taking exercise rather than a true source of insight. But, when used appropriately, in-product surveys can be a source of identifying gaps in your product offering.
Ideal for: understand user motivations and actions just before / after using your product/feature – this can lead to uncovering new problem areas and segments that can be exploited.
Pitfalls: sampling bias – users that respond tend to be either extremely happy or extremely dissatisfied.
Complement it with: user analytics to overcome bias, support and sales interactions to check similar feedback.
User analytics comes from recording all the actions users take when using our products. Should be the daily dashboard of a product team to get a pulse of how the product is used – specially when you get to 1000s of daily active users. Before having that usage use it with caution.
Ideal for: identify friction points in the user journey, size the importance of product features
Pitfalls: over reliance on user analytics might lead to local optima by focusing on successful paths / journeys.
Complement it with: in-product surveys to understand motivations behind the observed behaviours; support tickets to see alternative paths users are taking.
When users reach out to your support team, it usually means they have encountered a major roadblock in their workflow. Ignore this source of feedback at your peril.
Ideal for: understand where the product experience is failing. Documentation gaps, bugs, experiential deficiencies etc.
Pitfalls: over reliance on this source of feedback focus the product team on the existing customer segment and blinds them to new opportunities to acquire different types of customers.
Complement it with: user analytics to size the problem and in-product feedback to understand what leads to edge cases.
Sales calls are a great way to listen to customers. In a B2B product that is focused on the Enterprise segment, it is an opportunity to listen customers express in their own words what challenges they are facing and what are they looking for.
Ideal for: identify what prospects value most and what challenges they are trying to overcome.
Pitfalls: not listening to the actual conversation and relying on Account Executives to recall what was said. Relying to much on this source leads to over investing on features that drive acquisition, at the expense of existing customer’s experience
Complement it with: in-product surveys to understand if new features complement the workflow of existing users.
Session recordings are an ideal way of gaining empathy for how users use the product.
Ideal for: find where users might be confused when navigating; looking for points where users are accomplishing tasks outside of the product
Pitfalls: similar to user analytics, over reliance on session recording can lead to local optima
Complement it with: in-product surveys to understand motivations behind the observed behaviours; support tickets to see alternative paths users are taking.
If you are entering a new market and problem area, it’s tougher. The focus at this stage should be understanding the market and formulate a proposition that will resonate with customers. Focus on understanding the circumstances potential users live in, what are the market dynamics. Do this without spending too much obsessing about fine grained details. Focus on launching as fast as possible to really go deep into user problems. I find desk research a powerful tool at this stage and I’ve used material from: analysts, professional organisations, regulatory agencies, etc.
If you have the budget to pay to recruit users or subject matter experts – this is the time to spend it.
Ideal for: understanding how the competition position themselves and general trends in a market
Pitfalls: confusing what the competition does for what users need…
Complement it with: blogs and user reviews to understand the users point of view.
Ideal for: understanding what your users talk about and what they are concerned about
Pitfalls: sampling bias – what people might be discussing in public is not necessarily what occupies their mind.
Complement it with: market reports to get the point of view of who is serving the market.
Ideal for: understanding what competitor’s users value and what they dislike.
Pitfalls: big sampling bias risk. Many reviews are pushed and controlled by the marketing departments of your competitor. The worse reviews are driven by disgruntled customers.
Complement it with: market reports
Wait … there is no users, how can we use “user analytics”? If you are launching a new product into a market chances are people are searching for solutions to their problem. Leverage SEO tools and search analytics tools to understand what people are searching for. Get a landing page up for the product and A/B test the proposition…
Ideal for: testing a proposition before developing a product
Pitfalls: like with any analytics you get little insight on why people are doing what they are doing.
Complement it with: dedicated blogs and podcasts to understand why people are searching what they are searching
It goes without saying that structured, live user interviews cannot be substituted and that as a complement to these sources it is invaluable. However, users’ time is an expensive resource, and we cannot afford to waste it when we can learn from other sources. Too much product development advice emphasises “talking to your users,” which can paralyse product development teams and make them feel disempowered. They may think that every single feature requires full problem discovery, solution validation, and so on. Yes, talk to your users, but only when you have a question that cannot be answered reliably in any other way.
Written by
