Testing assumptions before you build
Every product idea rests on a stack of beliefs: that people have the problem, that they will notice the solution, that they will understand it, that it can be built in reasonable time, that it makes sense for the business. Most of these beliefs are never checked. Testing the riskiest ones before building is the last and most practical stage of discovery.
Four kinds of risk
Marty Cagan describes four risks every idea carries, and they make a good checklist:
- Value: will people want it and choose to use it?
- Usability: will they be able to figure out how?
- Feasibility: can we build it with the time, skills and technology we have?
- Viability: does it work for the business — legally, financially, operationally?
Teams tend to over-test usability, because prototypes are fun, and under-test value, because it is uncomfortable to find out that nobody cares.
Listing the assumptions
The simplest way to surface assumptions is to walk through the idea step by step, as the customer would experience it, and ask at each step what has to be true. For an automatic reminder sent to clients with overdue invoices:
- Users want reminders to go out without checking each one first.
- Clients respond to reminders that come from the invoicing system rather than from a person.
- Users will find and switch on the setting.
- Reminder emails are delivered and not treated as spam.
- Automatic reminders do not damage the user’s relationship with their client.
Doing this with the whole trio produces a longer and better list. Engineers spot feasibility risks that nobody else would, and designers notice where people will get lost.
Choosing what to test
You will not test everything, so put each assumption on two scales: how important it is to the idea’s success, and how much evidence you already have. Assumptions that are both important and poorly supported go first. In the example, the fifth one — whether automation harms the relationship — might sink the whole idea, and so far it is pure guesswork.
Cheap ways to test
The goal is the smallest test that would change your mind. Some options, roughly from cheapest:
- Existing data. Sometimes the answer is already in the logs or support history. Check before running anything new.
- A one-question survey shown in the product, about a recent specific behaviour.
- A prototype test. Clickable mock-ups put in front of five or six people reveal most usability problems.
- A fake door. A button or menu item for a feature that does not exist yet, measuring how many people try it. Tell the people who click that it is coming and thank them; leaving them confused is poor form.
- Doing it by hand. Send the reminders manually for a handful of customers and watch what happens. Sometimes called a concierge test.
- A technical spike. A day or two of throwaway code to see whether the hard part is actually hard.
Decide what success looks like first
Before running a test, write down the result that would make you continue and the result that would make you stop. “At least four of six participants complete the task without help.” “At least 3% of users who see the menu item click it within two weeks.”
The thresholds will be somewhat arbitrary, and that is fine. Without them, any result can be read as encouraging, and almost every result will be.
Acting on the results
Each test ends in one of three places: the assumption holds and you move on, the idea needs changing, or the idea is dropped. Dropping is a success, not a failure: you have just avoided building something that would not have worked, at a fraction of the cost.
When enough of the important assumptions have held up, the idea is ready to leave discovery. What happens next is the subject of Product delivery, end to end.