“I will never forget the shock of the first week, September 2012: our surveys had indicated big demand coming up (70 percent of 250 target market respondents had said they would buy the product)…People really liked the idea! Then we got knocked out of La-La land back into reality. On the first day, we had exactly 3 orders.”

Michael Bohanes’ account of his startup’s demise provides a cautionary tale about the gulf between receiving a positive response to a market research survey and the complex process of using customer insight to create a valuable user experience. Bohanes founded Dinnr to provide same day delivery of ingredients and pre-planned menus which could be cooked at home. However, with the benefit of hindsight, he identifies how the flaws in his initial research and his inability to establish an objective distance between himself and the product led to skewed answers.

“We were not solving anyone’s problem. I should have found that out in my initial market research…However, we committed the big mistake of presenting people with the idea and asking them if they liked it and would buy it. And when people said yes, we thought they meant ‘launch it and I will buy’.”

What actually happened will be familiar to many who have undertaken customer research – the respondents behaviour didn’t match their own predictions:

• Instead, prospective customers told Bohanes, the founder, what they thought he wanted to hear because he was obviously personally enthusiastic about the project.
• Furthermore, they were overly optimistic about their own future behaviour, leading them to provide answers based on how they hoped they would behave rather than how they actually behaved.
• And, of course, they did what almost all users do and tried to imagine how the product would be received by others rather than concentrating on their own response.

These 3 traits are common, particularly when your user research focuses on showing participants a concept and then asking them a series of questions about it.

What could he have done differently to obtain a more accurate picture of users’ behaviour and a more reliable way of connecting those behaviours into the right design decisions? There are a few simple principles which underpin useful customer research:

• Use multiple research methods to examine behaviour from different angles. Bohanes’ account suggests he focused on a single research method – a customer survey – conducted by a mix of questionnaire and a few one-to-one interviews. As a result, the answers he received were all subject to the natural biases detailed above. Instead, he could have diversified his research approaches to include methods like ethnographic observation – which might have shown him earlier on that there was a difference between users’ actual behaviour and their survey answers – and co-creation sessions, where you can explore users’ real preferences by challenging them to show you how they’ll interact with versions of an idea rather than just asking them how they think they’ll react.
• Hire an expert. By conducting much of the research himself, Bohanes may have thought he was prudently saving part of his limited budget, but in reality he was wasting time on gathering ineffective data. Investing in user research expertise would have helped him avoid basic mistakes like the ones he learned with the benefit of hindsight and, crucially, allowed him to separate himself from biasing the results, allowing him to question both the answers that were being received and the type of questions that were being asked. To be clear, he should not have been absent during the research, but he should have been an observer, not the conductor.
• Establish an ongoing cycle of user research. Bohanes, like so many startup founders, treated user research as a one-off bridge to cross in an essentially linear journey. It was something they did at the start to confirm an existing hypothesis and set their product requirements, which were then passed to an external team for development. Startups succeed when they are able to constantly monitor user behaviour and employ the resulting insights to drive an iterative development process. With this in mind, the larger the gap between those conducting the user research and those developing the product, the less likely it is the outcome will reflect users’ true requirements.

Michal Bohanes detailed account is worth reading in full, as it also covers lessons he learned about his own skills and attitude as founder and how he liaised with external design and development teams.