How do we enhance mobile customer research methodology?

Ahead of the MEX 2009 conference, I want to dedicate this issue of the newsletter to exploring MEX Manifesto Statement #3, entitled ‘Customer research methodology must be enhanced to close the reality gap’.

The objectives of this article are two-fold:

2) Second, I want to reach out to the MEX community and ask you to contribute some of your own ideas by posting a comment to the MEX blog (below). I’d like to hear about any and all approaches, however experimental. Also very keen to hear some quantifiable success stories of how this research has translated into commercial benefit. I’m hoping this will give the mobile UX community an opportunity to expand its collective knowledge in this area.

The full MEX Manifesto statement reads: “We believe…the industry must enhance its methodology for understanding customer experience and translating that knowledge into better mobile products. The continuing disconnect between the lifestyle of real customers and the experiences they’re offered points to an urgent need for new research methods and new ways of using that research within the product management structure.”

There’s more background to this, some Stat Spots and questions to get you thinking on the official MEX Manifesto page.

So what’s the issue here? What’s wrong with existing methods?

In talking to people throughout the industry, it seems the problems fall into 4 major categories:

1) Unique mobile conditions

It is difficult to adapt existing methods to the unique challenges of the mobile environment. Replicating and tracking someone’s natural behaviour on a PC is quite easy in the lab - they are sitting at a desk or on a sofa, just as they would be normally. This enables methods like eyeball tracking, filming hand movements etc… While it is possible to do something similar with mobiles, you come up against the fundamental issue that users are performing the tasks in a lab environment very different to the real world in which mobile devices are typically used.

2) Cost

Few companies have the resources to do user research on an ongoing basis, so if it is included in a budget at all, it is as a one-off event at the start of the development process. Time and again user experience practitioners tell us the most valuable user insights often come in the middle of the project and by maintaining a constant feedback loop, but budgets are often incompatible or insufficient for this.

3) Capitalising on insight

I can’t count the number of times I’ve heard this story: the research team spends months on a study and gathers a huge range of information about users. They put together a presentation for the big project ‘debrief’ with the executive team and a week later most of the insights have been forgotten or ignored because it isn’t compatible with the development roadmap.

4) Sharing and duplication

Looking at some of the products which emerge from the mobile industry, it would be easy to imagine very little user research is conducted. However, this simply isn’t the case. Handset manufacturers, network operators and platform providers invest considerable time and money in both internal and external studies. Unfortunately, very little of this ever leaves the companies’ offices, even when it could be of considerable benefit to their partner networks. We all know how dependent companies within the value chain are on each other, yet often organisations working on the same project don’t share user research data with each other - the result is missed insight opportunities, costly duplication and a lack of collaborative understanding.

None of these problems are insurmountable. Indeed, the answers are often quite apparent once you start asking the right questions within your organisation. The most important thing is to make sure you are asking those questions at all, asking them early in the project and then ensuring they don’t have to re-learnt every time you start a new development.

Here are a few ideas which might be useful additions for enhancing your user research toolkit:

1) Abstract simulation

Rachel Hinman of Adaptive Path (who will be speaking in response to this Manifesto statement at MEX), told me about an interesting experiment she was involved in recently. The objective was to get designers thinking about user reactions to handset form factors in a very different way. To achieve this, they asked participants to simulate mobile phone interactions using a variety of abstract objects, including things like silk flowers. I’d definitely classify this as sitting at the extremes of user research, but it struck me as an effective way of shocking designers into forgetting their pre-conceptions and thinking about form factors, materials and human interaction in a new light.

2) Time perception contrast

‘Time to complete’ metrics, where an interviewer measures the time it takes for a user to achieve a set objective, are a standard part of usability testing. However, they only tell half the story. There is often a difference between the actual time taken and the user’s perception of that time. For instance, ‘latent’ periods - when the device or service seems unresponsive and isn’t providing any user feedback - are often perceived as lasting much longer than they actually do. Next time you’re planning a time-to-complete study, try comparing actual completion times against the user’s subjective estimates of each action they’re asked to perform.

3) Big data for the little guy

When you’re 4 people working out of the boss’ house, buying in global research on usage trends from the likes of ComScore, Nielsen and Mediacells may seem way out of your reach. However, all of these companies frequently share useful subsets of their data in the public domain, so as a first point of call, make sure you’re on their announcement lists. There is also a growing trend for companies with large mobile internet user bases to share a considerable amount of information on usage trends in free, regular reports. Mobile advertising network Admob publishes its Metrics report most months, summarising usage trends across around 6000 mobile publishers in 160 countries. Opera, the mobile browser company, publishes ‘State of the Mobile Web‘ on a monthly basis, based on the behaviour of the 20m customers using Opera Mini every month. Crisp Wireless has a quarterly report (free, but reg. required) on usage trends from 200 premium web-sites, mainly based in the US.

4) On-device behaviour tracking, for free!

When I first heard about Flurry I had difficulty believing they were really offering the service for free. Quite simply, the company provides a remote tracking technology which allows any developer to track a huge range of user actions within their applications by embedding a few code snippets. For instance, you could use the platform to keep track of the typical sequence of actions a user goes through before they decide to upgrade from the free trial version of your iPhone application to the premium edition. Flurry started life as the developer of a popular Java-based email client and created this technology to enable them to iterate improvements to their application more quickly. They’ve now changed their business model and are offering this analytics technology, currently free of charge, to developers of iPhone, Android and Java apps.

5) Event level digital diaries

Another MEX speaker shared one of their personal user research methods with me the other day. They often use a hybrid approach linking quantitive analytics to qualitative digital diary entries. This approach is not uncommon in user research, but their particular technique ensures they can link specific user events with how the user was feeling and why they took a particular approach.

6) Executive-level immersion

This relates specifically to the 3rd problem identified in the overview above - capitalising on insight. Delivering a report on user behaviour to your executive team, no matter how great your writing and presenting skills, makes you a ‘middleman’. You’re effectively making yourself an agent acting on behalf of the user and the entire tone of the engagement with your executive team is coloured by having to ’sell’ your ideas. Charlie Dawson, partner at strategic consultancy The Foundation, spoke at the first MEX conference on the importance of ‘Immersion’ in the user research process. Citing his experience of working at board level with Foundation clients like O2, Marks &amp Spencer, Virgin Mobile and Volkswagen, Charlie recounted how the most effective and long-lasting commitments to user-centred thinking came after they involved the executive teams directly in the research process. By immersing the strategic change makers in the reality of their user’s worlds, the research teams were able to show executives the importance of customer-centred design at first hand.

Now it’s your turn.

Please add any suggestions you have for enhancing mobile user research methodology by commenting on this article on the MEX blog (below). I encourage you to forward this around to as many of your colleagues as possible so they can submit their ideas too - let’s see if we can get a really valuable list going.

I’m looking forward to hearing Rachel Hinman’s response to this Manifesto statement when she keynotes on the topic at MEX on 19th - 20th May 2009 in London. True to the spirit of MEX, we’ll be using Rachel’s presentation as inspiration for a number of practical breakouts, where everyone in attendance will have the opportunity to participate in responding to the Manifesto and enhancing their collective knowledge.

Hope you’ll join us for the conference. The event sells out in advance every year, so please register early to guarantee your place.

Read related articles

March 20, 2009. Posted by Marek Pawlowski in the 'Opinions, ideas and new thinking' category. 4 comments - join the debate below.

4 Comments - join the debate

Marek Pawlowski replied:

Help build a list of research methods for the mobile user experience community. Please post your ideas, success stories and wishlists as comments on this article.

March 20th, 2009 at 12:43 pm. Permalink.
Mariana Oliveira replied:

It’s not easy…

I’m here more to learn than teachand I must say the post showed some great insights.

Regards

March 20th, 2009 at 8:57 pm. Permalink.
Tamara Gruber replied:

Thanks for including the Crisp Wireless Index in your list of resources. Each quarter we look at mobile web browsing and mobile search trends across our Publisher Network, representing over 200 premium sites that attract over 30 million unique visitors per month. A new report will be out soon and if you register beforehand, it will be sent to you automatically. This report will look at trends throughout 2008. http://www.crispwireless.com/analytics/register

March 22nd, 2009 at 6:00 pm. Permalink.
arona replied:

I am enjoying your article. It’s quite informative.

March 24th, 2009 at 9:15 am. Permalink.