Measuring the user experience


For something to be improved, we must first be able to measure it. This is certainly true of user experience, but there are numerous challenges in trying to evaluate something so complex and varied. What kind of metrics can we use to measure the user experience of mobile services and devices? Is the ultimate arbiter of good experience the fickle and diverse opinon of an individual customer’s mind?

This is a question we’ve been looking at in some detail as part of the manifesto for our MEX conference and we’re very interested to hear about the different approaches the industry is taking.

Point 5 of the manifesto reads: “User experience performance must be measured if it is to be improved. It must be constantly tracked through quantitative and qualitative methods. We think organisations throughout the value chain are failing to recognise the importance of understanding customers because quantifying the return on investment is too difficult.”

One idea we had was to compare handsets with similar feature sets against usage data for particular applications. For instance, we could identify all the handsets with a 2 megapixel camera and then measure which handsets generated the most MMS or web album photo uploads.

Another possibility we considered was awarding a score to a handset for the number of features it was capable of performing and plotting this against how many of them were actually used. A device may have 10 features but if only 3 were being used it would achieve a low score. In contrast, a handset with only 5 features but where all were used would produce a high score. The higher scores would tend to imply a more useable handset and therefore a better overall experience.

However, these approaches are still flawed and simplistic. In the latter example there is no accounting for the myriad other factors which comprise a good experience. A user may be quite content utilising only a few of their handset’s capabilities if they purchased the device simply because they felt it was fashionable or they had received good customer service from the retailer. How can we measure these opinions? After all, they may play an important role in determing whether or not a user becomes a repeat customer or recommends the product to their friends.

There are numerous techniques and a wide range of consultancies who can conduct this kind of research. Some use laboratory observations while others track users interacting with products in their natural environment. Results can be derived from third party observation or obtained directly from the users by asking them to complete surveys. Others can be sourced by tracking individual actions like keystrokes or download times.

None of these approaches, however, come close to capturing the complexity of a customer’s overall feeling when they use a device. In an attempt to better understand these emotions, some companies are now employing researchers who trawl web forums collecting opinions posted about products and tracking the ‘noise’ generated by online conversations.

One of the biggest challenges is eliminating the filters which distort feedback. These range from the ability of an observer to perceive how a user is really behaving to the time delay between a user completing an action and then reporting to a researcher how they felt while doing it. One of the key objectives in user experience analysis, therefore, is getting access to unfiltered usage data.

A Dutch group has developed a system known as SocioXensor which attempts to address this. It comprises software installed directly on the mobile device and a server comp0nent. This allows researchers to send survey questions to users in their natural environment and in response to particular actions. For instance, a researcher could program the system to send a short survey to a user immediately after they’d finished sending a text message or making a voice call, asking them how they felt or what could have been improved.

SocioXensor also collects data on other aspects of the user experience. In addition to opinions from the users, it is also able to track objective information such as their current location, their proximity to other devices and raw data from the applications themselves – e.g. keystrokes and events. The result is a comprehensive picture of a user’s feelings, activity and context.

Crucially, it also gets as close as possible to eliminating the filters of third party perception or the inaccuracy caused by the delay of recall.

You can read more about SocioXensor here.

When we look at our manifesto statement, however, it is evident that even this level of access to a user’s emotions and behaviour may be insufficient. It is obvious that for user experience to improve it needs to be measured, but this must be in a way which translates easily into figures which can demonstrate the return on investment. If an organisation can’t prove the money it is spending on understanding users is translating into better results, user experience budgets will never increase.

We believe several quantitative and qualitative methods need to be combined to provide an accurate picture of user experience throughout the product lifecycle. To prompt a debate on this topic, we’re suggesting the following approach. We don’t necessarily thinks this answers all of the questions, but hopefully it provides a starting point for further discussion.

– Start by looking at how users behave when they’re using other products or techniques to achieve the result targeted by your product. If you’re developing a mobile news reader, this could mean observing people consuming information from traditional newspapers, accessing news web-sites, watching TV news and – obviously – any competing mobile news applications. Track as broad a group of users as possible, recording both subjective commentary (how are they feeling, why do they need the news at that moment etc…) and objective data (how often do they do it, how long does it take them to achieve the result they’re looking for, how quickly do they move between different stories etc…).

– Involve your target users throughout the design process, from concepts to prototypes and testing. Establish an evaluation framework which allows you to feed a mixture of objective and subjective information back into the development work.

– Make time for live, commercial, public testing of the product and establish clear targets for results. Benchmark these against previous development projects. In the most basic terms, you can demonstrate a return on investment if the amount you’ve spent on understanding users is outweighed by the additional revenues generated in comparison to previous projects where less was spent on user experience research.

– Continue this benchmarking throughout the full commercial deployment and the ongoing versioning process.

– Establish tests which enable you to calculate other important aspects of return on investment: how many customers are recommending your product to friends, thereby saving you money on marketing. Are you support costs lowered because fewer customers are struggling to use the product?

Please feel free to post any ideas and feedback using the comment links below. If you’d like to take part in the debate at the MEX conference on 2nd/3rd May 2007, you can register on-line by clicking here – register before 12th January and benefit from a 20% discount.


2 Comments

Add yours
  1. 1
    A.J. Bayne

    I have invented a mobile phone (pat. pending) that presents advertising to persons STANDING NEAR a mobile phone user when he speaks on the phone in a non-handsfree mode (screen is positionable to face outward). While the advertisement is played on the outward facing screen a digital coupon related to the advertisement may be transmitted via Bluetooth to handsets nearby. The coupons redemption provide a measurement of the effectiveness of the advertisement. In this way the advertiser reaches more persons per presentation,and are able to present for a longer period of time (i.e. the length of the call)
    – A.J.Bayne

  2. 2
    Paula Thornton

    There’s a much longer answer here, but your intents are great and an indicator as you described might be a nice ‘calling card’ (it’s what people think they want to see), but in the end, it’s more about Evidence-Based Design. This requires that all reasearch (which includes incidental research data, like evidences from the feedback loops) be treated as a corporate asset — synthesized and managed. A new dimension of activites — Design Research — are facilitated by a Design Research Steward, who collects and makes evidences available via a shared corporate space (ala. a Design Research portal). This unique individual also watches for patterns and items of interest across initiatives and regularly reports on unique findings (ala. a blog). Other items would include a variety of ‘metrics’ (rates of change, behavioral comparitives, etc.). Design Research will be a core competency for businesses that value a competitive advantage.

+ Leave a Comment