It reflects two customer trends: firstly, the ability to research purchases on a mobile device is making consumers more informed and confident enough to buy relatively high value items without speaking to a salesperson. Are digital accessories the convenience snacks of our increasingly virtual lives?

Secondly, there is an expectation of immediacy at all times. The notion of store ‘opening hours’ seems quaint and dated when supermarkets are open 24 hours a day, orders can be placed online in the middle of the night and vending machines sell you consumer electronics at the touch of a button. When was the last time you waited on the advice of a local merchant to buy a digital product? Indeed, how often do you actively avoid physical stores because you know the service is unlikely to meet your expectations?

This vending machine is located beside the ATMs on the way to platforms 15 – 19 at London’s Victoria Station (ironically, many years ago, this area used to be dedicated to BT’s fixed line payphones). It sells basic mobile phones, mobile broadband USB sticks, memory cards and a selection of headphones. Note that a pair of Apple iPhone headphones costs GBP 30, whereas an entire Samsung mobile phone costs GBP 20.

It is sponsored by Sandisk, the memory card manufacturer, and features two digital displays: one at the top (displaying an advertisement for mobile broadband in the photo above) and one above the keypad.

A selection of low end handsets:

iPhone accessories:

Keypad detail:

Here’s another example, this time embedded in the frontage of a jewellery store at the Westfield London Stratford shopping centre, located adjacent to the Food Court. It is worth noting the Food Court at Westfield remains open after the retail stores have closed. Hence the poster proclaims: “Want it, buy it, wear it…24/7 vending machine.”

Unfortunately, however, the piece of white A4 paper taped to the screen of the machine reads: “Vending machine not in use. please see in store for details. sorry for any inconvenience caused.” [The mis-capitalisation is their own].

It seems the process of explaining failure to the customer is still very much an analogue task, whether it be on paper or in store (which was also closed when I took the photo). Disappointment and mistrust seem to be amplified when the failure of something digital is flagged so untidily with paper.

This is the first in an occasional series sharing how MEX use observation and immersion to understand user experience. Some observations shape our consulting work, some simply confirm existing hypotheses and others go on to become major themes we explore through MEX Pathways at our events.

The way in which design responds to context, however, lags this new availability of contextual data points. This is due partly to the relative complexity of creating contextually responsive digital experiences and partly to a misconception among designers that users are willing to change their context in order to gain access to particular products. In reality, such life altering products are few and far between. Most experiences, and particularly those distributed across multiple digital touchpoints, can be made better by embracing contextually responsive principles.

Responsive design is understood today as the ability for digital services to adapt their look, and sometimes their content, to match the display capabilities of the user’s access device. Simple as that may sound, the best examples remain remarkably complex to achieve, but should also be recognised as merely a first step towards much broader potential for adapting design in response to context.

Display capability is just one factor which might influence the way a service manifests to users, but there are myriad others:

• Location
• Light
• Proximity
• Time
• Motion
• Sound
• Temperature
• Tools
• Connectivity

The usefulness of each element is determined by how well a designer translates sensor data into a response meaningful to the user. For instance, a string of GPS co-ordinates accurately reveals where a user is on the surface of the earth, but nothing about what that location means to the user.

However, other data could be combined with GPS co-ordinates to make a contextual assumption about the user’s needs. The service could pull a location-based picture count from Flickr and, if the user was in a place where a particularly high number of photographs had been taken by others, it would be reasonable to assume they are standing close to a sight-seeing attraction. The service could, therefore, prioritise the camera app icon over others, adding it to the lockscreen or making it appear larger than other icons in the menu.

Another obvious example concerns light. Most navigation services offer at least two colour schemes, one suited for daylight and another with more muted tones for the night. Few currently switch automatically between these modes, but there are numerous contextual clues which could be used to determine if it was appropriate to make an automatic switch.

Many devices are now equipped with embedded light sensors; however, data exists to help even those which aren’t to make the same contextual judgements. It would be a simple matter to pull sunrise and sunset times from a web API or, as devices gain closer integration with cars, to activate night mode on the device at the same time as it is activated on the car dashboard.

The complexity of responding to context increases with the number of data points used to inform contextual decisions. It is complex enough to achieve a success within a single dimension, such as ensuring a service looks visually correct within the display limitations of a user’s device. To achieve success across multiple dimensions of context is, using today’s methods, orders of magnitude more difficult. It also carries the risk of trying too hard and ending up with an experience where the user feels the design is doing too much to ‘second guess’ their needs.

However, the rewards make it worth trying. Contextual response improves the usability of existing services, but also holds the possibility of developing entirely new experiences.

Achieving success is more likely when designers take a modular approach to creating service elements. The practical steps required vary considerably depending on the type of service; they are as diverse as the huge range of service experiences enabled by mobile technology. With no set formula or process, how deeply a ‘modular mindset’ is embedded in the development team becomes more important than following a prescriptive framework.

Content, interactions, business logic and any other elements which comprise a service should be made accessible in their most basic forms. By making each element distinct, designers gain flexibility to adjust the way components work together to respond to certain contexts. Not all of these need be used today, but there is an opportunity to ‘future proof’ against further improvements in contextual understanding by taking modularity to an extreme. When a new contextual data point becomes available, the service architecture will be able to respond.

New dimensions to contextual understanding continue to emerge and the pace at which they become available will only increase. However, the first rule of contextual response will never change: the ease with which the user can correct the mistakes of the ‘guessing engine’ will always be as important as the quality of the engine itself. No matter how detailed the data, no matter how many dimensions of context it spans and no matter how regular the user’s behaviour, it is impossible to get it right every time.

Unless the user has a simple way to correct these mistakes – and feels the engine is learning when they do so – they will quickly disable any contextual capabilities, or abandon the service entirely. You need only look to traditional, person-to-person customer service scenarios to understand why: it is frustrating when users feel the customer service representative doesn’t understand them, but even more infuriating when they don’t listen to and remember the user’s corrections.

The predictive text and auto-correct engines essential to typing on cramped touchscreens provide an example. The majority of users report higher satisfaction with engines that are less accurate, but easy to manually correct, than engines with higher accuracy but which make manual intervention more difficult.

The need to feel in control of manual intervention closely mirrors the way users feel about privacy. The benefits of contextually responsive services must be clear and significant to overcome the wariness users feel about sharing the data which enable contextual experiences. On balance, most prefer not to try at all if the potential benefit is marginal. This kind of irrational fear (and the majority of users freely admit it is irrational) is the most immediate stumbling block to widespread adoption.

By clearing explaining the benefits and pre-empting concerns over risk, users can be persuaded to try new things. However, ‘contextual response’ as a concept means little to a customer. It is more effectively presented as an integrated part of an overall service offering. In the visual example above, where the colours used by a navigation service would change automatically, a simple and direct marketing story about ‘Night Mode’ would be more effective than, for instance, asking users “Would you like to share location and light sensor data with this app?”

The starting point for successful contextual response is a mindset which prioritises modularity in all aspects of the service architecture. Subsequent results depend on how effectively user research is used to determine which service elements can be improved through contextual response. And, ultimately, ensuring the service learns from its inevitable mistakes and the user remains in control throughout will ensure good experience.

‘Context’ is in development as a MEX Pathway and will be explored extensively in the coming months, including at the next MEX event. I’d love to hear from those working in this area – please get in touch to share ideas.

The implications for designers are significant. The majority of digital interface design, from PC to mobile phone, is built on the assumption of a single user at any one time. Many interface conventions break as this expands beyond two or more users trying to interact with the same device at the same time.

MEX first identified this trend two years ago from user research observations undertaken after the launch of the first iPad. Almost as soon as the iPad appeared in the shops, multiple users could be seen gathered around single devices in cafes and on trains. It subsequently became MEX Pathway #3 at the December 2010 MEX, where Kate Ho of Interface3 led the working team on this subject and delivered an inspirational speaking session (video below):

It was also one of themes of my November 2010 article in User Experience Magazine, entitled ‘Mobile UX trends on the horizon‘.

I was reminded of its prescience while chairing MobileMonday London’s 19th March panel discussion on tablets. It came up in several questions from the audience and points made by the panel, both in the context of simultaneous usage and several ‘owners’ asynchronously sharing a single tablet within a household or office.

Asynchronous usage is a relatively simple concept to design for: it is easy to imagine tablets supporting multiple user profiles, ensuring the device presents each individual with their own set of apps, preferences and content. Indeed, a similar concept – Digitag – from designer Aaron Rustill was voted MEX Innovator of the Year back in 2009, and subsequently featured on TelecomTV.

However, simultaneous multi-person user interfaces (SMUIs), present a more complex design challenge.

There are physical considerations: how, for instance, should an interface adapt depending on whether users are sitting side-by-side holding a device between them versus laying the device on a table while sitting opposite each other?

Interaction conventions must be re-thought too. Which user should have priority if there is a conflict between the virtual interaction elements they’re using on-screen? Is it possible to identify which interactions are driven by which user and, if so, how should that be recognised visually within the UI design?

The exploration of MEX Pathway #3 at the December 2010 MEX generated a set of 9 insights into the implications of SMUIs:

1. Simultaneous multi-person user interfaces (SMUIs) compliment personal UIs. Platform design should recognise that the same device may be used in both simultaneous multi-person and individual user scenarios. Products should be able to switch easily between these modes.
2. SMUIs rely on parallel rather than serial interface metaphors. Events and actions should be designed to happen simultaneously.
3. The concept of a user interface having an absolute top, bottom, left and right has less relevance in SMUIs. Actions must orientate themselves according to multiple users, who may be physically located on opposite sides of the device.
4. SMUIs bring new meaning to our understanding of social computing, by allowing a shared digital experience to also be a shared physical experience.
5. The number of sides and corners on a device imposes a proportional physical limitation on the number of simultaneous users.
6. Touch is currently the preferred input method for SMUIs, with the possibility gestures may emerge in the future. The fixed nature of physical controls makes them unsuitable for SMUIs.
7. SMUIs can exist outside single devices. Larger SMUIs can be created by physically-linking several devices and manufacturers should consider this modularity in future form factors.
8. SMUIs are best suited to activities such as content manipulation and organisation. Creation and research activities continues to favour personal interfaces.
9. SMUIs should make it easy for participating users to contribute content from their personal devices and take away the results of their shared interaction.

Amy Godsell, the MEX illustrator-in-residence for the December 2010 event, also produced this visual summary of the key themes emerging form the Pathway (click for the full size image):

This Pathway is likely to return at a future MEX and I’d love to hear ideas or see concepts from anyone working in this area. Please get in touch.

Barnes & Noble Nook Tablet (by MobileTechReview)

Amazon Kindle Fire (by MobileTechReview)

Barnes and Noble Nook Touch (by The Verge)

Samsung Galaxy Tab 7 Plus (by MobileTechReview)

You might also be interested in my review of HP’s ill-fated TouchPad and WebOS.

By James Whatley

To give this some background, about a year ago I gave a talk at MEX entitled ‘Inspiring new forms of creative expression through mobile devices’ (you can watch it here). I blogged about it shortly afterwards and I’ve since gone on to mention the work on other sites too; it was a great day of learning.

The point is, the way we interface with our devices (and the environment around us) is a big thing to me and it’s something that keeps me curious day in, day out. From TED talks with Jan Chipchase, to the social object theory of Jyri Engestrom – mobile is key.

That, and I’m a massive comic book geek.

Using that statement as a jumping point, this kind of technological advance is of course hugely exciting. Of course it is.

However, with my mobile and ‘tech’ head on, I can’t help but just see this is a fun feature or a nice-to-have (as opposed to a must have). The video above doesn’t really do much to sell it to be honest. My feeling is: ‘Well, that’s cool!’ but after that, why I would do it again? I wouldn’t…

But, as the accompanying article suggests: “Users can watch video trailers of books they see in stores, as well as 3D animation, recaps, and other augmented reality extras by holding their phones up to comics.”

Trailers? From books?! Okay, that’s awesome (as long as the comic book store owner doesn’t mind packs of smartphone users gathering around, waving their phones across the stands – let’s remember the practicalities). Marvel is projecting towards the idea of building “DVD and Blu-Ray extras into the comic book itself.” Again, awesome.

How do I see this playing out? Well, let me give you a real world use case.

My favourite penciller is a chap named Adam Kubert (above). If I was able to buy one of his books that enabled me to see either an interview with him about his process or, better yet, a short video of how he drew the piece I’m looking at – simply by waving my device over it – then suddenly that becomes a much easier sell. Would I pay an extra pound for the ‘augmented’ version? You know what, I think I would. But, as ever, it would depend on the features.

Finally, the first thought that sprung to mind when I saw that 3D Iron Man strutting his stuff in front of the comic book was ‘Okay, cool – but how could it extend the story?’.

We live in a transmedia world; cinematic blockbusters are played out over web, mobile, books (and comics) and film. With the threads of these stories splaying out across these different platforms, how long will it be until we get to fully integrated books with AR-only, story-propelling components? Imagine one comic book panel building up a huge superhero battle, the next merely showing the aftermath but, when you open your Marvel AR (mARvel?) app, suddenly the whole thing springs to life: a three minute, fully animated story especially for mobile users?

That would be immense. And, perhaps, the next evolution in the way we continually strive to connect the digital with the tangible.

Well done Marvel. I applaud you.

James Whatley blogs at whatleydude.com and is @whatleydude on Twitter.

The original MEX Pathway #8 on new forms of creative expression can be found here. Also, the recent MEX article entitled ‘Hockney, iPad and creative expression‘ contains additional thoughts on this subject and a summary of the insights which emerged from May 2011 MEX working sessions on this Pathway. Jason DaPonte also contributed valuable ideas on the notion of how mobile technology might change storytelling during his November 2010 MEX session – watch the video here.

It is remarkable that, just two years after the iPad’s introduction, it has become an important medium for an artist such as Hockney and one capable of drawing new crowds to the venerable institution of the Royal Academy. The response to the exhibit has been extraordinary. It has been busy every day since opening in January, such that the gallery has extended its hours in response to demand, staying open until midnight on Fridays and Saturdays.

It is attracting visitors who rarely frequent galleries. When I attended it was almost uncomfortably full with young and old, Londoners and tourists alike, from all walks of life.

The body of work itself is magnificent. Its central theme is the detail of changing seasons in rural England. Hockney captures a country rarely seen by tourists self-confined to the bounds of central London. The locals who inhabit the woodlands, lanes and fields of Hockney’s pictures will find familiar scenes cast in a new light.

I hope the prominence of the exhibition will encourage exploration of England’s many rural regions, particularly among those who visit from abroad and previously returned home convinced the stone heart of London was representative of the country.

The iPad pictures are one of several mediums which reflect the breadth of Hockney’s talent. They sit alongside oils, charcoal, photographic collage, video and sketchbooks filled with pen and ink.

Hockney has a history of experimenting with new tools, but his use of the iPad deserves special attention. It is unusual for new technology to be so readily embraced by a 75 year old and speaks to the universal appeal of the device.

The iPad works are arranged in chronological order, enabling visitors to chart the artist’s changing use of the device. It is a prolific collection, averaging one picture a day over a couple of months. The experimentation with techniques is clear to see. In some Hockney builds up colours and shadow with many individual line strokes, in others he uses touches of a digital airbrush to create clouds. Some rely heavily on colours blended with a finger tip.

Most remarkable of all, however, is the scale. The pictures are huge, yet printed at such fine resolution that even close-up it is hard to discern they are made of pixels rather than pigments. To achieve this, Hockney transposed his established technique of working on canvases in sections. Each overall image was created by zooming and scrolling around a large virtual canvas, allowing the artist to construct each part of the grid in fine detail, so that it could be printed at size without pixelation.

Hockney’s comfort with working in sections while maintaining a sense of the overall image makes him an ideal candidate to experiment with the iPad. It will be interesting to see whether the increased resolution of Apple’s new version impacts his future work.

The use of the iPad is central to the wide appeal of this exhibition. It has attracted the interest of those who might have been unmoved without this novelty. Most importantly, however, it has inspired a sense of possibility.

As I left, I overhead a mum asking her son, “Are you going to have a go with the iPad on the way home?”

It seemed to sum up the factors which combine to make the iPad a popular canvas for creative experimentation, be it music, drawing or photography. The mobility of the iPad would allow it to be used on the train ride home, its accessibility meant it would be easily mastered by a child and its sociability would keep the family engaged during their journey.

We initiated a MEX Pathway, #8, entitled ‘Inspire new forms of creative expression through mobile devices‘ in February 2011 and the iPad has been an important enabler of this theme. There was a noticeable trend towards supporting creative experimentation as a use case for tablets at Mobile World Congress this year, with Adobe releasing new tools for both iOS and Android, and Samsung featuring a sketch artist on its stand.

Hockney’s work, despite its many virtues, is not a new form of expression, but rather a successful example of transposing existing techniques into the digital sphere.

The MEX Pathway #8 working sessions in May 2011 arrived at a number of conclusions about the future of mobile devices as creative tools. Here’s an extract from the Pathway summary:

1. The embedded person-to-person communication capabilities of wireless devices make them ideal tools for new forms of creativity reliant on shared interactions between multiple people.
2. The proliferation of wireless devices is still being driven by their primary role as communications tools, but this is enabling new digital creative capabilities to filter into the mass market by stealth. Examples include capacitive touchscreens for digital painting, processors for music creation and cameras for digital photography and video. Users can experience new forms of creativity first through the generic capabilities of their mobile devices and are then more likely to buy specialist tools if they decide to pursue an activity further.
3. New sensors are facilitating new forms of creative expression, including ambient light detectors, dual microphones, gyroscopes, digital compasses, GPS, thermometers and capactive touchscreens and other surfaces. Once included in a device, developers often find additional uses for sensors beyond the manufacturer’s original intentions.
4. Mobile devices allow creative activities to happen closer to the user’s source of inspiration, simply by virtue of the fact they are almost always nearby. Users can create a digital painting when they come across a beautiful landscape or record their dreams when they wake up because their mobile device is always within reach.
5. Creativity is innate. Interfaces which feel most natural and similar to the physical interactions we already have with other humans are most likely to facilitate creative experiences on digital devices. Haptics adds tactility, an essential element for natural interfaces.
6. New forms of creativity can bring users closer to the old and familiar. For instance, input from motion sensors could be used to ‘age’ digital photographs in the same way a favourite photo in a wallet would develop a patina over time.
7. The potential for new forms of creativity is limited by people’s tendency to categorise experiences according to their stage in life. Douglas Adams said: “Everything that’s already in the world when you’re born is just normal; anything that gets invented between then and before you turn thirty is incredibly exciting and creative and with any luck you can make a career out of it; anything that gets invented after you’re thirty is against the natural order of things and the beginning of the end of civilisation as we know it, until it’s been around for about ten years, when it gradually turns out to be alright really.”
8. Creativity is deeply personal. 10 people in the same room, equipped with the same tools and asked to draw the same thing will all produce different results.
9. If you’re trying to encourage users to experiment with new forms of creativity, remember three rules: make it easy to find, simple to use and fun.

‘David Hockney RA: A Bigger Picture‘ is at the Royal Academy until 9th April 2012. A word of advice: if you’re planning to visit (and I thoroughly recommend it), allow plenty of time to queue, or if you’re lucky enough to know a Royal Academy member, ask them to take you as a guest: you’ll bypass the queues and their membership allows you and them to attend without additional charge.

Measures of technical prowess dominated Mobile World Congress. The event’s vocabulary was characterised by quad core 1.5 Ghz processors, 41 megapixel cameras, 720p displays and LTE network speeds, all resulting from intense competitive pressure at the high end of the device market.

Users, however, will struggle to see the difference.

My lasting impression of several days immersed in the industry’s most advanced technology is of a sea of generic black slates, each with slight differences at the component level, but ultimately just variations on a theme unchanged for several years: imitating the iPhone and iPad.

Apple, meanwhile, are poised to release a new template tomorrow.

Some at the event put forward the argument that differentiation has shifted into the virtual sphere and that users are satisfied simply with faster, cheaper glowing rectangles through which to consume services. I don’t subscribe to this line of thinking. The body of evidence from user interviews and observations suggests people are still driven by physical form and, crucially, how component and software innovation is used to create integrated, meaningful features.

A longer specification list does not in itself correlate to better user experience. The need to reiterate this in 2012 suggests, for all the lip service the industry pays to the important of user-centred design, much work remains to be done within the MEX initiative to espouse better practice more closely aligned with the reality of users’ lives.

I spent my time in Barcelona looking for examples of this approach and, amid the noise of technical cliches, interesting threads emerged.

Damian Dinning of Nokia showed me how, if you look beyond the 41 megapixel headline, the PureView 808 has actually been designed to enable new types of close-up photography through a combination of technical innovation and new interaction sequences. It also comes in a gloriously bold red colour, part of a palette which seems to speak of a new confidence at the Finnish manufacturer.

Emporia revealed how their ongoing commitment to refining products through multi-layered user research, in partnership with Cambridge University, is making mobile services accessible to those previously excluded by technical complexity.

There was a noticeable growth in the intended use case of tablets, expanding away from pure consumption activities to embrace more creative endeavours. The re-emergence of the stylus as a credible accessory provided evidence of this, as did new applications from the likes of Adobe. A central focus of Samsung’s stand was an artist-in-residence, sketching aesthetically-pleasing models while the slightly jaded mobile industry mob looked on.

The majority of experiences remained reliant on visual innovations. However, Immersion’s HD haptics, SRS’ audio technology and HaptiMap’s research efforts all provided examples of how mobile experiences can engage our senses of hearing and touch. Firing designers’ imaginations and driving the development of better tools for integrated visual, audio and haptic design remains one of the MEX Pathway priorities.

Jason Flick of YOUi Labs demonstrated how gaming industry inspiration can make speed relevant and a clear user experience enhancement.

The unique visual qualities of Qualcomm’s Mirasol displays prompted thoughts about their role in another MEX Pathway initiative – quiet design – particularly in view of their exceptionally low power requirements.

Matt Plested of Alloy showed how even a market as seemingly commoditised as USB modem sticks can be reinvigorated by user-centred design principles, showcasing the work they did for Sierra Wireless.

Innovation in software interfaces was harder to find, not least because the small application developers at the forefront of UI experimentation were noticeably absent. Ixonos’ multi-pane sliding UI and Nokia’s PureView slide-to-zoom gesture were the standout exceptions.

A discussion with Liat Rostock of EyeSight broadened the horizon of interface innovation beyond touch and into gestures, but also revealed the nuances of user behaviour and technical challenges facing this approach.

I’m sure many of you were there too. What did you think of it all? Email me, I’d love to discuss.

My full archive of notes and photos from MWC is here, or individually listed:

• Gestures in a mobile context


• Searching for new UI metaphors


• Mobile creativity unleashed


• Prioritising relative speed


• Sound experiences outside the visual canvas


• Maps appropriate to context


• Refining through multi-layered user research


• User environment driving technology


• Displaying visual quiet


• Subtle interaction and sensory engagement


• Finally found form factor flourish


• Consuming devices move to create


• Copy fast, fail soon

Command line interfaces were direct: pressing a single key produced the corresponding character on-screen.

The introduction of the mouse resulted in a more abstract relationship: small hand movements and finger clicks, performed some distance from the screen, were translated into graphical sequences.

Currently the growth of touch is making interaction tangible again: users place their finger on what they want and get an instant response.

Gestural interfaces are best understood within this continuum of interaction development. They are more abstract, placing the user further from the content, and reliant on movements derived from metaphors rather than direct manipulation.

Liat Rostock, Marketing Director of EyeSight, demonstrated what’s realistic within the limitations of today’s mass market mobile devices. EyeSight have built their technology to work within the confines of limited processing power, battery life and camera capabilities, such that it can recognise arms-length gestures from the VGA front facing cameras common on mid- and high-end smartphones. At ‘living room’ distance – perhaps a few metres – a 1.3 megapixel camera is required. Both work best at a frame rate above 30 per second, but can function at 24.

EyeSight have also started to develop a basic gestural vocabulary. A left-right-left wave of the hand activates the input system to track a specific user. Bringing your hands together stops the camera from tracking you.

In testing, it is immediately apparent that the optimal vocabulary changes depending on the user context. A user holding a mobile device in one hand will require a different set of gestures to a user sitting in front of the home TV with both hands available. Rostock also stressed the need to reflect local cultural considerations, citing the example of a closed fist, which is inappropriate in a number of Asian countries.

(We looked at this topic as part of the Pathway #12 (‘Apply knowledge of brain processes for more effective mobile experiences’) working sessions at the last MEX event).

There are further challenges. This kind of camera-based tracking requires a certain amount of light, although Rostock claimed the glow from a screen is sufficient for arms-length interactions, while the illumination of a table lamp is enough in a room scenario.

User behaviour varies, particularly the speed and accuracy with which individuals perform gestures. Children tend to be fast and expansive, while the elderly are slower and more restrained.

Over time, I suspect a layer of software intelligence similar to the engines used for predictive text will improve recognition. EyeSight is already using video analysis, both manual and automated, to better understand how the system can be tailored to individuals.

Cameras are also inherently power hungry. EyeSight are conscious of this and are working with CEVA to pre-integrate into their DSP chip to minimise processing power requirements, but the action of image sensing itself will always be relatively power intensive. There is, perhaps, a role for combining with other sensors here, using proximity or sound levels to determine when a user isn’t present and switching the camera into sleep mode during these periods.

In practice, EyeSight’s demonstration was impressive. It recognised swipes to silence and answer calls. Both Rostock and I could control the same device through its ability to track multiple people. Most impressively, it was able to recognise one or more finger tips at a distance of a couple of metres, allowing reasonable accuracy for detailed on-screen navigation. I don’t imagine this kind of mouse pointer-style interface as the most useable application for the technology, but as a technical feat the ability to recognise finger tips opens up many possibilities.

There were glitches. The demonstration systems needed to be reset a couple of times and there were moments when the recognition performed poorly. However, these tests were being performed on older hardware in an exhibition booth filled with background movement.

Performance aside, there is also the question of whether gestural interfaces represent a good user experience on mobile devices. Firstly, consider whether they are ergonomically sound: will it caused discomfort to perform in-air gestures with one hand while holding device with the other? Second, think about the alternatives: what are the scenarios where users couldn’t or wouldn’t want to interact with touchscreen or physical keys?

I remain to be convinced on the ergonomics question, but the second consideration – demand – has a more obvious answer.

Gestural interfaces are well suited to partial attention or partial capability environments where some of the user’s cognitive or physical capacity is otherwise engaged. The in-car environment, sporting activities or simply the growing number of users who interact simultaneously with several digital touchpoints in the home. In all of these instances, gestures – which generally have a lower cognitive loading than visually-reliant touch interactions – may allow users to multi-task more easily.

Not to mention, of course, the large number of users who for reasons of climate or occupation find themselves wearing gloves incompatible with capacitive touchscreens or too inaccurate for small buttons.

Gestures will be most effective, however, when combined with appropriate audio or haptic feedback. One of the biggest advantages of gestural interfaces in the mobile environment is the ability to perform actions without the visual precision required by touch. However, users naturally expect confirmation of an action and, without an audible or tactile response to a gesture, their instinct will be to seek a visual confirmation. This would negate the multi-tasking benefits of gestures.

Interaction designers, therefore, need to rethink traditional metaphors when optimising for gestural input. The ticks, ‘OK’ buttons, colour changes and highlights which work when you have the user’s full visual attention are simply not appropriate to gestural interactions. Perhaps in their place we will see light flashes, more sophisticated audio confirmations or haptic vibrations.

These issues link with MEX Pathways #9 (Expand mobile interactions with the neglected dimensions of sound and tactility) and #12 (Apply knowledge of brain processes for more effective mobile experiences).

A few years ago, small app developers would court network operators and handset manufacturers, hopeful of a licensing deal to access the mass market. Today, new sales channels – app stores – dominate. Success is determined more by a developer’s own ability to target customers than by distributing through operator decks or handset bundling.

There were two examples of UI innovation which impressed me. One was Nokia’s new ‘slide to zoom’ gesture on the PureView 808 handset (more on that in another MEX article). The other was Ixonos‘ multi-pane tablet UI.

Sami Paihonen, Head of User Experience at Ixonos, explained the behavioural insights which drove the design. The Ixonos team had observed the trend of tablet usage combined with TV viewing and seen how the complexity of navigating between application silos focused on a single function was a source of frustration. For instance, a user watching a sports game would have half their attention on the TV screen and the other half divided between flicking through social media feeds to chat with fans, a web browser to see scores and stats from other games and their team’s own branded app.

Ixonos’s solution was to build a a multi-pane, user customisable canvas to bring together all these elements in a single app. The canvas’ total conceptual size is larger than the device screen, but instead of panning around it, users slide a cross-hair axis to reveal as much or as little of each pane as they require.

There are several benefits to this approach. In a scenario where the user’s attention is already divided between TV and mobile device, it minimises further attention division within the device itself by centralising everything within a single interaction language. In addition, the user gains control over which components are most important to them and can adjust their focus between chatting with other fans or keeping on top of the latest game stats.

Ixonos is patenting this UI and using it as a template to build apps for sports clubs. They’ve already done it for a Finnish ice hockey team and an as yet unannounced, but very well known, European football club.

Paihonen also demonstrated a new handset reference design they’d created, built around two central concepts: speed and natural, paper-based metaphors. Photos peeled upwards like sheets of paper torn from a pad of Post-It notes, while message feeds were printed on a concertinaed ticker tape, which could be folded with a pinch. Despite the skeuomorphic techniques, the design was refreshingly minimalist, with few of the textures and cliches which dog the corkboards, leather journals and whiteboards typical of the genre.

Most impressive of all, however, was how Ixonos optimised the 3D rendering so that the interface worked just as smoothly on Paihonen’s high-spec reference handset as it did on an off-the-shelf, single core 800 Mhz Samsung device running Android.

Nokia’s PureView 808 in red polycarbonate is a stunningly bold visual statement. The intensity of the colour combines with a bulging, muscular form that is the antithesis of today’s generic slim slates. I cannot stress this enough: it is red in the loudest sense of the word. There will also be white and black varieties, but Nokia’s decision to offer such vibrant colouring should be applauded for experimentation and commitment to variety. Some users will hate it, but I suspect more will see it as brave and refreshingly unusual.

It is hard to look beyond the headline number everyone is talking about: 41 megapixels. No one expected this, not least because much of the technology had to be developed specifically for the 808. According to Damian Dinning of Nokia, who has clearly taken great pride in crafting this product over 5 years, it required not just a new approach to the sensor, but also working with Carl Zeiss to create optics 10 times more accurately than required for an SLR. As light passes through a lens of this size and this close to the sensor, the potential distortion from lens imperfections would be amplified, so higher production accuracy was essential to ensure the image reproduction remained true despite the form factor limitations.

Despite the obvious attention generated by such a large sensor, it is other parts of the user experience which will make the difference as to whether the 808 succeeds or is remembered as a technical monster with a single, grotesquely over-developed signature feature. Talking to Dinning, it was quickly apparent the minutia of user experience have not been ignored. He was, for instance, just as vocal about the changes to made to the interaction sequence for zooming as he was the camera hardware.

The level of detail it is possible to capture with the 808, and the way in which that detail is captured, made it necessary to re-imagine what zooming means to the user. Nokia’s approach has been to eschew the traditional magnifying glass metaphors and even the newer, but ergonomically questionable pinch gesture, in favour of a single finger slide. This gesture simultaneously selects the camera’s area of focus and then resizes a box to determine the zone for magnification. Crucially, the user’s finger never leaves the screen until the gesture is complete, allowing them to adjust the desired result with greater fidelity than a pinch or step-by-step zoom.

This new interface is central to the major use case Dinning imagines for that massive 41 megapixel sensor: shooting small, zoomed areas of detail at a distance. This is where the 808 excels. Once a small area is selected, it can be captured at 5 or 8 megapixels, but using the 41 megapixel sampling to deliver higher overall quality. This is achieved by analysing several captured pixels for every individual pixel which actually ends up in the photo. The software looks at a number of possibilities on your behalf and then chooses the best combination of them to distill into each and every building block.

This mode enables users to easily capture new viewpoints, while a more advanced mode – accessible through the software menu – opens up the possibility of snapping full 38 megapixel size photos (there is an explanation (PDF) of why it is 38 and not 41).

Dinning highlighted the importance of this split level user experience. The default interface and settings allow users to start taking good photos or videos immediately. The additional level of complexity remains hidden until required for more advanced flexibility.

The Nokia brand has become synonymous with quality photography, not just on the 808, but across its range. When I talk to end users it consistently emerges as the photographer’s choice, a position it has captured from Sony Ericsson over a 5 year period, led by the N series and Carl Zeiss partnership.

I asked Dinning whether, given the importance of imaging to Nokia, he felt there was a particular ‘house style’ to pictures captured with its cameras, in the same way aficionados can identify a photo taken from a Leica or a Polaroid. “Natural with a little vibrancy,” was his answer, confirming this as the aim of the complex software tweaking used to optimise each Nokia camera phone. It is, in many ways, an awesome responsibility: these seemingly technical tweaks are the filter through which hundreds of millions of users will record important moments in their lives. On Flickr alone, there are more than 43 million photos taken on Nokia cameras.

The 808 is a Symbian OS device, running Nokia Belle Feature Pack 1. It is the fastest, smoothest and most logical Symbian user experience to-date, but idiosyncrasies lurk and first time users – particularly those switching from iOS – will encounter moments of frustration. That said, Symbian brings advantages. It is the most power efficient open OS, so users will enjoy the best battery life (the 808 also has a user replaceable battery, unlike the sealed units common in many of the latest devices). Symbian’s video and music support is unparalleled: users can expect most media files to run instantly, without the need to transfer them to a companion PC for transcoding, something even the most powerful iOS and Android devices still struggle with.

My biggest concern was the marriage of Nokia’s impressive image capture technology with the relatively low pixel density of the 640 x 360 display. Before spending some hands-on time with the 808 I was worried the screen wouldn’t do justice to the photos, limiting the device’s utility without a companion PC for editing or viewing. However, the high contrast AMOLED, assisted by Nokia’s Clear Black Display (CBD) trickery, is adequate and there is no noticeable disconnect between the quality of the camera sensor and the on-screen reproduction.

The 808 provides a good overall mobile user experience for most users, but there aren’t many standout reasons to choose it if you’re confident your usage will be confined primarily to consuming: viewing emails, browsing sites or interacting with social networks. However, if you have an inclination towards creating – capturing, enhancing and sharing photos, video or audio – the 808 is peerless. This is a device for the creative spirit buried in all of us: the part of you which would choose the vibrant red colour scheme over the safer black or white.

The 808′s predecessor, the N8, enabled new forms of creativity by combining high quality image capture with the immediacy of mobility. This creative output will be raised to a new level by the 808 PureView.

These are, of course, the objectives of the Pathway #8 initiated a year ago within in the MEX initiative.