Beyond the rectangle: charting progress towards next generation UIs

Beyond the rectangle: charting progress towards next generation UIs


The majority of digital interfaces will eventually break free of rectangular frames. The prison walls are coming down. We’re not there yet, at least not for most people, but soon. Soon enough that those desiring a stake in this future should be planning for it.

This article is about connecting a few dots, some of them quite disparate, to hint at the eventual picture which might emerge.

The technologies which will enable this are diverse. On the one hand, ever smaller projectors pared with motion sensing will allow interfaces to materialise on new surfaces, objects and even seemingly out of thin air.

Interactive Light, an experimental project by argodesign, provides a good way to envisage this. It is also a valuable, early contribution to the community of lessons they learned about the nuances which could define good experience in this field.

On the other hand, advances in microphones, speakers, headphones and audio processing are challenging the notion of visual primacy which has dominated digital interface design from the beginning. Necessary first steps, such as wireless headphones and equipping smart speakers with microphones good enough to cut through background noise, are well underway with the adoption of products like Apple’s AirPods and the improvement of Google’s Nest Home speakers. Accordingly, the number of digital interactions which take place entirely through the medium of sound is growing and each of those interactions is becoming more meaningful in what it can achieve.

An example which shows the scale of ambition in this area is Bose’s AR platform. It packages Bose’s noise cancelling technology and motion detection into products like sunglasses and headphones to enable audio-centric, multi-sensory experiences. For instance, responding to in-ear point of interest suggestions with a simple nod or a shake of your head.

We should also pay attention to near-to-eye products, like Snap’s latest Spectacles and the persistent rumours of Apple’s digital eyewear, most recently predicted for launch in 2023. There’s a reason why so many technology companies, from established giants like Apple and Microsoft to startups like Snap and Magic Leap, are collectively investing billions in this area. It is likely to be the first way in which large numbers of people experience visual interfaces outside the traditional rectangular screen. Simply, the closer you bring the electronics to the eye, the easier it is to control the quality of the experience. At least in theory. Significant challenges remain to solve some basic issues of comfort, usability and miniaturisation, but it is still likely to happen sooner than airborne interfaces capable of manifesting in our physical environments.

Of course, there’s a big, unanswered question here. Why is any of this good for people? Simply because we can doesn’t mean we should. However, I think there are clues all around us as to why it might be useful and they’re bound up with some of the bigger existential questions of our time.

Humans have shown themselves to be very fond of rectangles. Books are rectangles. Most of our buildings are governed by rectangular form. The tiles on our walls, the shape of our furniture and the way we package our products are all constrained by this regular, four-sided shape. But has it worked for us? Where else in all of nature’s diversity, evolved over billions of years, do we find rectangles occurring?

If as a species we are finally questioning the extent to which the human approach – particularly the industrial age – has been at odds with the world’s natural balance, might we also consider that digital interactions which better reflect the whimsical, multi-sensory and incomprehensibly beautiful diversity of natural phenomena are an opportunity for better designed experiences? We’ve lived through the age of skeuomorphism in digital interface design, with its attendant desk drawers and notepads, but perhaps the next epoch will be characterised by principles of biomimicry?

This to me is the fundamental underpinning of any desire to liberate interfaces from rectangular screens. The great irony of locking our digital world into these limited spaces is that we ourselves have become captives of these rectangular prisons. Given the amount of time we spend visiting these compounds, walled in by the dimensions of iPads and smartphones and TVs, could it not be argued that we too are living the life of inmates?

I should stress this is not an argument against digital progress itself. I am enthusiastic about the potential for digital to deliver many more benefits than it already has, and in many more areas of life. Yet we are reaching the limits of how this pervasive potential can be achieved while it continues to be mediated through glowing rectangles which demand our attention and dictate the terms of engagement through their prescriptive form.

Image of Dolce and Gabbana's Rome store interior (from Dolce and Gabbana web-site)

It should also be an opportunity to question some of the basic premises which attend the term ‘interface’. Earlier this year, Dolce and Gabanna opened a boutique in Rome where the walls are digitally decorated – filled with virtual murals which shape people’s experience of the space. This is an interface, of sorts. It extends the definition into a territory which overlaps with ambience and decoration. Performative architecture is by no means a new concept (Michael Trudgeon wrote about it for MEX twelve years ago in this piece entitled ‘Push button ambience, performative architecture and mobile communication‘), but the recent developments in projection and motion sensing are bringing it closer to realisation.

Once interfaces are no longer synonymous with flat canvases limited by an X and Y axis, we can also start to review the governing metaphors and visual conventions of the WIMP (windows, icons, menus and pointers) world view which has influenced everything from the early PCs onwards.

Lightfield display by Continental and Leia

Automotive component supplier Continental and lightfield display start-up Leia are collaborating to bring floating, depth-based interfaces to vehicle interiors. It is a bold ambition in concept, but it is interesting to note how the announcement also remained tethered to the past. It talked of floating stop signs – icons, essentially – albeit ones which hover in the air, in a way that feels anchored to the conventions of a previous generation.

Designers of the future will need to question the relevance of such static symbols. Might not natural metaphors, like mist descending or a soundscape with softly rising tones of warning, be a more appropriate way to manifest the concept of an approaching stop sign or dead end? The capability will certainly be there to materialise such things, but who will encourage the ambition and creativity of designers to leave behind the conventions of an older world of rectangular screens?

For me, the starting point is to look beyond rectangles, pages, windows and canvases. Instead, I’m trying to embrace a conceptual model in which digital experiences exist all around us as floating orbs. They are inherently multi-dimensional and inclusive of multiple sensory elements. They may change in size and form. They may merge to create larger overall experiences or separate into smaller, more distinct ones. They possess gravitational properties which might at times bring them closer to the orbit of our attention and at other moments let them drift further away. There will be times when we interact with orbs as external observers, holding them at conceptual arm’s length, and others when we step inside them for a more immersive experience.

I wrote more about orbs in an August 2016 article, ‘Canvas to orb: an inflexion point in digital experience design‘. Revisiting the theme today I can see the new dots which have emerged in the intervening three years aligning on a path leading to a quickening of these developments. If you’re thinking about this area or have been working on things which relate to it, do please drop me a line. I’d love to compare notes.

2 Comments

Add yours
  1. 1
    Marek Pawlowski

    Marko Balabanovic followed up on this article with two links of interest.

    The first is Active Cues ‘Tovertafel’, described as a ‘magic table creating moments of happiness for people living with dementia and all those around them’. Like argodesign’s Interactive Light project, it uses a combination of motion sensing and projection, but is focused on helping people with specific conditions, developed using a co-design process to understand their needs. There are also versions in the works for those with learning difficulties and children with autism. Take a look at https://tovertafel.co.uk, where there’s a video showing the system in action.

    Marko also shared an excellent reference piece he wrote as part of his work at the Digital Catapult in London, with a wide-ranging overview of developments across all the components needed to realise the AR ‘metaverse’. The listing of multiple tools which developers and designers might use to create this kind of world is particularly useful. You can find it at https://medium.com/digital-catapult/who-will-own-the-metaverse-48c2912d7aae.

  2. 2
    Marek Pawlowski

    Caitlin McDonald followed up with a link to an article on the workspace of the future, which is a precursor to her forthcoming report on the subject:

    https://leadingedgeforum.com/research/reconfiguring-the-collaborative-workspace-making-the-most-of-time-space-attitude/

    I particularly liked the fig. 2 illustration at the end, showing ‘Steve’s ideal workspace’. I was struck by how many potential interconnections there could be between all those different touchpoints – and how unprepared the tech industry is to make those seamless in the first instance and, eventually, crafted experiences where the process of interconnection actually adds something to the overall capability. A few (Microsoft, for instance) are nibbling at the edges of all this, but the reality today remains fragmented, despite how clear it is becoming this is how people want to work. There’s an opportunity here for solutions which think about UI canvases with no borders.

    p.s. Caitlin provided the inspiration for our January 2019 edition of the MEX dining club (https://www.mobileuserexperience.com/dining-club/) in London, when our theme was:

    How does digital design influence our physical working environments?

    What tools and processes do teams need to ensure digital natives and non-natives collaborate effectively? What makes teams cohere in a remote-friendly world? Research has shown workers in open plan offices spend less time collaborating face-to-face than their cubicle-bound counterparts. What kinds of physical and digital boundaries do organisations need to spur more effective collaboration? In a world where attention traps are increasingly common, how do we set effective boundaries for ourselves and our teams?

+ Leave a Comment