Application Developer Perspective: Part 3: Testing


This is the third in a series of posts describing what I see as the primary issues affecting the current user experience.

In my previous post, I commented on the poor user experience when users attempt to set up or use an application or service. In Tom Peter’s book, Re-imagine, Tom makes a point that offering an excellent product or service is no longer enough. These days, it’s no longer exceptional for stuff to work. Excellence has become the norm. It’s actually the exception for something not to work. People have high expectations. This is where many mobile products and services fail. Unlike other consumer products and services, they don’t always work.

A recent UK Which study, showed that there’s a one in seven chance that a phone is going to develop a fault, usually in the first 6 months. As an example from the mobile software application world, users are often disappointed when they upgrade their phone only to find software they have previously purchased no longer works. This is to become a big issue when OEMs release phones based on version 9 of the Symbian OS as this will introduce a binary backwards incompatibility. In the short term there will be a shortage of applications and consumer uncertainty as to what programs will run on which phones. However, as I will show later, the gains in using the new OS will probably justify the means.

Testing has a profound affect on whether our products and services work. In the hardware and software worlds there’s always a rush to be the first to market. Testing is complex. Each phone has it’s own performance, localisation, screen size and possibly other custom characteristics. When you add in phone firmware versions, operator, protocol, and network variations, the number of test combinations becomes huge.  Hardware vendors can only test a subset of the possible combinations. Software writers become constrained to only testing a subset, for example only testing new products with current devices.

Problems compound other problems. For example, the quality control of the Java Runtime Environment (JRE) on phones is poor. Different devices have different problems, especially in their thread and memory management implementations. You have to work around these problems one device at a time. As Steve Procter, founder of mobile services company iTagg has said

"It’s an absolute nightmare, like in the early days of multiple Internet browsers, you need to test on ten phones just to have a core everyone is happy with,” … “The cost issue with testing is such for brands that people’s enthusiasm is knocked back a bit when they get a quote.”

In terms of application development, two things have happened as a result of all of this. First, there has been an upsurge in the number of porting and test houses. The second is a move to signed or certified applications. The idea is that network operators (and users!) only want well behaving applications on their phones.

Symbian signed and JAVA verified are the latest incarnations of signing. Some argue the former doesn’t equate to application quality. I don’t agree with this. Symbian applications are in fact tested to ensure they don’t misbehave when used as described in the their user guide. There are anomalies within signing itself such that signed applications may present confusing messages to the user on older phones where the Symbian certificate isn’t already on the phone. The user needs to download the certificate onto earlier S60 and UIQ phones – but they don’t know this. There are a few people who criticise JAVA verified, for its high costs and high failure rate. Also, not all phones, for example some Nokia 6600, 6230 and 3220 handsets, will actually allow JAVA verified applications to be installed. You get the error, "Installation Security Error, Unable to install". This is because the root certificate is missing. If some people are right we are approaching a time when all phones will be locked down and most applications would need to be signed. Two things may happen with time. One is that mobile programming will become a marginalised, specialised activity much like, for example, device driver development. Some people will do it because they have to and are willing to suffer the pains. The other possibility is relaxation of the strict controls. We have already seen this with Orange and 3 in UK who released locked UIQ and MS Smartphone products only to open them up later, via OS updates, when there was a backlash from developers.

Up to now, excessive personalisation has cost us excellence. I see the main problem as device fragmentation. There are too many handsets catering for every taste. Not enough has been made of using the same hardware in different physical casings. Not enough has been made of the ability of software to personalise devices. Not enough has been made of standardising operating systems across OEMs. Hopefully, the new versions of the Symbian OS (or Windows Mobile – slim chance, I know, but don’t discount it yet) and its takeup on mass-market phones will solve some of these problems. At the end of the day, I think the the solution may be more about standardisation and less about testing.

This is not to say personalisation isn’t important. Returning to Tom Peter’s book ‘Re-imagine’, after we know we have excellent products and services we should start to think about how to differentiate them from other offerings. To make a difference, mass produced items need to appear to be tailor made. Personalisation is the topic of my next post.

This post was written by Simon Judge, Freelance Mobile Developer.

No related posts.

1 comment

Add yours

+ Leave a Comment