Which Devices Should I Test With?

Oct 29, 2013

Like a kid in a candy store …

An illustration of a stack of smart cell phones

Every time we go to the mall, the kids pull us to the candy store with the floor-to-ceiling tubes of colored candy. The kids quickly grab their bags and scavenge from each of the bins until their bags fill up. They usually don’t even get to the second aisle of candy.

As testers, we do the same thing with new technology, gadgets, and new devices as kids in a candy store. We say to ourselves, “We need some of each device for testing.”

Device Differences

Smart phones have flooded the market; worldwide, Android alone is manufactured by over 23 companies, producing over 250 different device makes and models of phones. The Unites States alone has over 15 manufacturers building over 100 different makes and models. However, that’s not the end; Androids come in over 10 different operating system versions with multiple screen sizes and densities.

While Android and Apple devices are about 92% of the US market, there are many different makes, models, and operating system versions such as Blackberry, Nokia, Sailfish, Microsoft, and Firefox, just to name a few.

Developing a test device selection plan that is updated on a regular basis will help your organization whittle down the list of devices to a manageable number.

Developing the test device selection plan requires you to:

  • Know your customer
  • Research and look for trends
  • Develop a test priority list

The rest of this post gives details for each of the steps above.

Know Your Customer

When you develop your application, keep the customer in mind. Different demographics have different characteristics and devices, and they use these devices quite differently. For example, my kids grew up using the tablet and now do everything on their smart phone. My mom on the other hand just started using a smart phone and mostly uses it to make calls.

One way to help determine your customers is through web analytics tools that create reports of Web site traffic and traffic sources–federal agencies should check out the Digital Analytics Program if they are looking for web analytics programs. From these reports, we are able to see exactly what make, model, and version of mobile device each person is using to access your application. Keeping this in mind, we have a better idea where to focus our testing when we make changes.

What are the demographics, wants, and needs of the users of your mobile application?

New phone makes, models, and operating systems are continually being released. How can you spot the trends and keep up with the ever-changing combinations of devices so you can plan and acquire the proper devices to use for testing?

Tools such as StatCounter, a free online tool can help to spot trends in data, such as changes in mobile OS, mobile browser, and screen resolution. StatCounter can show you that user trends from the United States and from other areas of the world are very different. This is yet further proof that you need to know your customers and where they are coming from.

Develop a Test Priority List

Since we don’t have time and resources to test every combination of make, model, and operating system, we need to develop a priority list.

Developing a mobile device priority list is like deciding what candy to purchase. Developing the proper list in the correct order can take a bit of time, but it is the best way to most efficiently test on the “most important” devices.

Each organization should conduct a monthly meeting with analysts, developers, testers, and any other stakeholders to agree on what devices their application will need to function properly on and be tested on.

The list should be developed using inputs from:

  1. Customer information – Your customers’ demographics, special needs, and considerations
  2. Mobile trends – Using tools such as StatCounter look at the world and US statistics and trends that reflect your needs
  3. Trends on your site – Using web analytics tools to see what your customers are using to access your application

Next, you should develop your list. Create a chart with the devices you will test with based on the research. Be sure to include make, model, and operating system.

Finally, now that you have your list, make sure the devices are available to the testers.

With the Internet, anyone can use a Web application anywhere in the world, but organizations do not have the time or resources to test every possible device. By developing a mobile device priority list, your organization can develop a standard list of supported devices.

Don’t be like kids in a candy store and think you need to test every make, model, and version of phone on the market. Develop a device selection plan.

What is your strategy for selecting devices to test? Give us your comments on the discussion tab.

References

The Mobile Tester Newsletter is sent bi-monthly to the volunteer testers in our Federal Crowdsource Mobile Testing Program. Each newsletter contains a feature article, and interesting trends and statistics from the world of mobile testing. To receive our next edition, sign up to become a tester.

Oct 29, 2013