Experts comments on testing mobile apps on emulators and real devices

Expert’s comments on testing mobile apps on emulators and real devices:

“The problem with any web-based or simulated environment is that there are some tests that are difficult to impossible to do,” says Uriah McKinney, QA manager at developer Übermind. “For example: trying to stress test with multitasking; what happens if you tap two buttons; can you crash the device,” says McKinney. “That’s hard to test in a simulated environment.”

 Chuck Hriczko, application developer at Accella, which develops applications for various mobile environments, “You have to test on multiple real-world devices, since they have differences. For example, Samsung’s new Nexus S comes with Android 2.3, which no other phone has, and it has hardware that no other phone has, like a gyroscope instead of an accelerometer.”

“Using the computer as an input device is so different from using a touchscreen,” says Big Nerd Ranch’s Conway. “It’s not about how it looks; it’s about how it feels.”

But while individual devices can be inexpensive, acquiring many devices quickly stops being cheap. “We can’t test on every device,” says Lance Parker, president  of iTag.com. iTag’s mobile security app (for locating lost phones and other security activities) is currently available for Android and BlackBerry, with iPhone and Nokia versions in the works. “We have to move pretty quickly and nimbly. It’s almost impossible to test on every platform or device, and put out a release.” For their Android version, says Parker, “We have a handful of Android devices, about 8 to 10, and we may buy another few Android phones next month.”

“The general strategy I use is bounds testing,” says Übermind’s McKinney. “Know what the bottom and upper limit are, and test those. If there are specific devices or carriers you want to target, get them.”

Equally, focus on the most popular devices. For example, says Big Nerd Ranch’s Conway, in terms of Android devices, “Based on TweetDeck’s tracking of device and OS, the majority of people were sticking with 1.6 or 2.2.  So if you get the more common devices, chances are it will run pretty well on those other devices. And you have to take a leap of faith, that based on the performance and how it looks on this device, that it will look and run OK on these other devices.”