Strategic Issues in Mobile App Testing

Smartphones and tablets are our window to the world and ultimate assistants, impacting the way we think and behave. Mobile software market is on the rise. Companies struggle to show off their products as fast as possible. If the quality and performance requirements are not met, soon after the release first downloads, unkind feedbacks from the customers follow. Meanwhile, testing budgets and time do not grow, and deadlines go steeper. In-house team may explore an app on several available devices, finding the critical bugs. But in order to receive coveted five star ratings in the app stores, many organizations decide to introduce dedicated talents and tools, offered by outsourcing teams. Together with good testing partner, they come up with truly scalable mobile testing strategy tailored to their audience.

Here are some aspects to put together when building such strategy

1. Unique mobile-specific challenges

Unlike desktop or web, the nature of portable devices is vastly fragmented. Even running on 10-15 popular devices, let’s say, on OS and platform versions released within couple years, results in hundreds of combinations, overwhelming the in-house team. There are many special test scenarios, mobile app testing intends to cover. Above all, neither incoming calls and messages, nor operation in motion, must interfere expectable user experience. People tend to delete power-demanding apps, so, observing battery consumption is important. Backups and recovery plan should be checked for a case if connection is suspended or charge level going down.

How fast is the application’s response time at different hardware, network types, memory parameters? Will the user having limited data plan be able to download it? Do network failures, switching between cellular, 3G, 4G and Wi-Fi, work with other popular products cause its crashing? How does it adjust to fluctuating signal intensity? What happens to user’s data when they upgrade from the store? Are temporary files deleted automatically? Those factors, natural for mobile world, only scratch the surface of possible challenges.

2. Emulators and tools

An emulator is a software that mimics the interface of a certain handheld device, browser or OS, giving insight on how the interface works and responds to input. New source code base is deployed to an emulator much faster than to a physical device. This cuts time between testing feedbacks and bug fixes. At early development stages, virtual devices allow to quickly check different application components with cost efficiency in mind. Such tools rapidly detect simple bugs covering the sheer volume of app’s features. Needless to say that the emulator costs significantly less compared to purchasing the real gadgets. However, its preparation and setup still requires effort and bandwidth.

There are more shortcomings. Though emulators mock up the most popular devices, they will never become available for all mobile brands and models. Virtualized testing does not consider contexts emerging when end-users run the app in live conditions. Emulator cannot reproduce the performance level expected from an actual device. Finally, user experience itself on the latter may vary from designed scenarios to run on an emulator. Summarizing the above, an emulator is a go-to solution for prototyping, early debugging and logic validation.

3. Hardware labs and Cloud

But when it comes to compatibility and usability, it is impossible to release successfully without testing on actual devices. The bedrock of good mobile app testing plan is the market research aimed to pick those most commonly used. Only with real device we validate performance issues, mentioned above in variable network, battery and memory states. Add the inputs from touchscreen, non-typical inputs like gestures, and data from parallel systems like GPS navigation.

Physical testing gives a glimpse into human interaction with an app, which merely can’t be replicated on emulator. Companies who opt to validate compatibility and performance using their own resources, will need to invest in different makes and models of devices. That fleet will need to expand along with mobile market, as well as require maintenance and support.

Thus, financial constraints (and relative simplicity of their apps) lead many organizations to Cloud computing-based testing. Such environments are attractive: they give advantages of accessing Android and IOS devices and versions of their ecosystems on good speed, without need of maintenance. Another benefits are data backup and recovery, unlimited storage capacity, web-based interface, and possibility to run the same scenario on the number of devices simultaneously.

Not surprisingly, since the application is deployed on remote or third-party simulator, the user’s control is narrowed. Risk of sensitive data leakage increases, as nothing on the Internet is absolutely secured. Web connectivity problems also affect availability and work of such testing services.

4. Is your application truly global?

Usability testing conducted in the USA is insufficient if we plan to release globally. To gain a worldwide appeal, we need the insights on how our product works within the parameters of target group locations, i.e. to test with customers in the markets we expect to capture. Application needs checks for support national languages, currencies, network density, cultural pattern, even popular social media.

Crowdsourced testing platforms like Kickstarter are often used for localized exploratory testing, when testers creatively explore the app. That brings us to beta testing. It’s smart to search the first beta testers among allegiant users, met on your online forums, or those communicating with your company via social networks. People who frequently interact with your app often address to the help desk. Some of them would gladly become beta testers for the next version.

Limited launch is another way to engage unbiased local and/or beta testers. Distributing to only a specific geography allows testing new features before global release. However, to receive all value of beta testing, you need to integrate strong tracking and analytics tools with your app. It may be also useful to switch on and off different features, so that different users’ groups tested different things. This has proved more practical than asking people to report on their activity, to fill out questionnaires etc.

Whatever the case may be, cloud-based and beta testing are only a remedy to fill the gaps, not a cure-all solution. Valuable mobile testing strategy combines emulators and actual devices in the right balance, in an efficient, economically feasible manner. With an integrated scalable approach, mobile apps go to production, incarnating quality and usability. Last but not least, the fair testing provider should always be there when you need it, at any project stage; you are supposed to thrive together. Choosing such a partner properly might determine success or failure for your product.