Software development is always associated with extensive testing. This market is extremely competitive, and for many potential customers, nothing short of perfection would suffice if such a thing were possible. At the same time, startups want to accelerate their time to market and ship at least a basic version of a product to their audience as soon as they can. That’s even more true for companies that create mobile applications.

Mobile apps must look and feel great and engage users from the moment they see the app’s icon in the App Store/Google Play. According to AppsFlyier, 1 in 2 apps are uninstalled a month after being downloaded, and usability issues — problems with navigation, unclear interfaces, input difficulties, and so on — are the largest reason for it.

A digital product of high quality must work smoothly and deliver the expected results. To achieve this, rigorous testing must be performed to detect and then eliminate code errors.

In this article, we will take a closer look at the types, stages, and challenges of mobile application testing.

Types of Mobile Applications

There are primarily three different types of mobile applications, and they vary in their design, compatibility, and functionality across different platforms. Understanding the types of mobile applications is crucial for an effective testing strategy.

These types are:

Native apps: These are mobile applications designed and developed for a specific operating system (iOS or Android). They provide optimized performance, reliability, and access to device-specific features.

Web apps: Web apps, on the other hand, are accessed through a web browser; they do not need any installation on your local computer, which makes them platform-independent. However, it is completely dependent on the user’s device’s browser.

Hybrid apps: Hybrid apps combine the elements of web and native apps into a single solution. Additionally, they offer cross-platform compatibility while leveraging native device capabilities.

Testing methodologies for each type of mobile application may differ, as each type has its unique characteristics and requirements.

Mobile App Testing Process

There is no one-size-fits-all framework or approach when it comes to app testing. In general, it usually includes the following steps:

Test Planning

The first step is to prepare for the process: testers figure out what needs to be tested in mobile apps and how it’s best to test it.

Mobile app testing is quite extensive because of device fragmentation, the necessity of backward compatibility, and so on. It’s vital to prioritize at the planning stage: test the core functions of the app — on all levels — on core devices with several connectivity modes.

Test Analysis

For mobile apps, QA teams — apart from planning to test if the app works how it’s supposed to — usually also consider device combinations to test with the app; emulators & simulators they’re going to use; and UX differences on various mobile platforms they need to go through. So, the test analysis process includes figuring out testable features, conditions in which they have to be tested, and priorities for future testing. QA specialists create use cases for test creation: scenarios for testers to utilize to look for mistakes in the app (if they do manual testing) or to use in a script (for automated testing).

The iterative development cycle requires at least unit & integration testing done at the end of each cycle, by developers. They’re included in QA activities but usually aren’t considered in conversations about a step-by-step description of the process because the step-by-step format implies linearity.

It’s also vital to have guidelines for quality control to look for flaws in the app’s structure from the beginning and start doing field tests and run tests for various types of input sensors, devices, screens, and environmental conditions as early as possible.

Test Design

Developers and testers can effectively design high-quality test cases covering a wide range of scenarios by understanding various test design techniques, ultimately improving the overall quality of mobile apps.

The following are the different test design techniques used by testers to ensure the quality and reliability of mobile applications.

‍

Boundary value analysis (BVA): In this method, testers test the values at the edges or extremes of expected inputs, like checking if the app handles entering "0,"  negative values, or a very large positive number in a quantity field.

‍

Equivalence partitioning: This technique helps us divide the input values into various groups where the app's behavior should be consistent. For instance, test a login form with valid usernames and passwords, then invalid ones (empty fields, incorrect formats).

‍

Pairwise testing: It combines a test combination of two input values to uncover interactions that might cause issues. For example, test a shopping cart with different product combinations and quantities.

‍

State transition testing: It focuses on how the app behaves as it transitions between different states, e.g. switching among screens or logged-in vs logged-out. This helps identify potential problems during state changes.

Test Implementation and Execution

This stage includes the QA team collecting mobile app testing tools necessary for testing. Testing can be manual and automation-driven, the mobile app testing tools for both are used across various test levels, for various testing types.

‍

Test automation for mobile apps is a bit different from that for desktop or web-based software. There are test scripts, too, but to make the app on a mobile device “available” for testing from a computer, QA engineers need a so-called “driver” — a program-in-the-middle that “drives” code in IDEs to devices (or an emulator; or a “device cloud” — a cloud-based playing field for testing an app).

‍

There’s UI Automator, a driver for Android applications crafted by Google, and XCUITest, a similar tool for iOS devices and Apple’s XCode. Those are used for native mobile apps. For cross-platform apps and hybrid mobile solutions, testers implement cross-platform tools like Appium, which “transform” test commands to the language of “native” drivers of iOS and Android.

‍

Many other mobile app testing tools are available for automation testing, and they are becoming more accessible and widespread. If in the past, automated testing for mobile apps was a choice of companies who had large-scale projects and large budgets, now, even small startups find themselves suitable open-source tools for it: software for UI tests, a testing framework for security checks, other “drivers” for native and hybrid apps, and so on.

‍

After the tools are selected, the test execution follows. Post-release re-testing and upgrade testing go hand in hand with developers’ maintenance efforts.

Real Device vs. Emulator/Simulator in Mobile App Testing

Choosing between testing on real mobile devices vs virtual devices (emulator and simulator) is an important decision because each offers distinct advantages.

Mobile app testing on a real device is the practice of locally installing the mobile app on a real device and testing the app’s functionalities, interactions, compatibility with the hardware, and reliability. However, due to a wide variety of mobile devices to work with and test, it can be a bit challenging.

On the other hand, testing a mobile app on an emulator or simulator allows us to test the mobile app in a virtual environment. Emulators and simulators are software-driven, which means they are somewhat faster than real devices. They also allow to do parallel testing and test automation using external frameworks like Appium and Espresso.

For a successful testing strategy, it is recommended to use a combination of both real devices and an emulator/simulator.

Levels, Types, and Methods of Software Testing

To explain levels and types of software testing, we’ll be using classification from the 4th edition of Foundations of Software Testing: ISTQB Certification and ISTQB syllabus for mobile testers that provide a comprehensive theoretical basis for understanding these concepts.

Levels of Testing

Levels of testing form a sort of hierarchy for different test types that describe running tests. We first go through the “classic” level of testing, applicable to all software, and then we’ll talk about mobile-specific test levels that cover checking through the structure and behavior of cross-platform and native apps.

‍

Unit tests aim to check if the smallest parts of the code are working as they should. They’re performed by developers during the software development process, not by QA specialists or testers: and, apart from code components, they test database models, classes, and data structures.

Integration tests are done to confirm different modules of the app are performing well and interacting properly with each other. The main objects of the testing on the integration level are subsystems, databases, APIs, and microservices. Integration tests also detect flaws in how the app's interface is built. They are done by engineers and/or technical architects as well.

System tests aim to confirm the functional and technical integrity of an entire app’s system. Testers also check if the app’s working properly in the production environment (i.e. within a chosen mobile OS), if it behaves as expected according to specs, and so on.

Acceptance tests make sure that the app’s development process delivered its promises: to the stakeholders and to users — and check if it aligns with the business model that underlines its creation. In other words, various types of acceptance tests aim to confirm that the app a) works as it should for users, b) aligns with stakeholders’ specs, c) complies with regulatory requirements, and d) maintains high performance.

‍

‍Field tests represent a mobile-specific level of testing that ensures the app is working well in every condition. That includes testing the app within different types of connection (cellular, Wi-Fi), signal strength, models of mobile phone, etc. Field tests also include testing the app on target devices within different environments.

Testing for an app store approval and post-production. This is another specific level of testing that’s usually reserved for the final iterations of development cycles. Tests here are usually checklist-based and aim to make sure the app follows the guidelines from the target application store that’ll ensure the app’s release. This stage includes, for instance, checking if the app follows the UI requirements of the store, connects to the store's libraries and APIs, etc. That level also includes post-production testing: running tests for the app that’s published in the store already — for instance, tests related to updates and upgrades.

Mobile Test Types

Now, we’ll go through different test types.

Functional Testing

During functional testing, developers (for some of the types of functional testing) and testers determine whether software performs all the functions it must per its specifications. The functional mobile app testing method includes the following testing types.

Interoperability testing. Interoperability testing for mobile apps includes checking how the app handles interruptions; whether it is functional on different versions of operating systems; the way the app interacts with other apps (e.g. if API integrations are set up correctly), and so on.

Regression testing. Most applications are ever-evolving. Over the course of their existence, they get changed and updated multiple times. Regression testing is performed whenever source code is modified to verify that the software is resilient to the change and that no unintended impact is caused. The process can be partially automated.

Smoke testing. Smoke testing, also known as build verification testing, is a mobile app testing method that primarily runs tests on a new build to ensure the basic functionalities are functional and aims to find major issues that could potentially prevent further testing.

Sanity testing. Sanity testing is a subset of regression testing and is usually performed after smoke testing on a stable build of software. It ensures that the code changes made have not caused any unintended issues and that the basic features are properly functional as expected. This helps us ensure the integrity and stability of the applications.

Non-Functional Testing

Non-functional testing is checking how the system — in this case, an app — behaves.

‍

Usability testing. Usability mobile app testing helps identify shortcomings in user experience (UX). The goal is to understand whether the software is easy to navigate and simple to use from a user’s standpoint.

Performance testing. Performance testing is used to determine how fast, stable, and reliable an app is. The common metrics here are memory consumption, performance speed, data transfer speed, sustainability, and others. Some of the widely used subtypes of performance testing are:

• Load testing helps assess whether the app can be successfully used by as many users as it was intended for. It can help identify possible bottlenecks.
• Stress testing determines the system’s limits — the maximum number of users and the amount of workload it can handle before it crashes.
• Endurance testing tests the ability of the system to withstand a high workload for extended periods of time.
• Spike testing helps discover the system’s reaction to a rapid spike in user activity.

Compatibility testing. Typically, software is created to work on multiple devices and be compatible with different operating systems. For most mobile applications, being compatible with both Android and iOS platforms is a must. Compatibility testing ensures that an app works great on every device, no matter the operating system, screen size, and so on.

Recovery testing. This type of testing checks the app's ability to recover after a failure. Common reasons for failure may include hardware and network issues such as server unavailability, power outage, etc. These situations are simulated to observe the system’s behavior — for example, how it restarts and whether there is any data loss.

Security testing. Security mobile app testing is used to assess how vulnerable an app is to malware attacks or unauthorized access. Several types of security tests should be performed:

• Vulnerability testing is an automated scanning process. It is conducted with the use of various mobile app testing tools to detect configuration mistakes, bugs, anomalies, etc.
• Penetration testing goes one step further. Testers try to invade the system, look for weaknesses, and use them — without causing any damage. This is a manual testing technique.

Interrupt testing

Interrupt testing is one of the most important testing methods, simulating real-world scenarios where a mobile app during usage can get potentially interrupted by various factors, from a low battery notification to an incoming call. The goal of interrupt testing is to see how the app responds and recovers from these scenarios.

The following are some of the interruptions that are faced in a mobile application:

System notifications: These are the notifications or warnings from the device OS, such as low battery warnings, app update reminders, security alerts, etc.

User-initiated interruptions: These are the actions initiated by users, such as incoming calls, text messages, notifications from other apps, etc.

External factors: These are interruptions caused by external factors such as network loss, connectivity issues, device orientation, plugging or unplugging chargers or headphones, etc.

Localization and globalization testing

Localization and globalization testing are essential processes to ensure that the mobile application functions are seamlessly functional and accessible to different regions and cultures. Localization testing focuses on adapting your app for a region or culture, including translating text and formatting content according to local preferences.

On the other hand, globalization testing prepares the app for global audiences by ensuring it can handle various languages, date/time formats, currencies, etc.

Cross-platform testing

Cross-platform testing verifies that your app functions are properly functional across different operating systems (iOS, Android), screen sizes, and device types (phones and tablets). It improves the user experience and maximizes the app’s reach and usability. The goal of cross-platform testing is to identify and address any inconsistencies or compatibility issues that might negatively affect the user experience on specific devices or OS versions.

It is worth noting that there is no universal categorization of testing types. For instance, usability, performance, compatibility, and localization testing are sometimes grouped under the umbrella of non-functional testing. Other sources might also distinguish scalability, compliance, profitability testing, etc., as separate groups.

Mobile Application Testing Checklist

• Functional testing

• Compatibility testing with other OS versions, different shells, third-party services, hardware, and external connection devices.

• Interrupt testing

• Security testing

• Testing permissions (access to camera/microphone/gallery/etc)

• Data transfer testing

• Localization and globalization testing

• Usability testing

• Stress testing

• Cross-platform testing

• Performance testing

‍

Mobile App Testing Challenges to Expect

Test strategy — the way a company is doing testing, its main goals, approaches, and the ways it prevents risks — for mobile devices must foresee the challenges that come with the process. The number of variables and conditions against which an app must be tested in order to make sure everything works seamlessly — especially if that’s a sophisticated app, or an app that’ll be released for highly regulated industries — is, frankly, maddening.

‍

Of course, it’s futile to try to cover everything with testing; moreover, it’s impossible and unproductive. One of the major axioms of the process is that exhaustive testing is not real. But, if you know where the roadblocks are — where to look for flaws — there’s still a good chance of making it work well.

Global Number of Devices

There are dozens of mobile device manufacturers out there. Moreover, the number of different device models fabricated by some brands, such as Samsung, is enormous. Most phones run either on iOS or Android, but the differences between all the mobile models go beyond operating systems specificities alone. The vast array of phone units having different technical specifications presents a significant challenge.

This challenge can be tackled via many approaches. There are bug-hunting activities testers — and people who want to assist in the hunt from the company — can do across as many devices as possible. There’s choosing target devices — several models that are majorly used by the app’s audience (this cannot be done without some market & user research). QA teams also often opt for using “device clouds” and emulators; they, too, allow testing the app on different, albeit virtual, devices.

Different devices mean different screen sizes. A subset of the challenge is closely related to UI, and the way not tracking splits within a user interface across different versions of Android OS may result in users’ dissatisfaction and drop-offs.

Developers approach this challenge, for example, by employing modularized UI components or using responsive width and height instead of hard-coded values. QAs’ job then is to make sure it works. Various app testing tools facilitate this process, too.

The recent invention of foldable screen technology is an extra complicator. Now, testers have even more non-standard screen layouts to check. Google even released a guide on how to test apps on foldables to give testers a hand of help.

User Expectations

Meeting the constantly increasing demands of users is an additional issue. Testers tend to use the same metrics of, for one, performance, for years. But expectations are regularly raised by companies that improve speed, stability, and other software characteristics. So, competitors start having trouble keeping up.

To meet user demands, companies run surveys and examine the newly released apps’ functionalities. That’s why community managers and product teams must collaborate tightly post-release. Community is a rich source of feedback that gives direction for further testing activities.

Security Aspects

Application Security Testing (AST) is crucial for multiple reasons. Unfortunately, cases of identity theft, malware infection, crypto-jacking, and other security breaches are not unheard of and are even more common for mobile software than for desktop programs. Moreover, apps that handle sensitive user data or deal with financial assets are at a high risk of a cyberattack.

Some app testing tools can help with basic and medium-level intensity security testing. It’s also recommended to do exploratory and checklist-based testing using, for instance, known mobile security vulnerabilities and the latest exploits.

Battery Usage, Memory, and Storage

Users expect an app to consume as little battery power as possible. But it is frequent that software is fitted with vast functionality, leading to extensive battery usage.

QA engineers focus on the functional changes that can decrease battery usage without compromising performance. For one, the display is the most greedy user of energy in mobile phones. The common way to tackle the problem is to introduce a night/dark mode.

Besides, the energy consumption of software running in the background mode can also be optimized. For this purpose, developers defer unimportant operations of an app to limit background CPU and network activity.

For memory- and storage issues, engineers do stress, temperature, and security testing within the field testing level.

What Else to Keep in Mind While Testing an Application

Testing software is a crucial process that requires diligence, attention to detail, specialized knowledge, and access to the right tools. Here are some additional useful tips that QA analysts and testers should keep in mind:

1. Pursue maximum testing coverage

Testing coverage is the number of tests and their scenarios. The main objective here is to meet Functional Requirements Specification (FRS), Software Requirements Specification (SRS), and User Requirement Specification (URS). It’s important to make informed decisions about what tests to include while creating a test plan. These should be based on the budget and the priority of the requirements.

2. Proceed gradually

There is good practice for achieving maximum testing coverage within a tight budget. QA specialists should divide the app’s functionality to be tested into small modules. Then, a reasonable combination of tests and their scenarios is selected for each of the units. Such an approach helps avoid unnecessary or duplicate testing and increases the cost-efficiency of the project.

3. Try different scenarios

In real life, the software does not operate under ideal conditions. So, it is important to run positive and negative scenarios. The difference between them is that the latter receives the invalid/improper datasets as input instead of the correct ones. Conducting negative checks allows testers to predict software behavior and potential crashes.

4. Conduct server- and client-side testing

Most apps operate under the client-server model, meaning that they have a fully functioning app downloaded on their phone, which at the same time can contact centralized servers to get access to the latest and most relevant information. When running tests, it is important to make sure that both client and server-side code functions correctly.

5. Think outside the box

Sometimes the test plan does not have all the combinations of scenarios. It usually happens with innovative, feature-rich app development. Alternatively, some combinations of tests and scenarios may not work as expected for some reason. In any case, it’s sometimes useful to take a different perspective: for example, choose another framework or a mobile app testing tool, perform testing manually instead of automatically, etc.

6. Write your test cases early

Once again, the main goal is to ensure maximum test coverage. For this purpose, test cases should be done during the analysis and design stages of product development. This way QAs have enough time to verify the validity of all the requirements.

7. Test on real devices

Emulators save time and resources at the early stages of the development lifecycle. However, testing on real mobile devices is crucial when it comes to proper quality assurance. Only they can give truly realistic results.

Getting Started With App Testing

Application development cannot exist without quality control and extensive testing. Software testing is a rigorous, multidimensional process that starts by creating a proper test plan and outlining a strategy. If executed correctly, mobile app testing has a valuable impact on the quality of the final product, user experience, and cost-effectiveness of the project overall.

‍

The selection of proper app testing tools and strategies is important. However, the expertise of software developers and testers is what truly plays a paramount role in mobile testing. If you need help with QA, contact Diversido with your project’s details.

‍