Our Insights Post 10 mins Arran Kirkup Testing Mobile Applications – Get this right and all else will follow It’s no secret that any development team worth their salt has a robust testing plan in place no matter which platform they’re developing for. Tools like Selenium are routinely used for testing web applications and there are similar tools available for mobile application testing. With the high levels of market fragmentation in the mobile space, however, there are a number of challenges unique to this space which may require a more bespoke approach. Mobile Testing Challenges With the ever increasing number of mobile users and devices testing mobile apps becomes more and more complex. Testing a mobile application is significantly different from that of a desktop-based web application. The common challenges faced during mobile testing are: Comprehensive test coverage Managing fragmentation (different O.S version, processor, memory) Lack of test plan Time Pressure Lack of Physical devices Diversity in platform and OS A complete mobile testing application strategy includes device and network infrastructure, selection of target devices, and an effective combination of manual and automated testing tools to cover both non-functional and functional testing. For mobile applications some of the things you’ll want to think about testing are: Installation OTA Wi-Fi Data Cable Bluetooth Uninstallation Application logo Application logo Splash Low Memory Visual Feedback Exit Application Start/Restart of Application Mobile Application Testing Tools - Test Automation As mentioned above the mobile application ecosystem is incredibly fragmented in terms of both software and hardware. Manufacturers launch new devices and customised versions of the base OS (in the case of Android) which can cause issues for your applications. New versions of iOS and Android are released frequently by Apple & Google with the phased rollout of Android into a diverse ecosystem of third party device; which can be a key concern. To ensure that you are maintaining compatibility in a changing market it is recommended that you revisit your applications when these changes occur. One way to simplify this kind of testing (if you have a broadly suitable app) is to implement automated testing. This keeps costs down on highly replicable tests which otherwise would mean hundreds of hours of manual testing for applications with a long shelf life. However they do require a larger upfront investment in terms of training, tools and infrastructure so it’s worth taking time to ensure you’d see the requisite ROI to begin automating some or all of your testing. Some tools which may be suited to testing your application could include: Appium – An open-source mobile test automation tool to test Android and iOS applications. Developers can test native, mobile web and Hybrid mobile apps on this software. To run the tests, Appium uses WebDriver interface. It supports C#, Java, Ruby, and many other programming languages that belong to WebDriver library. Xamarin.UITest – This is Microsoft’s testing framework that enables Automated UI Acceptance Tests written in C# using the NUnit unit testing framework to be run against iOS and Android applications. It integrates tightly with Xamarin.iOS and Xamarin.Android projects but it can also be used with iOS and Android projects written natively in Objective-C/Swift and Java. Xamarin.UITest is the Automation Library that allows the NUnit tests to execute on Android and iOS devices. The tests interact with the user interface just as a user would: entering text, tapping buttons, and performing gestures – such as swipes Jenkins – Currently the leading open-source continuous integration (CI) server monitoring executions of repeated jobs, such as building a software project or jobs run by cron. Jenkins provides an easy-to-use continuous integration system, making it easier for developers to integrate changes to the project, and making it easier for users to obtain a fresh build. The automated, continuous build and test automation increase efficiency and productivity. Test Early, Test Often All too often testing mobile applications is left until the end of a project and is something of an afterthought. This should never be the case. From the moment your app exists as something that can be loaded on a device testing should be carried out frequently. Ahead of this though your testing team should be putting together a codified test script in preparation for their first testable build then testing on an ongoing basis. Until a ticket is marked complete & tested it should not be considered complete. This will often mean involving your test team sooner rather than later and potentially far sooner in the process than you may be used to. In order to write a rigorous test script your product owner should be talking your test team through the requirements of the application as soon as sign off has been agreed. It may even be beneficial to have a senior member of the testing team present in meetings where the product requirements are being agreed. In this meeting they can propose testing methodologies and get clarification from the client on areas which could be missed otherwise. To get started developers should think about not only how to test early but also how to build a standard testing procedure at the beginning. When done properly it can accumulate as a saving of hours in the actual development and bug fixing. The complexity of today’s apps is constantly increasing and the case for testing early and often becomes ever more relevant and imminent. Test on actual devices wherever possible Given the issues with device fragmentation in the mobile market it’s worthwhile to test on actual devices and across a range of devices wherever you can. Running your app on a certain version of the operating system doesn’t guarantee that your app runs fine on all similar devices with the same OS version. It is also important to note that not all problems faced are related to software. Hardware problems cause similar fragmentation and difficulty to get the maximal performance out of your app. For example, the major differences in any chipset’s performance capability can make your app perform differently on devices based on those chipsets. Another major issue can be different resolution displays affecting the way your app elements display but other display issues like the quality, density or color brightness itself can make your app look unexpectedly poor. Similar issues exist around memory with limited memory on older or lower-end devices. This can lead to snappy performance on the latest and greatest devices but a disappointing user experience for users who are using older phones or simply don’t want to spend the money on a flagship device. Another issue which can affect your app and is often less spoken about is the OEM customisation on vanilla Android. Nearly all Android device manufacturers have built their own UI layers, skins to empower their brand and differentiate their devices. To complicate this further with some manufacturers implementing their own app stores or even removing Google’s Play Store entirely (not by choice in the case of Huawei) the market is continuing to fragment further. Isn't it easier to test on an emulator? Particularly when it comes to usability and how your users are going to directly interact with your app not everything can be tested using a keyboard and mouse on an emulator. Certain interactions like pinching, zooming or scrolling are considerably different on an emulator. Think also of issues such as interrupts, battery consumption and how charging affects the app. Using an emulated or simulated environment for performance testing is often not realistic and results are meaningless. Performance lags are easy to expose using actual devices. There are two ways to potentially testing mobile applications against real devices. By maintaining a library of the most common devices in house which can be tested against or by using real devices remotely using tools like BrowserStack or Kobiton. Testing remotely can be useful regarding some of the issues inherent to emulating but obviously using these services may not flag up some real world issues. It’s wise to maintain access to at least several up to date devices to ensure testing can be completed against real world conditions as often as possible.
