Automated Testing on Real Vs Virtual Devices

There are key differences between automated testing on real devices and on their virtual counterparts (emulators). Let’s go through a comparison to assess the best path for your organization to take with regard to your test automation practices.

A closer look

Here, we explore the advantages and disadvantages of automated testing on real devices and virtual ones.

a. Advantages of real devices

1. Testing real world conditions (weather, location, network, etc.) are much easier and provide deeper insight into app functioning in a variety of situations
2. Screen resolution, picture brightness in different lighting scenarios, etc. are also easily  testable
3. Interoperability testing is far easier with an actual device
4. Device performance is faster than corresponding virtual models, which means that evaluating app performance will yield more precise results

b. Disadvantages of real devices

1. Conducting automated tests on actual devices is a relatively expensive affair, which would hinder projects that are restricted by time and budget
2. Because of the wide range of platforms and devices, it is difficult to efficiently carry out testing in a limited time period
3. During the app development phase for unit testing, real devices are much harder to connect to the Integrated Development Environment (IDE) than emulators, which would suffocate the project

c. Advantages of virtual devices / emulators

1. In situations with severe deadline pressure, getting access to real devices might be challenging, so emulators can serve as reasonable substitutes for testing in such cases
2. Budget constraints can impact the team’s decision to invest in real devices. Virtual environments can save money and time by allowing testers to continue with their tasks without having to spend on devices that may not in fact be entirely appropriate for the particular tests they want to execute
3. Emulators tend to be open source and easily downloadable
4. Web app testing is more convenient with emulators

d. Disadvantages of virtual devices / emulators

1. For extended periods of evaluation, emulators may not be appropriate because extended usage can be measured best on a real device
2. The costs may add up when your team discovers that their emulator may be incompatible with certain apps or app elements. In such cases, they might need to invest in software patches to allow testing to continue

3. Emulators may support only a limited number of mobile operating systems, which could render them useless in many cases

Now that we’ve summarized the pros and cons of each, let’s look at a few testing tools for the Android platform (the most common mobile OS today, so therefore an obvious option to consider).

1. Robotium: A free UI testing tool, Robotium is often labeled as “Selenium for Android.” However, you need to spend time with the program source code to run your test automation suite, and is not entirely appropriate for interacting with system software.
2. Ranorex: Can provide comprehensive reports along with pictures, but the element search facility is a little sluggish.
3. Appium Automation framework: Uses the WebDriver interface to run tests, and supports a number of programming languages, and also supports mobile web browser testing. However, its reporting facility is not very well implemented, and XPath support on mobile devices is poor.
4. MonkeyRunner: A popular tool where tests are written in Python, MonkeyRunner can run tests on both real devices and emulators. But a crucial constraint in this tool is that you need to write scripts for each and every device. Furthermore, the test need to be altered every time there’s an alteration to the UI of the program under testing.

Thus, it should be clear now that there are a number of advantages and disadvantages to using real devices and emulators, and the kind of tools we should be looking at for the most popular mobile operating system. QA teams need to assess their own project requirements and decide what combination of real / virtual would best suit their needs for a given app, and what sacrifices they’d be willing to make when choosing a test automation tool.

Cigniti’s Test automation accelerates regression test efforts in a cost effective manner and our certified test automation specialists have expertise in diverse frameworks and methodologies. Contact us today for your test automation services.