{
    "version" : "https://jsonfeed.org/version/1",
    "content" : "news",
    "type" : "single",
    "title" : "The Emulator Dilemma: Can Mobile Device Testing Be Completed Without Mobile Devices? |Digital.gov",
    "description": "The Emulator Dilemma: Can Mobile Device Testing Be Completed Without Mobile Devices?",
    "home_page_url" : "/preview/gsa/digitalgov.gov/cm-topics-button-component/","feed_url" : "/preview/gsa/digitalgov.gov/cm-topics-button-component/2015/06/09/the-emulator-dilemma-can-mobile-device-testing-be-completed-without-mobile-devices/index.json","item" : [
    {"title" :"The Emulator Dilemma: Can Mobile Device Testing Be Completed Without Mobile Devices?","summary" : "Government agencies have created a variety of apps to meet the needs of the public. As you join in on the mobile first trend and begin developing your shiny new mobile application, you will need to test it. There are a basic set of questions that must be answered: Does it function properly? Does it function properly","date" : "2015-06-09T11:10:14-04:00","date_modified" : "2024-04-02T09:45:13-04:00","authors" : {"david-fern" : "David Fern","awichman" : "Ashley Wichman"},"topics" : {
        
            "mobile" : "Mobile",
            "research" : "Research",
            "user-experience" : "User Experience"
            },"branch" : "cm-topics-button-component",
      "filename" :"2015-06-09-the-emulator-dilemma-can-mobile-device-testing-be-completed-without-mobile-devices.md",
      
      "filepath" :"news/2015/06/2015-06-09-the-emulator-dilemma-can-mobile-device-testing-be-completed-without-mobile-devices.md",
      "filepathURL" :"https://github.com/GSA/digitalgov.gov/blob/cm-topics-button-component/content/news/2015/06/2015-06-09-the-emulator-dilemma-can-mobile-device-testing-be-completed-without-mobile-devices.md",
      "editpathURL" :"https://github.com/GSA/digitalgov.gov/edit/cm-topics-button-component/content/news/2015/06/2015-06-09-the-emulator-dilemma-can-mobile-device-testing-be-completed-without-mobile-devices.md","slug" : "the-emulator-dilemma-can-mobile-device-testing-be-completed-without-mobile-devices","url" : "/preview/gsa/digitalgov.gov/cm-topics-button-component/2015/06/09/the-emulator-dilemma-can-mobile-device-testing-be-completed-without-mobile-devices/","content" :"\u003cp\u003eGovernment agencies have created a \u003ca href=\"/preview/gsa/digitalgov.gov/cm-topics-button-component/topics/mobile/\"\u003evariety of apps to meet the needs of the public\u003c/a\u003e. As you join in on the \u003ca href=\"/preview/gsa/digitalgov.gov/cm-topics-button-component/2013/09/30/mobile-first/\"\u003emobile first trend\u003c/a\u003e and begin developing your shiny new mobile application, you will need to test it.\u003c/p\u003e\n\u003cp\u003eThere are a basic set of questions that must be answered:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDoes it function properly?\u003c/li\u003e\n\u003cli\u003eDoes it function properly on the different mobile devices your customers are using?\u003c/li\u003e\n\u003cli\u003eDo all developers and testers need a collection of devices to physically test the application with?\u003c/li\u003e\n\u003cli\u003eWho can afford to buy, support and maintain all the different devices and device operating system combinations required for a mobile test lab?\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eYou are in luck: you have a lot of alternatives!\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThere are many types of tools that provide alternatives to building your own lab. Tools range from services that allow you to remotely access and test every possible combination of device and operating system to the many free tools that simulate and/or emulate the entire operating system or just the screen size and resolution. This article will focus on the latter, with information gleaned from \u003ca href=\"http://testdroid.com/news/rely-only-on-real-emulators-vs-devices\"\u003eTestdroid\u003c/a\u003e, \u003ca href=\"http://h30499.www3.hp.com/t5/The-Future-of-Testing-Blog/Emulators-vs-Real-Devices-for-Mobile-Application-Testing-The/ba-p/5506993#.VWyWC89Vikq\"\u003eHP\u003c/a\u003e, and \u003ca href=\"http://www.guru99.com/real-device-vs-emulator-testing-ultimate-showdown.html\"\u003eGuru99\u003c/a\u003e.\u003c/p\u003e\n\u003ch2 id=\"what-are-emulators-and-simulators\"\u003eWhat Are Emulators and Simulators?\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"http://www.quora.com/What-are-the-differences-between-simulation-and-emulation\"\u003eAccording to Quora\u003c/a\u003e, both simulators and emulators let you run software in one environment that’s meant for another. This allows us to see how our mobile application looks and functions without having an actual mobile device. The benefit is that both developers and testers can sit at their desktop computers and see how their application behaves without needing to purchase and configure a lot of hardware or actually deploying the application to a mobile device.\u003c/p\u003e\n\u003cp\u003eThe most popular emulators are \u003ca href=\"https://developer.apple.com/xcode/downloads\"\u003eXcode for Apple iOS\u003c/a\u003e (below, left image) and the \u003ca href=\"http://developer.android.com/sdk/index.html\"\u003eAndroid SDK for Android devices\u003c/a\u003e (below, right image). Both of these tools were built by their respective operating system vendors and are free, relatively reliable, and easy to use, but they can take a bit of time to get setup and properly configured. Keep in mind that you need to have enough physical resources on your computer to run these tools.\u003c/p\u003e\n\u003cdiv class=\"image\"\u003e\n  \u003cimg\n    src=\"https://s3.amazonaws.com/digitalgov/_legacy-img/2015/06/262-x-210-Figure-2-Xcode-iOS-Emulator.jpg\"\n    alt=\"Illustration of a computer monitor with an emulator on the screen.\"/\u003e\u003cp\u003eFigure 1: Xcode iOS Emulator.\u003c/p\u003e\u003c/div\u003e\n\n\n\u003cdiv class=\"image\"\u003e\n  \u003cimg\n    src=\"https://s3.amazonaws.com/digitalgov/_legacy-img/2015/06/299-x-210-Figure-1-Android-SDK-Emulator.jpg\"\n    alt=\"An emulator with a screen on the left and a keyboard on the right.\"/\u003e\u003cp\u003eFigure 2: Android SDK Emulator.\u003c/p\u003e\u003c/div\u003e\n\n\n\u003cp\u003eAnother option is the lightweight Chrome Device Mode and Mobile Emulation tool (below) that is built into Chrome version 32 and later. This allows you to see what your responsive design application looks like on a variety of different device screen sizes and resolutions.\u003c/p\u003e\n\u003cdiv class=\"image\"\u003e\n  \u003cimg\n    src=\"https://s3.amazonaws.com/digitalgov/_legacy-img/2015/06/450-x-292-Figure-3-Chrome-Device-Mode-Mobile-Emulation-tool.jpg\"\n    alt=\"An emulator with a preview on the left and a code panel on the right.\"/\u003e\u003cp\u003eFigure 3 - Chrome Device Mode and Mobile Emulation tool\u003c/p\u003e\u003c/div\u003e\n\n\n\u003ch2 id=\"who-is-using-emulators-and-simulators-and-to-what-extent\"\u003eWho is Using Emulators and Simulators and to What Extent?\u003c/h2\u003e\n\u003cp\u003eThe following chart, from a \u003ca href=\"http://saucelabs.com/resources/sauce_labs_testing_trends_2015.pdf\"\u003eresearch study sponsored by Sauce Labs\u003c/a\u003e [PDF], shows the percentage of people using simulators and emulators and to what extent:\u003c/p\u003e\n\u003cdiv class=\"image\"\u003e\n  \u003cimg\n    src=\"https://s3.amazonaws.com/digitalgov/_legacy-img/2015/06/600-x-307-Pie-chart-How-do-you-test-on-a-mobile-platform.jpg\"\n    alt=\"Graphic for a pie chart titled, How do you test on a mobile platform? Its three slices show that 29 percent use simulators or emulators only, 37 percent use real devices only, and 34 percent use a combination of real devices and simulators. The breakdown of that is: 12 percent mostly use real devices, 9 percent mostly use simulators or emulators, and 13 percent use emulators and real devices equally.\"/\u003e\u003c/div\u003e\n\n\n\u003cp\u003eIs this surprising to you? Which category do you fit into?\u003c/p\u003e\n\u003ch2 id=\"should-i-use-emulators-and-simulators\"\u003eShould I Use Emulators and Simulators?\u003c/h2\u003e\n\u003cp\u003eGenerally speaking, using emulators and simulators is a great, low cost way to increase your breadth of testing.\u003c/p\u003e\n\u003cp\u003eThese emulators and simulators are most valuable for developer testing: early stage testing, high level sanity and smoke testing. It is preferable to bring in a subset of “real” devices for testing in the final stages of development.\u003c/p\u003e\n\u003cp\u003eThe following are 5 categories of test classes that are much better tested using physical devices rather than emulators.\u003c/p\u003e\n\u003ch3 id=\"testing-the-usability-look-and-feel\"\u003eTesting the Usability, Look and Feel\u003c/h3\u003e\n\u003cp\u003eEmulators can be used to navigate through an application and look at the screen layout and presentation, but the true user experience, look, and feel can only be properly tested as a customer would: on a physical device. It is a totally different experience using a full-sized keyboard and mouse than physically navigating the mobile application using touch and gestures.\u003c/p\u003e\n\u003ch3 id=\"testing-hardware-components\"\u003eTesting Hardware Components\u003c/h3\u003e\n\u003cp\u003eHardware testing is one of the biggest challenges in mobile testing today. There are hundreds of makes, models, screen sizes and versions of mobile devices, and new ones are coming out weekly.\u003c/p\u003e\n\u003cp\u003eOne best practice would be to develop a prioritized mobile hardware device make, model and version list that represents the devices that the majority of your customers are using. Use these devices for regular testing and special testing such as memory usage, low battery, and interoperability.\u003c/p\u003e\n\u003cp\u003eOnce you have physically tested the priority devices, emulators may be used to provide some of the other screen resolutions and display sizes that may be popular but that you do not have on hand.\u003c/p\u003e\n\u003ch3 id=\"testing-related-software\"\u003eTesting Related Software\u003c/h3\u003e\n\u003cp\u003eYour mobile application never works alone and always interacts with other software, including multiple operating system software types and versions, multiple web browsers, and simply coexisting with other software installed on the device. All this software is competing for resources and needs to be properly configured in the operating systems.\u003c/p\u003e\n\u003cp\u003eOne best practice would be to develop a priority operating system and browser priority test list, similar to a Web browser priority testing list that you may use for testing your desktop Web applications.\u003c/p\u003e\n\u003cp\u003eOne great use for the emulator testing might be for testing your application on a new version of an operating system that is in beta or was  just released. While you may not be able to get the newest device or want to upgrade an existing device, these tools may give you a great quick check to see if your application has any issues when customers’ devices start upgrading to the new operating system version.\u003c/p\u003e\n\u003ch3 id=\"testing-networking\"\u003eTesting Networking\u003c/h3\u003e\n\u003cp\u003eThe uniqueness of mobile devices is that they use wireless and/or cellular networks. Testing networking and connectivity includes Interrupt testing, signal connectivity, network radio switching, OEM and carrier, and network interoperability. None of these types of testing can be completed correctly using an emulator, as the emulating software uses much, if not all, of the desktop computer’s connectivity.\u003c/p\u003e\n\u003ch3 id=\"application-specific-testing\"\u003eApplication Specific Testing\u003c/h3\u003e\n\u003cp\u003eEvery application is different. For example: if your application heavily uses native device functionality such as GPS, initial testing may be conducted using emulators, but different makes and models of physical devices should be included in later testing to prove that things are functioning correctly.\u003c/p\u003e\n\u003cp\u003eIf your application has responsive Web design (RWD) or is a native application, each has their own different challenges.\u003c/p\u003e\n\u003cp\u003eIn general, RWD applications use fewer native device functions, so emulators may be useful. For example, since the application is browser-based, there are many emulators that can be used to see the application on different screen sizes and orientations. Additionally, some of the many mobile browsers may be installed on your desktop where you can test for compatibility. But, even with RWD applications, don’t underestimate the importance of testing on the physical device.\u003c/p\u003e\n\u003cp\u003eNative applications are far more unpredictable and require much more physical device testing, as the emulators do not exactly mirror the operating systems your customers are using. For example, the plain vanilla Android version 5.0 that is used in the Android APK is quite a bit different from the Verizon or AT\u0026amp;T version your customers got pushed to their devices. With all these “splintered” versions, extra testing focusing on installation (new/upgrade) and uninstall testing on physical devices becomes so important and requires many different operating systems and versions.\u003c/p\u003e\n\u003ch2 id=\"conclusion\"\u003eConclusion:\u003c/h2\u003e\n\u003cp\u003eWhile emulators and simulators are great tools for the tester’s and developer’s toolbox, testing on physical mobile devices will always be a required part of the development and testing process. Keep in mind: the public will not be accessing your agency’s mobile product via an emulator!\u003c/p\u003e\n\u003cp\u003eThis is why we have been testing agency websites and native apps on mobile devices in the Federal Crowdsource Mobile Testing Program, and we have learned a few things that you would not learn on an emulator. If you want to help out a fellow agency you can sign up to \u003ca href=\"https://openopps.digitalgov.gov/tasks/94\"\u003ehelp us virtually test the beta site of USA.gov\u003c/a\u003e at your leisure, June 15-22, 2015.\u003c/p\u003e\n\u003cp\u003eThe goal of testing is to reduce the risk of releasing software that does not function properly for your customers. Testing your application using emulators is better than not testing at all, but testing on the physical devices that your customers use is always the best choice.\u003cem\u003eYou can also submit your mobile Web application for a future test cycle. The Federal Crowdsource Mobile Testing Program page provides information on how you can use this free service to test your products and is sponsored by the \u003ca href=\"/preview/gsa/digitalgov.gov/cm-topics-button-component/topics/mobile/\"\u003eMobileGov Community of Practice\u003c/a\u003e.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eDavid Fern is an IT Specialist and Technical Leader at the Social Security Administration’s Division of Quality, Testing and Validation.\u003c/em\u003e\u003c/p\u003e\n"}
  ]
}