{
    "version" : "https://jsonfeed.org/version/1",
    "content" : "news",
    "type" : "single",
    "title" : "Mobile Product Testing Guidelines and Resources |Digital.gov",
    "description": "Mobile Product Testing Guidelines and Resources",
    "home_page_url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/","feed_url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/2013/08/22/mobile-product-testing-guidelines/index.json","item" : [
    {"title" :"Mobile Product Testing Guidelines and Resources","summary" : "You have started developing your mobile product, but you may be wondering what and how to test. As with any form of software development, mobile testing should be done intermittently throughout all development stages. This article was developed as part of the Mobile Application Development Program to provide agencies with some general testing strategies, types,","date" : "2013-08-22T13:45:01-04:00","date_modified" : "2025-01-27T19:42:55-05:00","authors" : {"jparcell" : "Jacob Parcell"},"topics" : {
        
            "mobile" : "Mobile",
            "mobile" : "Mobile",
            "research" : "Research",
            "user-experience" : "User experience"
            },"branch" : "bc-archive-content-3",
      "filename" :"2013-08-22-mobile-product-testing-guidelines.md",
      
      "filepath" :"news/2013/08/2013-08-22-mobile-product-testing-guidelines.md",
      "filepathURL" :"https://github.com/GSA/digitalgov.gov/blob/bc-archive-content-3/content/news/2013/08/2013-08-22-mobile-product-testing-guidelines.md",
      "editpathURL" :"https://github.com/GSA/digitalgov.gov/edit/bc-archive-content-3/content/news/2013/08/2013-08-22-mobile-product-testing-guidelines.md","slug" : "mobile-product-testing-guidelines","url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/2013/08/22/mobile-product-testing-guidelines/","content" :"\u003cp\u003eYou have started developing your mobile product, but you may be wondering what and how to test. As with any form of software development, mobile testing should be done intermittently throughout all development stages.\u003c/p\u003e\n\u003cp\u003eThis article was developed as part of the \u003ca href=\"https://digital.gov/resources/mobile-application-development-program/\" title=\"Mobile Application Development Program\"\u003eMobile Application Development Program\u003c/a\u003e to provide agencies with some general testing strategies, types, tools and testing scripts. The information on these testing pages has been pulled from the Mobile Gov Community of Practice and private sector resources. Specific guidance on testing has been split into four broad categories.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2013/05/25/functionality-and-usability-testing-resources/\"\u003eFunctionality/Usability\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://digital.gov/2013/08/05/mobile-product-security-and-privacy-testing-resources/\" title=\"Mobile Product Security and Privacy Testing Resources\"\u003eSecurity/Privacy\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://digital.gov/2013/07/31/mobile-product-accessibility-testing-resources/\" title=\"Mobile Product Accessibility Testing Resources\"\u003eAccessibility\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://digital.gov/2013/08/05/mobile-product-performance-testing-resources/\" title=\"Mobile Product Performance Testing\"\u003ePerformance (how your mobile product affects a device’s ecosystem)\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eEach of these pages provides government guidance, further strategies, resources and tools like testing scripts and checklists. If you are in the early stages of mobile development, be sure to check out the \u003ca href=\"https://digital.gov/resources/mobile-user-experience-guidelines-and-recommendations/\" title=\"Mobile User Experience Guidelines and Recommendations\"\u003eUser Experience Guidelines and Recommendations\u003c/a\u003e which contain resources for the design stage, as well as our \u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/resources/mobile-sow-and-developer-qualifications/\" title=\"Mobile SOW and Developer Qualifications\"\u003esample Mobile SOW’s\u003c/a\u003e pages which contain sample contracting language for vendor testing. The Federal CrowdSource Mobile Testing Program offers agencies the ability to have their web-based mobile products tested on many different devices by federal employee volunteers through the Open Opportunities program.\u003c/p\u003e\n\u003cp\u003eIn June 2012, the testing group at NIH provided guidance on different types of mobile testing.\u003c/p\u003e\n\u003ch2 id=\"toc0\"\u003e\u003ca name=\"x-General Testing Strategies\"\u003e\u003c/a\u003eGeneral Testing Strategies\u003c/h2\u003e\n\u003ch3 id=\"toc1\"\u003e\u003ca name=\"x-General Testing Strategies-Test Driven Development\"\u003e\u003c/a\u003e\u003cstrong\u003eTest Driven Development\u003c/strong\u003e\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eAutomated testing of each piece of code to ensure that it does not generate errors\u003c/li\u003e\n\u003cli\u003eLimit time to trouble shoot entire application\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"toc2\"\u003e\u003ca name=\"x-General Testing Strategies-Emulators/simulators (Native Apps)\"\u003e\u003c/a\u003e\u003cstrong\u003eEmulators/simulators (Native Apps)\u003c/strong\u003e\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003ePossibly the most convenient and cheapest. Testing can be done on the developer’s laptop.\u003c/li\u003e\n\u003cli\u003eCan test design (e.g. layout, screen size), usability (e.g. button usage), functionality (e.g data input), etc.\u003c/li\u003e\n\u003cli\u003eMany app companies use simulators to identify bugs at the early stages of development.\u003c/li\u003e\n\u003cli\u003eNot always specific enough – not OS, device/hardware or carrier specific.\u003c/li\u003e\n\u003cli\u003eExamples of emulators/simulators:\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"http://developer.apple.com/library/ios/#documentation/Xcode/Conceptual/ios_development_workflow/125-Using_iOS_Simulator/ios_simulator_application.html\" rel=\"nofollow\"\u003eiOS Developer Library – iOS simulator\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://developer.android.com/guide/developing/tools/emulator.html\"\u003eAndroid SDK mobile device emulator\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://docs.blackberry.com/en/developers/deliverables/5716/The_BB_Smrtphn_simulator_447179_11.jsp\"\u003eBlackberry smartphone simulator\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"toc3\"\u003e\u003ca name=\"x-General Testing Strategies-Real world\"\u003e\u003c/a\u003e\u003cstrong\u003eReal world\u003c/strong\u003e\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eTesting can be done with a real device tethered to a workstation or a real device, on a real network, in a real location. Testers generally adhere to a \u003ca href=\"http://gsa.github.io/Mobile-Code-Catalog/testing.html\"\u003etesting script\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eTypes of real world testing:\n\u003cul\u003e\n\u003cli\u003eThird party\n\u003cul\u003e\n\u003cli\u003eOutside Party\n\u003cul\u003e\n\u003cli\u003eHiring an outside company to run all testing.\u003c/li\u003e\n\u003cli\u003eWill gain feedback from a variety of users using a variety of devices on a variety of networks.\u003c/li\u003e\n\u003cli\u003eExample:\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"http://www.deviceanywhere.com/\" rel=\"nofollow\"\u003eDeviceanywhere.com\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003eThird Party Monitoring — web services that track and aggregate errors, generate crash reports, logs, etc.\n\u003cul\u003e\n\u003cli\u003eAirbrake\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://www.crittercism.com\" rel=\"nofollow\"\u003eCrittercism.com \u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003eCrowdsourced solutions\n\u003cul\u003e\n\u003cli\u003eCan be through a free open-source environment or hired through a third party company that specializes in crowdsourcing solutions.\u003c/li\u003e\n\u003cli\u003eWill gain feedback from a variety of users using a variety of devices on a variety of networks.\u003c/li\u003e\n\u003cli\u003eExamples:\n\u003cul\u003e\n\u003cli\u003eUtest\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://www.mob4hire.com/\" rel=\"nofollow\"\u003eMob4hire\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://www.usertesting.com/\" rel=\"nofollow\"\u003eUsertesting.com\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eThe Federal Mobile Crowdsource Testing program provides this service for free to federal agencies.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003eIn-house (Using resources available on-hand)\n\u003cul\u003e\n\u003cli\u003eLimited to what is available.\u003c/li\u003e\n\u003cli\u003eCan be expensive if thoroughness is a goal.\u003c/li\u003e\n\u003cli\u003eTesting with a real device (tethered to a workstation).\u003c/li\u003e\n\u003cli\u003eTesting with a real device, on a real network, in a real location.\u003c/li\u003e\n\u003cli\u003eFeedback from a variety of users using a variety of devices on a variety of networks\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003eRepetition\n\u003cul\u003e\n\u003cli\u003eRepetition of testing through all stages of development\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003eTesting Updated Versions \u0026amp; Revisions\n\u003cul\u003e\n\u003cli\u003eEnsure application is not out of date for certain features (Android and iOS change frequently)\u003c/li\u003e\n\u003cli\u003eTesters should understand what is available in newest version and make comments for improvements on future revisions\u003c/li\u003e\n\u003cli\u003eMake testers feel like they can contribute to improvements\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003ePlatform Testing Resources\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"http://developer.android.com/guide/developing/testing/index.html\"\u003eAndroid testing\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eApple testing\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://us.blackberry.com/sites/developers/resources/simulators.html\"\u003eBlackberry testing\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch1 id=\"toc4\"\u003e\u003ca name=\"Types of Testing\"\u003e\u003c/a\u003e\u003cstrong\u003eTypes of Testing\u003c/strong\u003e\u003c/h1\u003e\n\u003cul\u003e\n\u003cli\u003eStability\n\u003cul\u003e\n\u003cli\u003eEnsure there are no crashes – not really a problem for apps submitted to the Apple Store because of their strict approval process, but other app markets may have looser standards.\u003c/li\u003e\n\u003cli\u003eApp should be bug and defect free.\n\u003cul\u003e\n\u003cli\u003eThe most frequent types of updates after app deployment are bug fixes.\u003c/li\u003e\n\u003cli\u003eBug fixes should not change the appearance nor the function set of the app.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003eUpdate policy\n\u003cul\u003e\n\u003cli\u003eAndroid – able to push out updates whenever needed.\u003c/li\u003e\n\u003cli\u003eApple, Amazon, Microsoft – updates go through same approval process as initial app submission. Apple allows for fast-track approval for updates, but is only reserved for apps that constantly crash or issues concerning security\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://digital.gov/2014/01/21/functionality-and-usability-testing-resources/\" title=\"Functionality and Usability Testing Resources\"\u003eFunctionality\u003c/a\u003e\n\u003cul\u003e\n\u003cli\u003eNeeds to be tested on specific/targeted devices to make sure it is functional on different pieces of hardware.\u003c/li\u003e\n\u003cli\u003eTest on different OS versions (latest and legacy OS).\u003c/li\u003e\n\u003cli\u003eTest on different carriers.\u003c/li\u003e\n\u003cli\u003eTest on different browsers (if a web app).\u003c/li\u003e\n\u003cli\u003eCheck our article on \u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2013/11/21/mobile-product-compatibility-and-functional-testing-101/\" title=\"Mobile Product Compatibility and Functional Testing 101\"\u003eCompatibility Testing\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://digital.gov/2013/08/05/mobile-product-security-and-privacy-testing-resources/\" title=\"Mobile Product Security and Privacy Testing Resources\"\u003eSecurity\u003c/a\u003e\n\u003cul\u003e\n\u003cli\u003eIs private data kept private?\n\u003cul\u003e\n\u003cli\u003eStored personal data is password protected and/or encrypted.\u003c/li\u003e\n\u003cli\u003eTransmission of personal data from device to device is encrypted.\u003c/li\u003e\n\u003cli\u003eLimit user privileges (i.e. limiting access to certain files).\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://digital.gov/2013/08/05/mobile-product-performance-testing-resources/\" title=\"Mobile Product Performance Testing\"\u003eLoad and Performance\u003c/a\u003e\n\u003cul\u003e\n\u003cli\u003eTest for lagging issues on lesser-ended devices/hardware.\u003c/li\u003e\n\u003cli\u003eTest download speeds on different networks in different areas.\u003c/li\u003e\n\u003cli\u003eBattery usage – does the app heavily drain the device’s battery.\u003c/li\u003e\n\u003cli\u003eScalability — ensure the app can handle the success scenario of getting used by a large group of people\n\u003cul\u003e\n\u003cli\u003eWeb services available to mimic load\u003c/li\u003e\n\u003cli\u003eUtest\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://digital.gov/2013/07/31/mobile-product-accessibility-testing-resources/\" title=\"Mobile Product Accessibility Testing Resources\"\u003eAccessibility/Usability\u003c/a\u003e\n\u003cul\u003e\n\u003cli\u003eCurrently, there are no clear Section 508 rules for native apps.\u003c/li\u003e\n\u003cli\u003eWeb apps \u003cem\u003eshould\u003c/em\u003e be fully accessible and Section 508 compliant.\n\u003cul\u003e\n\u003cli\u003eiOS\n\u003cul\u003e\n\u003cli\u003eAccessibility Inspector in iOS Simulator – checks elements for accessibility in an application\u003c/li\u003e\n\u003cli\u003eVoiceOver on device – screen reader for those with impaired vision\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003eAndroid\n\u003cul\u003e\n\u003cli\u003eEnable Accessibility on Android devices and activate a screen reader (TalkBack is preinstalled on many devices). Other screen readers and text-to-speech libraries are free in the Android Market.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003eBlackBerry\n\u003cul\u003e\n\u003cli\u003eApplications should be tested on a production screen reader or other assistive technology application that is installed on a BlackBerry device.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003eWeb Services\n\u003cul\u003e\n\u003cli\u003eWebaim\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eFinal product should be stable when first released to the public. A survey by Harris Interactive found usability and user experience is more important than brand name. Users that have a bad experience will tend to avoid other apps from the same developer.\u003c/p\u003e\n"}
  ]
}