AWS Device Farm FAQs

General

AWS Device Farm allows developers to increase application quality, time to market, and customer satisfaction by testing and interacting with real Android and iOS devices in the AWS Cloud. Developers can upload their app and test scripts and run automated tests in parallel across 100s of real devices, getting results, screenshots, video, and performance data in minutes. They can also debug and reproduce customer issues by swiping, gesturing, and interacting with a device through their web browser.

AWS Device Farm is designed for developers, QA teams, and customer support representatives who are building, testing, and supporting mobile apps to increase the quality of their apps. Application quality is increasingly important, and also getting complex due to the number of device models, variations in firmware and OS versions, carrier and manufacturer customizations, and dependencies on remote services and other apps. AWS Device Farm accelerates the development process by executing tests on multiple devices, giving developers, QA and support professionals the ability to perform automated tests and manual tasks like reproducing customer issues, exploratory testing of new functionality, and executing manual test plans. AWS Device Farm also offers significant savings by eliminating the need for internal device labs, lab managers, and automation infrastructure development.

Q: Do you store my app, tests, and other files on your servers? For how long?

Apps and files you upload are automatically removed after 30 days. Test results, screenshots, and videos are stored for 15 months. You can also choose to delete files and results at any time through the AWS Device Farm console or API.

After test execution completes, we perform a series of cleanup tasks on each device, including uninstallation of your app. If we cannot verify uninstallation of your app or any of the other cleanup steps, the device will be removed and will no longer be available.

While we continue to add additional cleanup steps and improve the cleanup process, it is possible for data to persist between sessions in some cases, especially if you make use of the device system outside the context of your app. For this reason, and because AWS Device Farm captures video and logs of activity taking place during your use of each device, it is recommended that you avoid providing or entering sensitive information such as account info (E.g., Google Account, Apple ID), personal information, and other security-sensitive details during your automated test and remote access sessions.

On iOS, we replace the embedded provisioning profile with a wildcard profile and resign the app. If you provide it, we will add auxiliary data to the application package before installation so the data will be present in your app’s sandbox. Resigning the iOS app results in the removal of certain entitlements. This includes App Group, Associated Domains, Game Center, HealthKit, HomeKit, Wireless Accessory Configuration, In-App Purchase, Inter-App Audio, Apple Pay, Push Notifications, and VPN Configuration & Control.

On Android, we resign the app. This may break functionality that depends on the app signature, such as the Google Maps Android API. It may also trigger anti-piracy and anti-tamper detection available from products such as DexGuard. For built-in tests, we may modify the manifest to include permissions required to capture and save screenshots.

Setting up for direct device access for Xcode and Android Studio

This feature allows developers to access their private fleet devices via their local host machines the same way they debug with real devices connected directly to the machines. On top of the ability for customers to perform the same tasks they do with real devices on Android Studio and Xcode today. Device Farm direct device access gives user the wide range selection of devices and OS versions. To learn more about private devices, please contact us.

Selecting devices

Q: Which devices are available in AWS Device Farm? How do you select the devices in your fleet?

AWS Device Farm has a large (and growing) selection of Android, iOS, and Fire OS devices. We add popular new devices as they are released by manufacturers. We also add new devices as new OS versions are released. See the list of available devices.

Q: Does AWS Device Farm have international devices from markets like Europe, China, and India?

We currently have international devices from India. We use market data and customer feedback to continuously update the fleet. If you would like to see a device that isn’t in our fleet, please let us know.

For Automated Testing, devices are selected through a collection called a device pool. Some curated device pools are provided automatically, but you can create your own pools, too. During execution, tests will be run against all devices in the specified pool that are compatible with your application and tests. For Remote Access, you select the desired device based on make, model, carrier variant, and operating system version. You can then optionally upload apps and other data as well as configure other device settings. Device Farm then locates an available device matching your request and displays the device’s display in your browser. You can then interact with the device and capture screenshots and video.

Q: Are devices able to communicate with other services or systems that are available on the Internet?

Yes. All devices have a WiFi connection with Internet access. If your systems are internal (that is, behind a corporate firewall), you can whitelist the IP range 54.244.50.32-54.244.50.63. All device traffic will come from those IPs.

While you can't test actual carrier connections, you can simulate connection types and conditions using the network shaping functionality. When scheduling a run, you can select a curated network profile like "3G" or "Lossy LTE," or you can create your own, controlling parameters like throughput, jitter, and loss. All WiFi traffic from the device will be shaped and manipulated for the duration of your tests according to the profile you choose. You can also simulate dynamic environments by changing network parameters from your test scripts.

Fuzz will perform fuzz testing on your UI immediately after launch. It streams random user input (touches, swipes, keyboard input) in a rapid fashion to your app. You can configure the number of events, the delay between events, and the seed used to randomize events. Using the same seed across test runs will result in the same sequence of events.

Q: I’d like to supply media or other data for my app to consume. How do I do that?

You can provide a .zip archive up to 4 GB in size. On Android, it will be extracted to the root of external memory; on iOS, to your app’s sandbox. For Android expansion files (OBB), we will automatically place the file into the location appropriate to the OS version. For more information, see the Developer Guide.

Tests are immediately queued for execution and usually start within minutes. If one or more devices are not available, test execution for those devices will remain queued until they become available. Testing on the other devices in your test run will continue.

Q: Does AWS Device Farm provide a way to run tests and get results through an API?

Yes. We have a plug-in for the Jenkins continuous integration environment and a Gradle plugin compatible with Android Studio. AWS Device Farm also provides programmatic support for all console features, including setting up a test and downloading test results through an API. For more information, see the AWS Device Farm API Reference. In addition to the API, you can access AWS Device Farm from the AWS SDKs.

Q: My tests generate and save additional log files. Will I see them in my AWS Device Farm reports?

If you write data to logcat (Android) or the device log (iOS), those log entries will be included in the report. AWS Device Farm does not collect any non-standard logs or other artifacts, although you may transfer files via your test script using the device's or device host's Internet connection.

Pricing

Pricing is based on device minutes, which are determined by the duration of tests on each selected device. AWS Device Farm comes with a free trial of 1000 device minutes.* After that, customers are charged $0.17 per device minute. As your testing needs grow, you can opt for an unmetered testing plan, which allows unlimited testing for a flat monthly fee of $250 per device.

Your first 1000 device minutes are provided free of charge.* This is a one-time trial and does not renew. Once your trial allocation is depleted, you will be billed at the standard rate of $0.17 per device minute.

A device minute is the billing unit. Device minutes are a measurement of the time it takes (in minutes) to install, execute, and uninstall your app and tests on every device you have selected for your test run. The unit price is constant regardless of the device, test, or application type. Device minutes are only billed for tests that complete without any device or system errors. Similarly, for Remote Access sessions, device minutes are measured from the time it takes to prepare a device to your specification to completely removing any apps and data you placed on the device.

You can dynamically grow and shrink your usage according to your business needs without any upper limits or up-front commitments.

Your first 1000 device minutes are provided free of charge.* This is a one-time trial and does not renew. Once your trial allocation is depleted, you will be billed at the standard rate of $0.17 per device minute.

Unmetered plans allow unlimited testing and remote access starting at $250 per month. Unmetered pricing is based on the number of device slots you purchase for each usage type (i.e. automated test or remote access) and device family (i.e. Android or iOS) and are priced at $250 per slot per month. Device slots correspond to concurrency.

For instance, if you purchase ten automated test Android device slots and schedule a run on 100 Android devices, Device Farm will execute your tests on up to ten devices at a time until all tests are completed on your selected devices. Purchasing more slots would enable you to get your results faster. Regardless of how many tests or remote access sessions you have in a month, you are billed at the flat rate of $250 per device slot per month.&nbsp;You can cancel your subscription for one or more device slots at any time and the cancellation will take effect at your next renewal date (the day of the month that you purchased your first active device slot). To learn more, see our docs.

Q: What if my testing needs change and I need to add or remove device slots?

You can add device slots at any time and they will be available to you immediately. You can also cancel your subscription for one or more device slots at any time and the cancellation will take effect at your next renewal date (the day of the month that you purchased your first active device slot).

Q: If I'm on an unmetered plan, can I still make use of metered billing?

Yes. When creating a run, you can choose to make use of your unmetered device slots or use metered device minutes instead. Because concurrency is not limited on metered billing, this gives you the flexibility of running tests faster than would otherwise happen using your device slots.

A private device is a physical instance of a phone or tablet that is exclusive to your account. Private devices can have custom, static configurations and run custom OS images. Each device is deployed on your behalf and removed at the end of your subscription.

Each private device under your account is considered a private device subscription. The monthly subscription price is tiered on the cost of the device and starts at $200/month. After the minimum subscription period, you can choose to cancel your subscription at any time. Please contact us for more information.

Yes. When selecting devices for a test run or remote access session you will see your private devices as well as public devices. You can also create device pools comprised of both private and public devices. For more information about private devices, please contact us.