How We Test Tablets

By
Sascha Segan, Joel Santo Domingo7 Jun 2014, 5:17 a.m.

Tablets straddle the line between mobile and home computing. For many people, they're becoming primary computing devices; for others, they're entertainment gadgets to keep the kids quiet in the car. Since we test dozens of tablets on every operating system available in the U.S. each year, we have a few different approaches to tablet testing.

Most notably, we divide our tablets by operating system. Android tablets, Apple iOS tablets, Windows RT tablets, and Windows 8.1 tablets all get somewhat different testing procedures, as all the testing applications and benchmarks won't run on every device. Most notably, Windows 8.1 tablets get much more comprehensive tests of productivity application performance. Here's how we test each type of tablet.

We have a test suite of music files in MP3, AAC, OGG, WAV, and FLAC formats at various bit rates. We play the music through the phone's built-in speaker and both wired and Bluetooth stereo headphones, if possible. Our video test suite consists of MPEG4 and H.264 videos in resolutions up to 1080p.

If the tablet has Wi-Fi, we try to connect to open and WPA2-protected networks. If it specifically promotes its Wi-Fi speed, we check speed using speedtest.net at distances of zero to 100 feet, every 25 feet, using a Meraki router. We check LTE speed by using Speedtest.net at multiple locations outdoors on the street in midtown Manhattan, New York.

If a tablet has GPS, we see if it can capture a GPS signal and show an accurate location both on a busy, crowded street and in an open park.

To test camera capabilities, we use an abbreviated set of our tests for digital cameras. For battery life, we use a rundown test where we play a 12-hour-long video on a loop with the tablet's screen set to maximum brightness and Wi-Fi switched on, until the battery dies.

Tablets with pressure-sensitive pens need specific tests of pen quality. We draw grids of crosshatched lines and grids of stippled dots with the pens, looking for gaps in the lines, uneven pressure sensitivity, or missing dots.

iOS

Apple iPads are compared extensively with the previous models of iPad, rather than to other tablets or phones. We start with roughly the same tests we use for Android tablets, above. For benchmarks on iOS, we run Browsermark and Sunspider to test Safari browser performance, Geekbench to check the processor speed and GFXBenchmark to look at 2D and 3D graphics performance.

For media, we play AAC and ALC music transferred from a Mac via iTunes, as well as both a transcoded 1080p H.264 video file and an HD TV show purchased from iTunes.

As each iPad typically comes with new, exclusive apps that show off that year's processor, we test the performance of both that year's featured app and previous years' featured apps on new and previous iPads, including launch time, frame rate, and UI smoothness. To further test the processor, we measure the time it takes to perform large file exports in iMovie on both the current and previous iPads.

We also test Wi-Fi reception and performance, checking it at 25-foot intervals from a Meraki router against previous iPads and other Editors' Choice tablets.

Windows

We test Windows tablets in a similar way to how we test laptops, beginning with PCMark 8. It is an overall performance test that supports the Windows 7 and 8 operating systems only. The PCMark 8 score is derived from a compilation of common tasks that users might perform in a typical day. These include photo and 4k UHD video editing, music, gaming, communications, and productivity tasks. We use the Creative test with OpenCL acceleration, which mixes everyday tasks with media and entertainment content creation. PCMark works in both 32-bit and 64-bit environments.

Next, we use a video-based battery rundown test, which supports all operating systems, including Windows, Chrome OS, Linux, Android, and OS X. To this end, we play a 24-hour long MP4 video file compatible with a wide range of mobile devices. We start the test, and run it for as long as a fully charged battery lasts, usually to the point that the system hibernates at under 5 percent battery capacity.

Before we start the test, we crank down the brightness of the screen to 50 percent and disable adaptive screen brightness. Wireless radios are left on, but unconnected to access points. A system qualifies for the "all day computing" designation if it lasts more than eight hours on our battery rundown test.

If the tablet has an internal and an external battery pack or a keyboard dock with a built-in battery, we test both batteries. We will test with the internal battery alone, and then with the internal and external batteries charged and connected to the tablet. These results are reported in our reviews and in our benchmark tables.

For 3D testing, we want to use tests that are challenging to each system. To this end, we chose Unigine's Heaven and Valley tests. Each test simulates movement through highly detailed environments. Heaven navigates a virtual manmade location, while Valley flies through an expansive outdoor landscape. Both tests add atmospherics to each environment, and can stress even multi-GPU gaming rigs.

At the medium settings, we use 1,366 by 768 resolution and set all quality sliders to medium or normal. AA (anti-aliasing) and AF (anisotropic filtering) are disabled, in order to give us an idea of how a system plays games when frame rate is more important to the player than ultimate content quality. For the maximum settings, we set the screen resolution to native (aka max non-scaled resolution for that tablet), and bump the quality settings to extreme or ultra. We choose the setting that is the most strenuous. AA and AF (if present) are set to 4X each. This shows us what happens when the player wants an immersive gaming session with the most eye candy turned on.

Finally, we use an industry-accepted synthetic 3D benchmark that that truly taxes the graphics subsystem. FutureMark's 3DMark, another DirectX benchmark, has built-in Android, iOS, Windows 7, RT, and 8 support. 3DMark has three built-in tests, one made for mobile devices (Ice Storm), one for mainstream systems (Cloud Gate), and one test that truly taxes gaming rigs (Fire Strike). We use Fire Strike in order to put an emphasis on high-end 3D animation, like the stresses that an AAA game would put on your tablet.

Cinebench R15 is a multi-CPU rendering tool that addresses multi-core processing, multi-threading, and OpenGL elements (on the graphics side). CineBench has the benefit of running on 64-bit versions of Windows 7, Windows 8, as well as Mac OS X. We run the CPU test, which measures 3D rendering in software. This test shows the added benefit of multi-threaded multi-core computing.

For video encoding, we're using Handbrake 0.9.9 to transcode a 1080p file ripped from a Blu-ray disc down to a preset suitable for viewing on mobile devices like the iPod touch and the iPhone. Transcoding a video file will save room by shrinking the video file to a size that better fits on your phone's limited internal storage. The test is timed, and faster is better. Handbrake runs on all current flavors of Windows and OS X.

Finally, our Adobe Photoshop CS6 test involves running a 12MP JPEG through a series of 11 filters. The system is timed to see how long each filter takes to manipulate the image, and the total time is compared with times from similarly equipped tablets. The Photoshop CS6 test runs on all x86-compatible versions of Windows and OS X.