Sleep Trackers in 2017

Three years ago, sleep specialist and physician Dr. Christopher Winter posted an article titled “Personal Sleep Monitors: Do They Work?”. Today, with over 500 options from more than 15 brands available, I was curious to conduct a similar study three years later on how the data from personal sleep tracking devices compare with each other. I wore 10 of some of the most popular sleep trackers (including one research-standard actigraph as a control) across 10 nights and then graphed the data against each other. If you’re only interested in the results, feel free to skip to the end, but here’s a rundown of the devices used, and what data I was looking at. I’m also including links to guides and source code if you’d like to make a similar chart yourself!

The Devices

In general, sleep happens in five phases, but these are often categorized by consumer-level devices into two to three stages: light sleep, deep sleep, and REM sleep. Most wearables and apps in the market do not claim to recognize REM sleep since it is virtually impossible to detect without much more advanced sleep studies using polysomnography, a type of test that records your brain waves, blood oxygen levels, heart rate, breathing, as well as eye and leg movements during the study. It requires the placement of various electrodes on your head and is usually performed in a clinic for just 1 night.

A child with various electrodes placed on the head for a polysomnograph.

Because polysomnography is very expensive and inconvenient, these personal activity trackers offer simpler, more affordable, and longer-term options for people like you and me. They typically work by using accelerometer sensors (how much you move around), noise sensors (how much sound you make), or some combination of both, but because they are less complex, they are not nearly as accurate as the devices used in clinical sleep studies.

Here’s a quick rundown of each of the devices I used in my experiment. *I originally wanted to include the BASIS Peak that "won" Dr. Winter's original experiment, but decided against it after the device was recently recalled due to safety concerns.

Was once named the top paid app in the App Store. Northcube AB does not clarify what thresholds are used for classifying sleep stages, so our chart presents our best estimates based on the given images from the app.

A unique system that tracks not only your motion, but also your sleep environment during the night. The data output from their app is a bit rough, so some of the data on our visualization is also estimated.

The Experiment

Gathering the data was simple in theory: I just had to use all ten devices simultaneously for ten nights. So, I wore all five wristbands on the same arm (Fitbit Alta, Jawbone UP, Microsoft Band, Microsoft Band 2, AMI MotionLogger), three phone apps placed side-by-side on my bed (Sleep as Android, SleepCoacher iOS, SleepCoacher Android), one phone app placed on my nightstand (Sleep Cycle iOS), one device clipped to my pillow (Hello Sense Sleep Pill) and one product that required two sensors clipped onto my pillow and placed on my nightstand (Hello Sense). It was, as expected, not very comfortable -- which might be why my sleep patterns seem quite restless over the 10 nights!

The five wristband devices and three of the phone apps on my bedside.

Extracting the data from the devices proved to be a bit of a challenge. If you’re interested in a detailed explanation of how I did this, you can check out my data extraction guide. It’ll walk you through how to unlock your minute-by-minute (or stage-by-stage) data from your device. This isn't always as straightforward as you would expect -- in fact, we were of the first to unlock data from the Hello Sense, and you can find the guide for that here (thanks Jeff)!

As for the data visualization, I used this d3 chart. I tweaked the margins and the legend just a tiny bit for this project, so if you'd like to check out my version, you can find it here (they are virtually the same).

The Results

Below is the visualization of the data I retrieved. There are a few things to note that you can find in the limitations section later. You can hover over sections of each night to see timestamps, and click through the links to different nights.

The Conclusion

While the data I gathered was better than random, the results from my experiment should still be taken with a grain of salt. My experience with these 10 devices may differ from yours, and each device has its strengths and weaknesses. If we consider only the results from the AMI MotionLogger as the “gold standard” for our accuracy study, the Fitbit Alta seems to be the most accurate among the other 9 in terms of sleep versus awake data. However, each device varies in price, convenience, design, popularity, and plenty of other factors more that you might consider differently based on your lifestyle.

If you’re interested in learning more about sleep tracking and would like to sign up for SleepCoacher, the Android and iPhone app being developed by researchers at Brown University, check out our interest form here! By signing up, you’ll receive an invitation to be part of the first group to receive personalized recommendations through our app. SleepCoacher works by guiding you through small experiments to find the optimal sleep environment and schedule for you. The more you use it, the better the recommendations get. It’s been presented at research conferences, and our algorithms have been adjusted to Our goal is to provide convenient, affordable, and accurate sleep improvement recommendations to help you figure out what works best for your lifestyle and habits.

In the end, we found that most devices were similar to each other if they were based on the same type of sensor (accelerometer vs. noise) and the device type (phone vs. wristband vs. other). Ultimately, sleep is best tracked through polysomnography, but these devices offer much more accessible and convenient options for casual users to track their activity and learn more about their habits during both the night and day. These devices offer some powerful data tracking tools, but it’s best to let the experts analyze your sleep if you have any chronic conditions. Our findings tell that these consumer-level sleep reports should be taken with a grain of salt, but regardless we’re happy to see more and more people investing in improving their sleep -- after all, there’s nothing better than a good night’s rest!

*Limitations

The chart is missing Fitbit Alta data on December 6-7 because my battery died.

The Band 2 lost contact with my wrist when I was asleep sometimes leading to gaps in the data on quite a few of the nights.

It was impossible to wear them on the same exact part of my arm every night, and so the different distances from my body may have affected the data slightly.

I could not physically turn on all the devices on/off at the same exact time, so the start and end times for each device are slightly different.

I was unable to work with a real polysomnograph for this study, so the AMI MotionLogger in this study is the closest approximation of a "gold standard" available.

5 of the 10 devices used in this study were all wristbands that used accelerometer sensors to compute sleep data, and they were closer to each other (and my body) than they were to the phone apps. They were also all worn on the same arm as the AMI MotionLogger, giving them a comparative “advantage”.