Band on the run

Did you ever hear the one about the five Microsoft engineers and the writer who walked into a Gold’s Gym?

No? Well, that’s probably because the results were nowhere near as funny as you or I might have hoped. The engineers crushed all of the burpees, bear crawls and box jumps doled out by a top Gold’s trainer. And we need not discuss how the writer performed or how sore he is while typing this story.

All five engineers and I wore the new Microsoft Band as we worked our way through the group circuit training. We compared heart rates and calorie burn after each round of team sit-ups and mountain climbers, adapting the reps to maintain our optimal heart rates. A few of the engineers discreetly glanced at emails and texts on their bands between sets.

For this team that created the device, the workout stood atop years of research, testing, iterating and inventing a whole new product category at Microsoft. In turn, the device and Microsoft Health, the new security-enhanced cloud platform that stores and analyzes all of this data, have transformed the engineers’ personal health and fitness.

Microsoft Band’s cutting-edge continuous heart rate monitoring provides a detailed calorie count and sleep quality measurements. With the inclusion of intelligent personal assistant Cortana on Windows Phone 8.1, the band also offers hands-free access to the web and your most important correspondence whether you’re at the office or at the gym.

Available on Windows Phone, iOS and Android, Microsoft Health’s open cloud platform stores, organizes and learns from your fitness information in order to give specific prompts to improve your fitness.

As Zulfi Alam, GM of Personal Devices, noted at the beginning of our workout, “Imagine you’ve set the goal that you want to get fit and lose weight as part of your exercise routine. Based on your burn rate and exercise over one week, we will soon be able to auto-suggest a customized workout plan for you. As you follow that plan – or if you don’t follow the plan – our technology will continue to adjust to give you the best outward-looking plan, like a real coach would do.”

Thanks to deep collaboration with Microsoft Research (MSR), Microsoft Health and Microsoft Band will only continue to add more capabilities. “Microsoft has the big data analytics and machine learning to attack fitness and productivity challenges in this way,” Alam said.

The path to building the band started back in 2010, when Alam was the General Manager of Xbox Accessories. He was approaching 40, becoming less active and gaining weight. “I started wondering, ‘What happened to the last 15 years?’ and I had the embarrassing realization that I was getting really out of shape,” he reflected.

Never lacking for motivation, Alam resolved to exercise with greater intensity. He discovered an affinity for the self-tracking aspects of CrossFit and its quantified competitive exercise aspects.

“I started feeling better at work: more awake and more alert,” he said. “My fitness, weight and quality of life improved dramatically.” Soon Alam encouraged other colleagues to join too.

Available on Windows Phone, iOS and Android, Microsoft Health’s open cloud platform stores, organizes and learns from your fitness information in order to give specific prompts to improve your fitness.

Brian Bilodeau, now Senior Director of Personal Devices, was working alongside Alam doing Business Development for Xbox Accessories at the time.

He was shocked into action when his father-in-law passed away unexpectedly in his 60s. Bilodeau said, “ I have to get in better shape. I don’t want to die young. I have two girls at home.’”

Bilodeau got into fitness and changed his lifestyle. He and Alam attended the same 6:00 am fitness class together, encouraging each other along the way. They learned a lot about fitness but also about the challenges of getting into better shape.

That same year, Microsoft launched the Kinect motion sensor for Xbox. Farah Shariff, now Program Manager for the Microsoft Band Algorithms and the Electrical Engineering teams, said, “I started looking into what other cool accessories we could make. Kinect sees you, but what about making something that senses you? And what about using Kinect technology for something that can leave the living room?

She was running marathons and half marathons at the time and remembered, “I became interested in building a fitness companion device. I started to incubate the project and was Employee #1 on it.”

The concept of a fitness and wellness sensor dovetailed with the fitness passion of Alam, Bilodeau and others on the team. The Accessories team got behind the project.

“Continuous heart rate monitoring was something lacking in the market at that point in time and we set out to solve that problem. First we went after chest straps,” Alam explained. “Between having to use glue or even shave your chest to achieve sensor contact, chest strap use required serious dedication.”

On another level, Shariff, Alam and Bilodeau had all found it a hassle to record exercise metrics. And Bilodeau wrestled with pre-planning for the next day’s workout in order to maintain new lifestyle habits.

The first iteration of the team’s fitness device, called Jewel, adapted Kinect sensor technology to create an optical heart rate monitor. It used light to track blood flow (rather than measuring electrical pulse like other devices). Alam said, “It could fit on the forearm or bicep and was a big step forward in terms of creating a continuous way for fitness enthusiasts to monitor their heart rate.”

Shariff said, “We soon realized that only if the device could fit on the wrist and, therefore, be worn throughout the day would it truly collect all of the data provided by continuous heart rate monitoring.” Alam and other Microsoft executives also envisioned that the device would be more compelling on the wrist. With the inclusion of productivity and communication features, they aimed to make not only a wearable personal trainer but also a wearable personal assistant. Productivity features would deepen the device’s connection with the consumer and free them from having to keep their eyes glued to a smartphone.

In the meantime, a few other optical heart rate devices for the wrist started hitting the market. “They were OK for running or walking, but wouldn’t work with non-repetitive motions, and would often take 30 to 60 seconds to recover,”

Nobody else has the big data or machine learning to attack fitness and productivity challenges in this way.

Alam said. “I knew we could improve on that by working with Microsoft Research. It would be a big challenge but we had to go for it.”

Alam took his proposal to Todd Holmdahl, CVP of Next Generation Devices, who has been at Microsoft for 20 years and was involved with innovations from the IntelliMouse and the Optical Mouse to Xbox and Kinect.

Holmdahl said, “The concept was organic and pushed from the ground up by the Xbox Accessories team. They had gotten into fitness; some of them worked out two or three times a day. I knew that passion would translate into a great product.”

“I wanted us to create something that would help people get more fit,” he continued. “But I wanted something with a whole bunch of productivity features too. A great product needs to give people reasons to fall in love with it more and more every day.”

Matt Barlow, GM of Marketing for Personal Devices, has been involved with many of the same monumental product launches as Holmdahl. He explained, “We always insisted that the band had to work not just with Windows but with iOS and Android. As we built Microsoft Health, we started thinking, ‘Why should Microsoft Band be the only device that feeds into and benefits from this service?’ We want it connect to any device customers are using to track their health and fitness. We want to remove any and all barriers to providing insights back to customers, regardless of the device they are wearing.”

Microsoft Health is a natural fit for Microsoft because of the company’s lineage in cloud platforms and health and fitness data. Microsoft HealthVault, which allows people to gather, store and share health information with medical providers on a security-enhanced platform, has existed for nearly a decade. Add in MSN Health & Fitness and Xbox Fitness and the company has a long history of helping people manage and learn from fitness data.

“We are simply building on our experience in a broader way with Microsoft Health,” said Barlow, “This is an easy-to-use service for everybody and it will provide insights to help people make better decisions and live a healthier lifestyle. And because we can really help people to live fitter lives, I feel like this is one of the most exciting things I’ve ever worked on.”

Yusuf Mehdi, CVP of Devices & Services Marketing, was an early adopter of fitness tracking and wearable technology and is a self-proclaimed “data junkie.” He said, “There’s a conviction here at Microsoft that we’re at the beginning of a couple of cool trends. One is that technology is going to become way more personal than even the phone in your pocket. It’ll be on your body and will help you to use all kinds of amazing functions without getting in the way of your life. Two is bringing all of the value of our cloud and algorithms to glean insights about fitness and wellness and productivity. As people collect more and more personal data, our machine learning will filter out the noise and surface meaningful insights from all of that information.”

As creation of the device moved forward, the Accessories team was still busy building a lot of other Xbox products. Alam and company would need to split off and form a new group to focus solely on what was to become known as Project K. Alam gave up the title of GM of Accessories in order to head up the new team.

Bilodeau stated, “A number of us from Accessories volunteered to follow Zulfi and work on the project. We weren’t assigned to it; we chose with our feet because we were all so excited about the challenge.”

“We would need to invest thousands of hours refining the algorithms until we made them magical,” said Alam. “Microsoft is great at machine learning and we would collaborate with MSR to make it possible to do productivity work on the small screen and also make sure that the sensors are as accurate as we could make them.”

Tim Paek, Research Manager/Senior Researcher in Intelligent User Experience at MSR stated, “Zulfi came to us and I was incredibly excited because I understood that a wearable could be the next step for Cortana as a truly personal assistant. The device monitors much more of your day, your fitness, helps you to be more productive and is filtering all this data you need to know. That is the future for virtual assistants.”

“We needed to create the best fitness device, but how often are you really at the gym?” Paek mused, “Most people spend the majority of their day at work. This device needs to be a great productivity device as well, and Cortana integrated with Windows can provide that.”

Paek and his team also worked around the clock to improve touch and gesture experience on such a small screen. He said, “There is lots of cool productivity stuff in the pipeline that is gonna blow people away.”

A second MSR team worked with the Project K team to optimize the sensors. “Once we translate what is happening with the body into data, we can turn that data into information and derive actionable insight to help you do great things,” said Desney Tan, Research Manager/Principal Researcher at MSR. “This is so exciting because we see it as the next phase of really personal computing.”

Tan and his fellow researchers, including Dan Morris, who arrived first and left last at our Gold’s Gym workout, were involved with two areas of the Microsoft Band. First, they developed smart activity recognition so the device can automatically sense and count your exercises as you perform them. They also enabled the device to identify if you are sitting, standing, walking or traveling in a vehicle. Tan said, “Most other devices make you manually enter information about what you are doing. We understand how valuable your time and attention is, so we set out to make the Microsoft Band sense and infer as much as we could. The better we do this, the greater a chance we have of empowering you.”

The second thing they did was to improve heart rate tracking for non-repetitive motions. Many exercises, such as a push-up or a pull-up are much more erratic than the repetitive rhythm of running. Holmdahl said, “Desney and company were doing state-of-the-art work with heart rate in no time and were able to enable continuous monitoring for all sorts of fitness activities.”

Once you have a high-quality screen on your wrist, you can be more in the moment. I no longer need to keep taking my phone out of my pocket.

As the inner working of the Microsoft Band came together, it was time to hatch the design. Shariff, who had been involved from day one, argued, “The form factor was based on the fact we need something that people can use throughout their day and make it a part of their entire life.”

Quintin Morris, the Senior Industrial Designer on the Devices team, also made a strong showing at Gold’s. Morris, who had previously designed the Xbox One controller, stated, “Whether it’s something you hold like a controller or something you wear on your wrist, we strive to make technology that disappears into someone’s life so it’s not intruding but is still super useful.”

This philosophy carried through the design process. Morris said, “We wanted to create something minimal and neutral. We weren’t aiming to replace your watch or make a fashion statement so much as make something that allows the user the greatest degree of flexibility to use the device to track their activity of choice.”

Morris points out, “The wrist is a very dynamic structure. It’s always changing shape and is different for every activity.” The team researched the geometry of the wrist to design for the widest range of wrists possible. As no comprehensive data set on anthropometrics of wrists existed, the team measured 12 cross-sections throughout the first third of the forearm on a thousand men and women.

“You can have the best sensor in the world,” said Morris, “but if it doesn’t fit right, the sensor isn’t going to touch the skin and won’t work.” Moreover, the sensor needs to touch the skin though a variety of different actions from curling a barbell to hitting a backhand in tennis. The 18.5-millimeter band was chosen as the form factor because, unlike a watch, it can fit both above and below the bump (scaphoid) in the majority of wrists.

In designing the device inside and out, Alam made sure that the team personally tested every detail – over and over again. “As we developed the band, we needed to measure every activity and gather thousands of data points,” said Alam. “So, we built a gym among the offices in Studio B on the Microsoft campus and we used it all the time.” The little gym ran frequent CrossFit classes full of kettle bells, rowing machines and medicine balls.

“The fitness functions help you exercise and become more fit,” Alam said, “And we also found that once you have a high-quality screen on your wrist, you can be more in the moment. I no longer need to keep taking my phone out of my pocket. Cortana and productivity notifications help so much. If my boss or wife need to get in touch, I can glance at what’s going on and choose to respond later or tap and say, ‘I’ll call you back.’”

“This is still our first play in the space but it is already the best instantiation of a mobile first, cloud first product we have at Microsoft,” stated Holmdahl. “If you choose, information that is immediate and actionable on your body creates a new platform of data about you in Microsoft Health. Over time, you will have the option to integrate Microsoft Health with your calendar and email information from Office, as well as location based information to continue to enhance your overall productivity experience.”

Barlow argued, “We’re committed to the Microsoft Health platform and making sure it’s open so that people have easy access to actionable insights no matter what phone or wearable device they use.”

“Microsoft Health is a rich platform for innovation. We are just at the beginning – and that’s what’s so exciting,” Mehdi weighed in. “Wearables will return people from the tech to the human side of life. We’re going to see a lot of passion around these devices.”

Back at Gold’s Gym, we tied up the workout with just enough time for everyone to make it back to the office for their 4pm meetings.

“The other day I left my band at home by accident, and felt lost without it,” Shariff laughed, “It’s no longer just a wellness product: it’s lifestyle companion. I use it to manage my fitness, to stay productive and to stay connected.”

“This is just the beginning,” said Alam as we compared our total calorie burn while walking out the door. “Microsoft Health will develop a physical substrate about your data and use that together with a digital substrate to provide you with contextually relevant information be it your feeling of exhilaration when parachuting out of an airplane, how you felt the first time you saw the Grand Canyon or your stress level the first time you presented to Bill Gates. I know that my heart rate topped 120 when I shared our plan for Microsoft Band with him.”

“Once the algorithms know enough about you and your biometrics in a steady state,” concluded Alam, “they will recognize patterns and opportunities to improve your health and fitness. These proactive insights, or the ‘Intelligence Engine’ as we like to call it, are what will differentiate Microsoft Health, Microsoft Band and our products in the years to come.”