Microsoft Patents AR Diet Glasses

Bringing the HBO comedy Silicon Valley’s “Shazam for Food” to life, Microsoft was granted a patent for augmented reality glasses that track when and what you eat.

Imagine a little voice in your ear telling you that actually you shouldn’t get that side of fries. Today, Microsoft was granted a patent for a wearable system for imposing dietary self-control. These are augmented reality glasses that can monitor what the wearer eats including calorie content and makes food recommendations based on nutritional needs as well as product and restaurant reviews.

Easy, accurate nutritional tracking has been a long-time goal for many companies and consumers, and this patent is just the newest example of tech giants entering the fray. Apple recently patented a system of RFID tags to track the nutrition of restaurant orders, for example. Microsoft previously filed several patents related to diet, including an allergy awareness system, and a platform to analyze restaurant menus based on user preferences.

Microsoft’s new patent, titled “Wearable food nutrition feedback system,” includes mixed-reality glasses that can monitor sight, sound, location, temperature, and motion. These sensors help the glasses identify food in front of the wearer and analyze its nutritional content. Eyeball tracking helps the glasses understand which items the wearer is considering. Based on the wearer’s needs — including any special diets, calorie restrictions, allergies, or personal flavor preferences — an augmented reality display can steer wearers away from or toward certain foods.

This Microsoft patent was filed on 10/8/15 and granted on 5/9/17, and is an extension of an earlier patent filed in 2012.

Beyond making recommendations in the moment, it can also track users’ caloric and nutritional intake over time. The glasses use gesture recognition to confirm which foods the wearer chooses, and then monitor sound and head movement to track how much the wearer actually eats (the “consumption event,” as the patent puts it). For example, if the glasses track a user’s hand picking up a hamburger, and then see the hamburger returned to the plate with a bite taken out, it will confirm what and how quickly the wearer is eating. On the other hand, this may mean you can’t try to fool it by staring at a plate full of broccoli while reaching behind your back for the pizza.

The ability to easily track caloric intake has been a challenge for current wearable fitness trackers, such as Fitbit. For now, many companies offer solutions such as manual input or barcode scanning. In addition, many wearables on the market focus on counting steps and other exercises — calories out, not calories in.

After identifying the foods using the “image processing engine,” Microsoft’s system matches them to nutritional data provided by the manufacturer, submitted by the restaurant, or scraped from similar recipes. As the image below highlights, the heads-up display can warn users about nutritional limits or make suggestions when the user is looking at the food.

Apart from nutrition, Microsoft also aims to integrate social data to make personalized recommendations. For example, once GPS data confirms the wearer has entered a certain restaurant, the device could pull in any reviews written by friends. Users could also choose to share more detailed information on their past meals with specific other users, or could pull up their own past information, as shown below.

[…] includes mixed-reality glasses that can monitor sight, sound, location, temperature, and motion. [8] Payton’s friends, family and even complete strangers pulled together to give this little girl […]