Layar on Google Glass: It’s Not Augmented Reality

Dirk Groten, Ronald van der Lingen
June 25, 2013

Photo by The Verge.

The following is a post by Layar’s R&D lead Ronald van der Lingen and CTO Dirk Groten.

Two weeks ago we got our hands on Google Glass, and we have not been sitting idle. We started hacking right away to see what we can do with Layar and this hot new piece of technology. Here are our findings from these initial experimentations.

Glass runs on Android, we have an Android version of Layar. Piece of cake?

When we learned that Google Glass is “just” an Android device with a custom interface on top of it, we of course wanted to know if the code base of Layar for Android would work. We were already skeptical about the usability of “true” augmented reality (AR) on Glass, but you never know for sure until you try. “True” augmented reality is when you see your reality modified by the digital layer that’s added to it. You look at a page in a magazine through your AR glasses or AR-enabled smartphone and the page appears different than in reality: an image in the page comes to life, a 3D car model is shown instead of the flat picture of the car or a “Buy” button is added on top of an ad for a perfume bottle.

“Layar just runs on Glass.”

While Google Glass is running Android, it is not really easy to launch android apps from the user interface. Currently, you need to enable debug mode, which allows you to sideload applications and launch them. To our surprise, clicking “launch” in our development environment resulted in a correctly functioning AR display showing the full camera preview with attached augmentations in the corner of our eyes.

However, the user experience was terrible.

The problem is that the display of Google Glass is just a small screen in the corner of your eye, that you specifically have to look at by looking up. It is not completely immersive like what you would need for a true AR device. So the experience when running Layar’s vision based AR unmodified on Glass is that of holding your phone above your normal line of sight, looking up and at the same time trying to hold your head as if you’re looking at the magazine or object through the camera lens.

This confirmed our initial expectation that Google Glass requires a completely different mindset from any other platform we operate on. Rather than simply showing the content as we would on phones and tablets, we need to look at other ways of displaying the vast amount of content created for the Layar platform.

Google Glass UI

To figure out what type of user experience would work on Google Glass, a good first step is to look at what Google Glass offers out-of-the-box. Here we see a very simple user interface with big text and not a lot of content. The content is represented as screens (also called timeline cards) on a single timeline. This timeline contains the history of all actions the user has taken and notifications the user has received in chronological order. By swiping back and forth on the touchpad on the side of Glass, the users can easily scroll to the history of cards.

When tapping the side of Glass to enable it, you are first shown the home screen containing the time and the sentence “Ok glass.” Saying this command allows you to start actions like “take a picture,” “record a video,” “send a message,” etc. Triggering these actions will add new timeline cards to the history. Timeline cards can have menu options that are shown when tapping the touchpad. Some common actions include reply, share and delete.

This simple interface is all the user sees on Glass. There is no real concept of apps like you are used to on phones and tablets. Third party apps are really just services that interact with the user’s timeline.

Mirror API

The only official way to develop apps (“Glassware”) for Glass is through the Mirror API. This is a cloud based API, meaning that the software is not running on the device itself, but as a service on the Internet. This service is connected to the Google Mirror API servers, which allows interaction with the user’s timeline.

The possibilities of the Mirror API are quite limited. The most common use case that is covered is allowing services to add notification cards to the timeline with news items, messages or other content (including photos and videos). Glassware is also able to add contacts that can be used for sharing. This allows the user to share photos and videos with third party services.

For Layar, this was sufficient for us to create a simple prototype service that allows the user to take a picture, share it with a “Scan with Layar” contact, perform the visual search on our servers and push back the results to Google Glass.

While this was a nice proof of concept, the user experience is far from ideal. Currently, scanning an image with Layar requires the user to first take a photo, and then explicitly sharing it with Layar. Ideally, this would be a single action “Scan with Layar” that can be triggered directly, but the APIs don’t provide ways to combine taking a picture and sending it to a service.

Another problem with this flow is the fact that the Mirror API by design is asynchronous. The timeline on Google Glass is kept in sync with the Google Servers, but sometimes this synchronisation is slow due to bad connectivity or other circumstances. This is ok for sharing photos to social networks (as it will just synchronize once a connection can be made), but for Layar, the user will expect to see results fast.

Finally, the types of content allowed through the Mirror API are quite limited and static. The rich content created by publishers on the Layar platform is nearly impossible to show in a nice and useful manner.

Glass Developer Kit

At Google I/O, a Glass Developer Kit (GDK) was announced that would allow developers to write Android apps for Glass. While details are not given yet, this is supposed to integrate or at least launch real applications from the standard Google Glass user interface.

We feel that this will open a lot of possibilities for us to create a better, rich experience. The key thing will be to keep the UI very simple, similar to the standard Google Glass apps, and use the extra API possibilities to improve the flow and directness of the interaction with the Layar platform. So for example, you will be able to just look at a page in a magazine augmented with Layar and see a list of web links and videos that belong to that page appear in the corner of your eye.

Another thing that will probably be possible with the GDK will be to expose our big collection of geo-layers in a useful manner using the built-in sensors.

Layar will be actively using the GDK once it comes out and will provide feedback through the Glass Explorer program to make sure that we can make a nice experience, so we will be ready when Google Glass hits the consumer market.

In Conclusion…

Google Glass is an exciting new platform that will bring some nice new possibilities. Quite a few AR companies have announced that they will support Google Glass. At Layar, we do our research before making bold claims which set impossible expectations. No, true augmented reality is not possible on Glass at the moment. And no, the current Mirror API will not enable an AR platform like Layar (or any other AR platform) to provide a good user experience to enrich the real world. But with some effort and using the new GDK, Layar will be able to create a great experience to enrich the world with the digital content created for the Layar platform.

17 comments

25 Jun 2013

Heel interessant. Very interesting. Muy interesante.

25 Jun 2013

For see through head up display with real AR capability get in contact with Explore Engage

Diddi Eis25 Jun 2013

Interesting, thanks for documenting and sharing your use-case experience. I was balancing the suggestion "give them some time" on my tongue since I didn't expected Glass to be much of a true AR-experience yet. However, permanent server connectivity will be a huge obstacle for many developers.

29 Jun 2013

Thanks a ton for sharing with us so much information on google glasses. I am an <a href="http://www.adstuck.com/augmented-reality">augmented reality</a> freek, so i thank layar for keeping me updated on the same.

Full operational, it can be used by all people, ALL, disabled or not, whatever the disability including blind peopleHere a version for blind, with just audio (also innovating) but we already tested with a HUDhttp://videos.tf1.fr/jt-we/le-mercantour-s-ouvre-aux-non-voyants-6649984.htmlThe system is able to "know" not only in what you are interested with but it is able to "measure" the level of interest and then search, aggregate, sequences, format exactly according your real time preference, indoors as well outdoors.Is that can be considered as an AR device ?We are calling it a "behavioral interface".

1 Jul 2013

We to are running into the limitations of what can be done with the API and Glass. We are developing a Google Glass and Augmented Reality Golf application @ http://iCaddy.com #iCaddy_ @iCaddy_﻿

Thanks for sharing your Google Glass experience!I agree Google Glass is not an AR device (yet) and I believe that is the point. AR is still in very early stages, there isn't a 'standard' definition of AR so far, therefore Google didn't want to take any risk coming up with an 'AR device' which probably would have had lots of downsides let's say. I think the idea was to introduce a new device which is not revolutionary but it is stable enough so that the people will start trying it out and give feedback for improvements.I also agree it's not fully immersive but it will be. You can adjust the lense back and forth, you can even personalize them before placing the order and this is just the beginning.All in all I agree with what was written in the post and I think everything will happen step by step. We should not forget Google is a company and a company makes money. If Google Glass is not suitable for AR then we will use it for something else ...they will maybe call it an IR (Improved Reality) device instead of an AR device...who knows.

8 Jul 2013

Everybody is waiting to see the GLASS and their awesome features sooooooon !!!!

12 Jul 2013

Well done. Keep it up!

6 Aug 2013

Thanks for sharing your Google Glass experience!I think the idea was to introduce a new device which is not revolutionary but it is stable enough so that the people will start trying it out and give feedback for improvements.I also agree it's not fully immersive but it will be. You can adjust the lense back and forth, you can even personalize them before placing the order and this is just the beginning.

8 Aug 2013

Interesting, thanks for documenting and sharing your use-case experience. I was balancing the suggestion .

Henri Dussault25 Aug 2013

For now this over the picture AR for what my younger girl called the Harry Potter News Paper Effect might not be possible until a transparent OLED lens completly covering your dominant eye is made available with all the disadvantages and dangers this will bring to your driving and walking. But this is only important for marketing and advertising purposes in paper formats wich will dissapear one day anyway.Other applications of layar like scanning a city area or a movie poster or the front of a restaurant and provide you with any interesting information about what you are looking at or a short preview or a tour or automatically bring you to the related web site would be just incredible to have on these glasses with a simple GLASS SCAN command.I am sure this is what everybody would enjoy and where the future lies.Imagine walking around with layar app permanently on and the glasses shows you points of interest through out the city or even lets you select what it will show you. Just like walking for real through google maps. Imagine adds comming up to your screen when you pass near your favorite store.When layar process gains speed this could be possible.

giuliano fann10 Sep 2013

Google glass opens up new possibilities for future Augmented Reality and 3-D experience. Thanks for the info.

17 Sep 2013

With havin so much content do you ever run into any problems of plagorism or copyright violation? My blog has a lot of exclusive content I've either authored myself or outsourced but it appears a lot of it is popping it up all over the internet without my authorization. Do you know any methods to help reduce content from being ripped off? I'd really appreciate it.

17 Nov 2013

Hello, I am a developer in suboceana team and we have increased augmented reality in our book "decouvrir autrement" compatible with googles glass (lending by AMA). it's not a fake. Here are the videos.