Today, I was very excited when Thomas Wrobel sent me a draft of, “Everything Everywhere: A proposal for an Augmented Reality Network system based on existing protocols and infrastructure.”

Thomas has kindly agreed to let me publish his draft, to open a discussion on this topic. The diagram opening this post (click image to enlarge) shows, “An example of how collaborative 3D-spaces could be shared over existing IRC networks.” It is from Thomas’ proposal.The full text of his paper is included later in this post.

“Can we try to avoid a browser war this time?”

Thomas notes in the closing remark to his paper:

“I am absolutely confident in my belief AR will become at least as important as the web has, and probably a lot more so. It will also face much the same hurdles and challenges getting established as that medium did. But, speaking as a web-developer, can we try to avoid a browser war this time?”

Thomas Wrobel has consistently posted insightful comments on how existing standards could be used for creating open augmented reality networks. But he expressed concern to me that his work and this paper not be overplayed:

“I’m hardly a leader, I’m just an amateur with a load of ideas on AR-related topics, some which might be useful, others might become unworkable. I don’t want anyone to get the impression this is how I think it has to, or should be done.”

I have brought/am bringing up this topic of using existing standards and infrastructure where possible for open augmented reality networks in all my interviews with members of the AR Consortium.

And I am finding agreement on a point that Robert Rice makes, “there is no perfect, ultimate solution *now*, but we have to do *something* to work from and refine/evolve.”

Thomas Wrobel makes what I consider some crucial opening suggestions. I take my hat off to him for thinking about this early, coming up with some clear, elegant, and practical ideas, and doing the work to articulate these ideas so others can participate in evolving them. Massive props for that, many times over.

Good ideas on standards at an early stage of a developing industry like augmented reality are like spring sunshine and April showers for new crops. No one knows what storms and pests the growing season will bring – but water and sunshine (open standards) are always a good start. And, personally, I can’t wait to see how this new industry unfolds (see Bruce Sterling’s Layar Conference awesome keynote : “At the Dawn of the Augmented Reality Industry.”)

Thomas Wrobel is:

“a web developer working for a small, brand-new company called Lost Again, which mostly works on ARGs (That is, the alternate reality games, not the augmented reality games, although there’s probably going to be big overlap there in the future). We developed two educational ARG games for the Netherlands with a company called res-nova.”

I have been following Alternate Reality Games through the amazing work of Elan Lee and Fourth Wall Studios. Like Thomas, I think the intersection of ARGs and augmented realities is going to be very interesting. Thomas wanted me to point out that the website for his company with Bertine van Hövell, http://www.lostagain.nl/, is just a placeholder for now.
“Probably be up fully within a week or two. And, “despite the logo, we aren’t an AR company [yet], or a travel firm. The logos supposed to represent being lost in our minds.”

Thomas has been thinking about the topic of an open augmented reality network for a while now. He is an artist also known as DarkFlame and his ARN network is included in this augmented reality concept for 2086 he did in 2006 (click on image below to enlarge).

Mez in her brilliant brainsplosion on social tesseracting takes on the very definition of information:

“Tish, when you ask Robert “…what is your approach to delivering a massively shared real time [augmented reality] experience that is like Wave not confined to a walled garden?” that’s an extremely relevant question + one that needs to be addressed while considering the entirety of the Reality-Virtual Continuum. I’ve recently finished a series of articles addressing this: the framework I’ve developed is termed “Social Tesseracting.”

I have recently begun exploring the Google Wave Web of Protocols which are nicely outlined in this post by J Aaron Farr which includes the very interesting diagram below (so more on Google Wave in another post).

But, as Thomas notes, while he demonstrates his ideas using IRC (Internet Relay Chat) they reach Beyond IRC:

“As mentioned before IRC has some drawbacks, which are due to its age or method of working. As such, future systems might yet prove better alternatives for a open AR network. One example of such a system is Google Wave. It shares many of the advantages of IRC (open, anyone can create a channel of data, different permission levels can be set and its free), while avoiding some critical restrictions. (The data can be persistent). I believe some of the ideas I’ve mentioned, and possibly even the proposed protocol string could be adapted for Google Wave or other future systems. I believe overall the principles are more important then any specific implementation to get to them”

Also Thomas pointed that while he uses markers to illustrate some of his examples, they are just a method for tracking. What he is presenting is going to be transparent to the methodology of registration/tracking.

Tish Shute: You mostly use marker based examples but there is no reason why the principles you are suggesting will not be just as relevant as we move more into using more sophisticated image recognition tools is there?

Thomas Wrobel: No reason whatsoever. I mostly choose familiar markers as something that could be used now, with a lot of coding library’s already established for them. I think for most future AR use, markers will go completely…especially outside. Either things will be done purely by gps, object recognition, or the (in the case of advertising) markers will look like normal posters.

However, I do think traditional markers might “cling on” as being used for non geographical specific stuff at home. After all, if you need some reference points for moving mesh’s about in real time…(say, when playing a board game with a friend on the other side of the world)….then there’s probably nothing that’s going to be more practical then some simple bits of paper or card.

Everything Everywhere

- A proposal for an Augmented Reality Network system based on existing protocols and infrastructure.

by Thomas Wrobel

The following paper is my vision of a open AR Network and potential methods to implement it with existing technologies. Specifically I’ll be focusing on a potential for a global outdoor AR network, although the ideas aren’t limited to that.

Of course I call it “my” vision, but I’m obviously not the first to have many of these ideas. I have been influenced and inspired by many things…

[Some of Thomas Wrobel’s influences – watched and played. Images from Mitsuo Iso’s Denno Coil (Click to enlarge) top, below from the game “Metroid Prime,” and Terminator, and the last from Buffy the Vampire Slayer!]

The AR Network.

When I speak of a future AR Network, I mean one as universal and as standard as the internet. One where people can connect from any number of devices, and without additional downloads, experience the majority of the content.
Where people can just point their phone, webcam, or pair of AR glasses anywhere were a virtual object should be, and they will see it. The user experience is seamless, AR comes to them without them needing to “prepare” their device for it.

From this point forward, I will refer to this future AR Network simple as the “Arn”.

The Arn should be an inclusive, and open platform where any number of devices can connect to, and anyone can make and host their own location-specific models or data.
It should allow people to communicate both publicly and privately, and not have their vision constantly cluttered with things they don’t want to see.

There’s two old, existing paradigms that I think can help reach this goal when they are combined.

The Internet Relay Photoshop.

IRC, or Internet Relay Chat was a chat system designed by Jarkko Oikarinen in the late 80′s.

Its a system where people meet on “channels”, they can talk in groups, or privately. Channels can be read-only, or open to all to contribute to. There is no restriction to the number of people that can participate in a given discussion, or the number of channels that can be formed. All servers are interconnected and pass messages from user to user over the network.

To me, this relatively old internet technology is a great template, or even foundation, for how the Arn could operate. Rather then text being exchanged, it would be mesh data (or links to mesh data), but other then that much of the same principles could apply.

People could join channels of information to view or contribute. Families could leave messages to each other scribbled in mid-air on private channels. Strangers can watch AR games being played between people in parks. People going into a restaurant could see the comments from recent guests hovering by the menu items.
None of this would have to be called up specially, if they are on the right channel when it was broadcast, they will see it.

The IRC paradigm becomes particularly powerful when combined with another one common to many computer users; that of a “Layer” in an art program, such as Photoshop or Paint Shop Pro.
As most of us know, layers allow us to separate out different components of a piece of art while editing, either to focus our attention on one piece, or to make future editing easier.

Now what if we simply have each “channel” of information represented as a layer?

Click to enlarge image above.

Having channels corresponding to layers is an easy and intuitive way for the Arn to operate. The user can login and contribute data to any channel, like IRC as well as adjusting the desired opacity and visual range of each layer, like they would a layer in Photoshop.

In this way they can get a custom view of the world, both with shared and personal AR elements visible at the same time.
They would not have to switch between various overlays to their world view, as they could see many at the same time.

Persistence of Data

With IRC or IRC-like system to communicate the data sent is mostly temporary data…broadcast on the fly from user to user and device to device. Retained in the users local logs, but not “hosted” anywhere.

I think for the majority of day to day purpose’s this is not so much a drawback, but actually desired for AR. Most casual communication doesn’t need to be recorded permanently in 3D space and, indeed, if it was, the cost of running such a service would increase exponentially with users and with time. Not to mention, our visual view of the world would get very cluttered very quickly. Imagine what your monitor would be like if it kept a history of every window you have ever opened and their positions!

So for most cases AR space should be treated like a 3D monitor letting us display many pieces of data from remote and local sources, and even to share them with others, but not being, by default, a permanent record for it all.

Most data will be analogous to pixels on a display, and if kept in records its only on the clients devices, not on the network itself.

However, occasionally we do want 3d data analogous to a web-page, such as (in the example above), the map layer. Data here should be persistent and visible to all that have that layered turned on. I see no reason why hosting this data needs to use anything else but standard web-hosting with the (read only) #channel on the Arn merely providing a route to the data.

As the user logs onto the channel, the server, using a chat-bot, can send them a list of meshes with location data attached, and the Arn browser can simply pick the data to display that’s local to them. (Note 1: By doing it this way around, it allows some degree of anonymity to be possible, rather then the server knowing exactly where you are and feeding the specific correct string to you.)

We simply need to establish standards so this data can be pulled up and interpreted.

For instance, this standard could be as simple as a XML string pointing to a KML file on a server. This could then be then displayed in the users field of view at the co-ordinates specified.

In this way permanent data tied to locations, such as historical overlays or maps, could co-exist on the same protocol as temporary data such as mid-air chat’s or gaming related meshes.

There is also no reason why this shared-space/personal spaces based on channels of data has to be restricted to things given absolute co-ordinates.

(Different ways to access the same mesh)

It could work just as well with Markers and thus relative co-ordinates.

This would be mostly useful for indoor use, letting people logged onto a channel see the same meshes as everyone else on the markers. Thus allowing multi-player AR games, or AR games with observers very easily.

For example; games like Chess could be played between people with no additional code needed; You simply have a set of markers for only your own pieces, and as you move them the channel updates with the new positions, which are displayed in place in your opponents field of view.

This sort of game comes “free” with just having a generic system of shared space supporting markers.

It would also allow AR adverts down the street or in magazines to be viewed by simply logging onto the right AR channel

If markers are designed with URL data in them, this could even be a prompted or automatic process.
“There is visual data in this area on the following channel; #ABCD would you like to view this channel?”

Pros and Cons of using IRC or IRC-like systems

Pros;

• Anyone can write a IRC interface software.
• Anyone can create new IRC channels without cost
• Channels can have read and write permissions set.
• Users can easily have multiple channels open at once.
• Already established with thousands of severs worldwide.

Cons;

• 500-or-so character limit. 3D data must be linked too, not sent.
• Slow update rate. Lines of data can take a whole second or more to send.
• Non-persistent. Good for a 3d-view, not good for storage.

An example of how collaborative 3D-spaces could be shared over existing IRC networks;

Click on the image to enlarge

While in the long run I would hope for a dedicated AR network to be developed, with greater flexibility with persistence of data, there is a lot that can be done with the existing IRC system to implement the ideas mentioned above.

Below I will show an example of simple, crude, pseudo-protocol that could be fairly easily implemented to create shared AR spaces broadcast across IRC channels.

Its important to note, the goal here isn’t to exchange the mesh data itself on IRC, its to exchange links to the data.

Exchanging the mesh data directly within the 500 character IRC limit would be very hard, and liable to errors.

It’s also a waste of network bandwidth, as many people logged onto the channel might not have that object in their field of view, so their clients should not bother downloading it. (it should be up to the client browsers when to anticipate and cache mesh data).

Proposed Basic XML link exchange for AR;

Principle;
As user creates or changes an object, the clients software posts a simple xml formatted string to
the IRC channel.
Anyone logged into that channel then sees that mesh displayed in the specified location.

This string allows other users client logged into the channel to automatically load the object from the URL and display it at the correct position in their field of view.
If the permissions are set to allow it, they could then move the object themselves, with the update being feeding back seamlessly to other users on the channel.

The objects posted are given an ID, which can be just the posters name, followed by a unique object number for that name. These unique ID’s would allow clients to track different instances of the same mesh, as well as making it easy to implement permissions. (if only the poster should be allowed to move this object, then the clients simply check if ID matches the user name posting the update. If its not, they can ignore it).

Next the objects need to be linked to a mesh.

The location of the objects mesh doesn’t have to be a fixed remotely-hosted url, it could be an IP address and port number of the user posting the mesh,hosted by the application posting the link to the channel.

The objects co-ordinates, likewise, need not be specified as absolute gps co-ordinates, but instead could refer to generic Marker.

Loc=”(49.5000123,-123.5000123)”
Loc=”Marker1”
Or relative to a marker;
Loc=”Marker4 (+0.0023,-0.0023)”
Or relative to a default plane;
Loc=”Default(+0.213,-0.123)”

The AR Browsers could then handle the association between the Markers pattern and its Name.
This way the markers are reusable, they do need unique markers to be printed for every new bit of AR they want to look at.
Users could just keep a set of generic markers handy, which they could simply assign to be Marker1,Marker2 etc for any AR use. (Note 2: As mentioned above specific makers could also contain a default ID name and channel built into their data, letting the Arn browser simply prompt the user if they want to see the model even if they aren’t in the right channel. This set up would be most useful for paper and even billboard advertising.)

The Default location could be a settable region, or marker, on the clients browser that defines a playable/user-able area in the field of view. Mostly useful for home use, this could typical be a square region on a users desk.

So, in the chess-game example, the client of the person making the moves simply updates the position relative to the Default every time they move their marker (which is tied to a chess piece mesh).
Then the (non-owners) clients software could automatically display it relative to their Default plane. This would make games like Chess, Checkers, Go or any other game involving merely moving objects about automatically very intuitive and easy to set up.

So by having meshes settable to absolute gps, marker-relative, or default-relative locations, reduces the bother necessary to experience AR content quite considerably, and makes “non-geo-specific” AR applications and games trivial to implement.

Next is permissions.

Mesh-permissions would be a simple string saying who else can update the data, if anyone.

By default you could only update or move your own meshes. (identified by the ID of first posting). If you attempt to update anyone else’s, their clients would just ignore it.

Thus in a game of chess, you can only move your own pieces. If you attempted to move your opponents (by reassigning your own marker to their pieces Ids), the clients would just ignore that assignment. You’d only be fooling your own system.
Likewise, when pinning a message in mid-air for your friends to read, no one else can change that message without your permission, although copying it would be easy. (Note 3: It’s important to note this sort of object-specific permission system is in addition to the global-permissions, or “user-modes” it’s possible to set for the IRC channels and users as a whole.)

Finally, as object data could change within all sorts of time-scales, the easiest way to keep everyone logged in up to date is to just have a time-stamp of when each model was last updated.

LastUpdate=”12/12/0000,2012:12”

This would not necessarily be the same as the XML string post date, because the models mesh might not be updated, but merely moved, and in such case the Arn browser shouldn’t redownload the mesh.

This sort of arrangement could be used as a standard today, and users wouldn’t have to constantly download special AR programs to view a single AR mesh.

In the long-term I would hope for more advanced methods to manipulate Arn-content online, analogous to Dom manipulation in web-pages. But for now, we should at least establish standard methods for devices to pull up meshes and overlay them in the correct position.

So, having a layered system could give the user a seamless blend of dynamic and static data with which to paint their world with.
I believe this is all relatively easy to achieve using modifications of existing web technology, combined with some basic graphics systems.
Local Data:

However, so far I have only talked about remote data.
What of programs originating on the device itself? This is, after all, how most AR software we have at the moment works.

I think, that just like the remote channels, local software should also be blended into the same list of layers. People shouldn’t have to “Alt+Tab” out of one view of the world, to see another.
They should be able to see both at once, if they wish.

For instance, if your playing a AR game, why shouldn’t your chat window be viewable at the same time?

If you have skinned your environment with a custom view of the world, why shouldn’t you also see mapping or restaurant recommendations?

So local data and remote data should be blended in the same view.
How can AR software – of which I hope, there will be thousands – seamlessly be expected to layer their graphics, not only with the real world, but with each other, and with online data too? Will games and software makers need to co-operate to allow their graphics to be integrated together with correct occlusion taken into account? A tall order, no?

I must confess though, my technology knowledge fails me here.

I can only guess special graphics drivers, or 3D APIs, will have to be developed to let programs share their 3D world with that of a Arn browser.
Maybe programmes should simply treat themselves as a local-sever which the browser can connect too, and let the Arn handle all the rendering itself (although I imagine many games designers would find this quite limiting).
So I leave it as an exercise to the readers to discuss and propose the best methods by which this vision of a layered world could be realised..

Beyond IRC:

As mentioned before IRC has some drawbacks, which are due to its age or method of working.
As such, future systems might yet prove better alternatives for a open AR network.
One example of such a system is Google Wave.
It shares many of the advantages of IRC (open, anyone can create a channel of data, different permission levels can be set and its free), while avoiding some critical restrictions. (The data can be persistent).
I believe some of the ideas I’ve mentioned, and possibly even the proposed protocol string could be adapted for Google Wave or other future systems.
I believe overall the principles are more important then any specific implementation to get to them.
Summary;

⁃ In order for AR to flourish the user shouldn’t need to download a separate application for each mesh they want to see.
⁃ Having url’s embedded into QRCoded markers which point to standard mesh files like dxf or kml would be a way to do this right now. The QR code would only have to be seen precisely in shot once, then its borders could be used like a standard marker.

⁃ An augmented view of the world needs to support visual multitasking, and having layers of information is the best way to do that.
⁃
⁃ Methods need to be devised to allow drastically different software to contribute to these layers, without restricting either the software’s rendering ability’s, or the users ability to pick and choose what layers of information he wants to see.
Last point;

I am absolutely confident in my belief AR will become at least as important as the web has, and probably a lot more so. It will also face much the same hurdles and challenges getting established as that medium did.
But, speaking as a web-developer, can we try to avoid a browser war this time?

20 Comments For This Post

Thomas, nice job working through things and coming up with a thoughtful approach. I’m not quite clear if you are advocating using IRC model as a method or if you are suggesting literally using IRC servers/client tech as a communications and sharing method.

I agree with many of your points, and I really don’t want to see a browser war either, and I certainly don’t want to see a walled garden battle. You make some great points here, for example, the difference between hosting certain data types/objects on a server somewhere, or keeping data (like chat) transient and not persistent in memory.

One of the suggestions I would make is a minor change…I think channels should consist of multiple layers (or rather, layers are encapsulated by a channel).

Thanks
Well, basicaly Im advocating giving it a go on IRC -right now-
And if the platform proves unworkable due to technical constrains we can shrug out shoulders and move on, having learnt from the experience.

But I do think IRC has a surprisingly large number of the propertys we need for a public, open, sharable AR world. And I really dont think it will take much to start experimenting on it.
Obviously, looking at end-to-end tech, theres better ways it could be done….but better ways that are here right now, with tens of thousands of severs worldwide?

“One of the suggestions I would make is a minor change…I think channels should consist of multiple layers (or rather, layers are encapsulated by a channel).”

Well, its a question of terminology here really.
An IRC, or IRC-like system is already divided into what they call Channels, so it made sense to me to simply have each of those channels as a layer.
Now, we could have groups of channels by topic, or a (ar) channel being actualy group of (irc channel) layers….really the difference becomes more verbal then tech.

I think the important thing is that data can come from different people,or different companys, and by displayed selectively as the end-user wishs.
Its of course, also possible, to block or selectively allow seeing mesh’s on user-level as well as a channel one.

Thank you Robert and Thomas for taking the lead in this. Jeremy could you add a link to the papers you published in the parsons information design mag, if it is available on line? I am trying to collect together the thinking in this area of open augmented reality networks.

Hmm…darkflame +1 for open systems thinking and trying to reuse exisiting, already pervasive protocols. However I need some clarification on a couple things, since you’ve said this is a draft I hope it’s ok to poke at it?

1. How are you defining AR here? Are you thinking of the traditional CG notion of overlay graphics objects tightly registered to physical world geometry (think texture mapping)? Or the looser, locative-post-it style that is getting so much airplay with GPS+compass apps like Layar & Wikitude? Your approach might be useful for the latter, but I don’t see it working in the more rigorous tightly coupled case.

Also, I guess you are excluding audio AR ala Soundwalks?

2. How does your model handle dynamic, interactive AR objects? For example, if a mesh is a simple message to Alice with an instruction to delete after she reads it, how does that work? Or, if a mesh is supposed to only appear when the audience is within 20 meters of it, how would you do that? (Although it’s not hardcore graphical AR, I suggest looking at HP Labs’ mediascape model for ideas about how programmable experiences can be authored and played: http://www.mscapers.com).

3. Timing stuff. From your description, it seems like a client would have to be listening on the right channel at the time a mesh event was published. Otherwise they would miss it? And what if a mesh is changing rapidly, do you just flood that channel with updates? What’s the temporal granularity of that? What if accurate timing is important to the application?

4. Object attributes. Seems like your objects want a lot of additional (possibly optional) metadata in them. Just a few examples: altitude (how high to display that writing in the sky?), effective radius (how close do I have to be to see it?), visual priority (to use your photoshop analogy, which layer is on top in a pile of co-located objects?). Etc etc.

Just a few thoughts…keep on pushing the open, web-based nature of things.

“Nice job, Thomas. From a bit more meta-view, how do we get resources to people to try ideas like this? We need a way to seed fund promising ideas to help speed up the development of the medium.”

Cheers
Well, I’m guessing most of the people with resources and in the AR field are pursuing their own ideas. They might take inspiration or some aspects of ideas posted, but if they are funding themselfs and putting the time in, they are naturally going to prioritize their own methods. (as they should)

Naturaly, I welcome anyone taking, adapting and using my ideas and I hope they do (if they are workable). But I think at the end of the day, if we are going to get anywhere its going to have to be US….the AR community, which starts working ourselfs on this stuff. At least, perhaps, to get a working proof of concept.

I haven’t the skills or time to develop my ideas fully on my own, but I would welcome co-operation with anyone else. My skills lie mostly in java codeing for websites. (Using GWT), but I also have action script knowledge, Delphi knowledge, some C.
I haven’t ever interfaced with an IRC network though, or implemented AR tracking.

“However I need some clarification on a couple things, since you’ve said this is a draft I hope it’s ok to poke at it?”

No, absolutely.
Id rather my paper be poked completely full of holes rather then ignored. Even dismissing possibly routes is progress in my book

“1. How are you defining AR here? Are you thinking of the traditional CG notion of overlay graphics objects tightly registered to physical world geometry (think texture mapping)? Or the looser, locative-post-it style that is getting so much airplay with GPS+compass apps like Layar & Wikitude? Your approach might be useful for the latter, but I don’t see it working in the more rigorous tightly coupled case.”

Both.
Is it not simply a case of progressively more accurate co-ordinates?

I think the location in the xml could be specified to an arbitrary number of decimal places and get more accurate as technology’s improve…or am I missing something here?

The update interval I can see being a problem for objects moving too quickly, mind you.
You would get a lag/jerkyness if you tried to skin yourself while walking. (No AR clothing yet…unless your wareing a marker)

But for static or slow moving things I think it could work no mater how tightly the CG is tied to the object.
Like, for example a historic image of a building that has since been demolished, or a map of the water-pipelines under the street, or star constalations. (hmz…maybe the last one would need a rotational variable added too, else youd need to reload the skydome mesh every time, rather then merely rotating it).
But I think in principle all these examples….as well as “air writting”…could be done.

“Also, I guess you are excluding audio AR ala Soundwalks?”

umm..yes. hmz.
I dont really think of it as AR, but I guess, to the letter of the word it is.
I guess the system could be expanded to include audio….rather then a mesh file being specified its a wav file tied to a location? Would be trival I imagine and could be added as part of the spec. But its not a priority too me.

“2. How does your model handle dynamic, interactive AR objects? For example, if a mesh is a simple message to Alice with an instruction to delete after she reads it, how does that work? Or, if a mesh is supposed to only appear when the audience is within 20 meters of it, how would you do that? ”

Well, in this system most meshes would be temporary things hosted on the device sending it.
When I was writting it all down, I was thinking of permisions in terms of letting people move the location of a mesh, or even change the link to its source. (and thus change what it looks like. So, if a mesh is localy hosted by one person, the other persons client could edit it, and post a new string back pointing to their edited version).

However, I must admit complete deletion didnt occur to me.
I can think of a few solutions, but these are just of the top of my head, so they arnt as thought though.

It could be done by simply letting that person edit, and then Alice’s client reposts the xml string, removeing the link completely or purhapes replaceing the url with just “(deleted)”.
Bobs machine could see the new link posted, see that the person is the one with edit-permisions, and then delete it.

Of course, Eve (Evsdropper) could use the the old posted link to see the message, before Alice has a chance to signal to Bob to delete it. Alices client would have already posted the string saying effectively, “this is deleted”, so all clients on the channel that are behaving themselfs it would stop existing for them. But, naturally, if Bob isnt there at the time this signal goes out , the link/hosted file will still be there for anyone to look at by clients not obaying the rules…..only, of course, if Bob isnt there the file isnt accessible anyway!

The problem would come if Bob lose’s connection at the time the deleted message is sent, and then comes back online. In this situation, perhaps Alices client should notice Bob is offline, and repost the string when he comes back online?

Overall though, as far as security is concerned, messages that “self destruct” on this system probably cant be trusted. The security comes from permisions of the channel itself, and maybe even the existance of the channel. Eve would have to know the name of the potentially private channel between Alice and Bob, and then Eve would have to crack the IRC permissions to gain entry. (and, purhapes harder, gain entry without being noticed?).

I’m not sure what security is like over IRC in terms of being able to invisbly snoop on other channels set to be private; probably not that good.
(However, I think there is some security in the fact that messages are transient rather then hosted.)

If more security is needed, purhapes clients could have a restricted list of IPs allowed to view a particular mesh; This list of IPs dosnt need to be posted at all. It would just work like a firewall for the client hosting the mesh.

I’ll check out http://www.mscapers.com now, then get back to you later on your last points
Thanks for the interest/feedback

Well, I’m obviously late to the discussion, but I want to raise a point that I don’t think is discussed enough.
The thing is, we are all for making AR as open as possible and letting anyone create his/her own content. That would obviously lead to an information overflow. I know that Robert Rice works on filtering spam. However, it seems to me that even more importantly we need to devise an “AR search engine”, pointing to layers that may be of interest to us, according to our “past searches”, current location, time of day (restaurants at noon) and so on. This becomes even a harder problem to solve when considering “transient AR” as Thomas proposes.

I have a post in a draft status about this issue for the last couple of months, but I clearly haven’t cracked this problem yet. Would appreciate your opinion …

@rouli – If the number of layers in popular areas get to be overwhelming, then yes, a layer search protocol will need to be developed. But I imagine that it would be similar to a google search algorithm.

I think we need to differentiate between “layer owners” and “content owners” similar to the difference between updating wikipedia and owning the site. Creating a layer should be slightly prohibative, so we don’t have overload, but adding information to the layer (restaurant reviews, directions, etc) should be easy.

@ThomasCarpenter – it can’t be like google, you are not looking for a textual string and there’s no linking for assessing a layer’s worth. best case scenario – it would be some image based search engine, that will take your current position and the image of whatever you are looking at, and will return relevant layers according to the number of “subscribers” they have (I imagine some people will turn some layers on permanently because they will be very useful to them), how much those subscribers are like you (like amazon’s recommendation engine) and how specific the layer is in describing your current view (a layer with a description of the mona lisa is more specific than a layer with details about the louvre once you are inside the museum).

“3. Timing stuff. From your description, it seems like a client would have to be listening on the right channel at the time a mesh event was published. Otherwise they would miss it? And what if a mesh is changing rapidly, do you just flood that channel with updates? What’s the temporal granularity of that? What if accurate timing is important to the application?”

Yes, this is the limitation of the IRC system.
The granularity of time cant be guaranteed below 1 second. In fact, a second cant even be guaranteed. So, it would be pointless to post quicker then that interval.

Even posting at that interval would be flooding the channel somewhat.
So certainly this system wouldnt be good for things that are speed-critical.

There could be possible “patchs” that could be done to the system to allow more rapid updates.
Currently I preposed the XML in IRC points straight to a hosted mesh of some sort.
I guess there could also be an INI file read at the same location, that contains a integer relating to the update interval. The client reading the file could then recheck it according to that interval independently of the IRC system, and the host could alter a location specified in that same ini as often as it liked. Effectively in this case the IRC is bypassed once the connection is established.

This solution does lack elegance somewhat though, and could be strain on the client if the mesh is both rapidly updated and viewed by lots of people.

(I also thought about putting an update integer in the xml string itself at one point, but thought the string was getting a bit long, and wouldn’t be a big improvement over the ini as youd still need many direct connections to the host)
–

As for being on the right channel at the right time, yes, that would have to be true….more or less.
Its quite possible to have chat-bots that could re-broadcast either the backlog or predefined data at certain intervals. This would be more usefull for Map-style channels or other mostly-static content.
Clients could even rebroadcast stuff back to eachother. (purhapes there could be a “request backlog” command of some sort, which clients could choose to obay, at the discretion of the clients settings).
Theres certainly a lot of expansions and refinements needed to my proposal. Patching away weakness’s, and expanding functionality.

I think the key to all this though, is I’m seeing the Arn less of a 3d form of the internet, and more like a screen. Any perminate data isnt “stored” on the screen, the screen is merely a route to the data stored elsewhere.

“4. Object attributes. Seems like your objects want a lot of additional (possibly optional) metadata in them. Just a few examples: altitude (how high to display that writing in the sky?), effective radius (how close do I have to be to see it?), visual priority (to use your photoshop analogy, which layer is on top in a pile of co-located objects?). Etc etc.

Just a few thoughts…keep on pushing the open, web-based nature of things.”

- Yes, your quite right. GPS + Altertude would have to be needed.
Of course, you could store that stuff in a KML file and use that for the mesh, but then you lose the flexibility of movement, and letting other people move it without redownloading.
Not having alternatude, and possibly rotation, was a big oversite on my part.

- With the exception of the ID and URL, I think all the other metadata could be optional. Even the location default plane relative the viewer. (purhapes usefull just to show someone an object…allthough in this case its not really AR at all).
Permissions could default to None, Timestamp to the point first broadcast.

==

Effective radius, however, I think should not be broadcast with the object, and instead default with by the clients viewing the data.

Their viewing/browser software could have a simple “range” setting for instance.
Possibly even with an auto-fade with distance (as many videogames used to stop objects appearing suddenly).
I’m quite keen to let the browsing software control the viewing options for the layer for two reasons;

a) The viewer should have complete controll over what they see. (If I value one layer more then another, I should be able to have it stronger in view)

b) Various clients will have different specs. Some devices might be able to have thousands of things in view at once, others only a handfull. Having the viewing range controlled at the device-end makes dealing wtih this much easier.

As for visual priority; Imho, it should be treated exactly like real life.
The distance determains it normaly, and if one object is inside enough, then you cant see it.
However, if each layer can be set by the viewer (like Photoshop) with an opacity, viewrange and possibly even style (wireframe/solid/textured)….this should allow a few things to be visible overlaping.

–

Sorry for my slowness replying, btw.
Your bringing up some good points, and many of these Im having to think about.
If I do an update to my paper, I will certainly be putting a lot of this in.

Theres no real downside with having thousands, or even millions, of channels, provided we only have to see what we want.

The fact its transient means most of it, by its very nature, wont be searchable…but I think thats an advantage. I see AR as primarly like being a computer moniter,but 3d and all around me.
I should be able to share my expirence with others, or keep it private….but by default the data shouldnt be hosted or searchable by others.

If it was,
Aside from the (huge) cost and privacy issues this would bring, it would also make a huge volume of searchable data absolutely rubbish.

Instead, I think its better for the vaste majority of it to vanish into the ether after use…just like pixals on our 2d screens.

However; Perminately hosted stuff could and should be searchable.

Thats quite correct, and companys like Google will have to develope means to do it.
The most popular/well kept channels should be ranked higher.

This is certainly a problem, but I think its solvable.

One way would be for perminately hosted channels (ie, those with their own sever) to have a standard chat-bot which not only broadcasts mesh data, but also (on request) can give google (bot) some information regarding the channel.

” it would be some image based search engine, that will take your current position and the image of whatever you are looking at, and will return relevant layers according to the number of “subscribers” they have (I imagine some people will turn some layers on permanently because they will be very useful to them), how much those subscribers are like you (like amazon’s recommendation engine) and how specific the layer is in describing your current view (a layer with a description of the mona lisa is more specific than a layer with details about the louvre once you are inside the museum).”

Actually, I should have read that first, you hit the nail on the head there.
I do think chat-bot like things would need to be used to comminicate some of this data back to the “search engine”. Obviously you couldnt trust them for reliable statistics, but you could perhaps have some tags for topics or to point the search system in the right direction.

As for an Amazon-like “you liked this so you might like this system” well..umm..

We would love to discuss how we can make this happen. We have implemented a bunch of these things..

As you know we have a layer approach. We don’t host any data. We just connect to the layer and get it. We made some specification on how to get information from a layer. Moving this into an open environment would be great.

Did you have a look at our API? Would love to hear what you think of it and how we could move into an more open environment. http://layar.pbworks.com

Should we sit down together and take some steps instead of talking about it?

re. Registration of AR content to physical reality.
Varying levels of precision are needed sometimes more than others. ie. a kml mesh of a building needs to be closely tracked to the real building to attach AR components; but on the other hand, a chat window only needs to be pointing at/near the users that are using it (and could move to make room for more important things)

re. Searching for good content/layers/channels/users
Thomas K Carpenter: “I think we need to differentiate between “layer owners” and “content owners” similar to the difference between updating wikipedia and owning the site. Creating a layer should be slightly prohibative, so we don’t have overload, but adding information to the layer (restaurant reviews, directions, etc) should be easy.”

I think I follow your logic, but we should NOT be architecting in barriers to development. This field needs all the user generated content it can get, as quickly as possible.
Overload is going to be a problem, just as it is on the internet at large, so we should be piggybacking on the research there. ie. the information load is only getting heavier-we need better filters
I would prefer to wed AR layers to social network mechanisms of ranking content, ie. layer == Facebook group, Twitter hashtag, Twitter user, Google Alert
I’d like to use these in the AR context the same way I currently use them-I’d ‘subscribe’ to an AR layer/channel/feed (#urbanexploration, say) somehow and whenever I’m near some information with a strong locative component (“This manhole cover leads to pretty underground grottoes”), it would pop to the top of my stack of incoming information.
This is somewhat like RSS feeds are now, but arranged by location and time, not just time. So a new piece of locative information may be ranked higher than older ones in the same location.

re. Would the content be searchable?
Darkflame: “The fact its transient means most of it, by its very nature, wont be searchable…but I think thats an advantage. I see AR as primarly like being a computer moniter,but 3d and all around me.
I should be able to share my expirence with others, or keep it private….but by default the data shouldnt be hosted or searchable by others.”

I don’t think this is likely, unless you’re just talking about being inside the context of your own mobile computer. Anything that travels over a public network (pure data network or social network) is going be highly valued in the aggregate, just look at the importance attributed to Twitter search results. With the coming hype and hungry startups, the odds are very good that someone will think it’s important enough to cache.
What will be the AR/locative style of search? We’ve evolved our idea of search to include realtime stuff down to the granularity of single sentences, and aggregating stuff on the fly with hashtags-what happens when this information has a location attached? Obviously Starbucks would like to know when you trash talk their advertising on your social network as you walk past the store..

Protocol/UX question: If I’m driving past the store, how do I tag the location of the store and not where I am when I press Enter? (or the Starbucks that I’m talking about, not the Starbucks I’m outside when I press Enter.)
We need some clever context sensitive computing here: My mobile hears me laugh/sigh/shudder in visceral disgust and marks that location and time. A few minutes later when I press Enter it reads “Starbucks” in my text, searches around for a Starbucks near me, ahead of me, behind me and works out that I’m not talking about visiting the one ahead of me, still driving and won’t be stopping at the nearby one, therefore I must be talking about that one I laughed at.

Something else to keep in mind: There are going to be times where you’ll want to see locative AR information whereever you happen to be, divorced from it’s location. Better design for that too.

Isn’t this IRC content url distribution system a lot like DNS? Maybe this discussion needs to reach over to the protocol designers camp, get this into a Cisco trade journal

[quote]We would love to discuss how we can make this happen. We have implemented a bunch of these things.. [/quote]

Yes, on investigation I see that! Its so hard to stay upto date with progress, but thats, of course, a good thing

[quote]
As you know we have a layer approach. We don’t host any data. We just connect to the layer and get it. We made some specification on how to get information from a layer. Moving this into an open environment would be great.
[/quote]

Yes, it would indeed, and its nice to see those in the industry keen on this direction.
I think Layers doing a great job with getting hosted data into peoples “view”.
Some of my ideas might just help allowing more transient data…such as notes in the air between people…work with Layer and systems like it.

“Did you have a look at our API? Would love to hear what you think of it and how we could move into an more open environment. http://layar.pbworks.com”

I have scanned over it, but I havnt had the time to invest fully into examining it.
I’m quite impressed from early impressions though.

“Should we sit down together and take some steps instead of talking about it?”

cnawan – Sorry for the late reply, been a bit overwealmed with responding to people accross various mediums, and Ive been busy with Lost Again as well.

“re. Registration of AR content to physical reality.
Varying levels of precision are needed sometimes more than others. ie. a kml mesh of a building needs to be closely tracked to the real building to attach AR components; but on the other hand, a chat window only needs to be pointing at/near the users that are using it (and could move to make room for more important things)”

True.
I guess any spec for the rotation should include a mode for “just face the viewer”
Like the old Doom-sprites.

“Overload is going to be a problem, just as it is on the internet at large, so we should be piggybacking on the research there. ie. the information load is only getting heavier-we need better filters
I would prefer to wed AR layers to social network mechanisms of ranking content,”

Do we need to worry about this at all though?
Surely ways to filter and rank layers will emerge on their own, just like the internet as large.
I think it will come about naturally.
My own site, Rateoholic, is a generic review/rating site. Adding the ability to rate layers and correlate them to peoples taste would be a fairly trival addition.
I expect a lot of methods will pop up.

“I’d like to use these in the AR context the same way I currently use them-I’d ’subscribe’ to an AR layer/channel/feed (#urbanexploration, say) somehow and whenever I’m near some information with a strong locative component (”This manhole cover leads to pretty underground grottoes”), it would pop to the top of my stack of incoming information.
This is somewhat like RSS feeds are now, but arranged by location and time, not just time. So a new piece of locative information may be ranked higher than older ones in the same location.”

Indeed.
Under the IRC like system this would be a trival mater of the client automaticaly logging into channels you pre-set as subscribing too.
Of course, with no persistant data in the severs memory, you would only get data since you joined, or that a bot would broadcast back to you.

But you certainly get easy access to the channels you want around you. And it would, again, be rather trival for the client to give you a highly customised field of view to suit your tastes.
eg. Some channels could appear stronger/more opacity then others.
Things with your name in, or from your friends could show up at a far larger range.

I think as soon as a standard is even vaguely established, there will be an explosion in user-options to try to give people the best experience for them.

Google Wave protocal, incidently, would allow the sorting of data by age as well as subscribing to specific lists too.

“I don’t think this is likely, unless you’re just talking about being inside the context of your own mobile computer. Anything that travels over a public network (pure data network or social network) is going be highly valued in the aggregate, just look at the importance attributed to Twitter search results. With the coming hype and hungry startups, the odds are very good that someone will think it’s important enough to cache.”

Maybe.
But your rapidly talking about an insane amount of data here.
IRC severs are run for free and dont keep long logs, if any, because it would get too costly.
AR load could see their data easily go up by a few orders of magnitude. To cache it would mean bigger facilitys and potentialy charging access fees.

Now, for sure, there *is* a lot of data that would be valuable. I can certainly see local buisness’s like restaurants being keen to get feedback from their customers.
But for most of these purpose’s it would be far easier for them to just participate in the correct layer themselves.
I’m sure lots of food-review layers would pop up, restaurants could see real time comments on their buffet (“Beef too salty”) from customers without having to search huge volumes of random conversations. This would also give them more current data then looking over logs.

Its also worth noting that under my proposed system a lot of the data wouldnt be accessible at a later date anyway.
IRC acts as a means to access the data, it gives a IP address and port number…not the mesh file itselfs. A temporary mesh is kept on the client broadcasting the message…as soon as thats gone, no one can access it anymore.

In order to cache this data, your not just talking about saving IRC logs, but also talking about downloading all the links within them. (and thats assuming the client broadcasting the link to the mesh, isnt also locking it to just be accessible by certain friends IP address’s).
Its certainly possible. But its not an easy or cheap task for a host to do. Especially not as it could be thrown off by something as simple as using a custom font (as the font is forming a 3d mesh, and its the mesh thats hosted, your talking about data-havesting using a type of 3D-OCR here).

Overall; I think your right, it -will- be done.
I dont think people could consider things they say totaly secure….they would need to trust the sever as much as they do their ISP or email provider for that.

But I dont think it will be automatic, and I think it will be far too costly to do generically for all data exchanged.

“Protocol/UX question: If I’m driving past the store, how do I tag the location of the store and not where I am when I press Enter? (or the Starbucks that I’m talking about, not the Starbucks I’m outside when I press Enter.)
We need some clever context sensitive computing here: My mobile hears me laugh/sigh/shudder in visceral disgust and marks that location and time. A few minutes later when I press Enter it reads “Starbucks” in my text, searches around for a Starbucks near me, ahead of me, behind me and works out that I’m not talking about visiting the one ahead of me, still driving and won’t be stopping at the nearby one, therefore I must be talking about that one I laughed at.”

Thats interesting stuff.
To be honest though, I’m not sure Id ever trust a system to get that right.
Maybe the software could display a list of options, with context-sensativity putting the most likely one at the top?
Like some predictive text systems.

This would also be of use while gps isnt accurate enough for a precise location.
eg. If your in a small shop, surrounded by other shops, and you want to review the one your in.
The GPS system might not be able to tell exactly where you are, so it could give a list and you hit enter on the correct one.

Obviously our positioning systems will get more accurate.
But then the degree to which we want to post stuff will also get more accurate.

“Something else to keep in mind: There are going to be times where you’ll want to see locative AR information whereever you happen to be, divorced from it’s location. Better design for that too. ”

True.
Wikitude just launched a feature for this.
I think the idea of a simulated/virtual location could work fairly well in any AR system.
It would just be an overide to the location the device is broadcasting.
So purhapes a “pretend to be at…” style function.

Idly, It would be nice if the service provider didnt know you were pretending to be elsewhere. But that might be impossible due to IP giving your aproximate location away.

“Isn’t this IRC content url distribution system a lot like DNS? Maybe this discussion needs to reach over to the protocol designers camp, get this into a Cisco trade journal”

Maybe, quite possibly.
I really don’t know much about how dns works, only what it does.
Certainly this needs to be handed over to protocol designers.
I think my concept is implementable as I wrote it, but I am absolutely certain it is far from the best way it should be done.

Honestly, I think nows the time for experiments.
I hope some reading this tries it out in their own software, just a baby step like a string from irc tied to a location, and anyone else with the software seeing it too at that location. Something as simple as that is enough to get started.

Thanks for all your comments and ideas, btw. Really got me thinking

14 Trackbacks For This Post

[...] UgoTrade posted a discussion of Thomas Wrobel’s proposal called, “Everything Everywhere: A proposal for an Augmented Reality Network system based on existing protocol….” If you’ve been following along recent AR debates , you’ll know that Thomas is [...]

[...] whether you have read Thomas Wrobel’s ideas for an open augmented reality network that I just published here on Ugotrade. The principals he talks about are very important for augmented reality to become a major part of [...]

[...] your feedback in the comments below or via @genebecker. I want to acknowledge @tishshute and Thomas Wrobel, who have been leading the discussion on using Google Wave protocols as a communication & [...]

[...] the background, including: The Next Wave of AR: Mobile Social Interaction Right Here, Right Now!, AR Wave: Layers and Channels of Social Augmented Experiences, Total Immersion and the “Transfigured City:” Shared Augmented Realities, the “Web Squared [...]

[...] UgoTrade a discussion is posted of Thomas Wrobel’s proposal called, “Everything Everywhere: A proposal for an Augmented Reality Network system based on existing protocol…” It is basing on his paper draft of his vision of a open AR Network and potential methods to [...]

[...] post by Bruce Sterling on Pachube Feeds, and Thomas Wrobel’s prototype design for open distributed augmented reality on IRC, were key inspirations for me when I began thinking about the potential of Google Wave Federation [...]