FlyWeb – Pure Web Cross-Device Interaction

FlyWeb is an experimental project we’ve been prototyping from within the depths of Mozilla’s platform division. It started as a side-project late last year, and since then a small, ad-hoc team has been working on implementing a “version zero” of the concept. We’ve been tinkering for the last 6 months on an implementation, and it’s getting to the point where we feel comfortable talking about it, in particular to web developers and hardware hackers that might be interested in the ideas it proposes.

Fundamentally, we’re aiming for one goal: making it easy for people or devices that are physically close to stream applications and content to each other. To do that, we want to make it dead simple for someone to stand up a “local area web server”, and for a person with a browser to discover and connect to that local area web server. What gets built on top of that architecture is up to you and other developers/hackers.

FlyWeb servers can live on a web-page loaded on your computer or smartphone, or they can live on tiny hardware devices on your network. FlyWeb servers are not designed to be accessible to “the internet”, but instead only to people on the local network (i.e., people already in physical proximity).

The design of FlyWeb is simple. A FlyWeb server is a web server that advertises itself on the local network using mDNS. In Firefox Nightly, we’ve added a small UI element that lets you list locally advertised FlyWeb services (off by default, and only activated if the dom.flyweb.enabled config pref is true). When you select a service to connect to, the browser creates a unique “UUID hostname” for the service, and routes all URLs with that host to the service. Additionally, we extend Web APIs with a new navigator.publishServer() function, which allows a web page to publish a FlyWeb server on the local network. The user will be prompted to allow the web page to do this.

There are two major classes of use cases this design supports. One class relates to interactions between browsers, and the other concerns interactions between browsers and “smart hardware” that wants to expose a UI to users. Both are handled with the same architecture.

Browser To Browser Interactions

Let’s say you write a multiplayer game designed for smartphones. I’m not talking about MMORPGs here, but casual games meant to be played by a group in the same room. Like a digital Catan, Scrabble or poker. You have a couple of options:

1. Write a smartphone app

This solution will work, but you’ll need to publish the app on an app store, and have everybody download the app before playing. If somebody wants to switch to a different game, they all have to download the other game. The app will need to use some native communication protocol for manually discovering and connecting the phones to each other.

2. Write a web-based game

This is also a reasonable solution, but comes with some problems. Everybody has to visit the game website, and possibly make an account on the website (or receive some sort of unique token) which they use to coordinate with each other. Communication between the players has to be bounced through the server the game is hosted on, which means games requiring low-latency input are difficult or impossible. Furthermore, everybody playing has to be connected to the internet.

FlyWeb approach

Both of the above approaches cause significant user friction. With FlyWeb, you can design the game as a web game, but instead of using the cloud to enable multiplayer, the game itself can host a local multiplayer experience. Here’s how that would work:

One person in the group loads the game website, and asks the game to start a new multiplayer session.

The game calls navigator.publishServer(), a FlyWeb API, to publish a local-area server. publishServer returns a Promise that resolves to a FlyWebPublishedServer object, on which the game binds event handlers for HTTP/WebSocket requests.

The friends all discover the game via their browser, and connect to it directly.

When they connect, the host web page’s FlyWebPublishedServer event handlers are called. The host web page is acting as a web server, and serves a copy of itself to all the connectors (and keeps track of them). The “client” web pages can make a web socket connection back to the host web page if the game needs low-latency communications.

Everybody plays together. As far as the game is concerned, communication between the “host page” and the players’ pages is accomplished using standard web technologies like HTTP fetch requests and WebSockets.

No one needs to download apps, no one needs to register accounts, or cross-reference unique codes or anything complicated. Only the browser hosting the game needs to be connected to the internet to fetch the host “server” page (and with new web features like Service Workers, even that doesn’t require an internet connection). The game can be built on pure web protocols, and everything is low-latency and playable. Basically, multi-user experiences can be created without ANY supporting infrastructure outside of a local network, and a browser supporting FlyWeb.

As an example, Justin integrated FlyWeb into a Unity WebGL-based car racing demo, inspired by the 4-player split-screen Mario Kart gameplay, and Kannan held a quick gaming session at the Mozilla Toronto office:

In our example, a laptop browser acts as “the console”, and smartphone FlyWeb clients act as game controllers. The smartphone browser discovers and connects to the game server hosted on the laptop web page, and receives a “controller UI” web page. This controller page on the phone establishes a web socket connection back to the host page, and uses that for sending touch and steering input back.

If you have an Android phone and a laptop, you can play this demo today (and download the source code and hack it, if you want). Just visit flyweb.github.io and follow the instructions, then visit the “Showcase” section of the page for a link to the demo. The game is hosted via GitHub Pages and the source code is available in our GitHub repo under the “flyweb-gp” folder.

You can also take a look at the “flyfile” and “photo-wall” subfolders in that GitHub repo for much simpler demos of web-based file sharing between smartphones.

Smart Hardware

Let’s say you are building a hardware device. Maybe it’s a fancy new thermostat, or your own home-made quadcopter. Great! Now you need a UI to control it. You have a few options at your disposal:

1. Write a smartphone app

This is probably the first thing that’ll come to mind, but it has a few problems. Writing smartphone apps is hard. You need to set up a development environment, download an SDK/build tools and all the other stuff for your platform of choice. You need to write the app, compile it into a bundle and load it on your phone. If you want other people to be able to use your app easily, you need to publish it on an app store (hoping it gets accepted) and maintain it. It’s a HUGE hassle.

Additionally, you need to build this app for every different platform you want to target. Want to access it from your Windows or Mac laptop in addition to your phone? Write another app. Not exactly the most convenient developer experience.

2. Build a web UI served from your device

This approach is a lot simpler. You don’t need to build a native app. The UI works on different platforms. You don’t need to host an app on an app store. You don’t need to build an app bundle to distribute for side-loading. But, accessing the UI is hard. You have to open up your browser, somehow figure out the device’s IP address, and type it in.

This approach is also error prone. If you later happen to access some other service with the same private IP, on some other local network (for example, 192.168.1.1:80), that service can steal the cookies stored by the first service. The services have to be careful about using caching because you might get the cached page for the first service when you visit the second service.

FlyWeb approach

FlyWeb’s approach is basically option 2, but with some extra magic. The first bit of magic is advertising over mDNS, and a browser UI to discover the service by name, instead of having to find out and type in an IP address manually. The second bit of magic is the unique hostname generation. Currently, every time you connect to a service, the browser generates a UUID to use as the hostname for that service. No two UUIDs will ever be shared between services, so we avoid issues like leaking cookies, or cache cross-contamination.

With the help of the amazingly talented Kate Glazko, we built a demo of this approach with a Parrot AR quadcopter and a Raspberry PI (controlling the copter and exposing a FlyWeb server), and presented it in June 2016 at a Mozilla All-Hands meeting. Here’s a video of that demo:

The source code for this demo is also available in our examples repo (under the flyweb-quadcopter folder). We’ve built other demos on hardware as simple as the ESP8266, which is a $5 WiFi chip with an embedded microcontroller. FlyWeb servers can run on extremely minimal hardware. Even tiny, low-power devices can present extremely rich UIs because the web platform lets them dynamically stream their control UIs to a far more powerful smartphone or computer.

What Next

Currently, this feature is implemented and exposed ONLY in Firefox Nightly (not in Aurora or Beta), and hidden behind a pref that’s off by default. The current implementation is basically a “version 0” implementation, which is enough to build some pretty amazing demos and get a taste for the potential of FlyWeb.

The team’s goal right now is to let early-adopter developers and enthusiasts play with this implementation and get feedback on the viability of the feature as a future “true” web standard. If you’re a web developer looking to create neat multi-user “local area” experiences, or you’re a hardware hacker looking for an easy way to give a UI to your creations, we hope you’ll take a look at this, play with it, and let us know what you think.

If you are excited by the potential of this feature, want to build things with it, or help with the implementation (our team has 2 people in it, we are not some massive project – this is very much an experimental “skunkworks” project), please visit us at flyweb.github.io. You can also chat with us on Slack at mozflyweb.slack.com (sign up here) or follow us on Twitter at @MozFlyWeb.

Caveats

Please keep in mind that this is a very early-stage implementation. There are probably bugs and security issues. If you DO experiment with FlyWeb, we advise creating a new Firefox profile just for that purpose and recommend that you DO NOT browse the general web using that profile. Use a separate profile only for running FlyWeb demos or your own code.

Kannan is a systems developer that has built everything from web tools for RNA sequence analysis, to JIT compilers for Javascript engines, to infrared stacks for embedded OSes, to experimental new web platform technologies.
Ask him about any of the above things if you have a few hours to waste getting your ears talked off.

The crashiness is troublesome. When does the crash happen with your nightly? Do you get prompted to allow the publish-server before the crash? Does it crash on nightly without “dom.flyweb.enabled” set to true (i.e. with the FlyWeb feature turned off)? I am very interested in tracking down and fixing this issue.

The crash happens after a client connects. The client receives an initial response (the pedals for the GP example or the photo wall ui for the other example appears in the client), then the host crashes. The host browser crashed on Win7 and Ubuntu 14.04 as well after connecting from either a separate device (Nightly on Android 5.1.1) or from the same device (using another profile and the –no-remote switch).

The FlyWeb remote example ran without a problem as far as I could tell. Also I was able to access a network printer and a home server gui (I guess because they broadcast themselves using mDNS), which was a nice surprise (no need to find out the IP of the device).

Cool. Yeah, the printer and other-service discovery is because along with discovering FlyWeb services (_flyweb._tcp.local), we also list (_http._tcp.local) services, which a bunch of printers tend to offer web UIs on.

Could you explain how this is different from other zeroconf stuff like bonjour/avahi? Is it just specific to web apps and in the “unique hostname generation”? Which I don’t totally understand, does hostname refer to the client device?

Well, zeroconf is a bit broader in scope than service advertisement. We’re just using mDNS+DNS-SD (i.e. Bonjour/Avahi, but those are brand names for implementations). FlyWeb services are just mDNS-advertised “_flyweb._tcp.local” services that point to a web server on the local network. There’s some extra protocol support for the advertisement to embed cert info for eventual encrypted transport support (currently we have self-signed certs sort of implemented).

The semantic expectation we add is that the browser, when it “establishes a session” with a given service, SHOULD assign a “hostname” for that service which cannot ever be used by another service. This generated “hostname” is internal to the browser, and serves as a surrogate for DNS-based name resolution. Currently we just generate random UUIDs for every single session that’s established. This addresses the origin-based security issues in a very brute force way: no two sessions will ever share hostnames, and thus will never have the same origin.

Looking forward, in the presence of public key info in the service advertisement metadata, we can assign a “stable” hostname to that service which still meets the criteria above. We do that by generating a service a name which embeds the fingerprint/digest of that public key. Later on, when connecting to the same service, we can reliably assign that same name after verifying that the service actually controls the keypair associated with the public key. This would give us stable naming, but also ensure that no service can masquerade as another service (without acquiring the keypair somehow).

The latest nightly (51.0a1) instead of the developer edition on desktop and today nightly update on my smartphone fixed my NS_ERROR_FAILURE errors.
Sorry guys, it’s was only a version problem in both cases :/

As a side question, I just tried using a “.local” adress on my local wifi network. It seems to work well on my Linux laptop, but I think it fails when I want to reach my laptop (running a “python -m SimpleHTTPServer 8000” for test) from my Android phone (with Firefox Android).

A quick search just showed me that mDNS support is not great on Android(*), but is it on the agenda of Firefox on Android to support it?

Are you using Nightly on Android? You have to get the Firefox nightly apk (from nightly.mozilla.org), and install that directly, then enable the “dom.flyweb.enabled” pref on Nightly for Android, and then use that browser.

Thanks for your feedback. Indeed, I was using stable Firefox on Android. I undertand that Flyweb requires nightly, but I thought mDNS was a separate and preexisting feature.

Also, the other day I wanted to report that the demos were crashing on my Linux machine (Debian testing). I was trying the GP demo, but in the same browser (one tab for display of the race, one tab for control). However, I’ve tried again today (with nightly 2016-09-17) and now it runs well. (Of course, it’s not really playable with mouse and keyboard, but that’s another story ;-) )

One powerful but simple application is using this for slideshow presentations.

It could be used in schools, colleges, conferences, business environments, etc – with or without a primary display for the presentation. Just show up, connect to the presenter’s server and go.
Potential features could include:
– audience feedback, such as polls
– allow someone to go *back* for something they missed on an earlier slide (on their device)
– easily save the whole presentation or take screenshots of just one slide

Especially useful in environments where not everyone can have a good view of the a main screen or if there is none.
Additionally, the large screen could be a Flyweb enabled TV or projector.

It’s quite ironic that you – as members of Mozilla – advertise Slack for communication, but not your IRC channel (#flyweb on irc.mozilla.org).

I thought one of Mozilla’s core principles is openness. Just look at https://www.mozilla.org/. Wouldn’t it be appropriate then to *first* list open ways of communication and *then* – if at all – closed data silos?

The icing on the cake is your app for inviting oneself to your Slack channel. Very creative – and telling.

Good point about the IRC channel not being listed. We’ll add that info to the main flyweb page. I don’t know what you mean by “app for inviting oneself” to slack. What are you referring to?

You’re reading more into the omission of the irc channel than you need to. There are two people doing the coding, debugging, website, comms, blog posts, and demos for this project. Things slip. Thanks for bringing the IRC channel omission to our attention.

Actually, the #flyweb IRC channel info is already on our website, we must have just omitted it accidentally from also mentioning it in this post. We also have the ability to “mirror” conversations between our IRC channel and our Slack channel so participants in either place do not miss out on anything. Our justification for creating a Slack channel was mostly to create another outlet for users to participate in the project who do not frequent IRC channels.

Also, I’m not sure what you mean either about our app for inviting new users to our Slack channel. The webpage we put up at https://mozflyweb-slack-invite.herokuapp.com/ allows *anyone* to join our Slack channel automatically without having to go through a back-and-forth invite process. You simply enter your email address there and you’ll be automatically “invited” (to use Slack’s terminology).

I think there’s a really cool application of this for in-flight peer-to-peer messaging within a small group of friends. Imagine the scenario where you’re on a group trip with your friends but you’re distributed over multiple seats in the plane. In order to chat with each other, someone could fire up a Wireless Hotspot on which the others can connect, and then the host can simply start have a pre-determined address which hosts a messaging application between all the individuals.

That’s a really cool use case! Our team has actually identified “offline travel” as a particular scenario where we think FlyWeb has the potential to enable all sorts of interesting applications like this. We often wrongly assume that users will have connectivity all the time, but as you know that is certainly not the case. Even in circumstances where you may have access to a 2G cellular network it can sometimes be just as bad as *no* connectivity. Another specific scenario that comes to mind where your app idea could be useful is for a group of friends who go hiking in a region that doesn’t have cellular service. They could still send messages/photos to each other via an offline FlyWeb app.

Another potential scenario for offline FlyWeb apps is when you *have* connectivity but it’s expensive to use (e.g. you’re in another country and your cellular plan charges you a lot for data when roaming).

As for Kartik’s suggested application, I’m actually working on something that might fit this really well called “Project Refrigerator”[1]. It’s a multiplayer drawing/chat app that uses FlyWeb to create a channel for nearby users to connect to. It’s still a bit too early to publicize broadly, but hopefully there will be a Hacks article about it in the next week or two. :)

I feel that at some point, the web will have to tackle its extreme reliance on TCP if it wants to be a viable platform for P2P use cases, which I think includes IoT. Instead of bolting TCP onto BLE or UDP-based protocols necessary for NAT traversal, I think we’d be better off if browsers and web apps had the option to support such protocols natively.

I like that you’re working on solutions to the certificate issue, and discoverability of local services. I just hope that the long-term plan doesn’t exclusively rely on LANs and an absence of multicast packet filter that would put mDNS out of order.