Native Web Apps Community Group

A community driven take on the concepts driving the Widgets and Device APIs. Collectively understood these technologies form the basis for installable web apps. Living in a secured context these applications give the web access to traditionally native capabilities.

Note: Community Groups are proposed and run by the community. Although W3C hosts these conversations, the groups do not necessarily represent the views of the W3C Membership or staff.

As well as being able to use a classic “http://domain.com” style origin, widgets can also use a custom scheme called the Widget URI scheme that typically looks like this:

widget://c1[…UUID…]a66/index.html

The widget URI scheme works like a “fake” HTTP server: sending back local files from inside a widget package (e.g., an image) by simulating a HTTP responses.

The misconception about origin arises because people don’t really understand what an origin is: they think it means http://some.url.com (i.e., a website). Thankfully, the HTML Standard makes it clear by defining an origin as:

opaque identifiers or tuples consisting of a scheme component, a host component, a port component, and optionally extra data.

Another ungrounded misconception is that widgets lack a security model.

Widgets, like most things “Web”, rely on the same-origin security model defined in HTML5. However, because the HTML same-origin policy is quite liberal and prone to cross-site scripting attacks, the Working Group also defined an optional security model for widgets called the Widget Access Request Policy (WARP).

WARP allows developers to declare up front which websites they are going to be communicating with, so the engine can block communication with websites that are not listed by the author.

By default, Widget engines that implement WARP will not allow widgets to access content on the Web (i.e., they have no ability to access anything on the network). So, if a widget tries to access an image on the Web, the widget runtime won’t allow it:

To overcome this default policy restriction, an author needs to declare what domains on the Web a widget will access. For example, if a widget needs to access the an image on “w3.org”, the author makes a WARP declaration in the configuration document (config.xml) of a widget like so:

If at runtime, the widget tries to access foo.com, the widget engine blocks that access because it violates the access request policy.

<!doctype html>
<!--
this will fail: foo.com not in declared in config.xml
-->
<img src="http://foo.com/hello.gif">

The WARP declaration can be used to do other useful things also. For instance, when the user installs a widget, the widget engine can tell the end-user which domains the widget may try to contact. Opera extensions partially do this, for instance.

Currently, the W3C Widgets PAG working towards towards finding prior art and working around Apple’s patents. The patent’s are pretty groundless (thanks again US Patent office!), so should be fairly trivial. If you know of some prior art that can help bust Apple’s patents around widgets, please leave a comment. All help is greatly appreciated.

In the case of widgets, there are now interoperable implementations of the Widgets XML Digital Signature specification. Although there is limited W3C Widget-based content out in the wild that make use of digital signatures, none of the implementers have reported interoperability issues thus far. So, this claim remains unfounded and those that spread it have been unable and unwilling to provide any tests that prove there is an issue.

It should be said that cryptography is hard irrespective of XML Digital Signatures. Very few individuals on this planet actually understand the tools, let alone the mathematics, and other things that make such systems secure. It is my belief that no matter what signature system would have been chosen for widgets, it would have still been hard for most developers to understand. I hold that our choice of XML Digital Signatures is OK (i.e., not great, maybe there is something better?) for the common cases for which widgets are used.

Proponents of JSONargue that XML is too complicated for developers and for implementers, while other more conspiratorial individuals claim that there is a “W3C strategy tax” at work. Proponents of this misconception ignore the fact that JSON can also easily be abused to make overly complicated data structures (see British museum’s near criminal use of JSON), and has a lot of restrictions that mirror XML’s complexity in many ways (e.g., JSON also needs to be “well-formed” and valid, some characters are reserved, only certain data types can be used, etc.).

Having said that, I personally can agree that XML, when used badly (e.g., RDF/OWL, XML Schema, XACML) can overwhelm authors to the point of wanting to rip one’s eyes out. But, the use of XML in Widgets has never been proven to be an issue for Widget developers. This is because the Working Group was well aware of authoring issues with XML authoring and so the configuration document was designed to be as simple as possible to author: the document format is mostly flat in structure, taking away most authoring complexity found in XML. In other words, we knew “the tools won’t save us”: so we designed everything assuming it would be written by hand.

Further, to make Widget’s use of XML as developer-friendly as possible, the Widget packaging specification provides a meticulously defined XML processing model, which handles error conditions gracefully, provide sensible defaults and fallback, and only halts processing in the most extreme of cases.

The processing model, which draws heavily from HTML Standard’s processing model, is also designed to be future proof: allowing the format to be easily expanded as needed through either proprietary extensions or through new W3C specifications.

The choice to use XML for W3C Widgets was for legacy reasons: all the Widget runtimes from when we first started the widget standardization effort were using XML.

The logic of the Working Group at the time was that if we used XML, then it would be easier for implementers to adapt their runtime and developer tools to use the XML W3C Widget format, and it would make it easier for existing content to be migrated to the XML W3C Widget format. Furthermore, when we started this effort in 2006, JSON was just appearing on the scene so it was not as popular a format as it is today (5 years on at the time of writing).

So, although we could have chosen to go against the grain in 2006 and used JSON, the parsing and error-handling models needed for either format would have been essentially identical: i.e., there is still little reason to choose one over the other, and choice of XML has helped other groups use the format for their own purposes (e.g., WAC defines a few additions to the format through their own namespace). If the industry moves towards Google’s and Mozilla’s use of a JSON format for packaged client-side Web applications, a new parsing model specifically for JSON will need to be standardized. This is not a big deal; it just needs to be written down by someone (and no one has found time to do that yet).

Regarding the mythical “W3C strategy tax”, certain individuals seem convinced that the W3C’s membership has some evil conspiracy to push XML on the World and seem convinced that W3C widgets are part of that conspiracy. As much as I would love to be part of some evil conspiracy (and get a few kick-backs from this “strategy tax”), there is no such conspiracy to my knowledge (and last I checked, I’m still broke but happy to receive kick-backs;)).

In spite of all this, I’m would still be happy to help standardize a JSON flavour of the widget configuration document format.

The widget packaging specification tried to overcome obvious limitations and complexity in those formats. For instance, the above formats all require special authoring tools (or make use of complex internal file/folder structures). I say explicitly because the Working Group looked at all the above formats when we were specifying the W3C widgets format. After a long hard-look at each, we concluded that none were fit for purpose.

To show why each one would not make a suitable format:

JAR: depends on the META-INF directory and defines files designed to load Java-based applications and libraries. Hence it is specifically designed around the needs of Java and the semantics of various aspects of the packaging format that don’t make sense for Widgets.

ODF: amongst other things, it requires that a special file (‘mimetype’) be found at byte position 38, making it extremely difficult to create a package without a special tool.

XPI: format (which itself reinvents JAR) makes use of RDF, which is notoriously difficult for developers to learn, read, write, and maintain. Hence, the working group concluded that XPI would make a lousy widget-packaging format. Furthermore, XPIs suffer from versioning issues, which causes them to stop working when Mozilla Firefox is updated.

By contrast, and by design, W3C Widgets require no special tools to make a Widget package. The format is designed to take advantage of Zip tools that have come bundled with all operating systems for at least a decade. The format puts very few requirements on authors: the only two being to include a config.xml (which can be almost empty) and a file to start the application (index.html)[1]. Furthermore, we designed W3C widgets so they were not bound to particular runtimes: in effect, overcoming the issues seen in XPIs and other platform-version-dependent applications.

Throughout the last few years, a number of misconceptions have been perpetuated about W3C Widgets. Over the next week, I will publish a number of post to address some of those misconceptions (hopefully setting the record straight):

I asked a friend, Jimmy Ti, if he could make a rough prototype showing if it was possible to implement the Widget URI scheme in iOS. The critical bit I wanted to see working was faking HTTP responses. Here is what he came up with:

It’s an Xcode project that reuses some sample code provided by Apple for Mac OS X, which Jimmy retrofitted to work on iOS. The app is very simple: it contains a WebView that loads a HTML file named “test.html”, the web page has a few tags and AJAX image loading that loads from “widget://”.

The app implements the widget:// scheme to return the images from local resources (the images are actually generated dynamically from the code, but it could very well be loaded locally).

If you have Xcode installed, you can simply open up the .xcodeproj file, and “Build and Run” the code on iPhone simulator. Works a treat! Huge thanks to Jimmy Ti for taking the time to do this! It’s a great proof of concept.

Ideally, next would be:

Write the code to support other content types: generated images are cool, but not quite what we are trying to achieve right now 🙂

Make widget:// the document’s location so that everything can be resolved relatively to the start file.

A little while back, I made a quick prototype showing how widgets could work with OAuth (with Twitter). The problem is that most widget engines don’t support something like Android’s Web View and usually can’t pop up Windows. Another issue is that OAuth authentication services usually use X-Frame-Options header, so that means you can’t just “popup” and iframe.

The goal was just to see what the developer experience was, what is missing, and what complexity was added (I used Chrome with Web Security turned off to get it to work like a widget). How it works:

As the end point of the OAuth flow (the success URL), I assumed I could not use a widget://.. so I used a real URL. A problem with widget:// is that it’s not supposed to be addressable from the outside world (supposed to be opaque).

At the end point, I did a PostMessage() back to the “widget” sending the authorisation token … nasty town, I know. I passed the widget’s id to twitter over SSL, which I used at the end pont to target the cross-doc message.

So, somewhat ignoring the fact that we were going to plot and scheme about what Installable Web Apps could be… and Web APIs… I’ve had a first crack at making widget:// behave more like a HTTP request. This should solve a bunch of problems, including allowing widgets to work with inner package XHR (YaY! jquery mobile FTW!).

This section describes how a user agent is supposed to respond requests to retrive resources from a widget URI. The purpose is to make responses “look and feel” as much as possible like regular HTTP requests.

To dereference a widget URI to a file in a widget package a user agent MUST apply the rules for dereferencing a widget URI.

The rules for dereferencing a widget URI are as follows:

If the request is not a GET request, return a 501 Not Implemented response and terminate this algorithm.

If the URI uses the scheme ‘widget’, but is not a valid widget URI, return a 400 Bad Request response and terminate this algorithm.

If the URI uses the scheme ‘widget’, but the authority does not match the one assigned to this package, return a 403 Not Allowed response and terminate this algorithm (i.e., prevent inter-widget content access).

If the user agent implements [[!Widgets]] (for the purpose of internationalization):