Rules for Responsible Reformatting: A Developer Manifesto

Editor: Luca Passani, luca.passani at Gmail dot com
Created on March 21, 2008, Last Edit, April 23, Latest Signature, March 5, 2009

This document is about a recent development in mobile web technologies, often referred to
as content reformatting.

The mobile web has never been a simple platform to develop for, given the fragmentation of the
underlying technology. Different devices, different browsers and different networks
have made mobile web development a challenge for programmers and content
authors from the beginning.
Content reformatting, which seeks to convert pages designed for PC users to be usable
from mobile phones may add another layer of complexity when not properly configured.

Some Mobile Network Operators (MNOs) are implementing reformatting proxies into their networks that
intercept and reformat all HTTP traffic to any HTTP server. This is being done without
safeguarding those sites which strive to deliver
a mobile-optimised user-experience, or, at the very least, not safeguarding them
sufficiently.

The introduction of reformatting systems can lead to the disruption of
mobile web sites for all of the users in their respective networks.

This situation is a concern for mobile web neutrality and has the potential to jeopardize
the mobile web as a platform in the years to come.

To this end, mobile developers of various nationalities and backgrounds, assembled
on the WMLProgramming list
at Yahoo Groups, have created this manifesto to make their position known.

Consequences of the Introduction of a Reformatting Proxy

Symptoms of a network that has deployed a reformatting proxy are:

Service is not receiving a device User-Agent string. In its place, one can typically find the User-Agent string of a web browser with the name of the proxy vendor somewhere.

The UAprof x-wap-profile header is removed.

The carefully designed appearance of a mobile site is disrupted.

Mobile users receive a chopped-up version of a web site, instead of the mobile-optimised site originally meant for them by the site owner.

Extra icons and/or links are added to headers and/or footers injected at the top/bottom of the content.

Advertisement of unknown origin may be added to pages.

Images are converted into lower quality versions of the original versions.

Revenues drop and, upon further investigation, the reason is determined to be the introduction of a transcoding proxy.

It is impossible to determine which version of ringtone, wallpaper, local application (such as J2ME, Symbian, etc) or other downloadable content should be provided.

Detecting a successful download and installation of a midlet may not be possible, making it difficult to manage J2ME midlet downloads.

What can be done?

While this document represents the viewpoint of mobile developers, it is not meant
to be hostile to proxy vendors and Mobile Network Operators who recognize
the importance of the ecosystem and the disruption that the introduction
of aggressively configured reformatters can cause to existing mobile sites and services.

To this end, developers have established guidelines that a reformatting proxy should respect,
in order to avoid reformatting mobile-optimized sites.

Vendors of reformatting proxies can subscribe to the manifesto when the following two situations are verified:

The default configuration of the vendor's proxy respects all of the rules illustrated in the Manifesto.

Vendor will notify customers and prospective customers of the existence of this document and
explain the consequences of not respecting mobile-optimised sites.

The first rule is meant to make sure that a reformatting proxy can effectively be
configured to respect mobile websites. The second rule is meant to pre-empt carriers' attempts
to blame proxy vendors for failure to respect mobile-optimized sites.

The recommended way to endorse the Manifesto is to add the following
sentence prominently in the documentation and in the configuration files of the proxy:

"[Vendor] supports the rules established by the Mobile Developer Manifesto
for Responsible Reformatting (available at: http://wurfl.sourceforge.net/manifesto/)
for its default configuration."

This sentence is meant to remind MNOs of the existence of a mobile ecosystem and the necessity
to respect it, a fact which may be unknown to MNO employees at different levels
in the respective organizations.

Mobile Network Operators are also invited to subscribe to the Manifesto in order to
express their acknowledgment of the rules contained in this manifesto.

Vendors and MNOs who sign the Manifesto will be entitled to a short statement to
better qualify their position with regards to the Manifesto.

How to Avoid Reformatting Mobile-Optimised Sites

As a general rule, transcoders should err on the side of non-reformatting, meaning that a reformatting proxy should refrain from reformatting in all of those cases when a site might already be suitable for mobile.
In particular, developers request that the following rules are observed:

No USER AGENT Spoofing: under no circumstances should the original User-Agent string be removed, modified or moved to a different header.

Note: It is considered acceptable for a transcoder to perform a second HTTP fetch of site content with a spoofed HTTP request, as long as the first request has been performed with the original headers and the proxy has positively identified that the website is NOT mobile optmised

Preserve headers: under no circumstances should a transcoder modify or delete existing HTTP headers. The addition of extra x-* headers is admissible.

Recognize Mobile-Specific MIME-types and Document Type Declarations (DTDs): documents served with the following MIME types:

MUST be considered mobile-optimised and no transcoding should be applied

Do Not Adapt pictures of mobile-optimized sites: mobile-optimized sites should not have their graphics adapted.

30Kb Limit: pages that have a total weight of 30 kilobytes or less MUST not be adapted.

Note: An exception can be made in those cases when none of the other rules apply AND the reformatting proxy is able to positively identify devices which cannot handle 30kb of content through a Device Description Repository (DDR).
In this case a lower threshold for reformatting may be adopted. The limit cannot be lower than 15 kilobytes under any circumstances.
A further exception can be made in those cases when the reformatting proxy can positively identify the web-only nature of the site by using more sophisticated techniques, such as: detecting the use of IFRAMES and other "web-only" tags and scripting. The limit cannot be lower than 15 kilobytes under any circumstances.

Do not reformat pages that return the following HTTP header:

Cache-Control: no-transform

This also applies to the case when the header is specified through the
<meta> HTTP-Equiv tag

The proxy should redirect the user to the [url] contained in the href attribute.

Transcoders MUST not reformat sites with URLs matching one of the following patterns:

m.*mobile.*wap.**.mobipda.*avantgo.*iphone.*wml.*wireless.*xhtml.*

Note: In the spirit of erring on the side of non-reformatting, vendors are also welcome to
support the following exclusion patterns, where "/" represents the first slash in the url PATH:

*/mobile/*/iphone/*/wireless/

Support for these extra exclusion patterns is only recommended

Whitelists are acceptable only as inclusion list to inform the reformatter of
sites which would normally escape reformatting, but which are marked as reformattable through
human intervention. Use of exclusion whitelists is not acceptable.

Reformatter vendors who abide by the rules above for their products' standard configuration
and who agree to notify their customers of the importance of behaving sympathetically in
the ecosystem are invited to sign the Manifesto.

Operators who need to deploy content reformatting solutions on their network are welcome to read this document and to derive the requirements for their systems from it.

Mobile developers are welcome to sign the Manifesto by sending an email to the editor and requesting that their signature is added.

Letizia Jaccheri, Professor in Software Engineering, Department of Computer
and Information Science, Norwegian University of Science and Technology

Robert Oberhofer, Phd, Mobile Consultant and Co-Founder JotInfo Inc.

Kanwar Dhaliwal, Mobile Consultant

Terren Suydam, Head of Mobile Development, SingleClick Systems

Adriano Fonseca, Partner and Co-Founder, TopSolutions.pt

Nihan Sin, Mobile Web Developer

Michael Dykman, Distributed System Architect and Developer.

Alexandru Stanciu, Co-Founder, StarTech.ro

Michael Maguire, Founder & CTO, Blue Whale Systems Ltd

Gabriel Balogh, Mobile Developer, MOBIcomp

Scott Beaumont, Co-Founder, Mippin.com

Steve Kamerman, Director of IT / Co-Founder, Tera Technologies Inc

Alfredo Pinkus, Applications Manager, CycleLogic

Ian Homer, Co-Founder, Bemoko

Tim Dedischew, Founder, MobileLeap Inc.

Alex Matthews, Technical Director, Marvellous.

Michael Stone, CEO, Amethon Solutions Pty Ltd

B. Vervoort, Co-Founder & Mobile Developer, InMovil Media Europa

Manuel Dominguez Sarmiento, General Director, Renxo

Idel Fuschini, Mobile Evangelist, Italy

Note:Questions about the Reformatting Manifesto can be directed to the Manifesto editor or publicly on
the WMLProgramming list
at Yahoo Groups.
Requests to sign the Manifesto can be directed to the editor via email at passani (at) eunet dot no