Pamela Jones asked me to comment in more depth on Microsoft’s response,
because I’ve had past related experience (e.g., with standards, XML,
and even OpenDocument specifically). Many people have made brief
comments about this letter, but she thought a more detailed commentary
would help those who are less familiar with this topic.
So, here’s my attempt at commenting on Microsoft’s letter
in more detail.

But first, a few disclaimers and general comments.
These are my personal thoughts, not anyone else’s, and I speak for no one.
I have tried to give references to justify many of my comments, though,
which I believe justify the statements well.
I’m independent; I have no financial stake in this outcome.
I am not interested in Microsoft-bashing, so for those who
were looking for that, sorry.
I do believe that governments should try to
create as level a playing field as possible, so that
suppliers can fairly compete for government business.
I also believe that it’s a very good idea to have standards;
we could not have a technological society without them!

Below is the Microsoft letter text (or summaries of it), which I’ve indented, italicized and put in bold text and
interspersed with my comments.

It’s a long letter, and commenting
it makes even longer.
Feel free to skip ahead to a few of the
discussions you may be particularly interested in
(I repeat key points in some places so that skipping ahead will make sense):

Microsoft respectfully invites you
to consider its responses to the proposed revisions to the Enterprise
Technical Reference Model-Information Domain published on August 29,
2005 (ETRM) which, as currently framed, mandates exclusive use of a
designated office document format within all executive agencies of
the Commonwealth of Massachusetts by January, 2007.

This is a good, conventional beginning of an appeal to government
officials, which is basically what this letter is.
Right up front they identify the subject they wish to discuss, which is
always a good idea.
At the end you’ll notice that this letter is cc’ed to
their boss, Governor Mitt Romney, presumably to try to get the
Governor to overturn his employees’ decision.

They don’t give the name of the format being discussed,
so I’ll re-state it here:
Massachusetts plans to use the OpenDocument format, as standardized by
OASIS.
Microsoft is unhappy about this, because they wanted Massachusetts to use
Microsoft’s new proprietary XML format also or instead.

They use the word “exclusive” here, which is interesting.
Literally stated, it’s true; Massachusetts
is mandating one standard to the exclusion of other specifications.
But that’s good;
you want to pick a single standard for a given purpose, to the
exclusion of others, as long as all suppliers can
implement the standard without unrelated restrictions.
If we had two
signal light standards, where red meant “stop” in one and “go” in another,
we’d obviously have bad results.
One of the biggest problems in information technology
is that in some areas there
are too many standards, instead of a single standard that
everyone can agree on and use.
Mandating a single standard for a given area is a very good thing, if
all suppliers can implement the specification
without legal, monetary, or other restrictions or discriminations.
Massachusetts has really a strong case for selecting OpenDocument
(the topic of this letter) as this standard, as we’ll see.

Be careful: the word “exclusive” is here, but
no product is being excluded.
As we’ll discuss, anyone can implement this standard
without restrictions, including Microsoft.
In fact, OpenDocument has already been implemented by several suppliers,
both proprietary software and
open source software,
using all of their normal licensing and business models,
so it’s demonstrably open for competition.
(Note: in this document
I’ll sometimes use the term “open source software”, and sometimes
use the term “open source software / Free software (OSS/FS)”;
in all cases I mean by these terms software whose license conforms to both the
open source definition and the
Free software
definition).

By the way, some news articles and other various groups
explained the issue very poorly,
suggesting that this decision was fundamentally about
“using open source software instead of Microsoft Office”
or
“throwing away Microsoft Office.”
In this introduction, Microsoft correctly identifies the real issue:
data formats.
There’s nothing in this Massachusetts decision that mandates or
even prefers open source software.
This decision does not even mandate that Massachusetts stop
using Microsoft Office, necessarily, though that decision is partly
Microsoft’s.
If Microsoft implements Massachusetts’
key requirement (OpenDocument), then great;
if not, that is a future possibility.
Data formats may seem like an esoteric thing to debate about, but
whoever controls the data format controls all the data in that format.
Once this letter is understood as a battle over power and sovereignty
over all documents (and the information in them)
ever created, the strong positions of these organizations all makes sense.

Microsoft strongly
supports the efforts of the Information Technology Division (ITD) of the
Executive Office for Administration & Finance (ANF) to bring the benefits
of XML to executive agencies of the Commonwealth of Massachusetts. We
recognize that governments are challenged to be fully accountable for
archived public records well into the future, and for ensuring that
government agencies can efficiently handle data and documents across all
technical and organizational boundaries. We share the opinion that XML
is the ideal format for data interoperability and storage, management,
and archiving of public records and endorse the direction to support
open and agreed-upon specifications for data interoperability within
government via XML standards. We share the proposal’s goals for data
interoperability across government agencies and for assuring proper
storage and maintenance of all public records. Consistent with this
viewpoint, Microsoft has been deeply committed to supporting XML within
Microsoft Office for a number of years and continues to work with many
governments around the world toward these goals.

It’s common in a disagreement to start with a list of things you agree on.
I suppose this can look like flattery, but it does have a value -- if
people can find many shared positions, then perhaps there’s a way to find
a solution that everyone can be happy with.

Why XML?

We should particularly notice Microsoft’s agreement that XML is where
everyone is going for storing editable documents.
This is a point of strong agreement between Microsoft,
Massachusetts, and most of the rest of the information technology
community (including office suite suppliers, other organizations who
manage or manipulate data in documents, and customers that use or
generate documents).
XML is where everyone is going, not because it just happens to be there,
but because XML
can help solve some serious government problems like data interoperability
and long-term storage.

That’s why staying with Microsoft’s
old binary formats isn’t a
long-term solution; they don’t support interoperability well, and
they’re horrific for long-term storage.
I have Office documents less than a decade old
that are no longer readable by current versions of Microsoft Office,
and governments think in terms of centuries.
Microsoft themselves are abandoning their old binary formats
as their primary storage format
(Office 12 will no longer use them by default); they plan to use a new
proprietary XML format that they have developed in isolation instead.
Other office suite suppliers are choosing to support
OpenDocument, which is also XML-based.

Compressed XML documents also tend to be much smaller than binary files,
which is great for archiving.
Loading and saving XML files tends to take longer than with
binary formats, but the advantages of being able to actually use
the documents later more than compensates
for that (in most people’s opinion).

But XML cannot provide these advantages by itself; specific XML-based schemas
have to be defined to store the data in a way that
maximizes data interoperability and long-term storage.
We are at a technology inflection point;
everyone agrees that XML is where we’re headed, and thus it makes
sense that Massachusetts has begun planning for that transition.

Microsoft agrees here that there’s a need for
“open and agreed-upon specifications for data interoperability.”
This is striking, because
these are some of the very requirements that Microsoft’s
proposed alternative fails to meet.
These issues show up throughout the letter, so let’s deal with them
them at length now;
otherwise, what the letter says (and doesn’t say) will be hard to understand.
Let’s look at the issues in two parts: “agreed-upon” and “open”.

Agreed-upon

Let’s start with the phrase “agreed-upon.”
Microsoft has developed its own proprietary XML-based formats,
first the XML format for Office 2003, and a later significantly modified
version that is to be used by Office 12 when it is released next year.
No one other than Microsoft has had any reasonable
way to “agree” on Microsoft’s specification.
In particular,
competing suppliers could not review Microsoft’s format in some
neutral forum where they could identify problems,
work together to fix them, and agree without pressure on a final result.
Users do not really count here; most users have no idea what is
in their format and cannot rationally agree to anything.
So, there’s never been an opportunity for agreement
on Microsoft’s format.
In contrast, Massachusetts’ choice, OpenDocument, has been
agreed upon between many office suite suppliers and large-scale users,
and vetted (agreed upon) by an independent standards body (OASIS).

This is a central issue.
Wise customers (especially large ones like governments)
don’t want a “standard” controlled
by any one supplier, no matter
who it is, because that tends to lock them into a single supplier.
Once a supplier has a monopoly, their prices tend to go up and their
service tends to go down... no matter who it is.
That is not because they’re “evil.”
In the U.S.,
commercial suppliers have a duty to maximize their profit;
raising prices and lowering service (overhead) helps do both.
To counter this, wise customers try to
use standards developed and implemented by a number of suppliers,
vetted and maintained by an independent body.

The European Commission's
Valoris report explained why vetting by
a neutral party is
especially critical for XML-based document standards like the
ones under discussion:

“It is quite trivial to add elements to an XML document that
place processing requirements and restrictions on the document,
thus preventing cross-platform processing capability...
While properly developed XML should in theory be platform-neutral,
experience has shown that suppliers who wish to maintain and
protect their platform’s market will go to extents to encode
elements that are capable of being processed only by their
own application suites.
The only counter-balance to this natural force is the development
of open, cross-industry, widely adopted standards that serve
to block the inclusion of application or platform specific encoding.”

Customers generally want the strong, supplier-neutral results that come from the
usual give, take, and consensus-building in standards development,
both for the original work and to control future direction.
Usually an existing specification that works is presented to a large
body of independent reviewers and competing suppliers, who work together
to discuss various improvements or subsets to make the result
appropriate for long-term standardization.
Massachusetts has selected other specifications that are good examples of this.
Adobe’s PDF has gone through ISO, it
has multiple independent implementations, and it has a specification
whose future is controlled by a neutral party.
HTML is standardized by the W3C, and again, has
multiple independent implementations and a future controlled
by a neutral party.
In contrast, no independent standards body has ever
reviewed Microsoft’s specification,
it’s not clear that there will even
be multiple independent implementations (and if there are they will
be unnecessarily constrained), and
there is no neutral party that can control the direction of later
versions of the specification.

“From an interoperability - transformation perspective,
MSXML fails miserably.
Every MSXML file has a binary key in the header.
The binary key holds the critical layout definition for styles,
or what would otherwise be called the ‘presentation’ aspects of the file...

If your purpose with XML is to have all your information in
a structured file format that enables the aggregation, reuse,
and re-purposing of massive stores of information,
the binary in MSXML is a deal breaker...

[It’s true that] XML is intended to be eXtensible.... [but if]
the eXtensions are not transparently declared, or worse,
critically important aspects like the layout definition
are locked in a binary key - and comes with a governing license
that threatens and looms over any reverse engineering attempts,
then this simply isn’t XML.

.. .
[I]nteroperability and transformation is broken right from the git go.
Because of the binary key in MSXML, [someone other than Microsoft cannot]
transform a MSXML file
to OpenDocument. And those transformation ‘filters’ Microsoft
promised the European Union in November of 2004? Forget it.
They don’t exist.

Since OpenDoc is Open XML, and an Open Standard,
it would be easy for Microsoft to perfect a transformation
from OpenDoc to MSXML.
But until and unless the reverse engineer experts break that binary key,
it is not possible to perfect in any useful way a transformation
from MSXML to OpenDocument...

Information can go in. But it can’t go out.
Least ways not without Microsoft’s permission.
This is one-way, arbitrarily limited interoperability...

With the key in place they can leverage their monopoly
on the desktop deep into the realms of
Open Internet servers and back end data-transaction processing systems.
The stockholders are going to be happy if they pull off this coup.
But how this works for the state of Massachusetts,
or anyone else for that matter is beyond me.

If Massachusetts were to accept this low level of
non-transformational, non-interoperable XML,
they would have to kiss their SOA goodbye,
and accept the cost of an all Windows,
all the time from now on far into the future.”

It is not clear to me exactly what version he reviewed;
perhaps this has been fixed, or will be fixed.
But that’s not the point.
The point is that
there are lots of problems that can
be embedded in a specification when there’s no independent review;
this is just one example.
The need for independent review is very, very real.

Back in 2004, the European Union’s
Telematics between Administrations Committee (TAC)
asked “Industry actors not currently involved with the
OASIS Open Document Format consider participating in
the standardization process”; Microsoft chose not to.
TAC also specifically stated that
“Microsoft should consider the merits of submitting [its] XML formats
to an international standards body of their choice.”
Microsoft chose not to, again.
Microsoft has every right to choose to do what it wants, but if
a supplier won’t do something that is important to their customers,
especially after being clearly told to do so,
customers typically start looking at alternative suppliers.

Open

When you think “open”,
think “open competition” or “open market.”
An open standard should allow competition
based on merit, instead of limiting customers’ suppliers to
one particular supplier or a subset of suppliers based on their
business model, development model, licensing model, and so on.
You should be able to replace one product that does the same function for
another, as long as they meet the same open standard,
and achieve at least the same basic function provided by the standard
(though some may perform better or have additional features --
think of plug-replaceable components).

The Valoris report (noted above) defines an open standard this way:

“The minimum requirements for an open standard are that the
document format is completely described in publicly accessible
documents, that this description may be distributed freely and that the
document format may be implemented in programs without
restrictions, royalty-free, and with no legal bindings.”

Indeed,
if a specification or its license is designed to make it very difficult or
essentially impossible for plausible competitors to implement it,
then it isn’t really an open standard.
The Valoris report notes, for example, that
widespread public review of a specification
is needed to ensure that a specification isn’t unnecessarily
difficult for another party to implement it.

“Because of its specific role in society, the public sector must
avoid that a specific product is forced on anyone interacting with it
electronically. Conversely, any document format that does not discriminate
against market actors and that can be implemented across platforms should
be encouraged. Likewise, the public sector should avoid any format that
does not safeguard equal opportunities to market actors to implement
format-processing applications, especially where this might impose
product selection on the side of citizens or businesses. In this respect
standardization initiatives will ensure not only a fair and competitive
market but will also help safeguard the interoperability of implementing
solutions whilst preserving competition and innovation.”

ZDnet concluded that,
“[when] open standards exist which are capable of supporting
the work the state does, this should be an unexceptional decision;
accessibility for as broad a range of citizens and [organizations] as
possible is a primary responsibility for any government.”

To be an open standard, anyone should be allowed to implement it.
And “anyone” means “anyone.”
A format that locks out a large part of the supplier market is,
obviously, not open.
In particular, now that open source software has become a major market force,
it is clear that open source software (using their most common licenses)
must be allowed to implement the specification, or
the specification is not an open standard.
Quoting obsolete guidelines doesn’t duck the issue; anyone is anyone.

Microsoft’s letter has many words (especially on page 9 and on) on
many things they’ve done.
And I agree that a number of things that they list in
their letter are indeed very good things, and some are
unusual steps for Microsoft.
Microsoft acknowledges they’ve applied for patents, but they proclaim
in particular that they grant use royalty-free, in perpetuity,
and that anyone can read the specification and its license.

Why would anyone have a concern?
Because that wasn’t the question.

Microsoft’s conditions on the use of
their XML format are fundamentally incompatible with many of their
competitor’s licenses or business models,
according to a large number of people
who have examined the conditions imposed by Microsoft.
They certainly fail to meet the definition of “open”
as defined above in the Valoris report (which required no restrictions
and no legal bindings).
There’s no practical way many of Microsoft’s competitors
could use Microsoft’s format given the license
Microsoft is imposing with it (in countries like the U.S. that
tolerate software patents).

Basically, if you choose Microsoft’s XML format, you have
decided against open competition, in perpetuity.
A naïve reader might not realize this, but Massachusetts
has been researching this issue for a long time and has learned
that the details matter.

For about two years,
Massachusetts has been working to identify its requirements,
discussing its options with others,
and asking people what the impact would be of various alternatives.
They found out that the impact of accepting these conditions
would be severe: they would probably
lock themselves into a single supplier or small set of suppliers
in perpetuity, and they
would certainly forbid real competition.
Massachusetts does not want to be locked into one supplier,
or a small set, in perpetuity; they want anyone
to be able to develop software for them to work with their data.
And they can’t keep this fundamental
right if they use Microsoft’s XML format.

This isn’t just my opinion; it’s the
considered analysis of people who have reason to know.
Groklaw
has posted a lengthy analysis by Marbux, a retired lawyer
(updated April 1, 2005). His detailed analysis found that
Microsoft’s specification excluded competition, in contrast with
Microsoft’s public claims. “Competitors are... effectively
precluded from bidding against Microsoft or its suppliers for
any... contract specifying use of Microsoft’s software file
formats.” He first noted that the patent license for the format
“is structured to be read restrictively, in Microsoft’s
favor... it states that: ‘All rights not expressly granted in this
license are reserved by Microsoft. No additional rights are granted by
implication or estoppel or otherwise.’ This is not the customary
‘all rights reserved’ phrase more commonly encountered... If
you cannot find words in the license explicitly stating that you have the
right to do something, you don’t get that right.”

Then, by
examining the patent license in detail, he found a number of omissions
and conditions that suppress competition: there is no integration
clause, no license for the schemas themselves, no grant of copyright was
included in the patent license, no commitment to delivering any future
changes to the schemas or right to develop software implementing them
under the same or more liberal license (this particular issue may have
been resolved later by Microsoft), no identification of the Microsoft
patents involved, no identification of third-party patents, no right
to sell or sublicense implementing software, a prohibition against
sale and licensing of implementing software, a prohibition against
software having functions other than to read and write files using
the specification without modification, no license to convert files
to and from other formats, no right to write files using the schemas,
vagueness and ambiguities will deter implementation by developers and
adoption by end users, and a discriminatory incompatibility with F/OSS
licensing, and discriminatory incompatibility with proprietary software
competitors. In short, he believes Microsoft’s license prohibits
effective competition from using the format.

This analysis is supported by others.
Richard Stallman, the author of the GNU General Public License (GPL),
states that Microsoft’s license was “designed
to prohibit all free software. It covers only code that implements,
precisely, the Microsoft formats, which means that a program under
this license does not permit modification... The freedom to modify
the software for private use and the freedom to publish modified
versions are two of the essential components in the definition
of free software. If these freedoms are lacking, the program is
not free software [as defined by the
Free
Software Definition].”

Indeed, let’s look at a particularly important example -- software
that is written and released using the GNU General Public License (GPL).
It turns out the GPL is a really good “acid test” for
determining if full and open competition is possible.
This isn’t an academic exercise;
there are already open source software office suite programs
that are licensed under the GPL, such as KOffice and Gnumeric.
That’s not surprising, because the
GNU GPL license is the most popular license, by far, for
open source software.

For example, SourceForge.net reported on November 10, 2003,
that the GPL accounted for 71% of the 45,736 projects it
hosted with open source licenses;
the next most popular licenses were the LGPL (a variant of the GPL) at
10% and the BSD licenses at 7%.
Billions of dollars worth of effort have been invested in software
released under the GPL.
These programs cannot realistically change to another license; they
have a massive amount of code, they reuse other code that requires the
GPL, and their development model depends on the GPL.

Dan Ravicher, executive director of the Public Patent Foundation,
says that “If [Microsoft has] rights and a license is needed, then
the term in the license that requires attribution by the licensee
of all of its downstream licenses is, in fact, not compatible with
the GPL.”

Even Jean Paoli,
senior director of XML architecture for Microsoft, acknowledged that
their attribution requirement might preclude any program that uses the
file formats from being used in open-source software
licensed under the GPL.
Paoli admitted, “The GPL may not allow code that is
attributable to another company like Microsoft to be included.”

Now, perhaps Paoli doesn’t know for sure, but that seems unlikely;
Microsoft has lots of lawyers, and people have been asking them point-blank if
really any competitor is allowed to use their format since at least early 2004.
Many believe that Paoli does not wish to admit that Microsoft’s XML
license is incompatible with the GPL, because that admission would
make it even more obvious that the license prevents real competition.

Such discrimination is unacceptable, runs counter to many customers’
policies, and it’s certainly not in the interest of customers.
It’s particularly absurd if such restrictions control access to
public documents (which is what government documents are).

If a specification cannot be implemented using the GPL, it
discriminates against open source software (because the GPL
is the most common such license).
If a specification discriminates against open source software implementations,
then it is not a specification that allows open competition.
This was not as big an issue decades ago, when large-scale open source
software systems were uncommon, but it sure is now.

Some examples from the U.S. federal government are probably instructive.
Back in 2002,
MITRE’s “Use of Free and Open Source Software in the US Dept. of Defense” (DoD) [PDF] found that
“banning [open source software / Free software (OSS/FS)]
would have immediate, broad, and strongly negative impacts
on the ability of many sensitive and security-focused DoD groups
to defend against cyberattacks.”
The report also found that the GPL so dominates
in DoD applications that a ban on just the GPL would have the
same strongly negative impacts as banning all OSS/FS.
MITRE noted that OSS/FS “plays a far more critical
role in the DoD than has been generally recognized.”
The
U.S. DoD has a formal policy
of neutrality between OSS/FS and proprietary software.
The U.S. federal government also has a position of neutrality, as
discussed in the
Office of Management and Budget memorandum M-04-16 of July 1, 2004.
Massachusetts is not bound by these federal documents, but they
have many of the same issues.
Using a format that cannot be implemented by OSS/FS programs,
using the most common license for them, is clearly discriminatory against
them and is not neutral.

The formal Standards Setting Organizations (SSOs), as
organizations representing the standards creators, consider a standard
to be open if the creation of the standard follows the tenets of open
meeting, consensus and due process.

An implementer of an existing standard would call a standard open
when it serves the markets they wish, it is without cost to them,
does not preclude further innovation (by them), does not obsolete
their prior implementations, and does not favor a competitor.

The user of an implementation of the standard would call
a standard open when multiple implementations of the standard
from different sources are available, when the implementation
functions in all locations needed, when the implementation is
supported over the useri’s expected service life and when new
implementations desired by the user are backward compatible to
previously purchased implementations.

Microsoft’s letter quotes the
Valoris
report from 2004, which said
“the MS license provides access to the schemas and full documentation,”
and also quoted some positive European Union statements about their
license based on the Valoris report.
But the Valoris report, page 51, said this as its primary
statement about the legal issues:

“The associated legal terms seem to create a lot of controversy.
As we are not qualified to make a judgement on this basis, we
will simply highlight the main points hereafter, and recommend
examining carefully the legal aspects of the license.”

Catch that?
The Valoris report was a market analysis with some
technical evaluation, and it was specifically not a legal analysis.
It is unreasonable to depend on the Valoris report for a legal analysis;
the Valoris report specifically
said it was not a legal analysis, the writers frankly state that they
were unqualified to do a legal analysis,
and they recommended that a legal analysis be performed as follow-on work.
We want to know if
any competitor can legally use the schema, not if people can read it.
Groklaw posted the most detailed public
analysis I know of, updated on April 1, 2005,
and it found a massive number of problems.
At the very least this license
discriminates against the most popular license for open source software;
not even Microsoft is willing to deny this.
And those who’ve examined this license
the closest have found a vast number
of restrictions in it that prevent or inhibit competition.

Note that this is fundamentally different than the issues surrounding the
old binary formats of Microsoft Office (.doc, .xls, and .ppt).
It was legal for competitors to implement these binary
formats; the problem was that
the specifications for these binary formats
were never published for competitors to use.
Because the specifications for those formats were not published,
and because Microsoft kept changing the formats internally,
competitors have had to spend many man-years figuring out those formats so
that they could read and write them and maintain interoperability.
But in fact, they have managed to do so; essentially all office suites
can read and write these formats.
No office suite, not even any particular version of Microsoft Office,
can read and write these binary formats perfectly, but
all are free to try.

The difference now is that patents are involved.
Microsoft has patented some aspects of its new XML format;
if these patents are truly valid, the law makes it
illegal to use Microsoft’s
XML format except under the terms Microsoft has created.
Many are especially concerned about this in the case of Microsoft,
because of the secret Microsoft documents now named
Halloween I
and
Halloween II
that were exposed to the world a few years ago.
These documents were developed in secret through a collaboration with key
people in Microsoft, and their bottom line was a
recommendation that Microsoft suppress competition by
“de-commoditizing” protocols (i.e., creating proprietary formats
that could not be used by others)
and by attacking competitors through patent lawsuits.
The use of a patent here does not necessarily mean that
is Microsoft’s intent,
but it does explain why Microsoft’s actions with
patents are watched so closely.
Since patents can be used to enforce
terms that are otherwise unenforceable, such patents
become a key issue.
The issue is not that patents necessarily
disqualify a format; the issue is that
patents can enforce terms that then need examination.

Microsoft hopes to have Office 12 out next year,
so there aren’t really any full, mature implementations
yet of the XML format that Microsoft wants Massachusetts to standardize on.
(
eWeek’s Steven J. Vaughan-Nichols, among others, notes that
“the so-called open Office 2003 XML formats aren’t going
to be Office 12’s formats.” -- they are incompatible.)
Are there multiple implementations? No.
How about one using the GPL (a really good acid test for competition)? No,
and unless there are license changes, there won’t be one.
So, there is no evidence that there can be
free and open competition using the Microsoft XML formats.

Bottom line on open, agreed-up standards: Competition permitted

This isn’t complicated at all, really.
The point of all this is full and open competition,
with a range of competitors actively in the market.
Are some valid suppliers prevented from entering the market, or
at an unnecessary disadvantage?

Few leading vendors ever really want to have an agreed-on open
standard, because that opens the market to competitors.
But once a technology stabilizes, wise customers demand standards that
let them pick between suppliers, and customers are the ones paying the bills.
This happens all the time, in all technology areas.
In this case, office documents have been around for decades, so it’s
a very mature technology and long overdue for an open standard with
a competitive market.

Massachusetts explained at their Sep. 16, 2005 “Town Hall” meeting
what being open required.
They included no or minimal legal restrictions
(so anyone can implement it),
it’s published and peer reviewed (so problems can be fixed and
it’s easy to implement), and
joint evolution (so future changes are not controlled by a single entity
or pair of entities, but by all).
These are good and necessary requirements, but I would add one more:
evidence of unfettered, active competition (in the form of implementations
with a variety of licenses).

It’s all too easy to meet some list of requirements, yet
fail to really provide what the customer wanted.
The best proof of competition
is in the pudding: are there
competing implementations, under a sufficient variety of licenses
(proprietary through GPL), that
proves that there is no discrimination?
Are those competitors reporting that they are disadvantaged by the standard?
HTML is a good example --
there are a variety of readers and writers of the HTML format
(with proprietary and open source software licenses, including the GPL), and
there is a neutral body (W3C) which created, published,
and maintains the standard.
PDF does very well by this measure, too.
There are multiple PDF readers and multiple PDF writers, in both cases
ranging from proprietary through open source software
(including those using the GPL).
ISO acts as a neutral body to review and maintain PDF specifications.
Later in this paper I discuss PDF in more detail.
OpenDocument is easy to prove the same way:
there are multiple OpenDocument implementations
already, some proprietary, some open source software (including those
using the GPL), with more on the way;
OASIS is acting as the neutral party supporting
peer review and joint evolution.

Obviously, it is possible to have competitors even when all others
are at a competitive disadvantage.
Which is why the other points Massachusetts raised
(no legal restrictions, published, peer reviewed, and joint evolution) are
still important to examine.
But the real point is to foster full and open competition.

Choosing OpenDocument does not mandate open source software, nor
does it mandate proprietary software.
OpenDocument is a choice that lets you choose.
Choosing Microsoft’s XML format locks you into one supplier for now
and possibly forever; it certainly doesn’t lead to a road with
full and open competition.
Even if it is legal to implement the format, which the legal
analysis earlier makes doubtful, and even if competitors implement it
fully (which may be technically difficult since it was not reviewed
through a consensus process),
the complete control of Microsoft’s
XML format by Microsoft into the future serves as a deterrent to competition.
Microsoft appears to have
a right to limit who can use their proprietary XML format in the U.S.,
according to current U.S. law; but customers have a right to avoid it, too.

If we take this part of the letter at face value,
by itself it disqualifies Microsoft’s XML format from consideration.
Microsoft’s format was never reviewed by an independent body with
representatives from multiple suppliers.
It can’t be implemented by all suppliers, either.
Even if there are suppliers who can and will use it, they
would be unwise to depend on it, because it would be directly controlled by
an organization with a financial incentive to harm them
(e.g., through capricious and unforewarned changes).
Governments have been asking Microsoft to change their approach to
their XML format for a long time, to no avail.
Microsoft no doubt has business reasons for its decision, but
Massachusetts has the right to make decisions that are best for it too.

Now that we understand some of the real issues underlying this
letter, let’s move on.

We have substantial
concerns, however, with the definition of “open formats” in the current
proposal. This definition mandates adoption of a single, immature format
for office documents throughout the Commonwealth’s executive agencies...

And here we come to the nub. They completely disagree with Massachusetts’
choices, and try to squeeze as many complaints as they can into one paragraph.

It may seem odd that they start by complaining about a definition, of all
things, but it makes sense.
As any government should, Massachusetts started its decision-making process
by identifying its requirements.
In this case, Massachusetts identified one of
their key requirements as the need for an open format, including
a definition of that term.
Microsoft is unhappy with this requirement.
By saying that the definition “mandates” the choice, they
acknowledge that Microsoft’s format does not meet
the Massachusetts requirement as currently defined, nor do they want
to perform the necessary changes so it would meet the definition.
Since they do not want to perform the changes necessary
to meet the requirement,
they want Massachusetts to change the requirement.

This kind of request is not necessarily unreasonable,
depending on the strength of their argument.
However, governments often identify their key requirements and try to
meet them directly, so the argument needs to be really strong.
The rest of the letter tries to give that argument.
In my opinion, their argument is inadequate.

They also complain that the format is “immature”.
All formats are immature, in the sense that there will be future improvements,
clarifications, and so on.
But OpenDocument has lots of evidence justifying that it
is mature for its intended purpose.
We’ll examine that evidence in more detail below, in two parts:
is the specification mature (in concern (iii)),
and will the products be mature in the timeframe Massachusetts is requesting
(in concern (v))?
Let’s get back to the letter.

and effectively requires deployment of a single office application
technology within those executive agencies.

Microsoft says here that OpenDocument would require deployment of
a “single office application technology.”
This is a bizarre argument, because it’s a good reason to avoid
Microsoft’s XML format (which they’re proposing instead).
To my knowledge,
there is only one application technology and one supplier
who has committed so far to Microsoft’s new XML format -- Microsoft.
Certainly Microsoft’s format has far fewer suppliers backing it
than OpenDocument.
This text seems to suggest that Microsoft’s XML format
should be disqualified, by their own rationale.

In contrast, there are multiple suppliers of OpenDocument already.
OpenOffice.org and StarOffice do share a technology.
But KOffice already implements OpenDocument
with a completely different implementation.
See my discussion about concern (v) for
more information.
Perhaps someday there will be multiple office suites that implement
Microsoft’s XML format, but since all the other suppliers are working
on OpenDocument, it’s quite reasonable to think that OpenDocument has
a much more promising future.

Besides, Massachusetts is clearly trying to get many suppliers to
implement this requirement by 2007, not lock themselves into
the choices that are available now.
Massachusetts has announced a requirement that
won’t be imposed for 16 months.
The most likely reason for this lengthy lead time
is to encourage even more suppliers to add support so
that they can compete for Massachusetts’ business.
This is how any big customer gets its needs met -- it announces
its requirements, with enough lead time so competitors can meet the
requirement, and then it picks the best available solution.
The text about concern (v) discusses implementations
in more detail.

Massachusetts could choose to use two incompatible standards for the same
thing, but nobody wants to do that.
The costs and problems are high, with very little benefit to
Massachusetts.
If Massachusetts used Microsoft’s new
patent-encumbered format at all internally, then
essentially they’d be locking themselves into a single supplier for
years to come to process those documents.
Choosing two formats gives them all the drawbacks without any
of the benefits.
Requiring a single format -- as long as anyone can implement it --
levels the playing field.

As such, this unprecedented approach...

I respectfully disagree.
It quite normal
for a government to determine its requirements,
pick a standard for meeting them, announce the standard,
and then make purchasing decisions;
governments do that all the time.
In fact, this announcement is unusually generous; it gives plenty
of time (16 months) for interested parties to implement the standard.
OpenDocument can be implemented by anyone (without restriction), and
was developed by an industry consortium of multiple suppliers.
This is the ideal case, actually; governments sometimes have
to make many more compromises to get things done, and are usually delighted
when they have such a clear option available to them.

It’s not unprecedented to change office programs, or even
to have a market-leading office program get dethroned.
Remember WordStar? VisiCalc?
It’s not even unprecedented to stop using Microsoft Office, if that
turns out to be the ramification;
other governments have done so.

In fact, the history of standards and governments has many similar examples.
The U.S. government imposes a standard for the distance
between rails in a railroad track, for example.
There was a significant one-time cost for doing this, but the continuing
advantages have been tremendous.

Having a lock on the current market is no guarantee of the future,
in general; change is certainly not unprecedented.
David Halberstam’s book “The Reckoning”
gives an example from carmaking:

“The US Big Three automakers thought that they could dictate what their
‘captive’ market could buy, but the US public proved that assumption
to be false, in the mid seventies. The only survivors from that era of
heavy, rear wheel drive land yachts (albeit in much reduced and much
improved forms) were the Ford Crown Victoria/ Mercury Grand Marquis and
the Lincoln Town Car. Every other passenger vehicle is some variation
of the K-car.”

It’s also quite quite common for a group of suppliers
to overtake a big market-leading supplier by
agreeing on a (more) open standard; the resulting competition tends
to produce superior products at lower prices, and customers quickly
flock to the open standard.
Look at the videotape standards war of VHS vs. Sony Betamax
in the 1980’s.
Sony was a big
company, trying to control an industry through a format
it created -- Betamax.
But the rest of the industry chose VHS, which allowed many different
suppliers (Sony tightly controlled who could implement Betamax,
while the VHS specification was far more open to implementation by
others).
The group of VHS suppliers quickly competed with each other,
while staying true to the standard, customers preferred formats where
there were competing suppliers, and as a group the VHS suppliers demolished
Betamax’s market share.

There’s even a slang term based on this;
“to betamax” means “to deploy a proprietary technology format
that gets overwhelmed in the market by a more open format that
allows multiple competing manufacturers.”
There’s more to that story, of course, but my point
should be clear: open standards, supported by competing implementations,
tend to win in the marketplace.
The fact that there’s a slang term for this complex situation
tells you something else:
this script happens repeatedly.

The market forces are clear;
often smaller competitors (who fear being
driven out of the business) will band together with customers (who fear
having a sole supplier) to develop and promote a standard that is not
(as) proprietary.
Eventually the broadly-created open standards tend to win,
because customers make the final decisions, and few customers want
to be controlled by any single supplier.
Having broad input into the
standard’s development also helps make sure that all important needs
were addressed, as well as enabling equal competition.

not only prevents impacted state agencies of the Commonwealth from using
many critical and well-established technologies, ...

And there it is! The real reason they believe
Massachusetts shouldn’t do this is right here.
Microsoft doesn’t want to support OpenDocument, even though they can.
If Microsoft won’t support OpenDocument, and
Massachusetts sticks to its requirements,
then Massachusetts may eventually need to migrate to
a different office suite other than Microsoft Office.
Microsoft doesn’t want Massachusetts to use the standard, nor
do they want to implement the standard.
I suspect many will read this text as a threat
to punish Massachusetts for daring to choose the international standard
instead of Microsoft’s proprietary specification.

This is circular reasoning.
Microsoft has the power to implement OpenDocument in Microsoft Office,
if they choose to do so.
It’s not a lack of money, certainly, and the specifications are free for
anyone to implement (including Microsoft).
Sixteen months is enough time to implement it, and if they don’t want to
do it themselves, they could
license the code from someone who has already done it.
It is not Massachusetts who is deciding
to stop using Microsoft Office; it’s
Microsoft who is threatening to not do business with Massachusetts.
Massachusetts does not want to be locked into any single supplier, so it is
choosing a format that any supplier can implement.
If Microsoft decides to not implement the requirements that its customers
demand, then Microsoft shouldn’t be surprised if customers switch to
suppliers more willing to meet customer needs.
It’s how competition works.

Wise customers choose markets with many competing suppliers.
The competition acts as an impetus for both innovation and cost reduction.
Think Betamax, for example.

Massachusetts’ Frequently Asked Questions (FAQ) makes some
interesting points here.
This selection won’t really impact the current well-established technologies,
such as Microsoft Office; as they note,
“use of existing MS Office licenses is allowed as long as
agencies use a method permitting the saving of documents
in Open Document Format.”
They apparently already have a process for doing this.
Their policy also
limits “the applicability of the Open Document Format standard
to documents created by the Executive Department in the future”
so they won’t have to convert what already exists.

but also runs afoul of
well-established procurement norms without due
consideration for the enormous costs and technical challenges that stem
from the proposal.

Not at all.
We’ll talk more about procurement norms as part of
concern (i), but a few comments are appropriate now.
If there were only a few days to discuss this, I agree that’d be a problem.
But this has been going on for a long, long time.
Paula Rooney reports in InformationWeek that
“The state and software company have locked horns on this issue
for more than two years.”
There were major statements in January 2005, and really all through the year.

We simply do not believe that the proposed mandate
for this exclusive document format is the best solution for achieving the
Commonwealth’s laudable goals.

It’s always fair to state that you have differing beliefs, so this
is perfectly reasonable.
But clearly, we need to look into why they believe that,
so let’s look at their key concerns.

Microsoft’s key concerns are as follows:

Now we have a short list of “concerns,” followed later by more detail.
It’s hard to respond to a claim without seeing the supporting information
behind it, so I’m going to show not only each of their concerns, but
I’ll extract some details from the rest of the letter to see what
they really mean.
Unfortunately, this letter is not well-organized.
In the front, it has a list of 7 concerns numbered (i) through (vii),
but the details of its arguments are numbered 1 through 5 with different
headings.
Since the organization of the rest of the letter doesn’t exactly
match the list of key concerns given in the front, it’s
more awkward than most letters like this.
Let’s soldier on, looking at each of their 7 concerns and then
the closing text in the front part of the letter.

(i) ANF did not provide sufficient time for review and comment on the
proposed policy, nor a robust process for addressing comments. Due process
requires much more, particularly given the unprecedented nature of the
proposal and the potentially adverse consequences it could provoke,

This clearly maps to the details listed under “reason 1,” which is
later in the letter. Reason 1 claims that the
“Executive Office for Administration & Finance did not provide
sufficient time for review and comment on the proposed policy, nor did it
provide a robust process for addressing comments.”

Microsoft’s letter gives more details, and quotes various laws,
which I don’t want to bore you with.
In short, this claim doesn’t seem to hold water.

“There’s no question that Microsoft’s Office XML Reference Schema
was once on Massachusetts’ “list,” and then, as a
matter of public discourse--some might say democracy-- removed
from that list. Going back to the notes from
Massachusetts’ June 9, 2005 meeting with industry--
referred to by Quinn as the Open Formats Summit--it would
be impossible for anybody, Microsoft included, to assume
that any format that was once on the “list” was guaranteed
of staying on that list. In the very last section
of those meeting notes, the next steps were clearly defined as:

Identify a continuum of acceptable open document standards for the Commonwealth.

Revise the standards and publish the revision for public comment; then finalize the standards.

These action items practically scream out that, at the time, Massachusetts
was undecided on a standard and that the due diligence for choosing one
was far from over...
Microsoft’s team on the job--Stuart McKee, Brian Burke, and Leslie
Tan--didn’t have to look far to see some of the criteria for openness
that the State was considering. It was displayed in a footnote on the
bottom of the June 9, 2005 meeting notes and stated as follows:

‘Specifically, [Ken] Krechmer, [Fellow, International Center
for Standards Research, University of Colorado] notes, standards
creators typically consider a standard to be open if the creation of
the standard follows the tenets of open meeting, consensus and due
process; implementers of an existing standard would call a standard
open when it serves the markets they wish, it is without cost to them,
does not preclude further innovation (by them), does not obsolete
their prior implementations, and does not favor a competitor; and
users of an implementation of the standard would call a standard open
when multiple implementations of the standard from different sources
are available, when the implementation functions in all locations
needed, when the implementation is supported over the user’s expected
service life and when new implementations desired by the user are
backward compatible to previously purchased implementations.’

[Noting that Microsoft’s format would fail this test,]
With the writing on the wall, the question that sticks out in my mind
is why didn’t Microsoft sound its internal fire alarm sooner?”

It’s certainly not a last-minute decision; the issues and their
ramifications have been publicly discussed for a long, long time.
It’s not just Massachusetts;
a lot of work was done by the European Union in 2004
(and probably even earlier), and governments talk together about things
like this.
There’s absolutely no evidence that any new information is forthcoming;
the issues at this point are well-understood.
Decision-makers are paid to make decisions, not to stall forever.
At some point, there comes a time to make a decision; not making
a decision is, in the end, a decision.

In their FAQ, Massachusetts explains why this didn’t require
a regulation review process -- because it’s not a regulation.
“In issuing the draft Final ETRM version 3.5,
ITD is not proposing a ‘regulation’ as that term is
defined under the state’s Administrative Procedures Act,
Mass. Gen. L. ch. 30A. Section 2 of that Act defines ‘regulations’
for which agencies must conduct formal notice and comment proceedings
to exclude ‘(b) regulations concerning only the internal management
or discipline of the adopting agency or any other agency,
and not substantially affecting the rights of or the procedures
available to the public or that portion of the public affected
by the agency’s activities’. Under the Final E
TRM 3.5, ITD would require use of Open Document format
only for documents create by Executive Department agencies.
It would not require use of such formats by citizens, businesses,
and other governments who communicate with the Executive Department.
ITD is a control agency with no regulatory authority over any citizen,
business or governmental entity outside the Executive Department.
In that the Final ETRM 3.5 does not constitute a ‘regulation’
under the APA, ITD was not required to engage in formal rulemaking
in connection with the issuance of this standard.
It offered an information notice and comment proceeding voluntarily
and in the spirit of cooperative leadership that ITD promotes.”

Their FAQ discusses several other legal points, justifying their
belief that the law was followed.
It appears that Massachusetts is quite within the law.
In fact, it appears that they went far beyond all
legal requirements; they had an unusually lengthy public comment period.

It’s important to note that this is not a procurement, anyway.
Eric Kriss made this very clear at the Sep. 16, 2005 “Town Hall” meeting,
and it was clear before that time too.
Massachusetts is warning suppliers that a procurement will come in the
future, and that a particular standard will be required in that procurement.
This is just what suppliers want to know, so that they can make their plans.
By letting suppliers know the requirements ahead of time, suppliers can
plan and work to meet those requirements, if they want to be considered for
the procurement that will come later on.
It makes no sense for Microsoft to complain that they’re being warned
about a future requirement;
instead, Microsoft should be thanking Massachusetts for giving
clear direction and a long lead time.

(ii) the proposed policy would create significant costs and problems
for state agencies, for the private sector, and for its citizens,

Well, any decision can involve costs and problems; even
not doing anything can involve costs and problems.
To figure out what they mean, we’ll need to look at the details
later on in the letter. Here’s what they say in their
“reasons” section, which gives more details:

2. The proposed policy would impose enormous costs on the Commonwealth of
Massachusetts and on its citizens and the private sector.

If the proposed policy were put in place, the fiscal impact on
the Commonwealth, its citizenry, and the private sector would be
substantial.

First, there would be significant, and entirely unnecessary,
costs incurred by all state agencies, departments, cities, counties,
and school districts to procure new software applications that support
the OpenDocument format for their individual users. Many state agencies
already have licenses for Microsoft Office and other software products
that do not support the OpenDocument format, and the expense already borne
by these state agencies for Microsoft Office and such other products’
licenses would be wasted by disallowing use of these products after
Jan. 2007.
As a result, costs to taxpayers would rise as executive
agencies would be forced to toss out software they have already paid
for, that they already know how to use, and that they can already use
for archiving in open standard XML formats.

Second, every state agency,
department, city, county, and school district would face enormous document
and/or application conversion costs and would need to invest in training
and support activity in order to make this conversion, with potential
risks arising from conversion errors in these public documents.

Massachusetts’ FAQ has some simple answers to this.
Most obviously, this directive
“applies only to the Executive Department, and then only to
documents created by the Executive Department.
Implementation plans will take into account the need to
maintain interoperability through the use of a variety of acceptable formats.”
Yes, they already have licenses;
“use of existing MS Office licenses is allowed as long as agencies
use a method permitting the saving of documents in Open Document Format.”
They already have such methods available, which gives them a graceful
way to implement this plan.
Besides, they note that they will implement this in a phased approach.
By doing things in phases, they can dramatically reduce the effort.

Though it doesn’t appear in their FAQ, I know that a survey found that
far more people just read documents than write them.
I would certainly expect that most documents created by the Executive
Department would be read more often than modified by those outside of it.
In that case, this issue is completely
irrelevant; people will just want the PDFs anyway.

Possibly most important, it’s not a question of cost vs. no cost;
it’s a question of where the money should go.
Their FAQ notes that
“the Executive Department will face significant costs
over time in connection with office application upgrades.
There is no evidence that migrating to office applications that
support Open Document Format will be any more costly than
upgrading current applications.”

(Footnote)
The impact of this proposal extends far beyond Microsoft. For example,
agencies that have chosen to make use of Corel software, such as the
Massachusetts court system, will face similar challenges, and it is
unknown how the proposed policy will adversely impact existing guidelines
in place for such agencies, such as the Massachusetts court system’s
electronic submission guidelines.

We’re talking here about Massachusetts’ executive branch.
The judiciary was not in the executive branch, so this is irrelevant.
In fact, Massachusetts’ courts determined about 30 years ago that,
by the rules set in Massachusetts’ Constitution,
one branch of Massachusetts’ government
could not set the information technology rules for another branch.

(Footnote)
Some may argue that lower licensing costs associated with software
technologies supporting the OpenDocument format counters the cost
associated with the migration. Recent Gartner analyst reports, however,
have documented examples where organizations who have closely evaluated
the issues conclude that a move to alternative software has no defensible
ROI; in fact, those organizations have concluded that the preferred
approach was to maintain and continue deployment of Microsoft’s most
recent software technologies. See: A Financial Institution Sees No
ROI on Desktop Linux. In many cases, companies license technologies
for “free” or at very low sales price in the hopes
of making money on
other related products and services including sales of complementary
proprietary software and hardware and service contracts. There are a
number of examples of government entities that migrated away from some
of the software that will be supporting the OpenDocument format due to
total cost of ownership (including testing and installing, and training
costs) among other factors.

This is a completely irrelevant study.
Massachusetts is requiring a file format, not any particular product.
The referenced article talks about “Desktop Linux”, and this whole
footnote is talking about GNU/Linux -- which isn’t even the topic.
The issue being addressed by Massachusetts is
a standard file format (OpenDocument)
that can be used on Microsoft Windows,
Apple Macintosh, GNU/Linux, Unix, whatever.
GNU/Linux is an interesting topic, but irrelevant,
unless you wish to note that Microsoft’s file format is
not supported on Unix and Linux (and thus inhibits the free selection of
products).
However, that’s yet another argument for
avoiding Microsoft’s XML format, and
using OpenDocument instead, if that is the point.

Even if this were relevant (it’s not),
reusing ROI or TCO studies is inappropriate.
The purpose of a TCO or ROI study is give an answer for a particular
organization; such studies are so sensitive to the specific
organization’s situation that you
usually can’t reuse them for anyone else.
Others’ ROI or TCO studies can be helpful to point out what might
happen or might be true, so as “examples”
it’s fair to say it’s possible.
But they are useless for proving what will happen for someone else.

Even if this were relevant (it’s not),
and even if reusing ROI/TCO studies made sense (it doesn’t),
real TCOs and ROIs for governments look completely different from commercial
organizations.
So in particular they can’t be reused for this case.
The reason is that
governments plan to be around for centuries, and they plan to have to
deal with their old documents in those future centuries.
When your time horizon is in decades or centuries, you have to assume that
you’re going to need to upgrade many times, and that you’ll have to
change suppliers many times.
Today’s
transition costs matter, but are dwarfed in these timeframes.
What standards will help you handle supplier changes many times over centuries?
Probably not one controlled by a single supplier.
And governments have requirements, like open access to all their
constituents (and not just the rich ones), that a commercial company does
not need to address.

Besides, a real TCO has to look at the whole picture, not just the
office suite software.
As Carr notes, Massachusetts is
“launching a serious and comprehensive initiative to replace
its fragmented, inefficient set of traditional information systems
with a modern, coherent, and flexible IT architecture that allows
data to be shared and reused easily.”
The cost reductions from replacing a fragmented set of systems with
a flexible architecture that allows full data sharing will probably dwarf
any additional costs for the office suites themselves, if they even have
an additional cost.
And the flexibility of using any supplier, which Microsoft’s
XML license does not grant, will probably lower the costs
of implementing this.

But it’s actually pretty unlikely that the suites will cost more.
In fact,
it appears this move will also save them money on the suites themselves,
even though that is not the primary purpose.
At a
Sep. 16, 2005 Town Meeting,
Eric Kriss, the state’s Secretary for Administration and Finance,
revealed some very interesting numbers.
The state operates about 50,000 desktops, mostly
Windows 2000 and Office 11.
They roughly estimate that to upgrade to
Office 12, which Microsoft is offering as the way to implement their
format, would cost $50 Million
(including software licenses, upgrading operating systems as
needed, newer hardware in some cases, and training).
In fact,
an Office 12 upgrade
will apparently require them to upgrade a bunch of operating systems too,
and since its user interface has substantial changes, there will be
a substantial training cost no matter what they do
(the alternatives may actually have a lower training cost too).
They estimate the hypothetical
cost to install OpenOffice.org instead (one of the
programs that implements OpenDocument) as $5M with comparable components.
Even though these are crude estimates, the scale of the difference suggests
that even getting better pricing from suppliers and discovering
an unaccounted-for cost is not likely
to make Microsoft the less expensive bid.
That obviously includes more than the purchase price, because OpenOffice.org
is available at no cost, in perpetuity.
Even more tellingly, it’s reasonable to believe that many of the
$5M costs are one-time costs, not per-upgrade costs; it would presumably
cost much less for all upgrades after that.
Even a great vendor discount can’t hide the cost difference.
When you factor in upgrades every 2-3 years, over centuries, the
difference becomes extreme.

Besides, as Kriss notes, cost is not the key reason for this decision.
Sovereignty is.
They’re willing to pay more, if it’s necessary to meet their
fundamental needs.
If cost was the only reason to make a decision, we would all
be driving the cheapest car available.
Sometimes it is worth paying more, to acquire something else.

Third,
extensive work would have to be done deep within the IT infrastructure
of the Commonwealth
to manage conversions of “non-compliant” documents and mapping of
processes that work well today to new, untested systems. On a daily basis,
state agencies would need to work with private sector organizations
and citizens to devise ways to convert documents back and forth and to
troubleshoot problems. One could also assume that other branches of the
Commonwealth’s government would incur substantial expenses in order to
adapt IT systems to be able to interface with the overhauled systems
of the executive branch.

As noted above, no conversion is needed; this policy
only applies to new documents.
There are freely-available programs that do this conversion,
if it’s needed (OpenOffice.org even has a built-in
“bulk converter” does that does it very painlessly).
In fact, it’s hard to not notice that Microsoft’s own letter
was quickly translated to OpenDocument format when it was posted by
Massachusetts.

And again, this argument can be turned around.
If switching to an XML format is this hard, you certainly want to choose
a format that everyone can implement, because you don’t want to get
stuck with a single supplier or a small set of suppliers.
Otherwise, you risk having that supplier raise their rates once you
have committed to them.

Fourth, new costs and problems will also be
imposed on those doing business with the state, including organizations,
businesses, and citizens, as the proposal could take away their choice
of the software they may want to use to interact with the government to,
for example, bid on a government project, submit filings, or correspond
with government officials. Further, Massachusetts companies who currently
sell products that do not comply with the proposed policy to Massachusetts
agencies will be cut off from a major customer base.

The way to allow choice is to pick a specification that all suppliers can
implement. OpenDocument meets that criteria, Microsoft’s does not.
If the supplier chooses to not implement the international
specification, that’s the supplier’s choice,
not the customer’s.

Indeed, the proposal
itself acknowledges the current pervasive deployment throughout impacted
agencies of technologies not compliant with the proposed policy and
the magnitude of the resulting costs that would be associated with the
migration effort: Given the majority of Executive Department agencies
currently use office applications such as MS Office, Lotus Notes and
WordPerfect that produce documents in proprietary formats, the magnitude
of the migration effort to this new open standard is considerable.

This will be a big change;
Massachusetts has lots of office suite applications.
They will need to be upgraded, replaced, or have extensions/plug-ins
added to support the new format.
How extensive it will be will depend on the suppliers, to some extent, and
will greatly depend on wise planning.
But big changes often have the biggest payoffs, if they’re handled well.

Note that Microsoft concedes here that their current Office suite
produces documents in proprietary formats.

There
is simply no principled basis for causing the foregoing costs to be borne
by the Commonwealth, its citizens, and the private sector, ...

Not true.
Massachusetts has repeatedly emphasized why it needs standards that
are widely vetted by an independent body, and that are implementable
by all.

At the Sep. 16, 2005 Town Meeting,
Eric Kriss summarized the reason very simply: sovereignty.
Kriss noted that without public documents, you don’t really have
a democratic government, and increasingly many government
documents are only available electronically.
The fundamental question here is, who owns Massachusetts’ documents?
The supplier, or the state?
If it’s the state, then it must be able use its data
however it chooses, using any supplier’s products.
“Public documents must be as open and as unencumbered as possible...
because it’s a democratic imperative.”
I.E., if the state wants to be able to exercise its rights, then its
data needs to be in a format where it can actually exercise its rights.
Microsoft is preventing this free exercise, by not allowing its
XML format to be reviewed and controlled by an independent body, and by
only licensing their format so that data in the format
cannot be used by many of its competitors.
Massachusetts has determined that this limitation is incompatible
with Massachusetts’ sovereignty, and thus must be rejected.

Marc Wagner’s “Microsoft and public access” noted that this debate
“has one component that no one in the public sector
should be dismissing -- that of access to public information.
Obligating a citizen to download a free document reader
(one that runs under any operating system available today)
is one thing, but obligating a citizen to own a Windows-based system
and MS-Office is quite another. So what if 400 million people
use MS-Office everyday? We can say with absolute certainty
that there are people in Massachusetts who own
Macintosh computers and Linux computers and UNIX computers.
I didn’t see anything in David’s article that suggested
the Commonwealth of Massachusetts objected to
Microsoft protecting its IP.
All they were saying is that if Microsoft wanted them
to use its document formats, that Microsoft needed
to make those formats accessible to those who did not
own Windows and Office. Seems reasonable to me.”

The practical, specific needs for this are spelled out in
articles by
Nicholas Carr.
Carr describes their needs this way:

“I assumed [this] was just another case of anti-Microsoftism, an
assumption backed up by the press’s ‘Massachusetts Dumps Office’
headlines. I was wrong. This isn’t about Microsoft. It’s about a state
government launching a serious and comprehensive initiative to replace
its fragmented, inefficient set of traditional information systems
with a modern, coherent, and flexible IT architecture that allows
data to be shared and reused easily. The adoption of OpenDocument as a
standard is just one element of the state’s ambitious plan. As described
in a comprehensive technical paper, called the Enterprise Technical
Reference Model (ETRM), the state aims to make a transition ‘from siloed,
application-centric and agency-centric information technology investments
to an enterprise approach where applications are designed to be flexible,
to take advantage of shared and reusable components, to facilitate the
sharing and reuse of data where appropriate and to make the best use of
the technology infrastructure that is available.’”

Carr argues that:

“The proposal for adopting consistent, open standards for document
formats needs to be understood in the context of this broad, long-term
effort. The state, like other government bodies, is struggling to
meet three particular challenges with regard to the large number of
digital documents it produces and stores: (1) sharing the information
in those documents seamlessly among diverse agencies (overcoming the
incompatibilities of its various existing systems and data formats), (2)
ensuring the accessibility and integrity of those documents in perpetuity,
and (3) guaranteeing that information can be dispersed quickly in the case
of a disaster or other emergency. (The problems that governments had in
responding to 9/11 and Hurricane Katrina, some of which can be traced to
an inability to share data among diverse computer systems, reveal the
urgency of this last goal.) To accomplish these goals, it is requiring
that all government agencies use a common set of non-proprietary,
XML-based data standards, including OpenDocument for office documents.”

In short, Massachusetts believes it has fundamental principles that
are the basis for such costs, if they turn out to be costs.
This approach is necessary to maintain the State’s sovereignty.
So let’s keep reading.

particularly
given a) the significant flaws with the OpenDocument format, and b)
the availability of more cost effective alternative ways to achieve the
Commonwealth’s principal data interoperability objectives. These issues
are discussed in turn in the following two sections.

Microsoft has had many years to prove that there are significant
flaws in OpenDocument, and it has still failed to do so.
OpenDocument’s predecessor has been in use since 2000,
it has received detailed multiyear review from multiple experts,
and it’s received worldwide public peer review as well.
Are they really saying that Corel (Word Perfect),
Sun (StarOffice), OpenOffice.org,
IBM, Adobe, KOffice, and others can’t figure out how to store documents,
even though they’ve been doing it for so many years?
They can say it, but there’s no evidence to believe it.

No doubt OpenDocument will be improved as all such formats are.
For example, many people (including me!)
believe that the format for spreadsheet formulas should
be described in more detail (there’s material in the
specification, but more detail would be very helpful).
But all agree
that the format is able to exchange the necessary information,
and Microsoft’s XML specification is no better in this area anyway.
Indeed, all evidence suggests that OpenDocument is perfectly up to the
job that Massachusetts has for it.
Microsoft has failed to demonstrate that there are
“significant flaws” that are so serious that it cannot be used by 2007.

For a cost-effective alternative, we need to see one.
Microsoft’s old binary formats are probably the best alternative; at
least they’re widely implemented.
But they do not provide the processing or long-term storage advantages
of XML, and they’re not even specified publicly.
Even later versions of Microsoft Office occasionally fail to read this format;
it’s clearly a failure for long-term storage.
The sooner Massachusetts moves
away from the binary formats, the faster they can use
XML processing and stop losing data.
Even Microsoft is abandoning their old binary format; no one believes
that the binary format has a long-term future.
Microsoft’s XML format
is proprietary, with no independent vetting and with a license
that forbids many competitors from implementing it,
as I noted before, so it’s not a real alternative.

Now let’s go back to the list of concerns in the front of the letter...

(iii) the document format designated in the proposed policy is new to
the marketplace, still subject to potential revision, and not widely
deployed or tested in a wide variety of product or usage scenarios,

This seems to partly map to this later assertion later in the letter
(remember, they didn’t organize this letter clearly):

3. The OpenDocument format is immature and not widely accepted in the
industry or public sector, and mandating the adoption of this
format would present affected state agencies with significant
technical obstacles.

This text suggests a lack of understanding in how standards,
customer plans, and industry adoption work.
There are two parts to maturity: (1) is the specification mature enough to
be implemented, and (2) will implementations be mature enough
for their intended use when they will be used?

If you step back and see how things normally work, it’s actually
quite straightforward.
Here’s the normal progress of events when a standard begins to develop:

Independent standards bodies, gathering various suppliers and users,
work to create a specification that
can be implemented by the various implementors.
Usually they start with a specification of something that’s
already in use and closely meets the needs of standard
(as a “base document”);
OpenOffice.org’s old XML format was used as the basis for OpenDocument.
Then they make changes based on the experience and needs of other suppliers;
those changes often make the specification far more
general and useful than any one of the original suppliers’ formats.
Usually a few of the eager suppliers actually start to implement it
before the specification is complete; this helps ensure that the
specification will be mature (implementable) when it’s finally
approved as a standard.
When it’s finally approved, that generally means that
the specification is mature enough to be implemented.

Customers examine the standard and, if they like what
it will do for them, talk with their suppliers and warn them that they
will need to implement the standard by some future date if they
want to supply products in the future.
Suppliers, in turn, decide if or when to implement the standard.
Suppliers who implement early may have an edge, since they’ll be out
to market first.
But sometimes customers don’t demand a standard;
suppliers will sometimes wait to
see if any customers really care about the standard, and only
implement it if anyone actually asks for it.
Suppliers who choose to not implement the standard may decide that they
can still have a market without it, or decide to withdraw from the
market altogether.
This is the stage we’re in now; customers are looking, and in this case
Massachusetts is
saying that they want what the standard can do for them.

Suppliers complete implementation of the standard, do tests, and
so on; at that point, the implementations are mature.

Customers buy the implementations that meet their standards
(and shun the products that fail to meet them).

There’s lots of evidence that the specification is mature enough to
be implemented:

The most obvious is that OASIS has formally approved OpenDocument,
over half a year ago (May 1, 2005).
Formal approval by a recognized standards body is usually considered
prima facia evidence that the specification is mature and
ready for implementation.

Multiple organizations who implement the products (office suites in
this case) cooperated to develop the standard.
That includes Sun/OpenOffice.org, Corel (Word Perfect), IBM
and KOffice.
Peer review by multiple organizations is one of the best ways to get
a mature standard, because it’s a lot less likely that important things
will be overlooked.

Multiple users who have stressing needs were involved in
developing the format. These include
Boeing (who create complex, large documents),
the National Archive of Australia (who need to be able to
retrieve documents long after development),
the New York State Office of the Attorney General (both), and
the Society of Biblical Literature (who need to manage
large multilingual documents).
Intel is developing a set of sample documents to be used as a test suite.
When diverse user needs are represented, the result is much more likely
to meet a variety of needs.

The group started with a pre-existing format that was already in use,
and already known to generally meet the group’s requirements:
OpenOffice.org’s version 1 format.
OpenOffice.org began as StarOffice, from the StarDivision company founded
in 1986.
The XML format itself was released in 2000, and has already seen
a great deal of use
(
an estimated 25 million users).

Office documents are not a new technology.
Markup languages have been used to transmit documents for decades,
including the predecessors of SGML, SGML, HTML, and XML.

The OpenDocument format has undergone two and a half years of
multi-supplier discussion and
public commentary, making a number of improvements over that time.
The first official OASIS meeting to discuss the standard was
December 16, 2002; OASIS approved OpenDocument as an OASIS standard
on May 1, 2005.
That’s actually quite a bit of time.

There are no real non-proprietary alternatives for an
XML office document format, which should speed implementation.
This is the only XML-based document standard accepted by a neutral
standard-setting body.
The closest “competitors” are
Microsoft’s binary format (which is binary and unspecified),
XHTML (which simply doesn’t have the functionality required),
and
PDF (which is essentially read-only).

Microsoft says that
“the open document committee of the OASIS umbrella
organization did not include any government representatives.”
OASIS’ umbrella organization does not have
government representatives,
but the OASIS OpenDocument committee formally included
the National Archive of Australia and
the New York State Office of the Attorney General.
In addition, the OASIS process allowed anyone (including any government
representative) to submit a public comment on their drafts.
Massachusetts has had ample time
to examine the specification themselves for suitability.
Governments will have one last chance to examine the specification
at the ISO level, too; ISO votes are by country, not by company.
At this point there’s every reason to expect ISO
will accept OpenDocument.

Microsoft says that participation in the OASIS standards process
was only from a “narrow set of companies”
and suggests that it cannot work across a
“range of organizations with varying infrastructure,
skills, requirements, and needs”; this is clearly untrue.
I see Sun/OpenOffice.org and IBM, yes, but I also see Corel, KDE/Troll Tech,
Intel, Boeing, the National Archive of Australia,
the New York State Office of the Attorney General,
and even the Society of Biblical Literature.
In addition there was the lengthy public comment period, when anyone
could submit comments.

Big IT companies like Sun, IBM, Corel, and Intel are widely-known, but
let me focus in some of the more interesting players.
Boeing isn’t usually considered an IT company, but creating
a plane requires
so much documentation that a single
plane couldn’t carry it, and those documents
are often very complex.
Boeing’s involvement greatly increases confidence that
complex documents with technical material can be handled.

The National Archive of Australia and
the New York State Office of the Attorney General have needs
strikingly similar to Massachusetts; it’s far more
likely that OpenDocument
will meet Massachusetts’ needs,
because similar organizations were involved in its development.

My favorite organization in this list, though, is the
“Society of Biblical Literature.”
Why? Because I would never have anticipated
that such a group would show up at a standards body... yet in retrospect
their involvement strongly validates the result.
This organization has to worry about complex multi-lingual documents,
with all sorts of internationalization problems.
Hebrew and Aramaic run right to left (not left to right), and they routinely
create documents that have multiple languages in the same document
with differing conventions.
What’s more, many of their documents are extremely long-lived (they
think in terms of millennia!), and
accuracy is absolutely critical (they believe that in many cases they
are dealing with God’s word).
(see this
introduction of their representative,
a simple
sample comment,
and this
example of an interesting need you might not anticipate).
If a document format can meet their needs for accuracy and
multi-lingual support, it’s ready to be
a true international standard.
Clearly, we have a variety of perspectives and
usage scenarios represented here!

In contrast, only one company was allowed to control Microsoft’s
XML format; if this is the criteria, Microsoft’s XML format is disqualified.

Microsoft complains that some companies were
“promoting their own technologies.”
They even note that there were two employees of Sun, and two employees of
IBM, on the committee on page 12 (though the participation of the many
other organizations and individuals is not mentioned).
Well of course each representative promoted their own technology!
That’s why neutral bodies, with representatives from multiple
suppliers, are so critical -- by creating a neutral forum, everyone’s
interests can be heard, and through consensus they can create
an impartial result.
One obvious retort to that is that KOffice (not associated with Sun or IBM)
managed to implement the specification first; if we assume that Sun or IBM
controlled everything to their liking, why was a completely different
organization able to implement the specification first?
But let’s follow the complaint; imagine that
we accept the idea that it’s bad when a company promotes their
own technology (I do not!).
It’s easy to notice that
there’s no neutral party with a multi-supplier effort covering
Microsoft’s XML format; by this logic,
we would be forced to presume that
Microsoft’s format is designed to only promote sales of their technology
(Microsoft Office) and is not supplier-neutral.
By this logic, Microsoft would disqualify its own format first.

It’s worth noting that there was a lengthy public review comment period
as well for OpenDocument.
Though I wasn’t on the committee that developed OpenDocument,
during their public comment period I read the specification myself
and sent in a few comments.
I was generally pleased with it after I reviewed it;
there’s a lot of good to say about it.
It’s actually quite clear to read, as these things go.
And the careful crafting, and review by many, shows in the result.
It’s smaller than it might otherwise be, because they reused
pre-existing standards instead of rolling their own
(that is generally a very good idea).
It isn’t perfect; no human product is.
But it’s a reasonable specification to use.

Microsoft notes that ISO could radically change OpenDocument.
In theory that’s true, but in actuality that’s unlikely.
OpenDocument is being submitted through the “PAS” fast-track process of ISO;
OASIS is one of the few organizations allowed to use this process.
This process rarely calls for any substantive changes at all;
this process is designed for documents that have already undergone
widespread, careful, and public review as OpenDocument has done.
Future work will no doubt add new capabilities and clarifications,
as they always do, but it’s not
expected to invalidate what has already been done.

What about implementations?
I’ll
discuss OpenDocument implementations in depth in concern (v).
The simple answer is that there are
already several implementations on the market, and many more are expected by
the time the policy comes into effect (2007).
As you can see,
Massachusetts is doing things in the usual way: an
international specification is created and approved,
governments and other customers then mandate the standard
starting at some future date (giving a timeframe),
and then suppliers have time to implement the standard in an orderly way
(knowing that it’s worthwhile to implement the standard).

Microsoft claims that Massachusetts’ ETRM
does not address issues such as
“embedded pictures, audio, video, maps, voice...”
At the Sep. 16, 2005 town meeting, Peter Quinn noted
it’s a matter of priorities;
“We are focused only on documents for now; it’s the
pressing issue; other media [will be addressed] in [the] future.”
Besides, OpenDocument already handles embedded pictures and audio
(including voice).
As for the rest, this is actually quite straightforward.
OpenDocument is just a “zip” file format, so it can contain
such information trivially, the
same way that bitmapped pictures and audio are handled now.

All formats are subject to potential revision, so that
concern in the letter is a red herring.
Microsoft hasn’t committed to never changing
their proprietary XML format, either.
In fact, Office 12’s XML format will be radically
different from Office 2003’s.
Besides, since formats are subject to revision, they
need to be controlled by a neutral third party to ensure fair upgrading.
That’s a condition
OpenDocument meets and Microsoft’s does not.

(iv) there are substantial technical challenges associated with
implementation of the proposed policy. For example, there are issues
associated with converting documents saved in the well-established,
existing document formats which apparently have not been considered,
including the possibility that the new policy will lock out citizens and
organizations which use software applications supporting these existing
formats from Commonwealth systems or services, or significantly change
countless legacy documents that are not fully supported by the newly
designated format,

In their FAQ, Massachusetts note that
“The Final ETRM Version 3.5 applies only to new documents,
and does not require conversion of documents existing before the
implementation date.”
Nicholas Carr has some interesting comments about conversion:

“Microsoft is,
in essence, arguing for the maintenance of the status quo. It says that
‘Commonwealth agencies should be allowed to choose the technologies that
best suit their needs, particularly in a context where, as here, multiple
open and competing technologies/formats are available and supported in the
marketplace, with many document conversion utilities already available
and with no licensing barriers to future conversion software.’ But this
reveals the flaw in Microsoft’s position. The state has determined that
the status quo is neither desirable nor sustainable. It believes that the
lack of standardization in technology and data is undermining its ability
to do its work effectively on behalf of its citizens. The state, in other
words, has made a conscious decision to endure short-term disruption in
order to achieve a flexible and efficient IT architecture for the future.
Indeed, when Microsoft points out the difficulties inherent in translating
existing documents in a variety of incompatible, proprietary formats
into a single open format, it is inadvertently backing up the state’s
argument. The only way to guarantee that the state’s document archive will
be accessible in the future is to move to a common standard today. If
translating all those documents will be hard now, as it no doubt will
be, imagine how much harder it would be if the state put the job off
for another decade or two. The status quo is the problem the state has
to solve; it is not a solution to the problem.”

“As a migration expert, I can state from personal and client
experience that it is actually not that hard.
It is certainly no more difficult than managing
an organization of Massachusetts’ size (approximately 60,000 desktops)
with as many as four different versions of Microsoft’s Windows
operating system and possibly four different versions of
Microsoft’s Office suite -- each of which has a different file format
that is not backward-compatible.
Additionally, any possible difficulties changing away from the
Microsoft file formats make the case itself for the immediate change,
for this is certainly a ‘pay-me-now... [or pay-me-later]’
situation...

Citizens who produce documents in an OpenDocument-ready
office suite applications, like OpenOffice.org or StarOffice,
do not need to convert their documents because they natively
save as OpenDocument files by default.
Statistics of common PC desktop workflow practices reveal that users
often do not access a majority of their old documents and, therefore,
most legacy documents don’t ever need to be converted.
More than 95% of legacy documents are simple, one-page memoranda,
notes, correspondences, short reports, or fax documents with minimal
formatting. These simple documents convert perfectly and naturally in one
click from the several different MS Office formats into the OpenDocument
format, once opened in an OpenDocument-ready office suite.
Trouble with document version compatibility is overstated by Microsoft
for obvious reasons. It is important to keep in mind that users of MS
Office have more difficulty with file format version incompatibilities
than users of OpenOffice.org or StarOffice, since the latter suites open
all legacy versions of MS Office documents.”

It is important to preserve important formatting, but
we also need to have reasonable expectations.
None of the formats
being discussed here are designed for
extremely high format fidelity (where the
the document layout is exactly
preserved across all platforms and computers, all versions, no matter what).
The Valoris report’s section 2.4 notes quite clearly that
the Microsoft Office binary formats,
Microsoft’s XML format (they evaluated its predecessor WordML), and
OpenDocument’s ancestor (OpenOffice.org 1.0 format)
are all designed to be medium-fidelity formats: important
layout information is preserved, but none guarantee the
exact same layout on all systems.
For example, Microsoft Word may reformat a document (with
new spacing, line breaks and so on) simply when you change printers or
upgrade a printer driver; it certainly won’t guarantee
the exact same formatting if you
transfer the file to a different machine (Windows to Mac or even
Windows to Windows), if you upgrade the suite,
or if you change to a different competing suite.
For high fidelity needs, PDF is a better choice; PDF is considered a
high fidelity format, though there are even cases in PDF
where fidelity is not preserved. But PDF is primarily
read-only, so it’s not very appropriate when you want to edit documents.
Microsoft implies that there might be reformatting problems with
OpenDocument if it did not handle some special-case
construct in Microsoft Office, but indeed, even
Microsoft Office does not guarantee the kinds of formatting
expectations this implies.
OpenDocument can handle today’s documents.

And of course, if there is a serious omission in OpenDocument,
it can be updated to fix it.
Microsoft has every right to submit comments, or join the
group, like anyone else; they are even a member of OASIS.

Microsoft’s XML format is less widely used than OpenDocument and its
predecessor, and
it has not received the same widespread review.
It’s fair to say that Microsoft’s proprietary XML format
is far less mature than OpenDocument.

ZDNet
puts this very bluntly:
“Does OpenDocument, which is the result of a lot of hard work
from people fully versed in contemporary corporate computing,
really fail at the very things it was designed to provide?” and was
very skeptical of this claim.
There were a lot of smart people involved in
OpenDocument’s standardization, many of whom understand
Microsoft’s binary formats very deeply.
I can say from personal experience that several of the products
that implement or will implement OpenDocument can open old Office
documents quite well, and they could not do that without
first understanding the binary format well.
So it’s reasonable to
presume that the OpenDocument specification that they created does a good job
with the important information in those binary files.
Even Microsoft’s Office cannot always open older Microsoft Office files!
Massive independent review is how you assure that a specification is
good for its purpose. OpenDocument has had it; Microsoft’s XML format
has not.
The claim that OpenDocument will be completely unable to handle the
documents already in existence seems unfounded.

(v) the policy would prohibit impacted agencies of
the Commonwealth from taking advantage of innovations and solutions from
a multitude of technology vendors, including vendors whose technologies
are now widely deployed throughout the Commonwealth, thereby denying
these agencies the benefits of future technological innovations,

which mostly maps to:

4. A preference for the OpenDocument format commits the Commonwealth to
a single specific technology choice, which contravenes well-established
public sector procurement practice, as well as various Commonwealth
statutes and regulations.

Microsoft clearly states here that it’s important to have technologies
from a multitude of technology suppliers, and I agree.
So, does OpenDocument allow multiple suppliers, or one supplier?
Are they really all one technology underneath?
And will innovation be helped, or harmed, by having a standard?
Let’s examine each question in turn.

Will anyone implement OpenDocument?

OpenDocument was specifically designed and licensed to allow anyone
to implement it.
there are a whole lot of implementations already, and in fact,
a whole lot of governments already use them.
The
OASIS OpenDocument datasheet
reports that the following organizations are already adopting
applications that support OpenDocument:
Singapore’s Ministry of Defense,
France’s Ministry of Finance and
its Ministry of Economy, Finance
and Industry, Brazil’s Ministry of
Health, the City of Munich,
Germany, UK’s Bristol City
Council, and the City of Vienna
in Austria.
According to the Wikipedia, as of Sep 26, 2005,
the following products already implement OpenDocument:
Abiword, docvert, eZ publish 3.6,
IBM Workplace, Knomos case management,
KOffice, OpenOffice.org, Scribus 1.2.2,
StarOffice, and TextMaker.
Corel’ is known to be working on an implementation of
OpenDocument for Word Perfect (though their situation is more complex
right now, as noted below).
Let’s discuss a few of them.

OpenOffice.org version 2 reads and writes OpenDocument, and its older
version 1.1.5 reads OpenDocument just fine.
I often use OpenOffice.org myself, and it works just fine;
Eric Kriss reports the same experience.
It’s very capable, and it’s free -- you can download it from the
OpenOffice.org website or
install it from CD-ROMs such as the
TheOpenCD.org CD-ROM.
Sam Hiser points out that
“OpenOffice.org and StarOffice with OpenDocument’s precursor
[have between them]
more than 25 million users around the world in more
than 75 different language versions of the software.
This means that OpenDocument has close to the magnitude of the number
of users as any of the several MS Office legacy formats, since the MS
Office user-base is fragmented across the new version and several older
versions of that discontinuous series of products.”
A CSC study determined that in 2004, 14% of the large enterprise office
systems market are using OpenOffice.org,
with over 16 million downloads and countless CD installations.
While many people still don’t know about OpenOffice.org,
Microsoft sure does.
Microsoft’s Form 10-K report ending June 2005
notes that OpenOffice.org’s competition is a significant
risk factor for them;
in fact, it’s part of the very first risk they identify.

KDE’s KOffice has been upgraded, and it now supports OpenDocument.
In fact, they implemented it first, before OpenOffice.org and StarOffice did;
independent implementations like this give you lots of confidence that
the specification is a good one.
Today, KOffice runs only on GNU/Linux systems, not Windows systems;
but its underlying Qt engine runs on Windows, and their developers could
easily accelerate porting work so that they can compete on Windows too.
The announcement of OpenDocument support gives them a reason to do so.
Inge Wallin’s
Open Letter to Alan Yates of Microsoft
says that he believes KOffice will be running on Windows within a year.

But what’s interesting is not what implements it now, but what
will implement it in time to be considered by Massachusetts -- in 2007.

I expect Corel to announce a Word Perfect implementation of OpenDocument
at some point, though when this will occur is unclear.
Corel has not made a formal announcement that Word Perfect will
implement OpenDocument,
but the evidence that this is likely isn’t hard to see.
Corel is an original member of the OASIS Technical Committee
on the Open Document Format, and Paul Langille,
a senior Corel developer, is one of the original four authors
of the OpenDocument specification.
Corel’s letter to Massachusetts
strongly supported Massachusetts’ selection of OpenDocument,
saying, “Corel strongly supports the broad adoption
of the open standards Massachusetts has outlined,
including XML, the OASIS Open Document Format and PDF....
Corel remains committed to working alongside OASIS and other
technology vendors to ensure the continued evolution of the
ODF standard and the adoption of open standards industry-wide.” Corel confirmed this strong endorsement of OpenDocument on October 25, 2005.
Many find it improbable that Corel would invest so much effort,
and say that they will work to ensure adoption,
without implementing the standard themselves.
After all, they’d be locking themselves out of that the market,
based on a letter that they wrote themselves!
More likely, Corel has been waiting to see if anyone would want
a product with OpenDocument.

No supplier wants to waste resources -- they want to know what their
customers want.
Corel need a financial reason to finish adding OpenDocument
support to Word Perfect... and this announcement gives them that incentive.
Massachusetts’ announcement is just what they’ve been looking for...
confirmation that there are customers who want (yea, demand) it.
Corel’s corporate statements have recently become rather confusing,
with one spokesperson saying they don’t have
any specific plans to release a version of Word Perfect
with OpenDocument support, while other spokesmen
are simultaneously saying Corel strongly supports OpenDocument.
But people with close insights into the company say that Corel
is already implementing OpenDocument.
Steven J. Vaughan-Nichols of eWeek says without caveats that Corel is
implementing OpenDocument.
IBM mentioned at the Sep. 16, 2005 “Town Hall” meeting (at 01:27) that
Corel and IBM were both implementing OpenDocument, and that they
routinely talked with each other while they were implementing OpenDocument
in their respective products.
After investing time to develop it, and more time to implement it,
it seems likely to me that they will release it.

Microsoft states on page 8 that “the policy clearly calls
for Corel and Microsoft products to be phased out...” which is
not true.
The policy requires that all products that perform a certain job use
the same standard for storing data.
Suppliers who meet the standard can be used.
Microsoft and Corel products don’t need to be phased out; they just
need to implement the standard.
Massachusetts has given every indication that they would like everyone,
including Microsoft, to implement the standard.

Microsoft’s failure to commit to meeting the standard
important, because they’re currently the dominant supplier.
But office suites are now a commodity; there are lots of other
products that do the job, if only you could share data between them.
With OpenDocument, you can use any office suite...
and suddenly the dominance of one vendor is not as relevant.
And as Massachusetts notes, there are various ways that people can
continue to use Microsoft Office yet store data in OpenDocument, which
can act as a useful transition approach if Microsoft chooses to
not implement OpenDocument.

In fact, I believe that it’s very likely that
Microsoft will eventually implement OpenDocument in Microsoft Office.
I’ll explain why at the
end of this article;
it’s my punchline.

Oh, one funny thing: If you go to Massachusetts’ website, they’ve
already posted Microsoft’s comments, in OpenDocument format.
The notion that there is no software to manage this is nonsense;
the software’s already available.
Anyone who doesn’t have such software can go
download OpenOffice.org, which is
free, very capable, and available for just about any operating system in use.
By waiting until 2007, Massachusetts can work with suppliers
and do all sorts of evaluations and tests to lower any residual risk.

Are all implementations actually the same program?

Microsoft’s letter, in their supporting text for point 4, says,
“The draft policy identifies four products that support the OpenDocument
format: Sun’s StarOffice,
OpenOffice.org, KOffice, and IBM Workplace. In reality, these products
are slight variations of the same StarOffice code base, which Sun
acquired from a German company in 1999. The different names are
little more than unique brands applied by the suppliers to the various
flavors of the code base that they have developed. In essence, a
commitment to the OpenDocument format is a commitment to a single
product or technology.”

This is wrong; Microsoft just didn’t do a good job fact-checking here.
It is true that StarOffice and OpenOffice.org share almost all the
same code.
This is a fact that is clearly explained by both StarOffice
and OpenOffice.org, in numerous locations, and not hidden anywhere.
Nor is Massachusetts likely to consider this a disadvantage, for it
still gives them two choices that share many technologies.
OpenOffice.org is available at no cost, for the price-sensitive
(and is an easy way to quickly get a reader and writer);
StarOffice costs money, for which you get some
additional capabilities and support.

But KOffice is demonstrably a completely independent implementation.
I asked David Faure (who is sponsored by Trolltech and very
familiar with KOffice) how much code
KOffice and OpenOffice.org shared, and his answer was,
“None at all!”
He also shared with me a lot of technical details that prove,
quite clearly, that they are completely independent implementations.
Here are some of those details:
“The KOffice support for OpenDocument
is brand new code, based on Qt’s QDom parser
for loading, my own XML writer for saving, and many KOffice classes that were
written in order to support the OpenDocument file format (style collections,
style stack, helper for writing out styles, manifest writer, list handling,
variables etc.) plus of course the application-specific code.”
For proof, you can go see the KOffice code, e.g.,
ZIP implementation,
XML writer,
Class for generating automatic styles on saving,
Class for loading settings,
and the
Class for loading styles.
Inge Wallin’s
Open Letter to Alan Yates of Microsoft makes the same point,
noting clearly that KOffice is a completely independent implementation.

And that is perfect.
The best way to be sure that a specification is really understood
is to have multiple independent implementations, just as we have here.

Perhaps when they say “technology” they mean “open source software.”
But that can’t be true; StarOffice is proprietary, while OpenOffice.org is
not, so their earlier statement makes even less sense.

What about innovation?

Microsoft claims that standards somehow prevent innovation
( “denying... benefits of future technological innovations” ).
This is nonsense, even bizarre.
Standards make innovation possible!
This isn’t just me;
IBM stated at the Sep. 16, 2005 town hall meeting that
common standards like this enable, not inhibit, innovation.
Let’s look at why:

Innovators need standards to build on.
If developers had to reinvent the wheel every time they made an innovation,
then nothing could get accomplished.
Standards for data formats let developers commoditize the non-innovative stuff,
and concentrate on the innovations.
We would never have had a World Wide Web if we had not first had an Internet;
innovations build on other innovations.

Customers need standards so they can transition to the new innovation.
Without standards, customers cannot switch to whatever products
provide the innovations important to them when they become available.
If a customer is into one supplier (e.g., through a proprietary format),
then they cannot switch to another supplier when they have an innovation.

Standards create an environment where innovation is rewarded.
If a customer is locked in to one product due to the lack of standards,
their supplier has no reason to really innovate.
After all, customers would buy the product and its upgrades whether or not
the supplier was innovative.

If all office suite products used the same standard format, then
it’s reasonable to presume that this would greatly increase innovation.
Innovation speeds up when there’s a vendor-neutral standard and
competing suppliers who implement it; just
look at past examples like Betamax or TCP/IP.
If Microsoft innovates while using a standard neutral format, then
everyone who wants that innovation can switch to or upgrade
to their product, no matter what they used before.

Innovation is often oversold, anyway.
I love great innovations, I support research, and I believe
supporting innovation is a good idea; I’ve even written an article about
The Most Important Software Innovations.
But people get software products to solve problems, not to get innovation for
innovation’s sake.
Innovation “just to be different” is often
a cause of problems, not a solution.
For example,
Joel Spolsky’s
“Joel on Software: User Interface Design for Programmers”
notes that
“consistency is a fundamental principle of good [user interface] design...
[but] there is a dark force out there that fights
against consistency, and that is the natural tendency
of designers and programmers to be creative.
Now, I hate to be the one to tell you “don’t be creative,”
but unfortunately, to make a user interface easy to use,
you are going to have to channel your creativity into some other area.”
That’s particularly true for mature areas like office suites,
where millions already have expectations of how things should work.
Many argue
that Microsoft is not innovative, but non-innovative companies
can still be very successful, as long as they solve customer needs.
Let’s face it; few people use Microsoft Office or any other
office suite because it’s “innovative.”
Many people choose to buy Microsoft Office because they want to read and write
files in Microsoft’s proprietary binary formats, not because of any
special feature in the product that’s unique to it.
Many people who have an office suite do not bother to upgrade for many, many
years,
because they just don’t need the capabilities of later versions.
Office suites are an extremely mature market, and few people use even
a significant fraction of today’s office suite capabilities.

Other comments

In this section,
Microsoft states that they spent over “five years
building its XML capabilities into its current generation products.”
They seem to be saying that everyone should use their formats, because
they invested a lot of money in their proprietary formats.
That’s not a good reason to use something; just because a lot of money
was invested in it does not make it appropriate for someone else
to use.
And more importantly,
the European Union, Massachusetts, and others have been repeatedly
telling Microsoft that they
do not want a proprietary format
and specifically instructed Microsoft
to not make them proprietary.
The whole point of XML is to enable interoperability.
Microsoft’s proprietary XML format
shuns neutral standards bodies and includes a
license that prevents customers
from manipulating their own data in arbitrary ways.
These limitations subvert the whole point of using XML.
Microsoft chose to invest lots of time in something the customers
said they didn’t want.
Now, as it’s getting ready,
the customers say they still don’t want it.
Suppliers that reject their customer’s requirements as
irrelevant often waste their
investment dollars; this may become yet another example.

Microsoft notes that the Commonwealth has statutes requiring the
“stimulation of competition.”
Good point.
They then amazingly complain that a common file format would reduce
competition.
Nonsense.
By requiring a single neutral standard everyone can meet, a level playing field
is created that everyone can implement, stimulating competition.

(vi) the proposal appears both inconsistent and discriminatory in that it
approves use of one “proprietary” document format as an alternative to
the OpenDocument format, while excluding others, and

This is referring to Portable Document Format (PDF);
more details of Microsoft’s argument are in
its details under “reason 5.”
Microsoft argues that the conditions for its format are
no different than PDF... but it’s not so.
It’s true that Microsoft has posted their specifications
for free download, just like Adobe.

But there the similarities end.
PDF has been reviewed by neutral standards bodies,
multiple times, and there is every evidence that future
evolution of it will involve other parties in a neutral setting too.
In additional, PDF has been implemented by a wide variety
of different implementations, from proprietary licenses to the GPL,
demonstrating that its license does not constrain
competition or use.
In short, there’s lots of evidence that there is
open competition in PDF implementations.
Let’s look at the evidence; it turns out there’s a vast array of it.

Massachusetts wants the specification to be reviewed by an
independent standards body;
this kind of review reveals biases for a platform or supplier.
Wikipedia briefly explains that this has already occurred with PDF,
and other information sources show this in spades.

For various reasons, the international standards community has
decided to standardize PDF by creating several different standards,
each of which is a subset of Adobe’s specification and
tailored for a different reason.
Adobe’s letter to Massachusetts briefly mentioned these different
standards.
Anyway, the main groups are:

PDF/Archive (also called PDF/A) has just been approved by ISO as
the standard ISO 19005-1.
This specification is specifically designed for document archives,
and is probably the most relevant standard for most of
Massachusetts’ intended uses.

PDF/X
is for reliable exchange of press-ready, high-end graphic information
(like color advertisements), and it’s defined by the ISO 15930 series
(more in a moment).

PDF/UA is ongoing work to define accessible characteristics for PDF
(so disabled people can access PDF files).

PDF/E is ongoing work to exchange engineering data.

Both PDF/UA and PDF/E are currently in the ISO working group phase,
under AIIM sponsorship.

PDF/X has actually been around for a while now as an ISO standard,
but some of its aspects seem to confuse people so I’ll
try to explain a little further.
The PDF/X FAQ
explains PDF/X in more detail.
The PDF/X format is a subset of PDF
designed specifically for reliable prepress data interchange.
There are three levels of PDF/X, with increasing capability but increasing
requirements on implementations;
normally the level is included in the name to avoid confusion .
One oddity is that the levels of PDF/X
(from the point of view of increasing functionality) are numbered 1, then 3,
then 2 (not 1,2,3); this historical oddity
was caused by the development process of CGATS and ISO.
People who choose option 2 have be technically adept anyway, so this
odd numbering (while a little unfortunate) doesn’t cause
problems in practice.
Anyway, here are the levels of PDF/X:

PDF/X-1a. The first PDF/X-1a standard went through
the standardization process years ago, and is now
published as ISO standard 15930-1:2001.
I’ll let the FAQ explain its purpose:
“The PDF/X-1a standard addresses blind exchanges
where all files should be delivered in [the color system] CMYK
(and/or spot colors), with no RGB or device independent (color-managed) data.
This is a common requirement in many areas around the world
and in many print sectors.”

PDF/X-3. PDF/X-3 is a superset of PDF/X-1a; it adds
the ability to contain color managed data.
Sometimes the CMYK color system isn’t good enough for print work;
“others are better served by transferring [color] data in other spaces,
such as CIELab or RGB with a profile attached.”
It’s important to note that
PDF/X-3 is a superset of PDF/X-1a --
a PDF/X-1a file meets all of the requirements of PDF/X-3
except for the label that says “I’m a PDF/X-3 file.”
ISO recommends that all tools designed to read PDF/X-3 should
also be able to read PDF/X-1a files, since there’s no
extra work in doing so and this makes PDF/X-1a files extremely portable.
The first PDF/X-3 standard (aka PDF/X-3:2002) is published
as ISO standard 15930-3:2002.

PDF/X-2. PDF/X-2 is a superset of PDF/X-3.
PDF/X-1a and PDF/X-3 define file formats for blind exchange.
Sometimes you don’t need blind exchange, and instead you’d like
additional control over file formatting.
PDF/X-2 (PDF/X-2:2003) enables an “OPI-like” workflow, and supports
data exchanges where there is more discussion
between the supplier and receiver of the file.
In this case there is a single “master” file that refers to others.
I looked at
ISO’s list of related standards
and found that PDF/X-2 has been published in 2003 as ISO 15930-5:2003.

But what about the future, and modifications to the format?
It’s true that Adobe has its own specification, posted on
its own website, and that Adobe could update its own specification.
But ISO has been updating things as well; there seems to be
excellent evidence that ISO is free to take what Adobe does
and accept or reject pieces, or even add new things.
In practice, Adobe and the other participants seem to be acting
collegially, but that’s a good thing.
Their process is not terribly surprising: Adobe creates a
modified specification as a proposal, and then the independent body
examines it and creates a vetted version of the specification.
This lets Adobe create new ideas, but it also lets people around the
world refer by name to a specific set that is internationally vetted
and approved.
For example,
the PDF/Archive committee noted that, since they had completed their task
of creating ISO 19005-1, they had
“begun work on ISO 19005-2 based on PDF Reference 1.6”
and were asking for anyone interested to join.
Clearly there seems to be no problem in working with neutral bodies into
the future to manage PDF.
I’ve found no
evidence that the neutral standards bodies are forbidden
from updating their specifications independently, or adding
completely different capabilities, if they found a strong need to.
If Adobe decided to abandon PDF, PDF could still go on.

Are there licenses that forbid arbitrary implementation of PDF? No,
the licenses allow anyone to implement it.
You can build partial implementations, for example;
you have to use a different
name (other than PDF) when referring to partial implementations,
but you can do it.
Permitting partial implementation is critical;
for many applications a partial implementation is all you need
(and can afford), and many projects have to implement
incrementally and thus must be able to have partial implementations.
There do not appear to be any traps or snares
in the license that would prevent someone from
implementing PDF.

Massachusetts wants anyone to be able to implement the specifications
they choose.
In fact, I think that’s the best test -- are
there multiple implementations, under a variety of licenses?
Wikipedia reports that
PDF readers include the Adobe Reader by Adobe Systems
(proprietary, no cost).
There are also open source readers,
including Xpdf for POSIX-like systems with the X Window System, derivatives
of Xpdf (including KPDF, GPdf, and Evince), and Ghostscript.
Ghostscript has historically had both a proprietary and GPL’ed branch,
so there has been an alternative proprietary implementation too.
And there are multiple writers.
Adobe’s Distiller is a wildly popular writer, but
Ghostscript includes ps2pdf (historically available under both a proprietary
license and the GPL), and OpenOffice.org includes a PDF writer
built into the office suite.
PDF readers are available for just about every platform out there,
including Microsoft Windows, Apple MacOS, Linux, Unix, and PalmOS.

So, in short, PDF is nothing like Microsoft’s XML format.
PDF has already undergone multiple reviews through
neutral international standards bodies.
Even more importantly, it
already has multiple independent implementations on a vast range
of platforms, using
both typical proprietary licenses
(like Adobe Distiller’s and Acrobat’s)
and typical open source software licenses (like the GNU GPL).
Remember, the reason to have the review is to be sure that arbitrary
implementations are possible, and that no platform-specific encodings
are hidden in there. The ability to have multiple implementations
on multiple platforms has been proven in spades.

Microsoft cannot say this about their XML format.
There’s been no
independent international review.
There are no multiple implementations under a variety of licenses, and
legal analysis says that competitors are severely constrained because many
common licenses used in industry are
forbidden by the excessively restrictive legal conditions.
Competition is constrained by their license and the lack of neutral forums.
Case closed.

(vii) there are
less costly, less limiting, non-preferential policy options to achieve
the proposed policy’s stated goals.

which sort of maps to this (except for the text about PDF):

5. The current proposal constitutes a significant and unjustified
departure from the Commonwealth’s prior policy, adopted earlier this year,
under which de facto format standards, such as Microsoft’s Office XML
Reference Schemas, could also qualify as “Open formats”.

The Commonwealth seems to respectfully disagree that
there are better options.
I think the Commonwealth made the right decision here, and it
can easily justify its decision.
In short, Massachusetts believes it should own the data it creates,
and be allowed to choose any supplier at any time to manipulate its data.
OpenDocument seems to be the only choice available to it
for editable office documents, given the
Commonwealth’s fundamental need to own its own data.

I already addressed most of this earlier.
Microsoft’s
letter points to lots of irrelevant details, while avoiding the
main point: Anyone can implement OpenDocument, without changing their
business model, licensing structure, and so on.
Microsoft’s XML format has never been submitted to any independent body,
the format stays in control of a single party (putting all competitors
at a permanent disadvantage),
and its licensing conditions make it impossible to select arbitrary
alternatives.

The front part of the letter continues this way:

... In particular, we respectfully recommend that the ANF:
1) reinstitute its prior definition of “open format” that properly
allowed for agency purchase of products based on openly licensed and
widely deployed de facto standards as an equally effective means of
fostering data interoperability,

This proposal makes little sense.
Massachusetts already examined this, and found out that Microsoft’s
XML format prevented competition.
This “open” seems to means
“open unless you’re a competitor I don’t like,”
which is not really open.
Also “de facto” is odd in this context, because this
entire discussion is about the XML-based formats.
Very few people use Microsoft Office 2003’s XML
formats, and Microsoft Office 12 isn’t even due out til next year
(and it will be significantly different).
Neither are de facto standards, in any sense.
The binary formats are a “de facto” standard, sure,
but everyone wants to move
on to XML; Massachusetts wants the processing XML supports, and
Microsoft is abandoning its binary formats too.

And that’s a deal-breaker. Massachusetts is trying to establish a
new and far more efficient process for managing its documents.
To do it, they need a format that anyone can manipulate.
And Microsoft doesn’t want to provide one.
I understand Microsoft’s business reasons for making that decision, but
it is fairly obvious why Massachusetts would not find that decision
acceptable for its purposes.

2) reinstitute its prior conclusion
that Microsoft’s Office XML Reference Schemas qualify as open formats
under the Commonwealth’s policy (under this approach, the OpenDocument
and PDF formats could also remain as viable alternatives), and

But Massachusetts doesn’t want two office formats. Who does?
It wants one, and only one.
There’s only one specification that is an international standard.
Only one that can be implemented by all suppliers.
Only one that already has commitments from many suppliers.
Yes, the current market leader doesn’t implement it.
But WordStar and Lotus 1-2-3 were once market leaders; when technology
changed, they lost that perch.
Assuming that the market leader will always be around is an
extremely dangerous assumption in technology, especially when your
records for all time are on the line.
Here, a new technology is finally ready -- XML -- which promises the
ability to share and manipulate data in novel ways.
But XML’s promises only work if anyone can manipulate the data.

Note that they imply here that somehow PDF doesn’t meet the criteria.
It does, quite nicely, as I explained in
concern (vi).
For example, PDF has both proprietary and GPL’ed implementations.
Microsoft cannot say the same.

3) incorporate a process into the ETRM that makes clear how additional
formats or standards may be added to the Commonwealth’s “accepted” list
as developments and innovations arise in the future.

They already have such a process; it’s embedded in
the definition of open format.
The problem is that Microsoft’s format does not
meet the criteria, not that there’s no way to add new developments.

It must have no or absolutely minimal legal restrictions attached to it.

It must be published and subject to peer review

It must be subject to joint stewardship

They were open to reconsidering Microsoft’s format, if they would
meet their requirements, but that window of time seems to
dwindling away. Massachusetts can’t wait forever for Microsoft to decide to
meet these requirements, they have problems that need solving.

In the next part of the letter,
Microsoft asks for more delay, after two years of discussion.
It seems to me that’s enough.
There’s absolutely no evidence that any new information is likely to surface,
and Microsoft has had plenty of time to make licensing changes.

They also demand inner details of Massachusetts’
cost and time estimates, which they have no right to know.
I’m sure Massachusetts has some estimated costs, and they have stated
in public forums some simple figures that show that they’ve considered
the issue.
But they’d be foolish to announce too many details in public right now.
If you walked into a car dealership and said, “I’m planning to pay
up to $50,000 for a car,” you can be sure that the cars you see
won’t cost $20,000.
Massachusetts can make some public observations, but I think they’ll be
coy when pressed for details.
Governments try to identify their requirements,
and then let the market bid to meet them.
By having competing bidders, governments tend to get better functionality
at lower cost.

Jason Brooks’ article “Massachusetts vs. Microsoft?” (eWeek, Sep 9, 2005)
opines, similarly to me, that Massachusetts’
“Plan to move toward open formats isn’t foolhardy
if the state thinks it can better serve taxpayers
by escaping proprietary lock,” and then says,
“I don’t blame Microsoft for doing what it believes is best
for its business -- but let’s not blame Massachusetts policy makers for
doing what they believe is best for their state, either... Massachusetts
has chosen to shift its default file formats to those for which any
developer or firm has an equal chance of building an equally good
application to create and consume these documents, thereby ensuring choice
and flexibility for itself and for its residents. Where’s the controversy
or zealotry here?”

In the end, Microsoft has written a long letter,
but I think it fails to make its case.
This is in some sense a surprising development.
Microsoft has the largest market share, and if it had
performed the reasonable actions its customers asked for,
it should have been able to easily meet their requirements and
just carry on.
Europe told Microsoft what to do back in 2004, including getting
involved with OpenDocument.
Massachusetts has been having a public dialogue since January 2005, and
apparently has been having private discussions even before that.
Microsoft’s license clearly limits how users’ data can
be used, yet supporting arbitrary processing is the whole point of XML.
Microsoft gambled that no one would be willing to choose a
standard that they didn’t already implement.
The gamble failed.

No doubt Microsoft will try all sorts of tactics to overturn this.
I think they’re unlikely to overturn this,
though it’s possible.

But whether or not they manage to overturn Massachusetts’ decision,
the dominoes are beginning to drop in lots of other places.
For example,
BECTA (British Education Communication Technology Agency),
is the UK agency in charge of defining IT policy for all schools
in the United Kingdom.
They recently published a comprehensive document
describing the IT policy for infrastructure in schools,
including a definition of the
standards for infrastructure for all the schools in the country.
In this new policy (pages 25-26), schools are required
to use software that saves files in open formats.
Look at their list:

Text documents

OpenDocument (.odt), plain text, RTF

Spreadsheets

OpenDocument (.ods), CSV

Databases

OpenDocument (.odb), CSV

Presentations

OpenDocument (.odp), HTML, SMIL

OpenDocument is notably present; notably absent are Microsoft’s
binary formats (such as .doc, .xls, .mdb, and .ppt) and both of
Microsoft’s proprietary XML formats.
This is no accident; BECTA explains saying:
“Any office application used by institutions must be able
to be saved to (and so viewed by others) using
a commonly agreed format that ensures an institution
is not locked into using specific software.
The main aim is for all office based applications
to provide functionality to meet the specifications
described here (whether licensed software,
open source or unlicensed freeware) and thus many application
providers could supply the educational institution ICT market.”
Groklaw includes an article discussing BECTA’s
decision to use OpenDocument in more detail.

But that’s just a drop in the bucket compared to what
is likely to come next.

What’s Next? Europe.

In my opinion,
the next big move is Europe, and there the choices are starker.
OpenDocument should sail through the ISO fast track process;
OASIS is one of the very few organizations who have been granted the
right to submit specifications directly to ISO,
and OpenDocument meets all the criteria
for a quick “yes” vote in ISO.
The European Union has already said that they plan to pick whatever
specification becomes an international standard; OASIS is
probably sufficient for some, but
the Europeans will probably wait until ISO accepts
OpenDocument before making a declaration.
There is only one real option: OpenDocument.
In contrast, Microsoft never even chose to compete.
Even if Microsoft removed all the license restrictions
and started a standardization process in a neutral body,
it would take years for a neutral group to go through the specification
(to find and fix problems like constructs not neutral to a supplier).
What’s more, there is no evidence that suppliers are
interested in doing all of it again;
there’s already a widely-vetted standard
for this purpose.

The European Union creates lots of documents, and is keenly
interested in having an exchange standard for them.
Their calculus is really obvious: do we choose the one and only
international standard, one with multiple implementations?
Or do we break our normal rules,
abandon our announced plan of 2004 to crown the international standard,
support a specification that many of our domestic
organizations won’t be allowed to implement, and
entrap our data in a specification controlled by a foreign company...
all to enrich a foreign company?
I think some have a visceral dislike for Microsoft,
and Microsoft has been in Europe’s courts quite often recently;
neither will help Microsoft’s case.
But many Europeans will be dispassionate decision-makers...
and end up with the same conclusions.
They’ll worry about transition costs,
of course, just like Massachusetts has.
But transition costs are a
one-time cost, while governments plan to be in business, handling documents
from their past, for many centuries.
As noted above, the transition costs appear to be surprisingly small, and
the only long-term alternatives are even more expensive.
They primarily use Microsoft Office’s binary format, not Microsoft’s XML
format... so now, while they do not have pre-existing commitments to
any XML format, is the time to make that decision.
Making a decision later will just raise the transition costs later,
and keep them from exploiting XML’s potential while they wait.
That decision will certainly involve lots of back-room politics, as
these things do.
But the likely outcome is already clear: OpenDocument.

If Massachusetts had not chosen to use OpenDocument,
I think the European decision would be the same.
But I suspect Massachusetts’ decision will embolden Europeans and
many other governments, enabling them to decide even
faster than they would have otherwise.
It’s easier to make a decision stick
if you’re not the first one to make that decision.

But even worse for Microsoft’s hopes to get everyone to use its
proprietary XML format,
the information Massachusetts has gathered and made public
will make deciding to use OpenDocument and not Microsoft’s format
incredibly easy for all other governments.
Governments worldwide
all have the same issues that led Massachusetts to this decision, and
Massachusetts has done a massive amount of work in analyzing the alternatives.
I think
David Berlind’s
September 26, 2005 post,
“Did Microsoft send the wrong guy to Massachusetts’ ODF hearing?”,
is particularly insightful on this point:

“Microsoft may see Massachusetts as just one state with 80,000 employees
across 173 separate agencies along with a handful of contractors that
it can let go. But, if you take a step back to look at the volume of
diligence that the Commonwealth undertook before it made its decision
final -- most all of which is public [--]
you can’t help but wonder whether
Massachusetts just created an online wizard that will make it easier
and less expensive for other governments to embark on similar projects.
Lest Microsoft think other governments aren’t paying attention, it may
want to consider who launched the Government Open Code Collaborative
(Massachusetts, Rhode Island, Pennsylvania, Utah, Kansas, Missouri,
West Virginia, and the cities of Gloucester [MA], Worcester [MA], and
Newport News [VA]) in June of 2004 and how the
current list of GOCC
members and observers has blossomed well beyond the founding group.
Not only is the GOCC clearly keeping an eye on things in Massachusetts...
(see this call for participation from the GOCC),
judging by a presentation
given by Massachusetts Information Technology Division (ITD) policy and
architecture director Claudia Boldman, the Commonwealth considers its
involvement in the GOCC an integral part of its overall open standards
initiative (not to mention that the presentation was given at the
Conference sur les Logiciels Libreset les Administrations Publiques in
Quebec, Canada and one can only imagine who else was paying attention!).
Take the digital ecosystem that lives around Massachusetts’ 173 agencies
and multiply that times some number of other states. Throw in some
cities and counties and then a dash or two of corporations that see
a reflection of themselves in what those governments are doing, and
suddenly, instead of a defector, Microsoft has an exodus on its hands.”

Lots of other states are seriously looking at these decisions
(see the Sep. 16, 2005 “Town Hall” meeting, 01:21).
And many other countries are discussing this, too; Harvard recently
hosted a meeting of many countries (ranging from Canada to China to
many others) discussing these very issues.

Is the data good enough to convince others to do the same thing?
Look how David Berlind views the quality of Massachusetts’ data:

“The clarity, the mission, the thoroughness and the goals under which the
Massachusetts officials were operating were the stuff that IT case studies
should be made of. This is a group of people that in no uncertain terms
(1) understood exactly what their employer’s needs were (for example,
preserving the state’s sovereignty), (2) figured out how best to meet
those needs from a strategic IT perspective, (3) anticipated the pain
points and costs and (4) very clearly articulated and communicated the
plan and the progress, soliciting feedback from all interested parties
along the way.”

What should really concern Microsoft is that Europe is usually a lot
more serious about standards than the U.S. is.
Europe has so many interoperation problems (due to multiple sovereign
states, multiple languages, and so on)
that they treat any standard as a Godsend.
Any improvement in interoperability due to a standard
is a real help, given the fundamental challenges they face.
As I noted above,
it’s almost certain that OpenDocument will become the EU standard for
governments; it’s a good standard, supported by lots of suppliers, and
there are no alternative international standards.
The EU uses supplier-controlled specifications when there are no alternatives,
but once there’s an international standard, controlled by a neutral
body and with at least one
implementation, the EU is usually quite purposeful (some
might say ruthless) in abandoning
nonstandard products and switching to the international standard.

Even back in 2004, the EU said that
“Where by choice or circumstance only a single revisable document format
can be used[,] this should be for a format around which there is
industry consensus, as demonstrated by the format’s adoption
as a standard.”
“Standard” here only means a “formal
standard vetted by an truly vendor-independent body” --
they are so serious about formal standards that
European governments have rules forbidding
them from even using the word
“standard” for a supplier-proprietary specification.
The “choice or circumstance” language looks more flexible,
but supporting multiple
formats raises costs, and the obvious alternative
prevents the use of arbitrary suppliers.
It’s not particularly likely that this “choice” will
survive in the long term.
At this point, it’s extremely likely that OpenDocument will become
the required document standard for all EU governments.
And it won’t be with a wink, while
accepting an alternative... their agencies typically
implement the standards once picked, all the way through.

The real question is, will Europe allow the sale of Microsoft Office 12
inside Europe if Microsoft fails to implement OpenDocument?
It’s not unusual to forbid the sale of products that
don’t meet international standards, after all.
Just try selling many other products in Europe
that fail to meet various international standards; their rules are
generally very reasonable and clearly defined in publicly accessible
documents, but they’re serious about them.
And Europe is already dictating terms involving Microsoft Windows to
allow competitors to compete; such a move would not be out of character.
If consumers routinely used Microsoft’s XML format, and then sent those
files to the government, the government would not be free to choose which
supplier to use in reading or updating those files.
At the very least, governments could require that only OpenDocument be
allowed for documents sent to them.
But it’s sometimes hard to predict when a document will go to the
government.
The obvious way to quash that problem is to mandate the OpenDocument
format, Europe-wide, for all documents, with some sort of phased-in timing
to deal with the old Microsoft binary formats.
That’s certainly interesting to speculate, and not outside the
realm of possibility.
Even if sales are permitted (I think that is likely),
and if the national governments require OpenDocument for their own use
(which I also think is likely),
and Microsoft refuses to implement OpenDocument,
then all the EU governments will abandon
Microsoft Office... and a lot of their citizenry will quickly follow.

It’s hard to imagine Microsoft continuing to
not implement the international standard at that point.
The risk of losing the entire EU market would quickly lead
to a worldwide collapse of its market share, and that
just isn’t worth it.
They’ll probably start with a quick rushed implementation,
which will be really bad (inevitable with rushed implementations),
and then later come out with a useful version.
The longer they wait to implement OpenDocument,
the greater the risk that they will lose significant market share.
It’s speculation, but it’s reasoned speculation.
Others have come to the same conclusion;
Steven J. Vaughan-Nichols says,
“OpenDocument is here to stay. Microsoft can either
gracefully accept and support it, or fight a losing battle against it.”
David Berlind of ZDNet says,
“My hunch is that there are plenty of government agencies,
both domestic and foreign watching this one and that,
in this game of chicken, Microsoft will not win.”
Lots of people are appealing to Microsoft to support OpenDocument now;
Massachusetts certainly is, and other organizations such as
ZDNet
are imploring the same.
OpenDocument
Fellowship has been running a petition for Microsoft to implement
OpenDocument, and they already have signatures from
thousands of people representing legions of computers
(as of October 29, they had 5780 signatures representing
171807 computers).
If that’s so,
the question will be, will they do it fast enough before they start
to lose significant market share because of this lack?
There are lots of smart people at Microsoft.
It’s my hope that they change course quickly,
implement OpenDocument well, and not suffer the
likely consequences of ignoring standards when customers and competitors
switch to them.

We’ve been here before, ten years ago.
Microsoft owned the client market in 1995, yet they were on the verge
of losing everything.
Why? They had completely committed themselves to their
own proprietary networking protocol.
But people didn’t want a proprietary protocol; they wanted the freedom
of the TCP/IP and World Wide Web standards,
which could be implemented by anyone and thus were a hotbed of innovation.
In 1995, Microsoft finally
abandoned their proprietary network format and embraced the industry
standards that anyone could implement (and that everyone else did).
This story of a giant’s proprietary standard
getting felled by a less proprietary standard happens over and over again.
The World Wide Web, TCP/IP, ASCII, Betamax -- they are all just a few
examples of the same basic story.
This isn’t about Massachusetts “rejecting” Microsoft;
I expect they’ll continue to work with Microsoft, and that they
will continue to use many Microsoft products.
And I expect that Microsoft Office will eventually implement OpenDocument
directly, because the market is already requiring it.