Is our idea of Open Standards good enough? Verifiable vendor-neutrality

by Rick Jelliffe

Open standards are clearly a good thing. Hurrah for open standards, etc. Nail my hat to the ceiling!

But anyone who has been involved in community and consortium committees where there are commercial rivalries engaged knows that the thing that kills or corrupts a standard is when the spirit of mutual accommodation is overtaken by the spirit of competition. When I look over the standards that I have been to one extent involved with, at ISO, W3C and tangentially at IETF and OASIS, the golden rule is that the standards that come out of a nasty process have problems. The rancour during the Open XML debates does not auger well either for ODF and Open XML, in this respect, but I am an optimist.

The trouble with the ideal that people seem have of "open standards" is the extremely pragmatic one: how do we trust the committee? Who appoints these elders? Now this is something that MS have brought up about OASIS ODF, that ODF people have brought up about ECMA TC45, and which will undoubtedly be brought up about ISO (though ever more tenuously) by one side or the other no matter what the result is at ISO, sooner or later.

I think the problem is that rather than talking "open standards" we need to be talking as much of "verifiable vendor-neutrality", if that is the goal for our public policy. It is nice that ODF and Open XML are open standards by the academic definitions, but it does not get us to where we need to be, and legislation based on mere "open standards" tickboxing will not succeed in getting vendor-neutral formats, if that is indeed the underlying aim. A standard may be as open as the grave yet not be good enough to the native format for an application, to bring up the current instance.

I have talked before about the need for profiles (to restrict extensible standards), and others are bringing up the natural progression from validation to test suites, but recently I am coming to believe that there has been a fundamental incorrect emphasis which self-defeats the open standards movements: the lack of scrupulous attention to the need for verifiable inclusiveness and fairness of process.

In other words (while not disagreeing that requiring "open standards" based on XML and ZIP is the best option now) the way forward for the EU and other governments is to direct and require that their application-suppliers participate in fair, mediated, format-harmonization standards processes (which is not the same thing as unification, and not the same thing as feature-leveling.) The boutique standards bodies, such as Ecma, OASIS, W3C are simply not constituted to be reliable here: they are democratic and two minor players will outvote one major one, which if done often enough will cause the major one to take off in a huff.

A company like Microsoft is famous for trying to keep effective control of its API. Some see Sun's JCP as the same thing, it is a rational approach. So it is simply futile to imagine it is feasible that a company will give up control of an important asset to its business rivals: this is an issue that we have seen time and time again in the W3C, and is a tricky one in general, because it is not government's role to solve this problem everytime. Some issues cannot be solved neatly or optimally or instantly, because of market forces (balanced markets or unbalanced ones!).

But the issue of public and archival formats for government and agency documents is clearly one where governments have a vital interest: the customer is always right. This is why I believe governments need to look beyond the current academic definitions of "open standards" and re-frame the issue as "How do we achieve verifiably vendor-neutral standards?"

Verifiable here means that there is a check in place that the committee proceedings did not discriminate against any player. Mere quorums and absolute votes are not enough. Vendor-neutral here means that the standard does not discriminate against any realistic players, either by making basic implementation too hard or by disallowing vendor-specific features or innovations or experiments, where appropriate. The only forum that I see that is set up for this kind of thing currently is ISO, where vendors can have committee input but only national bodies ultimately vote, but there may be some other approaches possible.

24 Comments

William
2007-07-09 06:12:05

One other aspect to vendor-neutrality needs to be intellectual safe-harbour provisions. You cannot implement an "open" standard if doing so opens you to lawsuits. This was true in the recent RAMBUS fiasco, and there are many who fear Microsoft's patent portfolio hidden behind "Open" XML. The recent sabre-rattling on the patent front seems to confirm this fear.

To address sentence 1, however. Yes, of course IP is important. But it is a well-understood issue, and the current definitions of "open standards" factor IP openness in. My point is that all that conventional thinking is still not enough, because it misses out on some important issues of process.

len
2007-07-09 14:19:41

One approach is participation agreements. To belong to the consortium (eg, W3C, Web3D) you sign the agreement. If you add to the standard, you consent to the royalty free unencumbered conditions. So one approach is to vette the standards authority for their conditions of service. This doesn't mean there is no IP involved. It means the conditions are explicit.

Another is to use the contracting process. Read the contracts and look for indemnification clauses. A customer procuring IP-ridden product can require the seller to assume all indemnity liability (if you get sued, the seller pays).

There are lots of ways around this. Unfortunately indemnity and participation agreements are two weak points of ISO. ISO warranties the process, separates voting from company memberships, etc., but it doesn't do much for IP. That is why partnerships with consortia and ISO can work well if:

2. The consortia has participation agreements that limit exposure but also contributions. Some don't want to use this approach because it means non-members can't contribute. This is the side-effect of getting protection from submarine patents or novices.

No free lunch. The big IP problems happen when a few folks decide to build a standard based on an open mail list with no process and no participation agreements. The way HTML or VRML will design simply will not work today, and had IBM enforced some of its patents or Silicon Graphics not explicitly allowed Open Inventor to be burgled, it wouldn't have worked then.

is from Doug Mahugh ( Microsoft employee ) talking about an "implementation" ( that requires Microsoft Sharepoint to work ) of OOXML by 3Sharp ( Microsoft Partner ) on hardware by Z5Technologies ( that runs Microsoft Windows XP+Microsoft Office ).

And guess what: Mahugh, 3Sharp and Z5Technologies are "reviewing" DIS 29500 ( OOXML ) at Incits-V1 to recommend the ANSI/USA position about it.

How many comments or concerns do you believe they will raise at this review process ? Yes, you guessed it: zero ( other members of Incits/V1 and other NB are surpassing the 200 comments/concerns mark ).

But it's understandable: we can't blame them to keep their business going, but the bad thing is that they are deciding for all of us if this specification worth being called "ISO standard".

Citing the recent words of a NB member: "something is wrong that needs fixing".

Rick Jelliffe
2007-07-09 20:19:24

Marc: Trying to hijack the thread, honey? "back to reality"? If the only permissible topic is "MS is evil" then a blog along the lines "How can we make it attractive for MS to be good" does not compute, I guess :-) But in the absence of anti-trust action, it looks like the only feasible way approach to me.

I don't understand your comment on "implementation". Is an implementation only an implementation if it doesn't use any libraries, components or services? What century are we living in?

On your point on errors. I would hope that a lot more than 200 corrections would be found. Indeed, if the standard I edited was anything to go by, I would expect at least a correction per page. In particular for systematic editing issues (should, shall, must, being the most basic.) Now the Open XML pages are not very dense (lots of boilerplate, 11pts not 10, illustrations, etc) but still based on my experience, I would expect and hope that several thousand (!!) individual corrections could be found for draft of this size. When I made my 1997 book, The XML & SGML Cookbook I reviewed it many times (as a person who used to work as a technical editor) and I had independent reviewers; even with that, running it through Word Perfect's amazing Grammatik program, I regularly found syntax errors in almost each page of text content.

This is one reason why specifications in natural language are so hard (XSD anyone?), and why constraint languages like ISO Schematron, allowing objective and executable/testable formulations for requirements, must be adopted more. There is an ISO requirement that specifications must avoid anything that is not verfiable, but natural language specifications don't give any tools to help.

On your point "they are deciding for us what an ISO standard is", are you saying that the ISO processes are not being followed? Where? Are you saying that the ANSI processes are not being followed? Where? Are you saying that the INCITS/V1 processes are not being followed? Where?

I think you have an altogether Utopian view of standards: many are the result of hard bargaining and negotiation and compromise and politicking, and it has always been like this. The best the process can do is make it fair and transparent, and the point of my blog was that the "open standards" definitions have a too-limited focus that ignores the possibilities for factionalism. The fact that people we don't like can get their technologies made into standards (at any of the organizations) is exactly why you have to view ISO technical standards as a library of possible technologies to be chosen from as appropriate, not a law mandating one technology at the expense of any alternatives. You don't like MS: you can use ISO ODF. Someone else has to use Office: they can have ISO Open XML.

Your presumption that a committee member is not doing their job unless they raise issues is incorrect. In usual committees, some people get involved in review (and agitation), some people supply expertise (and calmness), some provide institutional memory (and consistency), some people are out of their depth (especially when they are single-issue blow-ins) or are treading water until they figure out how things work, some people have specialist interests only (e.g. Math), some people find that others raise their concerns first and don't need to waste time with duplicated comments.

William
2007-07-10 06:32:37

Mr. Jellife:
I'm sorry if my FUD offends you, especially in the context of a blog post that I not only agree with, but whole-heartedly support. Perhaps your approach of shining a spotlight on only one elephant in the room at a time is a good one.

In your response to me you write "the current definitions of "open standards" factor IP openness in". This is true in theory, but there is one recent and sobering counter-example (RAMBUS) and very recently, Microsoft claimed that Open Source software violates 35 Office-related patents. The specific patents were not named. Your points above are very well-made, but in the context of OOXML, there are other red flags that need to come up when verifying vendor-neutrality - IP being one of the most topical.

If safe-harbour provisions for IP are built into the standards process I think it would be a crucial element of your suggested verifiable vendor neutrality.

2007-07-10 08:00:21

>hijack the thread

i'm sorry if my post was off-topic for you , i believe it wasn't; my point is that vendor neutrality won't be possible ( it will remaining an utopian wish ) if a corporation can easily game the system and the process of standardization.

I feel obligated to defend the quality and meaning of the word "standard" ( i believe now it is being tainted by economical interests and consensus[1] is not being guaranteed )

>I would hope that a lot more than 200 corrections would be found.
>Indeed, if the standard I edited was anything to go by,
>I would expect at least a correction per page.

IMHO, this is not the "spirit" of a fast-track process. In UK ( and a lot of others NB ) are 30 people doing editorial work that should have been done by the submitter ( i personally found +100 typographical errors in the specification, without mention other grave issues [i'm not a NB member] ).

My petition ( an utopia? ): keep up the quality, don't rush[2] standards ! they are very important for all people and all the world ( corporations, mid-size/little enterprises, developers, final users[3], governments, you, me ).

Anonymous: Sorry if I was snippy. I am saying "We need verifiable vendor neutrality because at the moment corporations do game the system" and you are saying "Vendor neutrality won't be possible while corporations can game the system". The difference, I think, is that I include competitors ganging up to block competitors as part of gaming the system.

In immediate terms, the difference is that I would see a result where Sun and IBM manage to get blocked a standard from Microsoft that people have been calling for for years as a case of abusing the system, while you think the opposite where DIS 29500 gets accepted as the abuse.

Yes, I have known Jon from ISO since before XML was a twinkle in his eye (and I like to think XML has some of my DNA in it too) when he was still with Novell, before Sun IIRC. Open XML will be a much better standard with his scrutiny and comments. But "Sun guy who founded ODF says Open XML (as used in competitor's product) draft is rubbish, tries to get it thrown out" is hardly surprising: he has to advocate his company's position (which he probably has formulated himself too.)

Your notion that commercial interests are "tainting" is difficult to understand in the context of a technology standard. Especially the idea that somehow ODF is free from this taint. Are you perhaps making some connection between ODF and Open Source that ain't necessarily so?

Doug Mahugh
2007-07-10 14:48:35

Marc, I understand the point you're trying to make, but I sure don't agree. John Peltonen and his colleagues have done lots of Open XML work, not just the Red Cross system. I think he's exactly the type of person the standards process needs more of: people who have actually used the technology being discussed. And the implication that we're rubber-stamping comments doesn't hold up. There's a long list of comments we all agreed are reasonable and useful changes/improvements to the spec. Yes, we've not submitted comments, but we already have more comments from IBM than we can possibly process in the allotted time.

As for the Red Cross system's dependency on SharePoint, I don't see how that makes it "not an implementation." Mindjet's implementation is dependent on their MindManager product; Altova's is dependent on their XMLSpy product; Datawatch's is dependent on their Monarch product; and so it goes.

len
2007-07-11 06:20:26

It's not that the system can't be gamed. It has always been a game. This is the nature of negotiated agreements. XML was a burglary of SGML. SGML relied on work in GML and Scribe and GenCode. You have to get used to the idea that just as one standard is almost completed, it's successor is in draft form. What ISO brings to the game is a set of rules for the referees.

Now one can demonize an opponent and turn them into a demon by play if not in fact. That won't help. Sun plays for Sun (obligated by fiduciary responsibility to do that), IBM for IBM (the same people who want badly to see open standards for document formats aren't very clear in their message about other formats such as virtual worlds), Microsoft for Microsoft (no examples necessary) and if you can't get your head around that you possibly don't need to consider yourself a professional player.

All that said, the game will be played with more rewards for all those involved if played in the open and that has been a process that has been improving steadily since the days of the XML SIG/ERB where a chorus of online experts (Rick and I included) watched a self-selected and barely recognized private group author and publish a so-called specification in lieu of the standard.

Beware the history coupled to the definitions. XML worked out because these were smart people who shared certain values and because they had almost 30 years of prior work (going back to Bill Tunnicliffe and GenCode) to carve up, and lastly because by comparison to the document application formats, carving up SGML was extremely easy (technically, not politically).

And no... XML did not make things simpler. We can take that up another time, but it is one of the bigger web propaganda sales.

What frosts me is a camp of users claiming some sort of moral hegemony over a technology in a market where the majority of users are using the product of a competitor and telling them they shouldn't have a public standard for that competitor's product. That is not just evil, it is stupid evil of the sort we've seen too much of lately.

Keep the process moving, keep it within the bounds of decent play, and then the market WILL sort out the ratios. But stop it with demonization and on the other side all you will have is more demons and smarter ones.

We don't invent technologies or standards. We breed them. That means not one round of play but generations of play.

Any Donkey
2007-07-11 08:50:55

Rambus Inc. is a modern day Dr. Mudd. In the end, Dr. Mudd was exonerated as will be Rambus Inc.

A critical aspect of an effective standard setting committee is the uniform application of the rules.

Kurt Cagle
2007-07-12 21:11:58

I would contend, however, that in the history of technology, a two standards state in a given technological niche is at best meta-stable, a saddle point. There are a fair number of these at the moment - ODF vs OXML, IE vs. Mozilla, HTML vs XHTML, XML vs JSON, etc. Businesses and products tend to stabilize at 90/10, but I suspect that with standards, the ratio is far more lopsided, once a standard's usage drops below a certain threshold it declines into oblivion pretty quickly.

Microsoft surely know this, and they know full well that while the usage of either ODF or OOXML is far lower than the usage of Word, the future of document standards will be XML based. They will not play fair on this, because to play fair is to lose that battle, and to cede dominance of the most important sector in their software strategy in the years and possibly decades ahead.

That's why I think that while the ODF vs OOXML debate will rage for some time, there will be only one standard left at the end of it. I'm not really sure which one that will be at this stage, but there can be only one (to quote the Highlander).

len
2007-07-13 09:59:27

A two standards state simply means the process is still open and that there are possibly two ways to get the same thing done for different locations. Saddle points like any minima can be shaken but it doesn't mean another saddle point won't emerge.

There are a lot of forces in play here and none seem able or willing to converge on a singularity. Regardless of the strategies applied, unless a dominant force loses energy resources, you can expect the game to go on for a long time.

The early web converged fast because it was an easily built low hanging consumable fruit. Once it became a centerpiece, it became a wax fruit symbolizing the real deal hanging in the orchard. The bifurcations are natural and inevitable. Welcome to the problem of the inverted control regime (the amount of energy needed exceeds the benefit of the output).

Predictable and predicted, so what we deal with now is keeping a civil process in place for the continuous and never-quite closing negotiations. And that is exactly what standards work was like before the web and will continue to be once the 'web' is an archaic term for global communications.

marc
2007-07-15 19:28:47

marc said:
"And guess what: Mahugh, 3Sharp and Z5Technologies are 'reviewing' DIS 29500 ( OOXML ) at Incits-V1 to recommend the ANSI/USA position about it.
How many comments or concerns do you believe they will raise at this review process ? Yes, you guessed it: zero ( other members of Incits/V1 and other NB are surpassing the 200 comments/concerns mark )."

Confirmed, this three V1 members, together with 12 more ( most of them Microsoft partners ) voted "yes with comments" at V1's Friday July 13th vote.

An important note: a "yes with comments" means "we approve the technical content of this standard and make this editorial or other comments" ( JTC1 Directives, pages 112-113 ). [1][2]

So, what this 15 people have done is simply ignored the +200 technical comments collected ( many of them agreed! ) during the +3 months of review process at Incits/V1 [3].

Marc: No, what is shameful is the attempt at INCITS to add requirements that no other specification has: requiring mappings to old binary formats, the claim that XML is not human readable despite it being plain text (and line numbering is a maintenance curse), the idea that you should not have local type definitions for attributes (...for goodness sake, is it a joke? ISO Schematron is entirely based on local typing. This comment seems to reveal a deep ignorance of the purpose and uses of modern schema languages), requirements for line-layout algorithms (what good is defining special cases when the base case is not defined?), let alone adding whole new behaviours to fit in with ODF.

So I would rate the INCITS material as being 99% good (some of the solutions suggested are really smart), but 1% are nutty/vexatious/weird and raise the bar startlingly: one could imagine that fully document the mapping to the around 26 versions of the .DOC formats would itself be substantially larger than the Open XML specification. Not to mention that it utterly subverts the whole point of having XML formats: to let us escape from binaries.

marc
2007-07-16 06:06:10

rick jellife said:
"Marc: No, what is shameful is the attempt at INCITS to add requirements that no other specification has: requiring mappings to old binary formats"

some points...:

i) the comment about binary mapping was contributed ( and justified with 2 pages of text[1] ) by Patrick Durusau, chair of Incits/V1 with a long background and record in this field and editor of many standards, a respected, neutral person and esteemed by V1 members. I trust his work and oppinions. Don't you?

ii) the same comment about binary mapping convenience was raised at BSI's technical group, together with 200+ technical comments, do you think this work is shameful too?

iii) "requirements that no other specification has", yes, for god sake! this is a very *particular* specification, with a *particular* goal, citing DIS 29500 text: "The goal is to enable the implementation of the Office Open XML formats by the widest set of
tools and platforms, fostering interoperability across office productivity applications and line-ofbusiness
systems, as well as to support and strengthen document archival and preservation,
all in a way that is fully compatible with the large existing investments in Microsoft Office
documents. (DIS 29500, Part 1, Introduction)"

What is the "problem" with this binary mapping ? Doesn't Microsoft, the creator of both formats, have one?

Marc: The problem with any binary mapping is that it belongs in some other standard: it has nothing to do with the goals or objectives of Open XML, which is to allow the information to be represented. ODF is about vendor-neutral formats, so under the same bad-faith assumption why shouldn't we require ODF provide binary mappings to all its source formats to prove that it works and to enable easy converters?

Patrick and I have emailed multiple times on these issues, and he knows that I think this one is rubbish, and that I think Open XML should not have requirements placed on it that other standards do not have. Don't dare try to make this me against him. Respecting the man does not mean agreeing with his every opinion.

If BSi raises the same comment, as a requirement, they run the risk of having allowed themselves to be nuanced into a set of vexatious requirements, too.For goodness sake, this is potentially thousands of pages, all of which would improve the specification of the XML parts not one iota.

Fred Arnold
2007-07-16 09:42:58

"verifiably vendor neutral" describes ODF. It's not even in the same solar system as OOXML. You're using an awful lot of words to make it sound like Microsoft is actually interested in a genuine open document format. They're not, and anyone who believes they are is naive. Or full of Kool Aid.

BTW, did you ever agree to accept a paycheck from Microsoft?

tiny grammar nit: it's "augur"

marc
2007-07-16 12:33:49

rick said:
"so under the same bad-faith assumption why shouldn't we require ODF provide binary mappings to all its source formats to prove that it works and to enable easy converters?"

Because ISO 26300 doesn't include a goal that claims: "to support and strengthen document archival and preservation, all in a way that is fully compatible with the large existing investments in [insert your favorite office product here] documents" (DIS 29500, Part 1, Introduction)

"Patrick and I have emailed multiple times on these issues, and he knows that I think this one is rubbish, and that I think Open XML should not have requirements placed on it that other standards do not have. Don't dare try to make this me against him. Respecting the man does not mean agreeing with his every opinion."

OK, you called V1's work shameful, don't blame me for this. Some members of V1 found and raised comments/observations about DIS 29500, most of this comments were agreed (or not) by consensus by *all* V1 members. Most of the technical comments processed and agreed could be fixed by ECMA of Microsoft. What is the hurry? what is the quality we expect for an ISO standard? Why 15 members of V1 didn't vote "conditional no" ? ( conditional no = dissaprove with technical comments, but if they are addressed, the vote changes to "yes" )

"one could imagine that fully document the mapping to the around 26 versions of the .DOC formats would itself be substantially larger than the Open XML specification. Not to mention that it utterly subverts the whole point of having XML formats: to let us escape from binaries."

ok, this can be addressed and agreed during the BRM. I don't think is shameful that a V1 member had raised this comment.

What i believe is shameful is this: 15 members of V1 didn't t make any comments during the review, and +10 of them are no-vendor neutral and seems don't care about standards quality ( they want a rushed standard because they have commercial interests on it ).

*This* is shameful

Rick Jelliffe
2007-07-16 18:54:54

Fred: You have missed the point of the article, I think. Neither ODF nor Open XML meet my criteria for "verifiably vendor neutral". What is the verification procedure to assure us that there was universal participation and universal agreement? None. The trouble with "open standards" as currently conceived is when people say "this is an open standard therefore is must be fair or good" as if there was some kind of democratic principle involved.

Now that we are having more regulations that use standards, it is important to understand the limitations of industry-committee processes. By analogy, it is one thing to say that "we should only have one government" if everyone votes; it is another thing to say "we should only have one government if only a handful of people get to vote for them".

Rick Jelliffe
2007-07-16 19:03:30

Marc: A "goal" is not a promise, it is an explanation. To say an standard for an XML format is no good because it is not a standard for a binary format is silly; like saying an apple should be an orange.

I complained about those items making it through the V1 process to the list. It is not inappropriate, I think, to raise issues that you think important to get resolution on at committee meetings, however, in the cases I mentioned I think the committee got it madly wrong. The trouble is that slipping in some of these impossible requirements becomes a poison pill that makes a "no with comments" vote impossible for supporters.

marc
2007-07-18 18:19:44

"The trouble is that slipping in some of these impossible requirements becomes a poison pill that makes a "no with comments" vote impossible for supporters."

May be the poison pill was dropped by Microsoft, stating this goal. They can propose to strike or change the current goal to not mention the "billions of documents".

But they don't want to do it, because this action could trigger the other concern: that OOXML overlaps in scope with ISO 26300 ( ODF ). This "billions of documents compatibility" is one of MS ( say, ECMA ) arguments used to justify this proposal.

Rick Jelliffe
2007-07-18 21:11:23

Marc: For a start they cannot change the text once it in progress. Second they have consistently endorsed editorial changes that come from the ballot and resolution process (indeed, that was the mantra to the so-called contradictions material). Third those scoping sentences are already up for improvement in other comments. Fourth, documentation of binary formats would go through a different working group at ISO.

But primarily, as I mentioned before, compatability in the sense that the information can be transfered is the design "goal" not a promise. How well a standard meets its goals is a different matter than what the standard specifies.

(In any case, using "poison pill" here makes no sense, unless you are implying that MS deliberately has been going through a charade here in the hopes of preventing DIS 29500? A poison pill is something that no-one will swallow.)

(In any case, there is an issue of the appropriate committee hwere. ISO/IEC JTC1 SC34 Document Description and Processing Languages is concerned with SGML-based documents and publishing systems such as abstract fonts. Standards for binary formats would go through some other committee, presumable ISO TC171/SC2 Document Management which PDF/A went through. That SC34 was nominated is a clear indication that in Ecma and JTC1's view, the standard is about markup not about binary issues.)

You are confusing a goal with content of a standard.

Shane
2007-09-10 14:51:05

My letter to Microsoft this morning....

Re Microsoft's "Open Standards".....

You two in this article - are just so full of shit.....

Tom Robertson
GM Interoperability & Standards
Microsoft

Jean Paoli
GM Interoperability & XML Architecture
Microsoft

http://www.microsoft.com/interop/letters/choice.mspx

All I can say with your lying and spin doctoring (what was originally called in Australia - spinning bullshit)...

Is that you have made me a complete advocate of the REAL open source software, and a true hater of American Corporations.....

I can't even be bothered to tell you to go fuck yourselves.... I am instead going to work 10 x as hard to sink you and this company for lying to me, and for trying to manipulate me.

In fact I shall start distributing Open Office CD's (at my own expense) to everyone in my own town.... and those beyond....

I am also going to start taking big shots at you two in the press...

And I am going to ask my government to ditch Microsoft - for lying and producing bad software.

These just some of many things I am doing to take on scumbag Americans, in their scumbag American corporations.....

Regards

Shane

Sign up today to receive special discounts, product alerts, and news from O'Reilly.