|
|
Subscribe / Log in / New account

When Is a Standard Truly Open? - When It's Universal, Reflections on Massachusetts and Microsoft's XML

November 30, 2005

By Pamela Jones, Editor of Groklaw

There are standards and there are standards. They are not all born equal.

What makes one standard open and not another? Massachusetts, when deciding to use the OpenDocument Format, as set forth in its Enterprise Information Technology Architecture (ETRM) document [PDF], set the bar here:

[Secretary of Administration & Finance for the Commonwealth of Massachusetts Eric] Kriss emphasized, however, that the state is not moving to open standards for economic reasons but to protect the right of the public to open and free access to public documents for the foreseeable future. "What we've backed away from at this point is the use of a proprietary standard and we want standards that are published and free of legal encumbrances, and we don’t want two standards," Kriss said.

A recent statement by the Governor's office in Massachusetts, expressing optimistic hopes that Microsoft's Office Open XML document formats will meet their standard for an "open format" someday raises two questions: Is the Microsoft covenant not to sue, assuming it is someday offered for their new version of XML schemas, and their plans to submit their XML to standards bodies ECMA and ISO sufficient to meet the Massachusetts requirements for openness? If so, what are the implications for the Internet? If not, will Massachusetts decide their bar was set too high, in order to include Microsoft? If the bar was set too high, which part shall we lop off?

Shall we say the public has no right to open and free access to documents their tax money paid for?

No. Massachusetts has a Public Records Law [PDF] that mandates [PDF] that "all people have an absolute right of access to public information". The law doesn't distinguish between paper documents and digital documents. The disabled have a right of access, but so do the rest of us.

Are proprietary standards acceptable if nonproprietary, universally available standards are available?

Can a government dictate which operating system you have to use to access those documents? More pointedly, can it favor one proprietary vendor and compel citizens to spend money on a proprietary system, when they already have a perfectly functional operating system on their computers already?

We already saw in the Katrina disaster what happens when a government agency enables Windows-only access. I wrote about that in "When Open Standards Really Matter: the Katrina Factor", which begins like this, trying to explain why standards matter:

If you have any doubts about the direction Massachusetts is following in requiring open standards for all government documents, consider what happened when Hurricane Katrina knocked out almost all communications except the Internet. Cell phones and walkie talkies failed, once again, just as they did in 9/11, as David Kirkpatrick tells us in an article in Fortune:

In the immediate aftermath of the hurricane, much of the region’s communication systems failed or didn’t work properly. Water and wind knocked out power, toppled phone lines, and destroyed cellphone towers. What systems remained were quickly overwhelmed. When rescue workers’ did have working equipment, like walkie-talkies, they often couldn’t connect with others on different communication systems.

Catch that? "On different communication systems." The same thing happened after the tsunami disaster in Thailand, as a report just released by the ePolicy Group reports:

"Responding agencies and nongovernmental groups are unable to share information vital to the rescue effort," the report recalls of the government in Thailand in the tsunami's immediate aftermath. "Each uses different data and document formats. Relief is slowed; coordination is complicated. The need for common, open standards for disaster management was never more stark or compelling."

Isn't it time, after so much suffering, to recognize that keeping people alive is more important than allowing private companies to lock in customers into proprietary systems that don't then work in an emergency? And why does the Internet always work, no matter who you are or what operating system you use? Because it was built, not on proprietary standards, but entirely on open standards. That's why you can send an email to me, even if you are using Microsoft Outlook. I don't run any Microsoft products currently, but because of open standards, I can still read your email, and in an emergency, we will not be disconnected because we are on "different communication systems."

Accepting Microsoft's proprietary XML *with its proprietary extensions* as the language of the Internet would certainly alter the openness and universality of the Internet. Proprietary, by definition, means it isn't universal. Is that what we want? If Massachusetts accepts proprietary extensions, it will inevitably be excluding some of its citizens, I think.

Can any government, including Massachusetts, in the Internet age, justify telling its citizens, by the decisions it makes, that they must stop using their operating system of choice -- one used by millions all over the world -- and instead purchase a proprietary vendor's product instead if they wish to interact with their government? In any case, in an emergency, it may not be possible to quickly buy a Windows operating system. People use what they have. Sometimes it's all they have access to; sometimes it's all they can afford. On what basis would you argue a government should do that? More pointedly, on what *legal* basis would you argue they can?

Should Massachusetts wait for a future, yet-to-be-developed proprietary standard, not yet published, on terms that are not yet fully known, controlled by a single vendor, when you have one that is vendor-neutral and is already available? On what basis could the Commonwealth justify such favoritism to a single company? In the FAQ on the Commonwealth's ETRM, they state this:

The Final ETRM Version 3.5 does not require that agencies use only one office product. To the contrary, it offers agencies many choices. Agencies may choose to retain their existing MS Office licenses, as long as they use a method to save documents in Open Document Format. They may also use one of the many office tools that support Open Document Format in native format--- OpenOffice, StarOffice, KOffice, Abiword, eZ publish, IBM Workplace, Knomos case management, Scribus DTP, TextMaker and Visioo Writer. Because the Open Document Format is an open standard, it increases the vendor pool available to state agencies by encouraging and permitting vendors not already in this field to develop products that support the standard. Adoption of the Final ETRM Version 3.5 will greatly increase competition among vendors for the sale of office applications to agencies.

What about legal encumbrances?

There are still questions about legal encumbrances on the MS XML and related licenses and EULAs. Is that going to be the part Massachusetts should wink at? They'd likely be sued if they tried, I think, and I doubt they will try, because it would be impossible to justify it in public, for one thing. That's the kind of thing that only works in back rooms, and nothing about this process will be hidden from the public's gaze.

On the issue of license clarity, obviously, if you can't understand the license, you won't dare to use the standard. I don't understand all the legalities of Microsoft's covenant not to sue and I don't know anyone who does. The danger is in establishing a standard that would, in effect, block Linux and GNU/Linux developers from participation. Do you imagine that could never happen? Consider your history.

Remember the SenderID flap? Who was to be left out in the EU Commissioner-MS deal? Do you really want to set up a world where Linux could never happen again, a world where the Linux we have is exiled to the backwaters of technology? Just being an ECMA standard doesn't mean it is open enough for *government* use. ECMA standards, by their own self-description, are for industry use:

Ecma is driven by industry to meet the needs of industry, generating a healthy competitive landscape based on differentiation of products and services, rather than technology models, generating confidence among vendors and users of new technology.

The needs of government are not identical to the needs of industry. It's up to industry to meet the needs of government, not the other way around. The government is the customer, after all, so they get to state what they need in a product and to choose whatever meets their needs best.

Finally, shall we have two standards?

That seems to be what they are leaning toward. The Commonwealth has already stated that the disabled can continue to use Microsoft's products, if there are no better options they like, so that might be one area of carve-out. From what I've heard, however, that will be a solved problem for ODF by the time of the 2007 rollout. So again, the issue is going to be: should a government favor a proprietary solution, when there is a universally available solution that isn't proprietary and excludes no one? Tim Bray has made a sensible suggestion:
The ideal outcome would be a common shared office-XML dialect for the basics—and it should be ODF (or a subset), since that’s been designed and debugged—then another extended vocabulary to support Microsoft features , whether they’re cool new whizzy features or mouldy old legacy features (XML Namespaces are designed to support exactly this kind of thing). That way, if you stayed with the basic stuff you’d never need to worry about software lock-in; the difference between portable and proprietary would be crystal-clear. And, for the basic stuff that everybody uses, there’d be only one set of tags.

This outcome is technically feasible. Who could possibly be against it?

Let's think about what we should look for in a standard, what elements justify even calling it a standard. I'd like to redefine the issue, if I may. I think the sine qua non for a standard isn't whether it's open or proprietary. If, for example, Microsoft can open up its XML to match ODF's, that's fine with me. The real issue isn't openness alone. It's universality. Let me explain.

In trying to understand fully the standards issue in Massachusetts, I came across a helpful list on Bob Sutor's blog dating back to September. I hadn't read it, and perhaps you didn't either, so I'd like to provide here the danger signs that a standard isn't as open as it needs to be. I think it's pertinent, because Microsoft folks are telling the world that by applying to ECMA, the conversation about whether their XML is open enough should be over. It's only just beginning, actually, as they will discover.

Consequently, let's list the elements Sutor provides that indicate there is a problem with a standard and you can judge for yourself:

In practice, I think it is probably easiest to tell when a standards effort is less open than you think it should be. Here are some danger signs:
  • control by a single vendor,
  • overly complicated license agreements,
  • license agreements that reserve certain special rights to individuals or vendors,
  • license agreements that prevent some kinds of implementations,
  • overly complicated procedural rules that can allow people to be less democratic than they should,
  • a history of disregard for backward compatibility,
  • costs of participation that exclude individuals or small organizations,
  • high costs of obtaining copies of the standards,
  • standard specifications not being openly available online, and
  • for XML-based standards, allowance for proprietary, vendor-specific extensions.

I see at least 6 items that would apply to Microsoft's XML. I'm sure I don't have to underline for you that the last is the deal-breaker, proprietary extensions. It's not a standard unless it's universally available and usable by everyone. The whole point of a standard is interoperability. Can everyone use it and have it work? Proprietary extensions, by definition, hamper interoperability.

The real question to me isn't just whether a standard is open. That's a continuum anyway, with various degrees of openness. The real question that matters is: is it universal? Can everyone freely use it? If not, should it be a standard?

XML Standards Impact the Internet, Not Just Massachusetts

When we're talking about XML, we're talking the Internet, not just about Massachusetts, because XML is the language of the future Internet. HTML was fine as the universal language for the Internet for its babyhood, but the future is XML, and so any discussion about standards for XML involve the Internet too. It's the next step, because it enables collaborative computing and universal data exchange.

No doubt Microsoft wants its proprietary version of XML to be adopted as a standard. But seriously, do you want the Internet's common language to belong, in essence, to one US company? And what a company! Ask yourself one question, and let your natural, first instinctive response answer: Do you trust Microsoft?

If you do, kindly tell me why, in light of their history. This is a company twice found guilty of antitrust violations, on two continents. What should that tell you? They had a version of the Internet too, remember? Happily no one was foolish enough to accept it. The theme of the Internet is universality, not proprietary walled gardens. Do you remember how Microsoft added proprietary extensions to HTML and degraded performance for competing browsers?

In the ETRM document, referenced above, the Commonwealth pointed out something very important:

Initiatives such as Homeland Security rely upon all parties adhering to Community of Interest XML specifications, defined by open standards bodies comprised of representatives from Government, Business and Technology Communities. Open formats for data files ensure that government records remain independent of underlying systems and applications thereby preserving their accessibility over very long periods of time.

Do you really think the Internet language should be determined by just one US proprietary company motivated by their own profit? The rest of the world will not go along with that, even if Massachusetts were to think so. And then how will they interchange data with countries and governments abroad? Massachusetts will be odd man out. Accepting Microsoft's version of XML with its proprietary extensions as the standard will thus result in serious issues ahead.

The Internet only works well if *everyone* uses the same language, and frankly, the whole world isn't going to accept Microsoft's proprietary XML as that common language. Can you imagine Europe agreeing to that? Or China? Or South America? FOSS developers? The next question is: Do we want two standards? Two Internets, in effect? That destroys the very purpose of the Internet, which is universality. Can there be justification for degrading the performance of something as important to us as utilities are for the sole benefit of a single proprietary vendor? I understand why Microsoft wants that, but why should you?

And there is a choice. ODF is here right now. There is no pie in the sky about it. It's not a year away. It's a universal standard, not vendor-controlled, which everyone can use, including Microsoft, without having to open up in any way themselves.

We're not just talking about Microsoft and Massachusetts, then. We're talking about the Internet, which doesn't belong to any company or any country. It's for everyone, and everyone uses it. Because it's universally available and usable, shouldn't the language of the Internet be universal too?

Index entries for this article
GuestArticlesJones, Pamela


to post comments

odf as a format

Posted Dec 1, 2005 9:55 UTC (Thu) by wingo (guest, #26929) [Link] (2 responses)

From what I've heard from some developers of Abiword and Gnumeric, the ODF formats end up being reflections of openoffice.org's internal structure rather than well-designed formats. One even said that Microsoft's spreadsheet format was better designed (licensing issues aside of course).

While Jones is correct in her assessment of the viability of MS' XML, the ODF grassroots appear to me to be made of OO.o astroturf.

Corrections welcome of course -- I'm pretty ignorant about the topic.

odf as a format

Posted Dec 1, 2005 11:31 UTC (Thu) by nathan (subscriber, #3559) [Link] (1 responses)

Whilst no specificly addressing spreadsheet requirements, this Format comparison between ODF and MS XML shows fairly clearly to me which format is better.

odf as a format

Posted Dec 2, 2005 20:43 UTC (Fri) by pimlott (guest, #1535) [Link]

Let me get this straight: You conclude which is a better spreadsheet format, from a comparison which, you acknowledge, doesn't address spreadsheet requirements, but rather focuses on superficial matters such as stylistic conventions and affinity to other standards? And you poo-poo the statements of actual developers of office software? That makes no sense.

Both MS Office and OpenOffice are gigantic, hyper-featureful systems with over a decade of history. It's hardly surprising that their document formats, even newer XML-based formats, reflect their labyrinthine internal structures. It's going to take some hard study to determine which is a better "open" office format, or whether neither is really suitable.

Why is it that as soon as XML enters the picture, complex and difficult issues of representation and semantics suddently evaporate, and all we have to argue about is whether or not they used mixed-content?

"The language of the internet"

Posted Dec 1, 2005 14:18 UTC (Thu) by kevinbsmith (guest, #4778) [Link] (3 responses)

This is mostly a good article, with some interesting ways of looking at the issue. However, I wasn't impressed with the last section, where somehow the XML format (roughly speaking, a schema) used for office documents gets confused with the XML "schemas" used for web pages and web services.

It's not a perfect analogy, but XML is more like an alphabet than a language. Imagine that all the world's human languages shared a single writing script, such as the Latin letters used in English. Microsoft might control the English language (office doc schema) that is built with these letters, but that has no direct impact on the French or Thai languages (other XML schemas) built with the same letters. There are thousands of uses of XML that have nothing to do with the office doc XML schema.

Unless Microsoft is planning to modify the rules of XML itself (the alphabet), this debate really is only about office docs. The outcome will *indirectly* affect similar discussions in other domains, but would have no direct technical effect.

"The language of the internet"

Posted Dec 1, 2005 14:39 UTC (Thu) by pjgrok (guest, #17472) [Link] (2 responses)

I think it will have the same type of effect that MS's
proprietary extensions to HTML had, because of its
dominance.

"The language of the internet"

Posted Dec 1, 2005 23:21 UTC (Thu) by giraffedata (guest, #1954) [Link] (1 responses)

kevinbsmith's point is that nobody is talking about Microsoft extensions to XML; the article is flawed. They're talking about Microsoft extending something else. I would like to know what that is, and if it is a standard that has a name.

XML is not a language for writing documents, like HTML is. But I see a lot of people use the term "XML" as if it is. I suppose it's the same kind of mistake that leads people to use "IDE" (the technology of packaging a disk controller with the disk drive) as if it were a protocol for attaching a disk device to a computer. (In particular, they're thinking of the protocol called "ATA", one of many protocols that are implemented by IDE disk devices).

So does anyone know: Is there some XML-based document standard that Massachusetts is considering adopting? Does it have a name? Is there a Microsoft alternative, and does it have name?

Microsoft Extensions

Posted Dec 2, 2005 13:39 UTC (Fri) by kevinbsmith (guest, #4778) [Link]

My understanding is that there is a non-proprietary XML-based document format named "Open Document Format", used by several office apps (word processors, spreadsheets, etc). Microsoft has their own similar (but proprietary) format named "Microsoft Office Open XML". Apparently MS is proposing turning their format into an ECMA standard, but they would still have proprietary extensions (presumably schema extensions, with new tags or new meanings for existing tags) that could only be reliably written and read by their own products.

Any product that reads or writes office documents is affected by this. And of course if MS wins this battle, it will be harder for anyone to push for governments to use other open formats like OGG or non-proprietary calendaring. This is an important decision, and it will have broad effects throughout the US and elsewhere, whether MS wins or loses.

But, to my knowlege, in the context of the Massachusetts/Open Document Format debate, MS is not saying they will mangle XML itself (the underlying rules of tags, attributes, etc) in any way that would have a technical effect on any XML format or app outside the office suite domain. So XHTML, the XML format used by all modern web pages, seems unlikely to be affected by the outcome of this battle. Likewise, web service XML formats, including blog feeds and SOAP transactions should be safe from harm here.

Interoperability - a bad analogy

Posted Dec 1, 2005 15:40 UTC (Thu) by Baylink (guest, #755) [Link] (1 responses)

While I love Pam to death,

> Catch that? "On different communication systems."

is evidence of a lack of understanding of interagency emergency communications; it's not a good analogy for the situation at hand, in most current cases.

The em-comms issue is one of assigned and provisioned communications frequencies. Field em-comm equipment is *not* frequency-agile; it's channelised to match the municipality's license.

So, even though most of those radios are narrow-band FM (APCO 25 digital is up and coming, but not there yet that *I* know of, and besides, it, too, is standard for precisely this reason), they still can't talk to each other, unless it's pre-planned and agreed.

But the reasons are administrative.

The *real* reason is that while interagency channels and coordination plans do exist, none of them were designed for a disaster on the wide scale of Kat/rita. Ad, frankly, most of those people who need to talk on them have only the most basic instruction on how to handle a communications net with that many people on it.

This, on the other hand, is why we train and licence hams... though the integration of those hams into the emergency response plan was also, I gather, somewhat catch-as-catch-can. Hams *do* train for precisely that sort of situation (traffic triage; 1000 people on the same frequency, etc).

I'm sure we haven't heard all the stories yet.

Interoperability - nope, it's right on point

Posted Dec 12, 2005 3:16 UTC (Mon) by pjgrok (guest, #17472) [Link]

Hi,

I love you too, but it isn't just a bandwidth-type of problem. Different systems were also a problem, which you can verify by reading a Berkman Center white paper (http://cyber.law.harvard.edu/epolicy/roadmap.pdf) and this Washington Post article.

I knew about it because a Groklaw member who is an expert in the field told me. He was in discussions about the interoperability problem with FEMA before the hurricane hit. They were asking him about open standards and Open Source, but unfortunately the storm hit before everything could get fixed and ready. The same problem showed up in the Tsunami disaster and on 9/11, something I knew because I was in that state when it happened, and it is something you never forget. Here is a snip from the Berkman paper, quoted in a news article, on how communications broke down there in the tsunami crisis:

A thirty-foot-high wall of water - a tsunami - slams into the famed resort islands off Thailand's southern coast. In one tragic moment, thousands of lives are lost, and thousands more are missing. In the race to identify victims and assist survivors, Thailand's government hits its own wall.

"Responding agencies and non-governmental groups are unable to share information vital to the rescue effort. Each uses different data and document formats. Relief is slowed; coordination is complicated. The need for common, open standards for disaster management was never more stark or compelling. The Royal Thai Government responded by creating a common website for registering missing persons and making open file formats in particular an immediate national priority."

This quote, taken from the opening of the white paper, "Roadmap for Open ICT Systems" by the Berkman Center for Internet and Society, Harvard Law School shows how open systems affect more than the business world ...

The issue you raise is a problem too. But the analogy is based on reality.

When Is a Standard Truly Open? - When It's Universal, Reflections on Massachusetts and Microsoft's XML

Posted Dec 1, 2005 15:57 UTC (Thu) by N0NB (guest, #3407) [Link] (1 responses)

Beware the Vapor!

Based on some of the pronouncements comin out of MA during the past week it appears the vapors are having their intoxicating effects. MS is now "welcomed" and "thanked" by high government officials. Meanwhile, those that have championed true competition in MA are now suddenly subjected to audits and other investigations.

Politics has always been a dirty vocation, but there is no question that American politics has become especially nasty over the past quarter century. There is no doubt that MS sees not only much money but much perceived prestige on the line in MA and they are fighting like a junkyard dog--to the death if need be. They know that they cannot allow MA to mandate ODF as it seals their doom.

We are fighting MS with logic and reason which is often no match for the nuclear weapon MS wields--money--lots of money. Money is being funneled into the appropriate accounts, back pockets, and palms of politicians and IT pundits. This is a full-on frontal assault. MS is playing for keeps. Are we?

EU vs US

Posted Dec 2, 2005 10:10 UTC (Fri) by kleptog (subscriber, #1183) [Link]

This reminds me of a news segment on here on Dutch TV where they were talking about EU lobbyists and what they should and shouldn't do and whether they should be registered. They were interviewing one of the firms that did lobbying and he was asked "What's the difference between lobbying in US vs in the EU?".

His answer was along the lines of: Well, the annoying thing is that you don't get anywhere near as far with money because the whole campaign financing angle simply doesn't exist".

The presenter asked surprised: How so, annoying?

He said: Well, you know, you have to put more work in to get the same result. You have to convince them with arguments.

To which my only thought was: Good! You should work for your money.

Pity I can't find the reference anywhere or his name, but I'll remember the conversation for a long time.

When Is a Standard Truly Open? - When It's Universal, Reflections on Massachusetts and Microsoft's XML

Posted Dec 8, 2005 13:49 UTC (Thu) by wolfrider (guest, #3105) [Link]

This is the best front page news I've seen on LWN. :) Good article.


Copyright © 2005, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds