On the Dastardly Evil of Proprietary eBook Formats and the Inherent Righteousness of Industry Standard Formats

If there is anything that the digital publishing industry loves more than complaining about Amazon invading small African countries Amazon's business practices, it is decrying Amazon's continued use of a proprietary ebook format.

This topic has been going around for at least six years that I know of, and it most recently came across my desk again last week. Joshua Tallent sparked this latest round of the debate last week when he explained over on DBW why proprietary formats are sometimes better:

Why do these companies choose to use proprietary formats instead of (or in addition to) the open source EPUB standard?

Because proprietary formats just work.

When an ebook retailer builds its own ebook format, it steps beyond controlling just the reading system or device to actually controlling the code that is loaded into that reading system. That level of control brings with it a level of certainty about what the retailer can expect to receive, in turn giving it more control over features, functionality, and—importantly—the quality of the reading experience.

He's not wrong; this is exactly why Kobo, Apple, and Google each use their own proprietary ebook format internally while maintaining support for Epubs loaded externally (in the case of Google, this is just a fig leaf).

Tallent's post is so obviously true that it is effectively unassailable. Nevertheless, someone managed to come up with a counter argument.

Andrew Updegrove responded to Tallent on Thursday with a post which twists Tallent's argument in such a way that it gave Updegrove a chance to rail at Amazon with an argument that I believe is fundamentally incorrect on several different technical details:

This is the same nonsense that has been propounded in every single media format that has ever been created where there is a dominant vendor that wants to stay that way. It’s what Microsoft has contended for decades about desktop software formats that underlie its Office productivity suite, and what we also saw in the early days of digital music. In each case, vendors (or, in this case, sales and distribution channels) have sought to gain, or retain, monopolist control and lock in their customers by fighting the creation and universal adoption of a standard that would allow freedom of movement and true competition in the marketplace.

...

Authors are the victims of this nonsense at both ends of the supply chain, because Microsoft still controls the desktop (I’ve written about the consequences of that reality here), and Amazon still controls the distribution channel to most readers. That means that instead of being able to simply create a book in your favorite word processor and publish it directly to every Internet bookstore with a few keystrokes, you’re stuck with a miserable experience preparing your files, and then uploading them multiple times at multiple sites with multiple issues to deal with at each.

It’s bad enough that this is the situation we’re stuck with, but it’s intolerable to have to read someone contending that we’re better off this way.

I see two fundamental problems with Updegrove's arguments, and I will take them in reverse order.

Updegrove blames the fragmentation of the ebook market on Amazon's insistence on using their own ebook format. Apparently the Kindle platform is solely responsible for Apple, Google, Kobo, et al each having their own upload requirements and platform quirks. (Yes, Jeff Bezos really is that powerful.)

That argument is obviously insane, but it is also fundamentally false on a technical level. It ignores the fact that one can upload an Epub file almost anywhere, the existence of tools like Sigil which can make that Epub, and the several ebook distributors like Draft2Digital and BookBaby which will upload the ebook for you.

Updegrove also misses the point that this debate is just as much about DRM as it is about ebook formats (though he does dismiss this point when it was raised in the comment section of his post). Should Amazon switch to Epub tomorrow, we would still have 3 walled gardens dividing the ebook market.

Those 3 walled gardens are controlled by the dominant DRM vendors: Amazon, Apple, and Adobe. (For the sake of simplicity, I'm ignoring the quirks in B&N's DRM and other lesser details.) Two of the walled gardens use DRM which is both exclusive and proprietary, while the third (Adobe) is equally proprietary but is also available to license - for an expensive fee, of course.

Updegrove thinks that Amazon adopting Epub would "allow freedom of movement and true competition in the marketplace", but he neglects to take DRM into account.

Apple will not allow ebooks out of its walled garden (oh, and they're sold in a proprietary format), and Adobe charges app developers, device makers, and ebook sellers an admission fee (making the garden expensive to enter).

If Amazon adopted Epub today we would still have those two walls dividing the market, thus rendering his desired goal moot.

Furthermore, should Amazon give up the Kindle format as well as the Kindle DRM, I suspect that little would change in the ebook market. As Chris Meadows points out at Teleread, many Kindle users aren't interested in mastering the technical skills involved in loading ebooks on to the Kindle:

The thing is, though, that for many Kindle users, even the process of sideloading e-books is beyond their expertise or even their comprehension. Even leaving aside user-unfriendly conduit software like Calibre, they have to plug stuff in, and open windows on their computer and drag and drop—it’s a nightmare! Hence, I have my doubts that many people would take advantage of “easy” interoperability even if it existed.

Just look at Baen. Baen has sold DRM-free e-books for a decade and a half, but even when it was allowed to email the books directly to people’s Kindles, that was apparently still too hard for most of its would-be customers. Baen kept getting asked why its e-books weren’t “on the Kindle” yet. It finally had to resort to making sweeping changes to how its e-book store worked in order to get its titles on Amazon’s Kindle store—because that was where all the customers were.

The marketplace had already chosen a standard. It’s mobi and Amazon’s derivatives thereof. That the marketplace’s standard is different from one a consortium has agreed to might be something the consortium should look more closely at.

I agree, consumers have the final say. They decided that Sony's proprietary Blu-Ray was better than HD-DVD, they've decided that Mobi works well enough for their purposes.

Standards may be great and wonderful things, but so long as consumers choose not to adopt the industry-approved standard, we might as well be mice voting to bell the cat.

Nate Hoffelder

Nate Hoffelder is the founder and editor of The Digital Reader. He has been blogging about indie authors since 2010 while learning new tech skills weekly. He fixes author sites, and shares what he learns on The Digital Reader's blog. In his spare time, he fosters dogs for A Forever Home, a local rescue group.

Related Posts

32 Comments

Actually, I’d be inclined to say not just many Kindle users, but the vast majority of Kindle users. People just don’t seem to get that even caring about technical matters makes them the rare exception.

You saw this back in the early days of DVD, when people complained about the black bars along the top and bottom of their screen. (And then in the era of HDTV when they complained about the ones to the left and right on older content.) I dealt with all of those people while I was working tech support for Insignia. They didn’t know anything about why these were the case. They didn’t want to know. It was too hard for them even to google it and figure it out.

You see it with e-books now. The average user doesn’t know and doesn’t care about any of the technical matters surrounding e-books beyond can they press a button and get it on their magic reading machine within a few seconds. The fact that they don’t have to care is the true genius of Amazon, and what allowed Amazon to expand e-books beyond the small core of early adopters who had characterized the market up to that point.

It’s a psychological blind spot a lot of us have, that we assume just because something is simple and easy for us, it ought to be simple and easy for everyone. It’s why the ability to do tech support is such an uncommon thing—it requires a knack for reminding oneself that, no, this thing you think is dead easy is the hardest thing in the world for them. That’s why they called you.

Again, the fact that Amazon was able to make buying e-books so completely dead simple for average, ordinary people is the biggest reason why they won the e-book war before any other combatants had even taken the field. And people just don’t get that anything even slightly more complicated than Amazon’s system will fly right over the heads of the vast hordes of people who need it to be that simple or they couldn’t use it at all—the very hordes who carried Amazon to its victory.

AMAZON does support epub the way he presents it.
Publishers big abd small can submit their ebooks as epub and Amazon will happily accept them…
…as feedstock.
The consumer will, naturally, receive whatever variant kindle format best fits their device. Kindle 1 & 2 owners will get mobi7, others KF8.
And never will they have to worry about which it is: it just works.

Kinda like streaming video: hardly anybody cares if Hulu, Netflix, or Prime uses Flash, silverlight, or whatever so long as it works. Once it stops working…

He’s not ranting about end user display, but only about authors’ inabililty to submit the same file everywhere. Which they can, in one way or another.
Which is why the publishers themselves don’t whine about the mobi format itself. To them it is a non-issue.
Some do handwring over walled-garden lock-in but not too loudly since the domination of the gardens is directly attributable to their own actions, both legal and… less than.

To the best of my knowledge, the Kindle is the ONLY e-book reader that doesn’t support ePub in addition to the manufacturer’s proprietary format. Literally everyone else displays at least this little bit of openness, *apart from Amazon*. So yeah, I can single them out just fine.

And sure, most people don’t care, just like they don’t care about using Microsoft’s proprietary office formats. But guess who gets called to fix everything whenever those proprietary apps and formats don’t work properly — the rest of us, who do care.

Because things DO break, and if you never cared and got stuck with, say, a discontinued DRM scheme… you’re helpless. Enjoy.

One major problem with Amazon supporting Epub is that it would double the number of headaches for Amazon’s own developers and support teams. If nothing else, Amazon is avoiding the hassle of having to say “oh, we support Epub, just not DRMed Epub” and having to explain the difference to angry customers.

This route is a heck of a lot less complicated for both readers and Amazon without actually restricting readers any more than if Amazon supported DRM-free Epub. After all, any store that sells DRM-free ebooks will have to support the Kindle format if they want to sell to kindle owners. And that means Amazon can make the format issue someone else’s problem.

The key point to consider is that despite “failing” to support epub at the consumer level, Amazon has not only survived, they have prospered and in some markets dominated. Evidence that consumer-level ePub support is not actually a requirement of the ebook market.
This renders the debate a purely political exercise, really.
It boils down to committee collectivism vs market entrepreneurialism.
And entrepreneurialism is “winning” the argument before the one court that matters: consumers.

So has Microsoft. But ask any Windows user if they were ever truly happy with the OS they’re using. I bet the answer will be, “it sucks, but that’s what everyone else uses, and I must be able to work with them”.

The free market doesn’t work. People do not, in fact, choose the best product. They choose what seems easiest to get at first, the shiniest bauble waved in front of their eyes, and then they’re stuck with it. Assuming they get a choice in the first place, and “entrepreneurs” hate competition more than anything else in the world. Even regulation is welcome… but only when it reduces competition instead of increasing it.

And sure, this entire discussion is a political exercise. But that’s what *all* discussion is. Absent that, we might as well let the One Party re-elect the Beloved Leader for another term, and be done with it.

What’s the “best” product? By whose standards is that judgment made? Taste is subjective in all things. The free market make not always work the way you or I or any individual taken in a vacuum would like in every case, but that doesn’t mean it’s not working at all. Back in the days when Apple was in danger of going belly up to Microsoft’s OS dominance, I was convinced Apple was the best product. That “free market failure” as I would see it per your standards, didn’t seem to stop Apple from growing into one of the most powerful companies in the world on the backs of offshoot devices that played a huge role in pulling people away from their desktops, even though they still lag behind Microsoft in that now much-diminished market. I struggle to see where the free market failed in any of that.

I think a better alternative to argue this as mainly a moral issue or not, would be to look at it in a perspective like in this paper, analyzing Amazon and Apple as frenemies in the reader market: http://hbswk.hbs.edu/item/7806.html. Here the analysis is based on multisided platforms/-markets theory and game theory.

For both the reader/publisher/author, the cost of proprietary formats is related to the cost of multi-homing, that is, licensing/owning, using and maintaining multiple platform devices/endpoints needed for reading/publishing.

If everybody supported the one, open standard, this cost would be reduced since, in principle, there would be just this one giant platform with different Us and affordanses.

This, of course, is not great news for platform owners that will meet a huge competition. But, still, in some scenarios it would make sense for platform owners to be frenemies with the open platform. Probably as long it is not to big, and you both compete with another, bigger platform.

If a platform is dominant enough, the cost of multi-homing will be seen as too high for both sides of the platform/market — no matter if the platform is open or not.

There are several issues that seem to be ignored in the debate. One of them would be the fact that the mobi format has technical limitations that do not exist in the epub format. So no, it’s not as easy as making an epub and then uploading it everywhere. For some months now I’ve been working on publishing ebooks, and I’ve hit plenty of walls due to Amazon’s proprietary format. Just visit the MobileRead forums and you will see.

Second, amazon’s proprietary format, DRM and market share, combined with their publishing policy means parts of the world are effectively not allowed to do business. Amazon now handles all ebook redistribution through KDP. KDP has a limited list of allowed languages. So when publishing houses and other rights holders require DRM on their products (important for translations), if the work is not in a language from the KDP list, you effectively can’t legally sell to a large share of your customer base. All you can do is cheat and pretend it’s in another language and pray the automated system doesn’t notice.

And last but not least, Amazon are greedy fuckers. They take a 65% cut of all ebook sales outside the Amazon Sales Territories (which is the US, Western Europe, Japan, India and a few more counties).

So in your own words, Nate: “Bleh. I know that Baen is merely increasing prices to cover Amazon’s cut, but the increase is likely going to make me more cautious about experimenting with new authors.” – thank Amazon for that.

If you’d read the whole phrase you’d see I mentioned the mobileread forum.

But aside from complicated stuff like those already mentioned, there’s simple issues like text flowing around an image that’s bigger than the character size. If you look on the forum you’ll find many more problems with fonts and other minor formatting issues that lead ebook developers to spending extra time on the mobi format.

Not to mention that the documentation from Amazon is lacking, to say the least, and development is often a trial-and-error process even for simple matters like pop-up footnotes.

If you’d read the whole phrase you’d see I mentioned the mobileread forum.

I did read the whole phrase. Mobileread is a big place, so saying “go look through it” is about the same as saying “I’m thinking of five numbers between 1 and 500,000. Guess which ones.”

Perhaps you could dispense with the snark when it’s not warranted. I asked a simple question. As an ebook developer myself, I may have had ideas to share. But, you know, it’s always fun to spend hours at Mobileread. Good luck.

lol I’m surprised you didn’t decide to publish this blog in Microsoft Word.

I expect companies to follow industry standards wherever possible and where it isn’t possible I expect them to structure the workflow so they can work in parallel with the industry standard initiatives and eventually integrate into the industry standard. When this doesn’t happen it introduces long term inefficiencies, complications in data life-cycle management, lack of competition and barriers to innovation. Companies don’t always behave well with others and it’s important to tell them we expect better from them. Otherwise you come off as just an Amazon schill.

It’s funny you should laugh; I published this in WordPress. While they maintain the pretense of being open source, the platform is about as proprietary as MS Office. WP makes arbitrary changes, they invent their own rules, and they adopt and abandon web standards on a whim.

Speaking as a professional WordPress developer, the WP team can’t reinvent (My)SQL. All the data generated by WordPress can be processed with 3rd-party tools; the format is public, self-documenting by its very nature, and they can’t lock it up. They also can’t replace HTML with something else. Or Javascript. (Actually wait, they could. But they aren’t.) And you just try looking at the source code of MS Office to see how something is supposed to work when the documentation doesn’t make it clear.

WordPress has many sins. But to accuse them of being as closed as MS Office is simply unfair.

Talk about DRM is missing the point from the perspective of publishers and independent authors. DRM imposes no burden on us. I’ve never even heard of situations where it has changed the appearance of a book. It’s simply tacked on by the retailer. As an author, fussing about it makes no more sense than fussing about whether my print book is shipped by UPS or USPS.

And yes, it is an issue for readers, but the entire ebook market is filled with issues for readers that aren’t being addressed. Do those who “buy” an ebook really own it? If so, why can’t they sell it or pass it along in their wills? That’s every bit as important as DRM, since it restricts what they can do with what they think they own.

Ebook standards are different from proprietary. The standards matter because you can’t get to standards that work with every outlet until you have standards that may require a bit of tweaking either at the author end or the retailer end. The existence of slightly different epub results on Apple, B&N, and Kobo are simply a stage in that process. Eventually, all those vendors will make their almost compatible products fully compatible.

We’ve seen that happen over and over again. PDF used to have issues, but it was an standard and those issues were eventually solved. The same was true for HTML, although the process was a bit more messy. I can recall when there was a movement to force web developers to get on the standards wagon and quit right just for Microsoft’s browser.

The existence of Amazon’s mobi and KF8 formats isn’t a stage in that standardization process. It’s a barrier to it. As long as Amazon rules over 70% of the ebook market with no standard formatting, there’s less pressure for the other 30% to standardize its products.

One more comment. Claims that Amazon’s proprietary formatting lets them make better-looking ebooks is utter bosh. Mobi still has the look and feel of text on a Palm Pilot of some 15 years ago. KF8 is so inadequate, Amazon had to make a big deal recently when it acquired the ability to do drop caps.

Epub has numerous inadequacies. I’ve certainly done my share of complaining about them. But it is still head and shoulders above Mobi/KF8.

Finally, what matters are powerful tools that eliminate the bother and expense of handcoding for a format. Open standards make it relatively easy for tools to develop those tools, one example being epub export by InDesign. Closed, proprietary formats that can change on a corporate whim make those tools hard to develop. In fact, even Amazon hasn’t upgraded its InDesign plug-in for so many years it’s now worthless. What Amazon can’t (or won’t) do for its own format isn’t something a third-party developer can do.