Publishing Is Broken, We're Drowning In Indie Books - And That's A Good Thing

I love books. Physical books. Books that sit in my lap and warm it like a sleeping pup. Three and a half years ago, I had an e-reader unwillingly thrust upon me. I ignored it at first; shunned it. Then one day I was packing for a long trip and it came on me in a flash that if I used the damned thing I wouldn’t have to limit myself to five pounds of books in my luggage.

Since then I read more ebooks than physical books. I buy a lot more books, too. Last year I noticed that books were getting cheaper, but the writing was getting worse. It started to get harder and harder to shop the Kindle store because I was either upset by the price of a book or the quality of its writing. Accidentally, I had stumbled upon the new face of self-publishing.

My experience reflects a profound and wrenching transformation of publishing that is shaking the industry to its roots. The beneficiaries of the existing order – major publishers and their most successful authors have become the most visible opponents of the turmoil that these “Indie” authors have introduced.

Which is too bad, because careful examination suggests that this period of chaos will eventually yield significant rewards for both authors and consumers. It even points a way forward for traditional publishers who have faced years of declining profits.

Is Indie Publishing Good or Bad for Authors?

I interviewed mega-bestselling techno-thriller author Brad Thor (whose new book Black List has already given me paranoid nightmares). Thor is unequivocal in his support for the existing system:

The important role that publishers fill is to separate the wheat from the chaff. If you’re a good writer and have a great book you should be able to get a publishing contract.

To me, it seems disrespectful…that a ‘wannabe’ assumes it’s all so easy s/he can put out a ‘published novel’ without bothering to read, study, or do the research. … Self-publishing is a short cut and I don’t believe in short cuts when it comes to the arts. I compare self-publishing to a student managing to conquer Five Easy Pieces on the piano and then wondering if s/he’s ready to be booked into Carnegie Hall

Why do mainstream authors dislike Indie publishing to the point where some even disagree with the coined term “Indie”? It comes down to worldview. Bestselling authors who are talented and hard working – like Thor and Grafton – are inclined to believe that publishing is a meritocracy where the best work by the most diligent writers gets represented, acquired, published and sold. But this is demonstrably untrue. The most famous counter example is that of John Kennedy Toole.

Many people know that Toole had his great American novel, “A Confederacy of Dunces” rejected by publishers and that he committed suicide at 31. They may not realize that Robert Gottleib at Simon & Schuster recognized Toole’s talent but believed Confederacy to be structurally flawed. Gottlieb did not think there was an audience for The Confederacy of Dunces without major revisions – revisions that would have changed the character of the novel. Toole refused to comply and eventually committed suicide.

Rejecting Toole’s work was a marketing decision that Gottleib made for Simon & Schuster. And it wasn’t necessarily the wrong decision from a marketing standpoint. Remember, Gottleib was the guy who acquired Catch-22 from Joseph Heller based on a partial manuscript. In the publishing world as it stood then and stands now, Toole’s work might have never found its audience. Without the advocacy of novelist Walker Percy – which helped generate the literary attention that allowed the book to win a Pulitzer Prize – the novel might well have failed.

Indie Success and The Publishing Lottery

Another reality that goes against the establishment view of Indie authors is that some of them have, in fact, gone on to sign very significant contracts with major publishing houses. A few examples:

Amanda Hocking wrote 17 teen supernatural suspense novels in her spare time and then self-published them, becoming the first Indie sensation before she signed a $2mm deal with St. Martins Press.

John Locke sold over two million copies of his Indie books before signing a limited deal with Simon & Schuster to get physical distribution for some of his novels.

E.L. Jameswrote the precursor to Fifty Shades of Grey online as fan fiction and self-published it on her own website before Vintage acquired it. She was named one of Time Magazine’s 100 most influential people for 2012.

Of course, there are always exceptions to the rule and these examples don’t necessarily prove that publishing is broken. But the conceit that Indie authors are merely a bunch of lazy hacks unwilling to face rejection ignores the fact that even the biggest proponents of the old publishing system admit that there are many talented published authors nobody has ever heard of. Have you read The Lion’s Eye?

And this line of thinking also ignores mid-list authors. These men and women have talent and persistence but write for small market segments. The economics of traditional publishing makes it very hard for either publisher or author to profit from a pool of only 10,000 to 30,000 readers, no matter how devoted. Amazon, on the other hand, allows Indie authors to keep up to 70% of their e-book royalties, compared to the 15-20% royalties that conventional book publishers offer authors on printed books - and mainstream publishing royalties are much worse for ebooks. Publishing independently can allow mid-list authors to make a reasonable living on writing.

Just like A-list actors, writers like Grafton and Thor are superstars – the exception rather than the rule. But exceptions exist on both sides of the publishing divide these days. One of the newer Indie stars is Hugh Howey, the science fiction author whose brilliant, dystopian novella Wool has sold more than 200,000 copies in the U.S. alone, sold overseas rights in fifteen countries and was recently optioned by Ridley Scott for a movie. (His newest book, I, Zombie launches today.)

When I asked Howey about the view that indie authors would do better to try to publish conventionally he compared the current system to a lottery:

When people think of traditionally published successes, they think of The Hunger Games and Harry Potter. This is what they compare to the self-publishing route. But those are lottery winners, the extreme outliers. In order to level the playing field and have a true comparison, you need to look at everything that gets submitted to the traditional machine – that means all the work that never makes it out of the slush pile – and compare that to all the self-published e-books on Amazon and elsewhere. Counting the top 1% from the traditional route and everything from the self-published route creates a weighted argument and is disingenuous. And calling cases like mine the exception and forgetting that this is also true of every book in the center aisle of the bookstore is also facile.

Tell me this: why is self-publishing antithetical to “honing one’s craft?” Who ever received writing advice in a rejection letter as sound as the worst 1-star review out there? There’s far more to learn from engaging the market with your product than there is in form letters that tell you not-a-single-frickin’-thing. What’s wrong with testing the waters? Instead of wasting one’s time writing query letters, why not work on that next manuscript instead?

There is something very odd about this war of words between successful authors on different sides of a tectonic shift in the publishing world: it doesn’t exist in many similar industries facing the same sort of technological upheaval. You don’t hear Christina Aguilera or Adam Levine knocking indie bands. Instead they joined a show called “The Voice” which aims to capitalize on the credibility of indie artists by finding journeyman artists and giving them a shot at major label contracts. Indie filmmakers are revered, not reviled, partly because they eschew the studio system and its constraints on artistic expression. And the art world seems keenly attuned to the idea that the next Georgia O’Keeffe might be producing revolutionary work somewhere out of their sight until she turns 30.

Publishing is different from other creative industries because the machinery has not yet adapted to the profound technological shift it is undergoing. But it’s not completely alone. Perhaps the best comparison is in comedy, where a new generation of comedians like Louis C.K. is changing the way that comedy is produced and marketed. As reported by Sean McCarthy in The Comic’s Comic, the comedian Patton Oswalt said as much in two letters he read to a comedy conference in Montreal this year. The letters are worth reading in their entirety but here is an excerpt:

I’ve been lucky enough to be given specials on HBO, Comedy Central, and Showtime. As well as I’ve been lucky enough to release records on major labels, and I was lucky they approached me to do it. And that led to me being lucky enough to get Grammy nominations.

I know that sounds like a huge ego-stroking credit dump. But if you listened very carefully, you would have heard two words over and over again: “lucky” and “given.” … The days about luck and being given are about to end. They’re about to go away.

Not totally. There are always comedians who will work hard and get noticed by agents and managers and record labels. There will always be an element of that. And they deserve their success. And there’s always going to be people who benefit from that.

What I mean is: Not being lucky and not being given are no longer going to define your career as a comedian and as an artist.

The Problem Today – No Rotten Tomatoes for Books

When I asked Olen Steinhauer, the bestselling spy novelist of The Tourist and An American Spy fame (and one of my favorite authors writing in any genre today) about the experience of buying and reading ebooks, he expressed some concern,

The e-book changes nothing from an author perspective – I’m a big consumer of e-books. But the “noise” of self-publishing is so vast. I worry that writing is becoming steadily devalued.

Indeed, the problem for readers is that regardless of which side you agree with in theory, in practice you probably love the idea of buying books for under $5.00 but hate the idea of having to sort through quite so much junk to find good books at that price. The question that divides Indie fans from the traditional publishing industry is whether a solid selection of good writing can ever be self-published for these low prices.

Consumer ratings should help sort out the mess, but they don’t. It seems that every author has twenty or thirty friends who are willing to write glowing reviews of his or her book, no matter how awful. And conversely, a mainstream author like Brad Thor finds himself targeted by scads of low ratings based solely on the high list price of his ebooks – which he does not control. The net effect is to make the peer rating systems useless.

So the big question for publishing is which way this paradigm will evolve. Will our future feature lots of small new interesting writers at low prices and a bunch of bestsellers for somewhat higher prices? Or will the chaos eventually yield to a higher pricing model where only the most stubborn and talented Indie writer can ever break through?

To understand the future of publishing we must examine three trends that have helped to rip it apart: eBooks, social media and low-cost self-publishing. But before that, we first need to understand the two big strategic decisions that traditional publishers made which left the door open for Indie books.

How We Got Here Part One: Hardcover Book Pricing

Without getting too far into microeconomics, I will suggest that traditional publishers set the stage for their current misfortunes years ago, when they developed the pricing model for printed books.

Books deliver three sorts of value: good writing, good stories (true or fictional) and social connection. By the last, I mean that reading is a social activity and we like to read what others are reading. So popular books are worth more to us than less popular books.

Ideally, every book should be priced so that its individual price maximizes its total marginal profit on the demand curve. In plain English that means that publishers should figure out the price above the cost of printing and distributing a book at which that particular book will make the maximum amount of profit. As consumers, we see this type of pricing all the time: such as when we buy wine. The price of a bottle of wine reflects the quality, availability and popularity of what’s inside, not just the cost of cultivating the grapes and making the wine.

The standard argument of publishers is that in order to find a few hit authors like Thor and Grafton, they have to pay lots of advances to untested authors who will sell few books. Therefore they set the price of all hardcovers to $25.99 or $27.99 because that’s the price they need in order to recoup their total investment. This is a fixed cost fallacy. The publishers’ logic assumes that demand is fixed: that you’ll sell as many copies of a book at $25 as you would at $5.00. That’s demonstrably untrue even in traditional publishing (paperbacks sell scads more copies than hardbacks), and it does not optimize revenue.

A simple example can show why. According to Olen Steinhauer, the production cost of an average hardcover printed in a lot of 50,000 and including distribution is around $2.00. If a hardcover by unknown novelist X sells 5,000 copies at $27, the publisher makes $125,000 in marginal profit. If that same novel sells 50,000 copies at $7, the publisher makes $250,000. That’s more profit to cover all of those advances to unknown authors. And the idea that “paperbacking” (selling the same product in a lower-quality format to scoop up a second round of profits) has the same effect is a canard. Most books never make it to paperback. Their value is already lost. And eBooks have largely replaced much of the paperback demand. In any case, the demand curve is not mysterious and unattainable – it can easily be established with testing, particularly online.

By creating and adhering to a rigid and suboptimal pricing model, publishers created a market opportunity for indie books. That window has been sitting open for decades.

How We Got Here Part Two: Publishers Turn Authors Into Marketers

Once upon a time, publishers had relatively small lists of new authors whose books they distributed to independent bookstores and small-format mall chains like Waldenbooks and B Dalton. Publishers fought vigorously for limited shelf-space and display.

Then in 1985, Border’s open the first superstore and by the early 1990’s Barnes & Noble followed suit. The new format dramatically increased both shelf space and display opportunities. Superstores also incorporated cafes and comfy chairs to allow consumers to evaluate books at their leisure.

On the publishing end, this allowed publishers to make bets on a more authors. But these expanding advances and production runs put real financial constraints on publishers because overall book sales weren’t exploding. Beyond that, profits were being squeezed by the increasingly popular practice (pioneered by Barnes & Noble) of steeply discounting bestsellers. This strategy assumed that bestsellers would draw consumers into bookstores and that turning the most desirable books into loss leaders would fuel more profitable sales. The net effect was to financially strain both publishers and booksellers. Then came the Internet and with it Amazon. Margins plummeted.

The publisher reaction was to cut staff and costs. Marketing support to lesser authors was one of the first victims. As a result, most new authors who make it through the arduous process of finding both an agent and a publisher are surprised to learn that it is the author who is responsible for marketing and promoting his or her own work. Trey Ratcliff’s story on GigaOm illustrates this brilliantly.

An entire generation of traditionally published authors has come of age learning to self-promote. Particularly for mid-list authors the burden of writing and marketing a book a year without much assistance can be crushing. Some publishing houses have trimmed back even further, limiting editorial assistance to new writers to proofreading and line editing rather than structural editing.

These authors feel less beholden to publishers and more independent. They have been forced to become entrepreneurs, but are not rewarded commensurate with their contributions.

By the end of the 2000’s, the elements existed for a revolution. A few gatekeepers tended an inefficient pricing model that consumers and authors both hated. A whole set of mid-list authors were increasingly not making enough money to write fulltime. And beyond that, a generation of new writers was growing up on social media. These writers are tempermentally less able to accept the gatekeeper model of publishing.

Still, it took three distinct technological advances to blow the model apart:

In January of 2011, Amazon.com reported that it was selling more ebooks than physical books in the U.S. market. This month, Amazon reported the same news for the U.K.

The trend is driven by two factors. First is the obvious convenience factor. Consumers quickly catch on that with an e-reader all you have carry with you is one thing. No more searching for that dog-eared paperback under the nightstand. Secondly, and perhaps more significantly, eBook readers seem to buy more books. Even traditional authors agree. “I love e-readers. I own one of every single e-reader,” Brad Thor says.

The biggest effect of eReaders may be that it enables the so-called “Long Tail” of publishing to flourish. Small books with a lesser audience don’t get squeezed off of bookshelves. Books that take time to find their audience can still flourish.The biggest change in the center of gravity in publishing is that traditional publishing operates in a world of limits – limited shelf-space, time and resources. The ebook reader eliminates that.

Electronic books also erase the physical quality gap that existed between mainstream books and the self-published books from the vanity presses of old. Everything looks the same as an eBook.

Advance #2: The Rise of Social Media

Most book sales are driven by personal recommendation. Social media – from Twitter and Facebooks to rating sites like Goodreads – just make it easier to recommend a book to lots of friends at the same time.

Hugh Howey says:

The effect of social media has been massive. Where social media is really powerful is in the area where the author has no control whatsoever. Word of mouth for WOOL spread because people were posting it on their Facebook page and Twitter accounts. It wasn’t through my efforts; but the efforts of readers. Many people think social media is about what the authors are doing not what readers are doing, and I think they miss the real driving force behind their sales.

Brad Thor, who was the host of a travel show on PBS before becoming a best-selling author, relies heavily on Twitter:

It keeps the relationship fresh while my readers are waiting for the next book. Think about how solitary my career is – I write and it all happens in my mind. Twitter is real human interaction and adult conversation for me.

Few authors have harnessed social media as effectively as Bidinotto, a former journalist:

Social media has been the great equalizer of advertising, promotion and marketing. This is essentially asymmetrical warfare. No customer going to Amazon knows what is traditionally published or independently published – and they don’t care. They’re interested in an experience that will educate or entertain them. Social Media allow the individual author to become a personality and establish real emotional bonds with his readers. I happen to really like my readers and I deal with them online all the time. By using social media to become a personality to my readers, I have not spent one nickel in paid advertising – and I haven’t had to.

Advance #3: Low-Cost Self-Publishing

Right around the time that eBooks were winning the war against the printed word, an independent bookstore in Manhattan, McNally Jackson, took delivery of a behemoth new printing machine. The Espresso press they installed in 2011 wasn’t meant for flyers or greeting cards. Instead it prints and binds entire books, quickly and cheaply. This allows McNally Jackson to mimic Amazon in offering millions of titles including out of print books. It also makes them more viable as an impulse destination because you’re sure to find the book you want there.

Print-on-Demand came at the exact moment that the economics of traditional publishing - including big print runs of 12,000 or more books - were looking less and less tenable. Although the per-copy costs are higher, print-on-demand creates no returns and leaves no unsold physical books. It also removes a major barrier for self-publishing: the upfront investment.

The parallel digital tool that emerged at the same time was the ability for authors to upload files to Amazon, Barnes & Noble and other sites and sell them electronically as eBooks. Critically, Amazon made the strategic decision not to ghettoize these works in some “Indie bookstore”. Amazon also created reporting tools that allow authors to track sales in real time. And finally, Amazon allowed authors complete control of pricing, while creating a royalty structure that incents authors to price above $2.99. Hugh Howey sums up the new reality best,

I don’t have to compete with the price of mainstream publishers. They used to have the price advantage with economies of scale and the realities of large print runs. Now I have the advantage because I have low overhead. Where I once couldn't compete with their physical price, they now can't compete with my digital price.

Three Predictions For The Future

There’s no knowing how this struggle between the old and new guard will turn out. If we consider the flaws in the pricing model for hardcover books as well as the strong economics of self-publishing for authors, we can at least guess that indie books are here to stay. Three predictions:

Prediction #1 - Indie Books Get Reviewed For Real

Rotten Tomatoes for movies and Pitchfork for music show that it’s possible to give consumers good online tools for judging quality. Kirkus has been using Indie publishing as a profit center, selling review services to Indie authors for the very significant price $425, which may cost many authors more than the rest of the publication process. Kirkus would do better in the longterm to dramatically expand their uncompensated reviewing to include a good selection of indie titles and thus become the tastemaker for a new generation of readers. Goodreads is the obvious candidate as an impartial arbiter, but the site functions more like a social network than a review site and might lose the battle to a more single-minded competitor. There are also several review sites struggling to establish a reputation for fairness amongst all of the chaos: Digital Book Today, Self-Publishing Review and IndieReader.com. Storybundle has an interesting business model that allows readers to set their own pricing and also attempts to filter.

Even Amazon and Barnes & Noble could improve the online experience by creating a corps of professional reviewers to weed out the Indie catalogue. Amazon offers a service like this for mainstream books called “Amazon Vine” already. It conscripts top reviewers to review specific new books in return for getting the titles for free. As it exists today, though, Vine is more of a revenue-focused promotional program offered to mainstream publishers than a tool for readers. A completely independent program that includes Indie titles and compensates the reviewer while preserving Amazon’s independence would be better. Maintaining a separate "grade" for professional reviews would be better still.

There is enormous pressure in the market to solve the “drowning in bad writing” issue with Indie publishing. It’s hard to imagine that a solution won’t emerge in the next 12-18 months

Prediction #2: Mid-List Authors Move Indie

The second near certainty is that mainstream authors who don’t reach the blockbuster sales of Brad Thor or Sue Grafton – but have an established audience – will choose the favorable economics of Indie publishing. One thing that mainstream and Indie authors seem united on is contempt for is the royalty structure that mainstream publishing houses apply to eBooks which is much less favorable than for printed books. This is foolish and counterproductive for publishing houses. Indeed, authors like Edgar-winning mystery author Lawrence Block and Margaret Muir have already embraced indie publishing for the obvious economic benefits. There are also many authors who started as self-publishers and are making a solid living in Indie-land. Many of them would be considered midlist or even minor celebrities if they had more visibility to the existing publishing world - I'm thinking of people like Robert Kroese and Rachel Thompson here.

Prediction #3: Mainstream Publishers Will Use Indie Publishing as a Minor League ... And Find a New Profit Model

Legacy publishers will be hurt badly by Indie books until they find a business models that co-opts them. One obvious plan is to revert to the old days when publishers made bigger bets on fewer titles and were really the marketing experts for their authors. In this world, Amanda Hocking and John Locke's engagement with mainstream publishers after achieving Indie success becomes the rule rather than the exception and Indie publishing becomes a minor league for publishers. By choosing authors who have already built an audience and know how to effectively leverage social media tools, publishers will mitigate their risks and improve their all-important bestseller ratio. They may also learn to price according to the demand curve and thus maximize profits.

Publishers also need to manage their backlists more effectively. Hugh Howey points out that in fiction many of the top-sellers are not recently published novels. Publishers are used to pulling titles out of physical circulation after a few months and focusing their promotional efforts on new authors. However a rich back catalog, all made available in electronic format will allow publishers to respond more quickly to emerging market trends.

The greatest irony of the current frisson over Indie publishing is that we’ve seen it all before. Ben Franklin would recognize this era. From the 16th to the 19th century, pamphleteering allowed unpublished hacks like Thomas Paine to espouse their views and argue their points cheaply and individually. Pamphleteers were accused of vanity, incompetence and even sedition. But the best of them survive in the literature of the Reformation, the English Civil War and the American Revolution. In generations to come, the same may be said of a few of the Indie authors publishing today.