Robert Cooper and Scott Edgett, founders of the Product Devel-opment Institute, estimate that “46 percent of the resources companies devote to the conception, development and launch of new products go to projects that do not succeed — they fail in the marketplace or never make it to market.“

Why is innovation so fraught with failure? Why do some companies succeed at innovation, and others, despite their best efforts, fall short? How is it that companies that were once the picture of innovation — like Kodak, for instance — fade away? It wasn’t that Kodak didn’t keep up with the times. It introduced digital cameras, inkjet printers and other accouterments of the digital age. It kept its innovation engine running, and yet that wasn’t enough.

Each and every enterprise has its own quirks and faces its own set of special circumstances. It’s impossible, and unwise, to generalize. But if there’s one factor that nearly always seems to separate winners from losers it is this: a willingness to take risks. In Kodak’s case, some have suggested that the problem was a reluctance to break free of its assumption that people were still as interested in printing out their photos. In this respect, Kodak was perhaps overly cautious, and in the process lost touch with its consumers.

That innovation is all about taking chances certainly helps explain why the failure rate is so high. JC Penney is taking all kinds of risks these days and so far the only notable result is declining sales. Maybe there’s something about the way Penney is going about innovation that is elevating the risk while suppressing the rewards.

Many have pointed to Penney’s too-quick conversion to “everyday low prices” as the culprit. For decades, the retailer had trained its shoppers to use coupons and wait for sales. Getting too far out in front of your consumers and changing too suddenly can indeed be a problem. Others suggest that Penney’s misstep is but a speed bump on the road to a grander vision of a reinvented department-store experience, and one that will take time to develop. This will be interesting to watch. After all, the retailer’s CEO, Ron Johnson, was a driving force behind Apple’s retail-driven comeback, and his appetite for insanely successful innovation should not be underestimated.

Procter & Gamble’s legendary culture of innovation is also worth considering, particularly given that its CEO, Robert McDonald, has recently come under attack because of weak earnings. The company’s stock has since rebounded, but P&G is not sitting still, and is promising “a new wave of innovation,” a raft of 30 new products that challenge the status quo by adapting to changes in consumer needs and retail environments. Jorge Mesquita, group president for new business creation and innovation at P&G, told the Cincinnati Business Courier that the focus was on “disruptive solutions that create new categories.”

This is risky — it remains to be seen how many of these 30 products, if any, actually hit their mark. But keep in mind that P&G has created a business unit whose sole purpose is to create and innovate, and has invested not only in people but also a process that is built to adapt and embrace new and altered streams
of business.

As we consider the risks and rewards of innovation, it’s important to keep one’s perspective, because clearly the odds are not in our favor. The average retailer stocks some 45,000 products, but only about 150 of those make it into the average household. That’s pretty sobering. The problem, for most companies, is that their business model and internal capabilities are premised on the production and distribution of goods. It’s all about efficiency.

When you go down that road, you may be developing products that the consumer needs, but the primary focus is really on maximizing the profit of the corporation based on selling more stuff. Consequently, products tend to be standardized so they fit neatly into an inventory that is tapped, with products delivered as marketing creates demand. That’s the old model of selling a product, and it has little or nothing to do with innovation. The new model of innovation instead starts with the consumer, and a shift from a product to an experience perspective.

Make Friends with your Consumers

Getting the brand experience right means thinking less in terms of the new and improved features and benefits of a given product, and more about the context of how people live their lives and where the product fits in. As Procter & Gamble and others have framed it: Whatis the product’s purpose? How does it serve consumer needs and improve their lives? The goal is not new and improved products; it’s new and improved lives.

This must be based on deep insight into what the brand experience actually means to the consumer, created on their terms — when and where they want to have the experience. It requires a whole new outlook on, and relationship with, one’s consumers.

Marketing remains a business locked into outdated and sometimes stereotypical, demographics-driven assumptions. We obsess over understanding Moms, Millennials and Multiculturals. Such understanding is useful, but it’s even more important to understand these marketing terms as fellow human beings. That guy in aisle six is more than just a user of razor blades; he’s our friend (or at least that’s how we should think of him). We certainly shouldn’t assume that his wife, sister or mother is doing all of the shopping and buyinghis blades for him. We need to understand what makes him tick as much as what makes him buy.

Our ability to make more meaningful connections — make friends — with our consumers has never been more potent, with the rise of social media and the powerful insights into daily life they provide. We can hear the consumer’s voice and feel their wants and needs as never before. Our ability to translate that understanding into meaningful brand experiences is essential to our capacity to innovate. It’s not really all that complicated. Just fill in the blank: My best friend’s life would be better if …

Hire Unreasonable People

The greatest limitation on our capacity to innovate is reasonable people. I don’t think anybody has put this better than George Bernard Shaw, who said: “The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man.”

In 1989, George Handy published a classic book, The Age of Unreason, inspired by Shaw’s quote. His essential thesis was that, given the rapid changes in the world, and the attendant disruption and discontinuity, it is now incumbent on individuals and society to be unreasonable to succeed in an unreasonable world. He wrote that “discontinuous change requires discontinuous upside-down thinking to deal with it, even if both thinkers and thoughts appear absurd at first sight.” To achieve this, he envisioned “villages of like-minded individuals, bound by a common purpose and managed by reciprocal trust.”

Given that it was written more than 20 years ago (and about eight years before Apple adopted “Think Different” as its advertising slogan), the book was remarkably prescient, and remains a valuable guide for the kind of culture that unleashes its people to not just tolerate risk and accept failure in the name of innovation, but rather inhale it as a way of life. Achieving this means hiring the right people and putting them in the right places, with the right incentives and motivations. It means providing the capabilities and core competencies to support those people because it can’t just be people; it has to be capabilities as well.

Doing so requires the right kind of leadership — the kind that inspires fun, celebrates success and takes the fear out of the organization. Otherwise, all you’ve got is a lot of happy talk about innovation that’s doomed to frustration and failure. Some people are born more unreasonable than others, and see a failure to take risks as failure, period. You can, and should, hire those people.

Start Spreading the News

Once you have an unreasonable organization and a sustainable business model that has moved from a product to an experience orientation, you need to develop a simple marketing program that drives equity and delivers brand promises, which is the only thing that gets you to profitable growth. You need to stay in touch with your consumer.

This begins with a deep understanding of what it is they are seeking from an experiential standpoint. It is about both the medium and the message. When Disney re-launched its stores in 2010, its president, Jim Fielding, stated his goal simply: “to create the best 30 minutes of a child’s day.” The entire Disney Store experience was designed from a kid’s perspective.

What’s interesting is that this message is timeless, but it is also built for change; fulfilling it requires continuous innovation. A similar principle supports BMW’s innovation mantra: “The Ultimate Driving Machine.” That’s a promise that only relentless innovation, and news about it, can keep.

Nintendo has also done a great job refreshing its brand experience, which is all the more impressive given the company’s limitations. It doesn’t have the resources to compete with its much-larger rivals who invest heavily in game consoles with ever-faster processing power and higher-resolution graphics.

So, Nintendo journeyed in an entirely different direction and in 2006 introduced the Wii, which re-defined “interactive” in the most literal sense by integrating the user’s physical movements into the games. It was a phenomenal success that opened up new markets with women and senior citizens. When,inevitably, it became “old news,” Nintendo scored again in the 2012 holiday season with Wii U, a gaming console that, among other things, doubles as a television remote control. It sold out almost immediately.

The master of news-management is, of course, Apple. The breathless anticipation with which each new product introduction is met is the stuff of legend. Apple manages this feat almost counter-intuitively; by keeping a tight lid on news about future releases, it feeds a frenzy of stories from the mainstream media as well as online chatter among its many fans speculating on what’s coming next from the wizards of Cupertino. The rest takes care of itself.

Make Magic with the Numbers

When innovation is cast within the context of making friends, embracing the unreasonable and staying in touch with “new news,” it changes the way you measure results. You move away from the traditional, linear metrics of sales and margins on products sold. That’s still important, of course, but you want to integrate that kind of hard data with results that factor in your consumers in total, and the extent to which they are engaged with your brand experience.

Once you’re measuring results by viewing the consumer in totality, you are also evaluating your success based not only on their purchase behavior but also how you have made their lives better. You need to answer the hardest question, and that question is why?

Why is this brand experience important to the consumer? Why is it not important? It’s not just about measuring the what and the rational factors, but also why the innovation is a big idea, or why it is not.

Measuring your success also means measuring all of your partners — whether they are research companies or agencies — as strategic resources that help you view the consumer by developing better insights.

The beauty is that this type of measurement feeds right back into the innovation loop. By incorporating how well you are listening to, connecting with, and delivering purposeful brand experiences, you have the information you need to refine, update — or upend, if necessary — and keep innovation running at full speed. There’s magic in the numbers, if you’re measuring the right things.

Without question, there’s a certain alchemy to innovation, and no fool-proof recipe. The risk of failure is the most daunting part of the equation, which is rife with intangibles. The most important determining factor is our own view of, and attitude toward, the dangers we face as we push forward.

I’m reminded of that great family of daredevil acrobats, The Flying Wallendas, who are famous for walking the tightrope without safety nets. Nik Wallenda recently carried on this tradition with a breathtaking tightrope walk across Niagara Falls. He made it, safe and sound.

His father, Karl Wallenda, was not so lucky. In 1978, at age 73, in 30 mph winds, he attempted to walk between towers at the Condado Plaza Hotel, in San Juan, Puerto Rico, and fell 10 stories to his death. Maybe it was his age. Maybe it was the winds. It may have been a faulty guide rope. At the time, his wife reportedly suggested it may have been something else, that perhaps her husband had made the fatal mistake of thinking about falling instead of thinking about staying on the wire.

It’s like that with innovation. It’s not for the faint of heart. It’s for those who derive inspiration from risk. It’s for those who think about walking the wire, not falling off.

SHARON LOVE is chief executive officer of TPN, a brand-centric retail marketing agency with clients including 7-Eleven, Bank of America, The Clorox Company, PepsiCo and The Hershey Company. She can be reached at sharon_love-@-tpninc.com.