Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

We ’ ve built on Eric ’ s model, because we believe that companies naturally go through five stages of growth. First, the need to get inside their customers ’ heads Then they need to make sure a small group of users keeps using the product Then they need to make sure that word can spread Then they need to feed some of their revenues back into customer acquisition, accelerated by the virality they ’ ve built. Then they need to scale.

THis is a simple test for whether a metric is real or bullshit.

Mike Greenfield and his co-founders started Circle of Friends in September 2007, shortly after Facebook launched their developer platform. The timing was perfect: Facebook became an open, viral place to acquire users as quickly as possible and build a startup. There had never been a platform with so many users (Facebook had about 50 million users at the time) that was so open to reaching them.Circle of Friends was a simple idea: a Facebook application that allowed you to organize your friends into circles for targeted content sharing. Mike notes now that it was basically, “ Google+ for Facebook ” (before Google+ existed.)

By mid-2008, Circle of Friends had 10 million users. Greenfield focused on growth above everything else. “ It was a land grab, ” he said. And Circle of Friends was clearly viral. But there was a problem. Too few people were actually using the product.According to Mike, less than 20% of circles had any activity whatsoever after their initial creation. “ We had a few million monthly uniques from those 10 million users, but as a general social network we knew that wasn ’ t good enough and monetization would likely be poor. ” So Mike went digging.

SegmentationA segment is simply a group that shares some common characteristic. It might be users that run Firefox; or restaurant patrons who make a reservation rather than walking in; or passengers who buy a first-class ticket; or parents who drive a minivan.On websites, we can often segment visitors according to a range of technical and demographic information. Then, by comparing one segment to another, we can learn things. If, for example, visitors using the Firefox browser have significantly fewer purchases, we may want to perform additional testing to find out why. If we see that a disproportionate number of buyers are coming from Australia, we might survey them to discover why, then try to replicate that success in other markets.Segmentation works for any industry and any form of marketing, not just for websites. As we ’ ll see, segmenting mailing lists is an essential part of improving e-mail performance; but direct mail marketers have been segmenting for decades with great success. Even if you ’ re selling to enterprise customers, dividing your messages by target audience, industry, or stage in the sales cycle—and then testing the results of those messages—is crucial.Cohort analysisA second kind of test is cohort analysis. As you build and test your product, you ’ ll be iterating regularly. Users that join you in the first week will have a different experience from those that join later on. For example, all of your users might go through an initial free trial, usage, payment, and abandonment cycle. As this happens, you ’ ll make changes to your business model. As a result, the users who experienced the trial in month one may have a significantly different onboarding experience from those who experienced it in month five. How did that affect their churn? To find this out, we use cohort analysis.Each group of users is a cohort—participants in an experiment across their lifecycle. You can compare cohorts against one another to see if, on the whole, key metrics are getting better over time.

Using cohorts is really, really important.

Using cohorts is really, really important.

This is a fundamental problem with evolutionary optimization. To illustrate, let me tell you why you ’ ll always have a blind spot. There ’ s a place in your eye where you can ’ t see things. The other eye compensates, and your brain fills in the rest, but you really can ’ t see part of what you look at.

The eye has evolved in nature in many, many different forms, often independently. Our eyes and those of cephalopods like the octopus look similar, but have very different histories. The human eye is “ backwards ” —the infrastructure is actually in front of the retina. That means the optic nerve runs in front of the part that catches the light. This is because the human eye evolved as a growth of the brain. On the other hand, the octopus eye has the optic nerve behind the retina, because it evolved as a dent in the head. Which means octopuses don ’ t have a blind spot. But you ’ re not going to evolve a better eye, because the intermediate mutations would be worse.

Imagine I gave an algorithm three wheels and asked it to optimize their configuration. It would arrive at some form of tricycle. But it wouldn ’ t go, “ hey, can I get a fourth wheel? ” That would be what a human would do. That ’ s a leap of faith.

If you work in a company of any significant size, you owe your org chart to an enterprising General Superintendent of the railroad era named Daniel C. McCallum. In the 1850s, railroads were a booming business. Unfortunately for investors, they didn ’ t scale well. Small railroads turned a profit; big ones didn ’ t.McCallum noticed this, and divided his railroad into smaller sections, each run by subordinates who reported back a standard set of information he defined. McCallum ’ s line—as well as other lines that copied this approach—thrived. McCallum ’ s model, inspired by his time as a soldier and the regimented hierarchies he had learned there, was then applied to other fields.McCallum was the first management scientist, introducing controls, structure, and regulations in order to reduce risk and increase predictability at scale. Companies like Google and Apple know this, creating their own advanced research groups such as the Google X Lab.

As World War II exploded across Europe, the United States realized it needed a way to counteract German advances in aviation—specifically, jet aircraft. The US military asked Lockheed Martin (then the Lockheed Aircraft Corporation) to build a jet fighter. Desperate times called for desperate measures: in a month, they had a proposal; less than six months later, working in a closely guarded circus tent, the first plane was built.This group became known as the Skunk Works, a title that ’ s synonymous with an independent, autonomous group charged with innovation inside a bigger, slower-moving organization. These groups are often immune to the restrictions and budget oversight that guides the rest of the company, and have the specific goal of working “ out of the box ” to mitigate the inertia of large companies.

Beforehand: get buy-inBefore you start doing customer development, you need executive buy-in. This may be implicit—if it ’ s your job to try and find new opportunities. But once you think you ’ ve found an opportunity, you need explicit approval from an executive. You want to know where you are on the BCG box, and where you ’ re trying to go. And you need to know what metrics your progress will be judged by. You need to know what resources you have, and what rules apply to you. This is like a pre-nup: better signed before the wedding.At this stage, you ’ re defining your analytical strategy, and the “ lines in the sand ” against which you ’ ll be judged. These may be goals for the whole company, such as margins; or they may be a growth rate that ’ s considered success. You ’ ll also need to define how you will adjust these metrics based on what you learn.Empathy: find problems, don ’ t test demandOnce you start doing customer development, remember that you ’ re testing problems and solutions—not existing demand. If you ’ re truly disruptive, customers won ’ t tell you what they want. But they will tell you why they want it. In 2008, Swiffer creator Gianfranco Zaccai explained, “ Successful business innovation isn ’ t about giving consumers what they need now, but about giving them something they ’ ll desire in the future. ” Customers weren ’ t telling Netflix they wanted to stream videos, but their patterns of usage, computer adoption, broadband deployment, and browsing told the company a need existed.This is a place for qualitative interviews. You should talk to existing users and customers, of course. But if you ’ re trying to grow market share, you ’ ll also want to talk to your competitors ’ customers, to distributors, and to everyone involved in purchasing the product. If you ’ re trying to improve growth rate, you ’ ll talk to adjacent customers. That ’ s what Bombardier did when it expanded from its cash cow snowmobile business to its rapidly growing jet-ski business.Skip the business case, do the analyticsAt some point, when it comes time to go beyond interviewing people, you ’ ll need to build a business case. Traditional product managers build profit-and-loss analyses to try and justify their plans: they create a convincing business case, and once someone believes it, they get funding to proceed. But a Lean mindset reverses this: you sell the business model, without a lot of prediction, and then rely heavily on analytics to decide whether to kill the product or double-down on it.This analyze-after rather than predict-before model is possible because many of the costs of innovation can be pushed later in the product development cycle. Just-in-time manufacturing, on-demand printing, services that replace upfront investment with pay-by-the-drink capacity, CAD/CAM design, and mercenary contractors all mean that you don ’ t have to invest heavily up front (and therefore doesn ’ t have to argue a business case at the outset.) Rather, you can ask for a modest budget, build analytics into the product, and launch sooner for less money. You can then use the data and customer feedback you get, which is vanishingly cheap to collect given today ’ s technology, to plead your case based on actual evidence.Stickiness: know your real minimumIf you ’ ve identified a problem worth solving and a solution that customers will want, it ’ s time to make an MVP. But you need to know what your real minimum is. As a big organization, you may have restrictions on data sharing, reliability, or compliance to which smaller organizations (who have less to lose) aren ’ t subject. You also need to know what your unfair advantages are.Consider, for example, the many meal pre-ordering tools on the market today. These mobile applications let you place an order from a food court restaurant, pay, and pick up at an agreed-upon time without waiting. The restaurants like it because they save precious time in the lunchtime rush, and the diners like it because it ’ s simple and they can browse the menu at their leisure. It ’ s like Uber for lunch.Now consider what would happen if McDonalds were to decide to compete by introducing an application. They might have franchise constraints, or regulations for restaurants located in airports, or state laws about disclosing caloric content. All of these would have to be part of the MVP.Offsetting this, however, is the huge amount of market control the company has. It could promote the app, and give away three hamburgers for free to everyone who installed it. The company would make back the money quickly in saved time at the cash register, and have access to a new marketing channel and untapped analytical insight into its customers.Intrapreneurs need to factor these kinds of constraints and advantages into their MVP far more than independent startups do.What ’ s more, as people start to use your MVP, you have to manage the beta process carefully. You may be interfering with existing deals in the sales pipeline, or creating more work for customer support. If so, you need to have approval for the roll-out and the buy-in of stakeholders. If you ’ re launching an entirely new product line, you may even have to camouflage it so you don ’ t cannibalize existing markets until you know it ’ s successful. This, of course, undermines your ability to use unfair advantages like an existing customer base.Viral from the startIf you ’ re trying to move upwards in the BCG box, your product should include viral and word-of-mouth elements. In a world where everyone has access to a mobile device, every product needs to have an interactive strategy. There ’ s simply no excuse not to find a viral angle to act as a force multiplier for growth. In fact, adding a viral component is one of the keys to moving dogs and cash cows up into question marks and stars.Revenue within the ecosystemYou ’ ll have less flexibility to set pricing and reinvest revenues in product marketing, because as you grow you ’ ll have to coexist with other marketing efforts by your host company. When Microsoft wanted to test its SaaS-based office suite, it could do so in a relatively controlled way. But as soon as it wanted to monetize the product, it had to contend with cannibalization and push-back from a channel that depended on license revenue.Your pricing may have to take into account channels, distributors, and other factors that restrict your freedom to experiment, because changes you make will have an impact on other products in the marketplace. Had Blockbuster entered the streaming video market, it would have had to deal with labor and real estate issues at existing stores.Scale and the handoffIn the final stages of intrapreneur innovation, the new product has proven its viability. It ’ s either stolen by settlers—who can help it cross the chasm and broaden its appeal—or the team that created it must transition to a more traditional, structured model of business and take its place among the other products and divisions of the host organization.Most of the time, the DNA of a disruptive organization isn ’ t well suited to “ boring ” management and growth, so you ’ ll need to hand the product off to the rest of the organization and find the next thing to disrupt. That means you really have two customers: the external one buying the product, and the internal one that has to make, sell, and support it.Ultimately, the intrapreneur must manage the relationship with the host organization as well as the relationship with the target market. Initially, this can be intentionally distant, but as the disruptive product becomes part of the host, the hand-off must be graceful.

Why might you not want to disrupt things? To understand this, you need to look at how large organizations plan their product and market strategy.The Boston Consulting Group (BCG) box, shown in Figure 54, is a simple way to think about a company ’ s product portfolio. This venerated model classifies products or subsidiaries according to two dimensions: how quickly the market is growing, and how big a market share the company has in that market.Products with high market share, but slow growth, are “ cash cows. ” They generate revenue, but they aren ’ t worthy of heavy investment. By contrast, products with high growth but small market share are “ question marks, ” candidates for investment and development. Those with both growth and market share are the rising “ stars ” . Those with neither—called “ dogs ” —are to be sold off or shut down.The BCG box offers a thumbnail of a company ’ s product portfolio. It ’ s also a good way to think about innovation. If you ’ re trying to change a company, you ’ re either trying to create a new product (hopefully in a growing market) or you ’ re trying to innovate to revitalize an existing product with the addition of new features, markets, or services.

The first thing you ’ ll notice is the title: the Goal is to Learn. This is important, because it reminded the entrepreneurs about what they were setting out to do. It wasn ’ t about building “ stuff. ” It wasn ’ t about adding features. It wasn ’ t about getting PR, or anything else. Learning was the measure of success.Next, they would fill in a brief update on their current status, focusing on the key metrics (qualitative and/or quantitative) that they were tracking. Notice how small this box is compared to the others.The Lessons Learned box is a quick bulleted summary of key learning. The title says “ and accomplishments ” because we wanted to give entrepreneurs a place to brag … at least a little bit. Not surprisingly, they ’ d include some vanity metrics in here and we wouldn ’ t focus a lot of time on them. The “ on track: yes/no ” benchmark is designed as a test of intellectual honesty. Can entrepreneurs really come clean on what ’ s going on, good and bad? If so, we could be much more valuable.Finally, we asked entrepreneurs to list the top problems they were facing at that moment. At most we ’ d include three problems prioritized in order of importance. This section of the Problem-Solution Canvas often elicited the most debate; but it was always healthy and critical for resetting everyone ’ s goals and expectations.

In this section, the founders re-list the problems, and include hypothesized solutions. These solutions are hypothesized because we don ’ t know if they ’ ll work. These are experiments that the founders will run in the next week. We always asked them to define the metrics they ’ d use to measure success (or failure) and draw a line in the sand. If engagement was the most important problem, they had to include possible solutions they ’ d experiment with to increase engagement, define the metric (example: % daily active users) and set a target. What ’ s the problem, how do you propose to fix it, and how will you know if you succeeded? That ’ s the core of the Problem-Solution Canvas.For us (as mentors and advisors) it was an extremely valuable exercise. The Problem-Solution Canvas is also useful for internal decision-making. It ’ s a level below Lean Canvas, focusing on very specific details in a very specific time period (1-2 weeks.)

Here ’ s an example of finding a cause. SOME years ago, executives at a Houston airport faced a troubling customer-relations issue. Passengers were lodging an inordinate number of complaints about the long waits at baggage claim. In response, the executives increased the number of baggage handlers working that shift. The plan worked: the average wait fell to eight minutes, well within industry benchmarks. But the complaints persisted.Puzzled, the airport executives undertook a more careful, on-site analysis. They found that it took passengers a minute to walk from their arrival gates to baggage claim and seven more minutes to get their bags. Roughly 88 percent of their time, in other words, was spent standing around waiting for their bags.So the airport decided on a new approach: instead of reducing wait times, it moved the arrival gates away from the main terminal and routed bags to the outermost carousel. Passengers now had to walk six times longer to get their bags. Complaints dropped to near zero. Turns out that long delivery wasn ’ t causing dissatisfaction. Time spent waiting was.

11.
Identify a key business problem,
pick the OMTM, draw a line in the
sand, and get started.

12.
Draw a new line
Pivot or
give up
Try again
Success!
Did we move the
needle?
Measure
the results
Make changes
in production
Design a test
Hypothesis
With data:
find a
commonality
Without data:
make a good
guess
Find a potential
improvement
Draw a linePick a KPI
The Lean Analytics Cycle

14.
Professional photography helps Airbnb’s business
Gut instinct
Concierge MVP
20 photographers in the field
Test results
Two to three times more bookings!
Back to the beginning
Use additional data to keep experimenting

18.
Draw a new line
Pivot or
give up
Try again
Success!
Did we move the
needle?
Measure
the results
Make changes
in production
Design a test
Hypothesis
With data:
find a
commonality
Without data:
make a good
guess
Find a potential
improvement
Draw a linePick a KPI
Remember this?

19.
“Gee, those houses
that do well look
really nice.”
Maybe it’s the
camera.
“Computer: What do
all the highly rented
houses have in
common?”
Camera model.
With data:
find a
commonality
Without data:
make a good
guess

20.
The only thing worse than bad
feedback is no feedback at all.
- Dave McClure, Startupfest 2012

22.
Virality stage:
Circle of Moms finds an
engaged market
• Stage: Stickiness
• Model: UGC
• Launched as Circle of Friends in 2007, it was a way for small groups to
interact atop Facebook’s platform; but when engagement wasn’t good
enough, the founders decided to dig deeper.

23.
The problem: Not enough engagement
• Too few people were actually using the product
• Less than 20% of any circles had any activity after their initial creation
• A few million monthly uniques from 10M registered users, but no sustained
traction

25.
Segments, cohorts, A/B, and multivariates
Segment:
Cross-sectional
comparison of all
people divided by
some attribute
(age, gender, etc.)
☀
☁
Cohort:
Comparison of
similar groups
along a timeline.
A/B test:
Changing one
thing (i.e. color)
and measuring
the result (i.e.
revenue.)
Multivariate
analysis:
Changing several
things at once to see
which correlates with
a result.
☀
☁
☀
☁
JanJan
FebFeb
MarMar
AprApr
MayMay

26.
Why use cohorts? Here’s an example.
January February March April May
Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50
Is this
company
growing or
stagnating?
Cohort 1 2 3 4 5
How about
this one?
January $5 $3 $2 $1 $0.5
February $6 $4 $2 $1
March $7 $6 $5
April $8 $7
May $9

27.
Two views of the same company
Cohort 1 2 3 4 5
January $5 $3 $2 $1 $0.5
February $6 $4 $2 $1
March $7 $6 $5
April $8 $7
May $9
Averages $7 $5 $3 $1 $0.5
Look at the
same data
in cohorts

29.
The problem with local maxima
(http://www.slideshare.net/bokardo/metricsdriven-design-4317168)

30.
The vertebrate eye is “backwards
and upside down”. Light travels
through cornea, lens, aqueous
fluid, blood vessels, ganglion
cells, amacrine cells, horizontal
cells, and bipolar cells.
The cephalopod eye is
constructed the “right way
out”, with the nerves attached
to the rear of the retina. No
blind spot.
Extension of the brain Invagination of the head

39.
Since large organizations
make data that
perpetuates things, you
have to willfully ignore
common wisdom at the
outset.
http://www.flickr.com/photos/jb912/8940173789

40.
Disruption may not fit neatly into
an existing part of the
organization, so you need to
incubate it internally for a while.

41.
Unlike a VC or startup,
when the initiative fails the
organization still learns.

42.
Big organizations might not
innovate at speed—but they
innovate at scale.

43.
The Lean Analytics lifecycle for an Intrapreneur
Empathy
Find problems; don’t test demand.
Skip the business case, do
analytics
Entitled, aggrieved
customers
Stickiness
Know your real minimum based on
expectations, regulations
Hidden “must haves”,
feature creep
Virality
Build inherent virality in from the
start; attention is the new currency
Luddites who don’t
understand sharing
Revenue
Consider the ecosystem, channels,
and established agreements
Channel conflict,
resistance, contracts
Scale
Hand the baton to others gracefully Hating what happens
to your baby
Stage Do this Fear this
Beforehand
Get buy-in Political fallout

44.
A large organization needs a
portfolio of investments,
judged by different criteria.
Treasury
bonds
Venture
capital

45.
A large organization needs a
portfolio of investments,
judged by different criteria.
Safe
Predictable
Evolutionary
Achieve the
business plan
Risky
Uncertain
Revolutionary
Discover the
business model
Times of
peace
Times of
war

49.
Build analytics into how you
operate the business
Three threes and the Problem-Solution Canvas

50.
Three threes
Three
assumptions
What big bets are you making?
•“People will answer questions”
•“Organizers are frustrated with how to run conferences”
•“We’ll make money from parents”
•“Amazon is reliable enough for our users.”
Three actions
to take
What are you doing to make these assumptions happen
(or identify they’re wrong and change course?)
•Product enhancements
•Marketing strategies
Three experiments
to run
What are you going to learn today?
•Feature tests
•Continuous deployment
•A/B testing
•Customer survey

51.
Three threes
Monthly
Weekly
Daily
Board, investors,
founders
Executive team
Employees
Strategy
Tactics
Execution
Three
assumptions
Three actions
to take
Three experiments
to run

52.
Three threes
Get more
people
Increase
answer %
Test better
questions
Change
the UI
Test
timings
Questions
from peers
Many people will
answer questions
Three
assumptions
Three actions
to take
Three experiments
to run