The Lean Startup Conference happened today in San Francisco. Here in Asheville, we livestreamed the conference at Mojo Coworking. Below are my notes from some of the talks, links to other people’s notes and links to other Lean Startup resources. I’ll be expanding this post throughout the week, so check back later in the week or follow me on Twitter for updates.

If you have other notes that you’d like to link from here, tweet them to me at @FastFedora.

My Conference Notes

Todd spoke about the way the U.S. Federal government has been entrepreneurial, including an overview of the Presidential Innovation Fellows program, which has five projects:

MyGov (@ProjectMyGov)
Reimagining the relationship between the federal government and its citizens through an online footprint developed not just for the people, but also by the people.

Open Data Initiatives (@ProjectOpenData)
Stimulating a rising tide of innovation and entrepreneurship that utilizes government data to create tools that help Americans in numerous ways – e.g., apps and services that help people find the right health care provider, identify the college that provides the best value for their money, save money on electricity bills through smarter shopping, or keep their families safe by knowing which products have been recalled.

Blue Button for America (@ProjectBlueBtn)
Developing apps and create awareness of tools that help individuals get access to their personal health records — current medications and drug allergies, claims and treatment data, and lab reports – that can improve their health and healthcare.

RFP-EZ (@ProjectRFPEZ)
Building a platform that makes it easier for small high-growth businesses to navigate the federal government, and enables agencies to quickly source low-cost, high-impact information technology solutions.

The 20% Initiative (@ProjectTwenty)
Creating a system that enables US government programs to seamlessly move from making cash payments to support foreign policy, development assistance, government operations or commercial activities to using electronic payments such as mobile devices, smart cards and other methods.

Tereza tried launching a company and failed. She discussed some of her mistakes, including:

Accepting a Vanity Metric in Term Sheet
VCs want some way to track you. Be careful. Tereza accepted a term sheet that included specific traffic metrics, when her business didn’t depend on traffic. Eric once accepted a term sheet that included how many press releases they would send out.

Preparing Metrics Only for Presentations
Creating reports of metrics only when preparing for a board meeting or presentation, instead of reviewing them regularly and using them to run the company.

Being Afraid to Pivot
It’s hard to tell your investors you need to pivot. It can cause a loss of trust. Investors have a third option beside pivoting or persevering. They can go and focus on other companies in their portfolio.

Used University Campus to Test
Just got owner’s to hand off their keys to a stranger.

Built an iPhone App
Built their prototype at Yahoo iPhoneDevCamp in a weekend.

Hacked the Hardware on a Car
Used off-the-shelf parts to show you could unlock a car with a cell phone.

Tested in a Larger Market
Entered a competition that gave them access to a larger market. Won the competition and had 1,600 car owners sign up in 24 hours–20% of the inventory Zipcar took 10 years to build. Now at 10,000 cars.

Take the romance of a motorcycle and combine it with the comfort of a car.

80% of financing went into proving there was a market.

De-risked market by building a test dealer showroom. When bringing potential customers into the showroom, 52% went on a test drive and 15.7% pre-ordered. The pre-order conversion rate was far greater than they needed to make the business work.

Built a prototype, which helped them determine the weight and de-risked the hardware components of the vehicle.

Worked with a company called Innovation Games that decided to build a consumer pass & play game, Knowsy, to learn how to build an mobile platform. Decided to get into the iPhone app store in 1 month when they hadn’t build an iPhone app before.

They had mockups of the screens. Then asked developer to build out his idea of the flow by sketching it out. That helped build a shared understanding.

Then Lana made a mistake. She spent the weekend making wireframes based on a scenario and making it all clean and annotated. Worked well, but then she became the bottleneck.

To solve this, she did a paper prototype. They played the game using sticky notes on paper. The team played the game together, and then they understood the game a lot better.

First product was postagram: send a postcard from your phone. Started with simpliest gift you can send in the world, a postcard. Now sending full physical gifts.

Built 9 products in 12 months on mobile using a team of 6 people.

10 Tips for Rapid Development

Build a Minimum Viable Product
First they build Postcards from Santa, which allowed a parent to send a postcard from Sanata to their kid. This helped them solve the billing & printing issues. Then they launched 99 cents Prints by Mail, where people could mail themselves photos. This helped them create a library of photos and image upload. They used these products to them launch Postagram.

Off-Brand Testing
Tested product off-brand & in Canada.

De-Emphasize Visual Design
Can’t afford to do great design on everything, especially in off-brand products. But they don’t de-emphasize interaction design, since that’s where their learning comes from. To speed the process & minimize costs, they let engineers do design for experiments.

Reuse Common Components

Buy Cheap, Disposable Users
For their experiments, they don’t worry about cross promotion or PR. They buy cheap users from Stumbeupon, Google AdWords, etc. to test their ideas. This allows them to kill the product more easily without affecting their core brand.

Be Willing to Kill Apps
Once an app is no longer serving your core mission, kill it. They are killing a product this month. You will have upset customers.

Use Android for Quick Testing
Android apps have no approval process. Android updates go live within hours. Apple updates take up to 7 days. Closest thing to the web in the mobile world.

APIs Are Everything
Both internal & external. Product engineers never touch their database. They build an internal API that all their product engineers use. This has then allowed them to make internal API external so partners can use the API.

Focus on Team Structure
One team focused on the internal API. Then product teams have one developers & one designers to build apps on that API. Off-brand tests might only have a single engineer and no designer.

Meetups measure their success using RSVPs. But RSVPs are the ultimately vanity metric.

The 3 myths:

People Give a Damn About “Lean”
No one wants to buy process. You can’t just sell others in your organization on the lean process. You have to start doing it. Started with a single usability session. Now do 600 usability sessions a year for about $30,000. Shifts the conversation about how you or I would use a product to how a user would use a product. People give a damn about doing their job better.

People Want to Test Things
People want to build things, not test things. Meetup had a culture where they wanted to take big bets. Started testing easy things. Tested in the sales funnel. Tested 10 things. 9 failed, but one got a 10% lift. Then expanded a little bit into subject lines, buttons, algorithms. What would you attempt to do if you knew you couldn’t fail? What would you do if you knew you would fail? Go after the things that will cause us to fail as fast as you can.

You Can Test Your Way to a Great Experience
At Meetup, they’ve made design an important part of the development process.

Live chat is Hipmunk’s secret weapon to get feedback from their users. From day one, they integrated Olark on their web site.

Most people wouldn’t have bothered to send an e-mail about problems they had with the product. But people were willing to chat about problems that they wouldn’t e-mail about.

They learned that people liked the basic layout through chat, and when they rolled out a feature that limited the number of hotel results, they heard an outcry immediately through chat and were able to roll it back rapidly.

It’s important that people manning the chat system can actually interact with the engineers who can change the product. Sometimes they have the engineers actually sit on the chat.

A startup is not a business, it’s a hobby. A business brings in more money than you spend.

Pebble went from hobby to business before they built their product by using Kickstarter. Kickstarter tells you if people will pay at a specific price point. But what you really need is a crowd-testing site. Lockitron & App.net both raised money outside of Kickstarter.

Selfstarter is an open source platform that let’s you run A-B tests on your price points by running fundraising campaigns on your own site, allowing you to test price points before you build your product.

They blind you to a lack of product/market fit, commits you to your original vision without agility and gets you invested in appearance, not reality. If you’re looking through your metrics looking for one that makes you look good, then you’re searching for a vanity metric.

Vanity metrics are success theater.

Vanity metrics attract the wrong type of people. They attract the type of people who want to be with a company that looks successful, not help a company be successful.

Metrics you should not be using:

Pageviews

New members

Total members

Unique visitors/visit

Percent growth

Conversion rate

Twitter followers

Facebook friends/likes

Characteristics of actionable metrics:

Messures success at your core business

Shows or directly relates to revenue

Tracks real, individual customers

Tracks cause & effect

Tells you what to do next

Can you replicate or expand these numbers? If you don’t know how you got that metric or how to do it again, it’s not a good metric.

Four most important metrics:

Revenue
Must be scalable, repeatable revenue.

Sales Volume
Are you making enough?

Customer Retention
Use funnel analysis and cohort studies.

Relevant Growth
Must capture your engine of growth. How the behavior of your customers creates new customers.

5 Whys help you do a post-mortem when things go wrong. But there are problems we have when doing 5 Whys.

One of the key problems with the 5 Whys is shame. Known in psychology as the Fundamental Attribution Error: humans dramatically underestimate the power of the environment when you’re not in a situation.

Solution: Adopt an economic, not moral mindset

Assume two factories. Both are missing their monthly goals. One factory has a broken machine. Questions this factory might ask are:

What is it costing us?

What will it cost to repair?

Can we kludge it to create a partial fix?

What are the risks if we delay the fix?

The other factory has an axe murderer who keeps killing their workers. For this factory, the questions don’t seem to make sense. How can we get the axe murderer to kill fewer people? Instead, the questions focus around:

Search for villains

Elevation of accusers

Mobilization of authority to mete out punishment

Most companies will treat failures like they are looking for axe murderers instead of trying to fix a broken machine.

Use humor to break people out of the moral mindset and bring them into the economic mindset. Some techniques to do this:

Always Share Worse “Bad Things”
Put things into perspective. Did we lose all our customers? Did the building catch on fire?

Mock Hindsight Bias to Cause an About Face
Avoid hindsight bias. If a step in the process was missed, the person might say, “I totally forgot, I won’t do this again”. Reframe it as “I was stupid once, I won’t be stupid again”. Let’s assume we’re going to continue to be stupid.

Relish Absurdities In Your System
All successful businesses have hacks someone in their process or codebase. Accept it.

“Broadest Fixes” vs “Root Causes”
Had a problem where a mistake caused a problem in the database, then their reporting system started generating 100s of e-mails a second. At which point their e-mail provider shut them down, so they didn’t discover the original problem for 3-4 hours. So there were two issues: the problem with the database and the failure of the monitoring. The lesson: There’s never a single cause. There’s always contingencies. Instead, ask the question: if we made a slight investment, which one would solve the most problems going forward. In this case, the monitoring was the clear investment.

Note: Audrey Reynolds (@helloaud) took the original notes for this talk while I took a break. I just did the re-formatting.

Scott spoke about how to do Lean Startup at scale. For the last 4 years, they’ve been converting Intuit to work on Lean Startup principles by creating innovators inside Intuit.

To do so, they’ve needed to change how decisions are made in company and change how leaders operate. Typically, large companies stop innovating and stop reinventing themselves. They make decisions based on hierarchy, politics and PowerPoints.

To fix the situation, you need to:

Move decisions from politics and PowerPoint to enable decisions to be made by the best idea that can find a market

Move decisions from a boss to customers voting with their feet

This involves changing role of leaders and making leaders apply the principles of Lean Startup to themselves as well.

Change the questions you ask employees from “What is your answer and analysis behind it?” to “What’s the fastest way to experiement and find answer?”

Leaders have 4 roles:

1. Set the Grand Challenge

For instance: “Create new businesses that improve the financial lives of Indians”.

For this, they used small 3-4 person empowered teams that choose a narrow problem to solve, eg “how do I know which market to take my produce to each day in order to gain the best price”. One idea was that since all Indians have cell phones, we could send them text messages with personalized market prices. They had product in a few days where they manually sent daily market prices via text messages. They used this to assess the reaction from customers. After 4 years they are giving 20% more farm income to farmers, have 1.2 million active users and are adding 20k farmers a week.

The team has now done 20+ major experiments. Experiments are hard to shut down because spending so little money.

2. Create the Systems & Culture

Install the systems and culture that enable people to run fast experiments, even junior people. Keep asking: how do we remove barriers? What speedbumps stand in the way?

For Intuit, they included the lawyers and built system that allows lawyers to remove barriers.

They installed 3 cultural principles:

Mindset
Shifted from a risk-adverse place of “no” to “how do we get to yes?”

Tools
Gave people tools that democratized teams. Asked how to reframe guidelines to empower teams to go out and experiment and take legal out of the way. Legal worked with teams as the guidelines developed to reduce the cost of failure for the teams.

Experiments
Made experimentation how you work. The legal guidelines themselves were experiments.

3. Learn from Experiments

Pull insights from experiments regardless of success or failure. Experiments don’t always turn out as expected. Often the market is trying to teach you something you don’t yet. Great leaders help team pull those insights.

In Lean Startup, the example is given about the Intuit mobile app for filing taxes. People love it now, but the initial concept was a document capture app for Turbo Tax Onine. When they took the concept to market, customers were amazed by document capture capabilities and kept asking “Why can’t I finish it on the phone”? Why do I have to go home and log in?”

They discovered customers relied more and more on phone, and were really excited about the idea of a start-to-finish app. The original concept had a positive reaction, but was mild by comparison.

This became their pivot point and they decided to go forward with mobile app. To test, they launched only in California for simple filers (those with no home or kids). They knew they had something when other customers complained they couldn’t file on their phone too.

4. Model the Right Behavior

As a leader, you must live by same rules and discipline yourself. Ask yourself questions like “I have a strong belief—what are the fundamental assumptions they are based on? And how can I experiment to test them?”

Lean Startup can uncork innovative juice in all of your people—young and old. It’s the only way to change people’s lives profoundly.

Q: What are the top 3 metrics you focus on at all times and what do they tell you?

Active users & engagement. But it’s not specific metrics. In the short-term, we monitor metrics for unexpected changes to see if something happened due to a recent change in the product. Otherwise, look at trends over time, such as the platform mix, which has changed over time. Learning the seasonality of the businesses.

Q: What methods or techniques do you use to prioritize new product features?

Formally, they have a quarterly & annually planning process. Informally, Drew goes down to visit his sister down at college and talks to her friends to learn what they do. Then he synthesizes the knowledge.

Q: Describe your first 25 hires.

First 10 were engineers. When they needed to hire business people, since they didn’t have experience to evaluate who were great business people, they hired consultants. This let them avoid not having a hole in their team, or hiring someone full-time and having that person suck. They were then able to learn what they needed in a business person before hiring a full-time person.

Panda Games started off 100% focused on the Android platform. Allowed them to describe themselves as “the Android guys”. No good games existed on the Android platform at that time and user adoption hadn’t hit it’s curve yet. But they could see where things were going.

Platform choices matter. Besides the tech mattering, distribution of the platforms matters and design works best when you optimize for a specific platform.

But then they pivoted. Why? They couldn’t get the monetization on the Android platform. The built a game with a social feature that let you share with your friends when you were playing the games, and the biggest question they received was: when is this coming to iPhone?

Changing platforms can be one of the most disruptive things you can do. Make sure you have the right people and plan enough time.

Vision matters. Your vision guides how you pivot and change your startup. Your vision allows you to make judgement calls.

Your vision is a constant. In fact, it is the only constant on which you can rely in a vast plain of uncertainty and doubt and hope. You need to hold true to that vision. Don’t try to pivot away from it. Hold true to it. Set a vision that others can see where you want to go. Pick a mountaintop. The path may vary, but your vision remains. Hold your vision. You may wander, but you will never be lost.

“A pivot is a change in strategy without a change in vision.” - Eric Ries

Netscape started out as a company to build the first software company for Interactive TV. Then when they went to the market & realized there wasn’t any such thing as Interactive TV, they pivoted to working on a product similar to Xbox Live but for Nintendo. Finally they settled on the Netscape browser.

If you are calling Eric Ries if you have product-market fit, you don’t have product-market fit.

Product-market fit is a feeling that things are running away from you. ”The key to being successful in life is to find a parade and jump in front of it.” – Jim Barksdale

You don’t really have a company until you have product-market fit. Then you have to start hiring, do international expansion, etc. Sales and marketing becomes more important, as you need to then go and start selling your product.

Predictors of product-market fit: one is really smart people that look like they are doing something crazy, but that have a small dedicated base of early adopters which are huge fans. 3-D printing is going through this right now. Another is when the employees within a company are enthusiastically using a product.

Three types of companies:

Founder-Market Fit
Where the founders have a strong need for the product.

Next Bench
Build for adjacent markets. The HP way. Where the voltmeters were built for oscilloscopes that used those voltmeters. Then those oscilloscopes were built to make the radio transmitters.

Building for Other People
Where the founder doesn’t have a need for the product, but others do. Need to have good product management practices.

Not every startup can be done lean. Ones that have audacious goals. For instance, the Macintosh. No one was asking for that. The Intel 8080 chip required a huge amount of upfront investment. When startups have significant technical risk, regardless of the market risk, then it’s difficult to do the startup lean.

Lean Startup methodology being used inappropriately by entrepreneurs to not take sales & marketing seriously. Hear from entrepreneurs that all they need to do is focus on product. If they build it, they will come. Entrepreneurs incorrectly extrapolate. They look at Salesforce.com and say they are only self-service, when they invested huge amounts into sales & marketing. Google has a salesforce. Or they take a consumer approach to selling business apps. Entrepreneurs think everything will be viral. The number of things that are actually viral are incredibly small.

Pivoting too early. Failures are acceptable as steps on the path to success. If a failure is a step through a search to find the right idea, that’s fine. But too often entrepreneurs give up too quickly. There’s a culture where people celebrate failure too much. The failure fetish gets old. Time to add a bit more stigma to failing. Marc admires entrepreneurs who succeed, and most importantly, those who persist through the challenges. Part of Lean Startup is helping people measure progress when they are in that long, flat part of building a business.

“Being an entrepreneur is like chewing glass, eventually you start to like the taste of your own blood.” – ?

Entrepreneurs almost uniquely have a problem with timing. They think they are too late to the game. In reality, entrepreneurs, by definition, live in the future. For example, 2012 will be the year of Software as a Service (SaaS), 13 years after Salesforce.com was formed. This year will be the tipping point where large enterprises start seriously adopting SaaS.

Do not go public until you build a fortress. Make sure you have all vectors of attack covered: technology, a full executive team, an established brand, sales & marketing fully up & running, predictability of revenue for several years into the future at least, and a strong IP position.

Last piece of advice from Marc: Lean Startup methodology is like discovering the theory of relativity. We now have a system and method and science for thinking about entrepreneurship. But Newtonian physics still matter. When you’re building a company, it’s great to be good at Lean Startup principles, but be good at the old school stuff too: sales & marketing, etc.

A key contributor to Lean Startup concepts often overlooked is Mark Leslie, the founder of Veritas and his Sales Learning Curve.

More Notes

Sketchnotes by Sacha Chau

Sacha Chau does sketchnotes of talks. Below are her notes from the Lean Startup Conference livestream. You can view her other sketchnotes on her blog. Toronto hosted a full Lean Startup Day for the livestream that included other talks. You can view all of Sacha’s sketchnotes for Lean Startup Day here.

5 pings

[...] notes via a Google doc and I also posted linked to some other summaries (and here and another here) and very complete notes (in one case, almost a transcript). The video and slides will be available [...]