Thursday, March 28, 2013

In Web Framework Benchmarks, there are some very interesting and surprising numbers around the performance of various web frameworks. Startup Founders really need to see these numbers. And I hope you are not running on Cake PHP when you see these numbers.

Many of us are putting more into the front-end and having the application logic and back-end exposed through JavaScript (JSON) APIs. In some ways, this frees us from worrying as much about the specific framework that's being used. I've found myself looking mostly at talent and time to market. But these numbers are causing me to pause a bit and really think about the choice of framework in terms of performance.

In looking at these numbers, seeing Cake PHP at 500x slower, Ruby-Rails and Django at 50x slower really surprised me.

I was also surprised by the performance improvement on dedicated hardware as compared to EC2 instances of roughly 10x.

Important Implications

Well I'm currently working with startup founders on their systems in JRuby, Django, PHP and Java.

Several of these are B2B applications with relatively smaller audiences. I'm feeling okay about our choices of frameworks that are slower and will cost more in terms of hosting and managing growth. The availability of talent was an important factor.

need to consider going with a higher performance solution. Most startups do not get a chance to move from one framework to another. It takes a lot of time and effort and the result is that you get to go through a whole new set of bugs only to get back to where you started but with a faster, more scalable application. Think about twitter - but they had lots of money.

Often we justify building an MVP in whatever framework is fastest to build or where we have resources that know that framework. You may get into the market, but just know that you are going to pay the price when you start to get traction.

In most cases, when you are building your MVP, you are trying to prove out certain startup metrics such as:

Cost of Customer Acquisition

Conversion Rates / Pricing

Viral Coefficient

in order to get those numbers in front of investors so that you have evidence of traction and can show that you can begin to build out more of the real product and begin to scale the business.

It is almost never the case that you are building an MVP to "show" to an investor the product itself. Yes, the investor may literally have said to you:

"That is something I'd seriously think about investing in when you have your product built."

But the reality is that they don't mean that. Two aspects to this:

You can let an investor see your product via a mockup or clickable prototype.

If you do build the MVP and show it to them, they will ask you about your metrics. They really want metrics, not a product.

A Mockup is Enough to Show the Product

Most investors can look at a mockup or clickable prototype and have a pretty good sense of the product. They may wonder if it can be built technically, but I (or other CTOs) can answer that question without building any code.

Cases Where a Mockup is Not Enough

There are a few cases where mockups or clickable prototypes may not be enough:

Usability, interaction design, etc. For example, the iPod won not because of better features and functions. It won because of interface, ease of use. To get an investor excited about another MP3 player at the time, they would have needed to play with the interface. That said, you likely could have still come up with some cheaper way than building the iPod.

Results of Algorithms. Often you can't tell if something like a search engine or matching algorithm is really going to have better results until you use it. You may have to build something out to show it working for someone to evaluate whether it really works better than what else exists.

I would guess that this represents less than 5% of startups.

Investors Really Want Evidence of Traction

So you built your MVP; you bring it to the investor; you demo it; and I will guarantee they will ask you:

"So how many users do you have?

How much is it costing you to get users?

How much are you making from your users?"

And other similar questions. Yes, they are happy that you have your product built and that does make it much more investable. But now that you have a product, you should be able to show that people want to and/or are willing to pay to use it.

Bad News? Not Really

At first you may be thinking that this is all bad news. Wow, now Tony is telling me that not only do I need to build my MVP, but I need to actually show evidence of traction. That's an even tougher hurdle. Yes, that's correct. Sorry, but that's the reality.

However, the good news is that for most investors, you can certainly change the question and get a lot more information without ever building the MVP. The real question you should be asking is "When I've built this product and show you the following metrics, would you invest?" The "this product" will be a well formulated mock-up or clickable prototype. If you don't have that, then you will naturally let the investor off the hook by saying, show me the product.

I actually think you can push the conversation pretty far with most investors and a few good mockups. No, you won't know if you will really get a check - most investors are hard to actually get a check from - but you will have a pretty good indication.

Additional Sources

But never forget that traction is necessary, but may not be sufficient, to lower the risk perception of investors, and assure an investment. The quality of the team, and overall financial health are equally important, as well as how your offering compares to competitors.

You can't raise money on achieving an MVP. Investors demand more than that.

As Steve Blank likes to say:

A Startup Is a Temporary Organization Designed to Search for A Repeatable and Scalable Business Model

The unfortunate reality is - an MVP is not the above! Yet most of the newly minted entrepreneurs I've met think their job is nearly done when they've found MVP - they think they can go build a pitch off their early MVP and raise money!

A startup does require MVP but it is much more than just MVP. The problem is that MVP means early adoption of product and its features, maybe even some who will pay. But it doesn't tell you how many people will do it in the long term and whether this can support the company (the people and operations within) that is behind it.

About Me

Dr. Tony Karrer works as a part-time CTO for startups and midsize software companies - helping them get product out the door and turn around technology issues. He is considered one of the top technologists in eLearning and is known for working with numerous startups including being the original CTO for eHarmony for its first four years. Dr. Karrer taught Computer Science for eleven years. He has also worked on projects for many Fortune 500 companies including Credit
Suisse, Royal Bank of Canada, Citibank, Lexus, Microsoft, Nissan,
Universal, IBM, Hewlett-Packard, Sun Microsystems, Fidelity
Investments, Symbol Technologies and SHL Systemhouse. Dr. Karrer was
valedictorian at Loyola Marymount University, attended the University
of Southern California as a Tau Beta Pi fellow, one of the top 30
engineers in the nation, and received a M.S. and Ph.D. in Computer
Science. He is a frequent speaker at industry and academic events.