Lean Software Development – build v1s and v2s

I was managing a software team and we were working several initiatives. The way projects were assigned was based on who was available, their skillset, and their developmental goals. This ended up resulting on two developers, let’s call them Mary and Melissa, being assigned to the same project.

Mary and Melissa had been working together a few weeks when I started hearing complaints in my one on ones with them about the other. Mary was complaining that Melissa was taking too long to do her part, and spending time on unit tests that didn’t make sense because things were in flux with the project. And meanwhile, Melissa was complaining that Mary wrote sloppy code and didn’t write enough tests (and even showed me comments in the code like “hack:….” and “to do: someone should add more error handling here”. Both of them had valid points and feedback for the other.

I spent the next few weeks coaching both of them on how improve and address the other’s concerns. With Mary I was focused on pushing her to write more tests and not just throw things together, and with Melissa I was focused on having her prototype something quickly and then add the polish after it was working. Both of them tried really hard, and both were miserable. It just wasn’t what they were meant to do.

These two ladies were just not meant to collaborate on projects together.

And this got me thinking: how could we change the process so both could write code the way they like to create (and that caters to their strengths) and would still allow us to meet our team objectives?

And from this was born the v1/v2 development process.

You see, there are certain types of developers that love building the first versions or prototyping – they are the ones that love hacking things together to get something working quickly. They love (and are best at) building the version 1 of a product. The other type is the one that loves building the second version. They see their code as a craft and write unit tests for everything. “Test coverage” and “beautiful code” are phrases often used in their vocabulary.

And of course this definition isn’t black and white, as there are certainly people who fall on both sides of the line, so it is more a spectrum.

Typically, the v1 type of person doesn’t like working with the v2 person, and vice versa – not because of personal reasons, but because of the way they think and create software. At the heart of it, the fundamental thing they enjoy about what they do is different.

By changing the way we thought about software, we were able to address this concern and actually make the team more agile and solve some business problems along the way.

The v1/v2 Process

When it comes to software development there are many ways to build and create products.

In my experience it is difficult to build the right product the first time. And by “right” I don’t just mean something customers will want, use, and hopefully generate revenue; but also the “right” technical solution. Often times the way customers use products can be hard to predict. For example, before you ship the first version of a product it can be difficult to answer questions like:

How quickly will the data grow?

How fast will writes need to be? Will it be feasible to write directly to the data store or do you need a queue to manage writes?

What will the throughput be, and is it enough?

Will the system scale with usage?

But even if you could reasonably answer these questions, or you build a system that takes all these things into account such that you can just “add hardware”, then it is likely that this effort was larger than if you had simply pushed to ship a first version quickly. And it is possible (if not probable in some cases) that such a system may be over engineered (it was designed to solve problems or scale in a way that may not be useful), or even have other unforeseen issues. While there are certainly pros to doing it right the first time, in practice it is very hard to get it right; and building the end all system will take longer, delaying the answers to all of the questions indicating you are moving in the right direction.

There is a popular movement called Lean Startup that advocates fast iteration and data collection to hone in on the right product and requirements as quickly as possible. It’s all about figuring out what customers will use and then building those things. And these same ideas can be applied to the technical ways we build products, too.

Build v1 fast, then build v2 right.

Similar to creating a minimum viable product (MVP), build the minimum viable technical implementation that meets the business goals (the product can meet current usage requirements and performance).

Most engineering teams have been taught to think about the software design and architecture from the beginning and build something that can scale. However, if you don’t have any customers yet, scaling really isn’t your challenge. So in some ways, designing for scale when you don’t have to scale yet, is solving the wrong problems.

Of course I am not advocating doing something stupid, or writing bad code, but don’t over engineer or solve for problems until you have them. For example:

Forgoing the deployment of a multi-node Cassandra cluster and storing the initial data in a single db (with backup) that is fast and easy to make arbitrary queries against.

Cutting corners on unit tests (because the units are in flux) or comments and documentation.

All if these short cuts will help you get something out faster, and while not necessarily best practices they do solve the issue to ensure you are building the right thing.

In many ways, this process is about looking at the problems differently, and instead of solving the “big” problem, focusing on speed and efficiency – getting the highest ROI (return on investment) for your development resources.

So you modify your development process to ship the first version quickly, and then once the v1 is out in the wild, plan to start on the v2 right after.

Then your v2 can address all the problems with the first version of the product. You can improve the usability, cut or add features based on customer usage and feedback, pick the right technology and libraries to power the product, and spend time writing acceptance tests and unit tests that are unlikely to change. And from a business perspective you get the product out there faster, so it is easy to understand if this product is going to drive revenue and success.

Why this process makes sense…

Reduce the risk of over engineering.As noted above, getting something out quicker and understanding how the product is used (or how successful it is in the market) will help confirm that your are building the right product and solving the right technical problems for your business. This allows you to mitigate the risk of building or scaling out systems that may or may not address the production issues.

If you build the wrong thing you can correct it.Since the product is truly the minimum viable product, you can mitigate the risk of wasting resources creating the wrong product. Furthermore, you can get the feedback earlier in the release cycle and pivot or make changes faster (since it is always harder to change a system with a plethora of moving parts).

Chances are you will get to market faster.Building something quickly will help you launch sooner, and this will help you test and determine the viability or your product faster than waiting for the bigger launch. This means you won’t spend time building software your customers won’t use or you can’t sell.

Staffing and personnel happiness.This is perhaps one of the biggest upsides to this method. Software engineers get to do what they love, and what caters to their strengths. Moreover, they aren’t forced to work with people that don’t share their same values, and they don’t have to work in a code base that they can’t sculpt into their desired piece of art.

So what are the downsides?

Total time to build is more than if you just got it right the first time.In this model it is likely that overall the total development time will be longer and cumulatively greater than building the product once. If you build the v1 on certain technologies (like say a MySQL DB that won’t scale the way you need it to), and then the v2 needs a different one (say a proper key value store for faster queries), then the team will spend time shipping and building expertise on one database, only to have to learn and manage another.

Operations can be a headache.Most seasoned engineers will tell you that building something really fast can result in operations nightmares if the usage or data grows in a way that wasn’t considered. And of course no team likes being operational and fighting fires. Of course, if this is a problem it also probably means that your product has achieved some adoption and success, so at least you are solving something that derives real value. However, even for the v1 it is still worthwhile to identify risks and potential bottlenecks, so in the event there are problems the team will have some ideas for solutions.

So you want to try it at home? Errr….work?

Like any software process or methodology it is important to assess if this makes sense for your business, product, and company – it certainly isn’t a one size fits all. However, it definitely has its merits, so feel free to steal the pieces you like 🙂

If you do embark on this journey though, here are some pointers from my experience:

Get management buy-in. This is really critical, because a key part of this being successful is always building a v1 and a v2. If you don’t have buy off and your boss is content to keep and operate the v1, your team is going to hate you. Seriously. No one wants to get stuck supporting crufty v1 software indefinitely. So make sure everyone knows and understands the plan ahead of time.

Measure everything. In order to build the right v2 you need to know what to build. Where are your bottlenecks? How is the data growing? Without this data then it is very hard to realize very many of the pro reasons to adopt this model. Make sure that you thing about what you need to measure, and how you will measure it before the product ships.

The v2 will take much longer than the v1. I have personally seen this process evolve across 4 projects (so not a large sample set), but in each case the v2 always took more than 2x the amount of time the v1 took to create and launch. Of course every v2 I have launched a lot of extra features and didn’t always leverage a lot of the v1 code. However, some people assume the v2 will be faster, since “you have already built is once before”, but I haven’t experienced this to be the case.

Have a tight feedback loop with your customers. Make sure it is easy to get insight into what people use, what they like and what they don’t like – that way you can build a great v2.

Make sure v1 & v2s are celebrated equally. Sometimes teams or companies celebrate the first launch of a feature in a much grander way than the next version. But in this case the next version is what you need to grow your business and is just as important, if not more so. Make sure you value each of these equally or else people will gravitate to the project with the most fanfare, not necessarily the one suited to their strengths and talents.

Be open to people being both v1/v2s. Sometimes I think I have someone pegged and then they prove me wrong. Be open to the fact that there aren’t two types of people, and some people are just good at everything… (I just wish I was one of those people!)

Hopefully some of these ideas prove useful for you and your team. If anything, let this anecdote inspire you to question the way you are doing things, and look critically at innovative ways to improve the way you build software.

And if you liked this or have other better ideas, definitely post them in the comments – I am always looking for ways and ideas to make us more efficient and effective as an engineering organization.

Comments

Do you use this model for all projects, or are there specific types of projects where you do/don’t do this? Would you try to figure out potential developers v1/v2 styles before hiring and would that strongly influence your hiring decisions?

With testing and doing things ‘The Right Way’ being as easy as it is with technologies like Ruby on Rails, I might argue that v1/v2 might cost more than the end result is worth. Then there is the http://en.wikipedia.org/wiki/Second-system_effect which I’ve experienced first hand and it’s not pretty!

I do like the idea that a first version is a prototype and should be considered throwaway after getting some customer traction, but in a world with great tools and frameworks I feel like this applies more to things like a developers first rails app or first e-mail campaign sender. Once someone has built one, they should be able to quickly and effectively build a well-tested one in the ‘v2 style’.

At http://highgroove.com, we tend to stick to the v2 way. If something crazy comes along like a new technology need in a project, a developer will do a time-boxed ‘Spike’ on it which looks a lot like what you’re suggesting for the v1 ‘style’, but we redo it in the v2 way before it gets integrated into the customer-facing project/prototype.

Great ideas. As an IT guy, I’ve been on the receiving end of v1 implementations that never got v2’s because the business kept requiring new features long after the wheels fell off the v1, so make sure you are clear with your biz folks about the plan.

@ Chris –
“Do you use this model for all projects, or are there specific types of projects where you do/don’t do this? ”
Generally I have done this for bigger initiatives (more than a month is my general rule of thumb), although I imagine the same ideas could be applied to smaller features.
This model also lends itself to new functionality or ideas – things that may require innovation or iteration. For example, if you are integrating a CRM or billing system into your application – for goodness sakes, take the time to do it right the first time.
I would also say that the project should probably have some customer facing or UI component, doing something like this with an external API or platform might be challenging, because a lot of people don’t feel comfortable on building on top of things that could change or won’t be backwards compatible (internal APIs and services are a different story though).
Just like anything – apply the idea and process where it makes sense 🙂

“Would you try to figure out potential developers v1/v2 styles before hiring and would that strongly influence your hiring decisions?”
Yes and no. I would say if you have a bunch of v1 people on a team, it might be good to tune your interview process to find people that are detailed oriented and care a lot about engineering and code quality (v2 type stuff) – not because of this process – but because you will build better software with people thinking about the problems from different angles. However, that should also apply to diversified experience, background, education, technologies etc – there are all sorts of studies that show you build better products when you have strong diversity on your team.
A lot of people can, and may want to, do both types of projects though, so it isn’t a hard, fast categorization – it is more of a spectrum. The process was actually born to accommodate a team so that we were catering to each person’s strengths and desires – and people really were happier.
In fact, I think this idea really resonates with people, because they see v1 or v2 qualities in themselves (I am definitely a v1 for example) – and most people know what they like and believe about “how” to build software. Tying it in with the lean startup ideas, and agile iteration around product, allowed us to make our development team happy and achieve business objectives at the same time.

@ John – thanks and totally agree – it is key to get everyone on board…the v2s always take longer than the v1s and so people need to be ready for that investment 🙂

Very good points! I guess I would add emphasis to v1 people sharing their findings and experience with the v2 team. On the plus side that gives them a new role during the early days of phase 2 which can be fulfilling while still working with the “v2 kind” of developpers…

Thanks for the detailed reply! I head you talk at Velocity Online, ended up here, and there’s a lot of great stuff.

Highgroove does things in a pretty unique way and it’s been really interesting seeing how other people approach similar challenges. This is the first thing like it that i’ve seen, as most people seem to just iterate (for better or for worse) on a single thing and only do a v2 when things get really really bad.

What I like most is the recognition that developers work and think differently. And, they do their best work when they a free to do so. Also, it still respects agile principles getting feedback often and using it to improve the product early in the development cycle.

Thinking about how this might apply to my job working on Microsoft Office. We’re in the curious predicament with a few unique characteristics:

– we know that anything we ship will have sufficient usage to expose poor performance
– we are set up around a multi-year development cycle (engineering processes, market expectations etc)

We certainly have v1 and v2 people, naturally shading towards v2 types as v1 types don’t gravitate to enormous companies. Within a given release cycle I think it would be difficult to have v1 types blaze ahead building stuff, get customer feedback and have v2 folks sweep up behind them.

@ dimdm and chris – thanks for the comments and I am happy to hear the information resonated with you!

@ shawn – I think that this is true in most things in life, no just work. When people are able to do the things they like and are naturally good at, they perform better and enjoy doing the task. The real challenge comes into understanding what they are best at, and sometimes people don’t know themselves. For example, I know so many great engineers that at some point will say they want to be managers when they really wouldn’t be a good fit in that type of role, etc.

@ Kevin – Microsoft is a challenge because in many ways they are not very agile – the history of the company is as a shrink wrapped installed software and so iterating quickly to just a subset of users and then measuring that performance, etc can be challenge. With web software fast iterations, A/B or multivariate testing, etc become so much easier.
In fact, for many web application companies (Facebook, Flickr, Amazon, etc) they have very minimal test infrastructure and process and rely on extensive monitoring and fast roll back.

That being said, I think some of the principles could be applied – maybe have prototypes of features or ideas and then send those to usability testing or a large set of beta users – that way you could get the valuable customer feedback earlier. And then have the product managers redefine what they build from there and that is what is built into the product. You would definitely need to move a lot faster and be really streamlined in that feedback and data gathering process (that is what is great about the web – real time data).

On your 2nd question, the v1s and v2s don’t resent each other at all. Part of it is how you manage it, people have to know what they are good at and be recognized equally for their hard work (if all the fan fare goes to the v1 people and the initial release, then you are doing it wrong).

The other thing that has come out in practice for us is that the v2 software (especially because we deal with TBs of data) tends to involve “scaling” problems and new technologies – and a lot of engineers get excited about learning to deploy new things (like NoSQL – we try not to build any first version of a product with a complicated architecture, only stuff we are familiar with and can build fast).

The v2s also tend to be team projects (whereas v1s often tend to be only 1-2 people) and so the man months are longer but the total time is similar (and having people on similar schedules helps).

And finally, more often than not, the v2 has new features and a new UI and gets as much fanfare (if not more) from our customers. And that is very encouraging, because many people are motivated by positive feedback from real people using their work.

And of course, we make it so we don’t necessarily typecast people into one flavor – remember v1->v2 is a spectrum, not black and white, and so allowing people to do both and understand what they like best is a really great way to mitigate any us vs. them notions.

1) I’ve been on some teams that were too lazy to make a better, cleaner v2 when v1 worked “well enough”. This leads to technical debt. BAD. Need management support to devote resources to the slow v2 iteration.

2) v2 Purists: at big companies with review processes (say, Google), there are some (but a minority) of notoriously difficult code reviewers that hold up changelists due to var names, punctuation, and other relatively minor style issues – the haggling can go on for days or weeks. This really frustrates the v1 folks because they just want to launch launch launch and see what sticks. I think your ideas about management buy-in can solve this problem, as the v1ers can have the mandate to move fast – but know that their code may have to be thrown away in a rebuild – they have to be okay with that.

@anthony Thanks for sharing the post! And both of your points are good ones.
When it comes to #2 I remember once having a conversation about how developers early in their careers would be so upset if someone changed their code, or heaven forbid deleted it. I mean, if you only have written a small amount of code, seeing it disappear is like erasing your career! But I think as you write more code, work on more projects, etc, #2 becomes less of an issue.
Thanks for the thoughtful comments!

re: code deletion – From my experience at YouTube, taking the time to remove dead/deprecated/obsolete/bad code gets you thumbs up from the eng elders. They like code pruning and maintenance as much as feature development. This sets up an incentive for people to work on these unsexy tasks. Otherwise, you’d have an organization where engineers only want to work on new stuff, leaving code maintenance to the style purists, who’ll turn around and be a stickler on code quality in v1s – leading to a what I imagine is a viscous cycle. I think setting up a culture of rewarding code maintenance will alleviate some of the v2ers’ concerns.

@anthony great points….I think that makes a lot of sense, but can be a challenge to drive/reward that behavior. The best tool in all of that is a really plugged in dev manager/leader advocating and rewarding those things….

topics

search this site

follow along via RSS

about me

Kate is known as one of the top technology leaders and CTOs. Her technical background is in creating and operating large-scale web applications. Her focus has primarily rested on SaaS applications and big data. She has extensive experience building and managing high-performance teams, and considers herself a fan of agile development practices and the lean startup movement. She is currently founding her own startup, popforms, but has held roles as developer, project manager, product manager, and people manager at great companies including Amazon and Microsoft. The last seven years she has been a VP of Engineering/CTO for companies like Moz, Decide (acquired by eBay), and Delve Networks (acquired by Limelight).
Kate is a keynote speaker, and she is also the curator of the Technology and Leadership Newsletter (TLN - www.techleadershipnews.com) and has a personal blog at katemats.com.