Topics

Featured in Development

As part of our core values of sharing knowledge, the InfoQ editors were keen to capture and share our book and article recommendations for 2018, so that others can benefit from this too. In this second part we are sharing the final batch of recommendations

Featured in Architecture & Design

Tanya Reilly discusses her research into how the fire code evolved in New York and draws on some of the parallels she sees in software. Along the way, she discusses what it means to be an SRE, what effective aspects of the role might look like, and her opinions on what we as an industry should be doing to prevent disasters.

Featured in Culture & Methods

Mik Kersten has published a book, Project to Product, in which he describes a framework for delivering products in the age of software. Drawing on research and experience with many organisations across a wide range of industries, he presents the Flow Framework™ as a way for organisations to adapt their product delivery to the speed of the market.

Featured in DevOps

The fact that machine learning development focuses on hyperparameter tuning and data pipelines does not mean that we need to reinvent the wheel or look for a completely new way. According to Thiago de Faria, DevOps lays a strong foundation: culture change to support experimentation, continuous evaluation, sharing, abstraction layers, observability, and working in products and services.

Author Q&A Continuous Digital and Project Myopia

Key Takeaways

Organisations need to shift to a continuous model in which technology development drives the business and the business drives technology

There are multiple problems when trying to apply the project model to software development - hence #NoProjects

Products are long lived and need to be maintained, projects are short lived and intended to end

A very high proportion of IT projects fail (anywhere from 30% to 70%) not because of incompetence or bad intentions but because the model is wrong

Author, speaker, practitioner and consultant Allan Kelly has recently released two complimentary books which address ways of working in modern digital businesses.

The first “Continuous Digital” addresses the way organisations need to structure themselves when “every business is a digital business” he presents a “Continuous Model, a world in which technology development drives the business and the business drives technology”.

The second, “Project Myopia”, is a companion book which explores more of the underlying theory of #NoProjects and explains why the continuous culture is so important.

An extract can be downloaded here. Continuous Digital can be purchased here and Project Myopia can be purchased here.

InfoQ spoke to Allan about the books:

InfoQ: Why did you write these books, what is the underlying problem or opportunity you are addressing?

Allan Kelly: Writing both books was a little bit of an accident. I'd long had this thing running around my head about how working on "projects" causes problems - I even remember a conversations with Mary Poppendieck about it as long ago as 2011.

Then about five years ago Steve Smith, Joshua Arnold and me started tweeting about #NoProjects. Twitter is an awful media for debate but it is great for floating ideas. It quickly became clear that there are multiple problems with the Project model and its application to software development. It also became clear that there were other people who saw these problems.

Vasco Duarte (Mr #NoEstimates) asked me to pull the arguments together. That long essay grew into Project Myopia which sets out the problems with projects.

However its not enough to set out the problems one needs to offer a better way, an alternative solution. Plenty of people on Twitter kept asking "So what do we do instead?"

So I started to set out the alternative. At first this looked at lot like my Xanpan method but, freed from thinking in projects two really powerful forces appeared.

First we live in a Continuous age: continuous improvement, continuous integration, continuous delivery and so on. And while businesses is obsessed with change what they really crave is longevity, you don't want you employer to go out of business, that is bad. Successful software continues - people use it an it needs to change. Unsuccessful software, software which nobody uses, doesn't change and dies.

Second, business is increasingly digital. Technology s no longer "back office" it is front and centre of the offering. For business people it is a big deal. The business and the technology are now inseparable. In that world the temporary nature of projects don't work.

InfoQ: Who are the books for, who is the target audience and how do you ensure you are not “preaching to the converted”?

Kelly: Quite possibly Project Myopia is preaching to the converted - I know many programmers who love it because it speaks to their frustrations. Still, I think those who believe in the project model should read the book and find their own answers to the critique.

Continuous Digital is different, this books sets an a new, alternative model for continuous digital business. There are two audiences here.

Those with management responsibility - and I include non-commissioned managers such as Scrum Masters and Business Analysts here. There is a lot for managers to learn here because it is the mental models these people hold that informs their decision making.

The second audience is tomorrow's leaders and managers: today's programmers, testers and analysts who will benefit from a new management model.

This audience is important for another reason: as we move to more self-organization and fewer managers there organisational decisions to be made and "management" work to do. Thus more people need to understand management issues. Paradoxically having fewer commissioned managers means that more people need to engage in management work.

InfoQ: Why did you write two books - what is different between them, and what order should they be read in?

Kelly: You can read the books in either order but if you only read one please read Continuous Digital - although it is longer, sorry.

Project Myopia is a look at why the project model is a bad fit for software development and how it damages software. You can think of this as the push to stop using projects.

Continuous Digital sets out a better way of working, this is the pull if you like. The book stands alone building its case not from the failings of the project model but form the nature of modern, digital, business. Such businesses are inherently based on technology. The business is technology and the technology is the business.

People ask "Is Uber a technology company or a transport company?" but actually Uber is something else. Uber is a digital business where the technology and product are inseparable. Spotify, JustEat, AirBnB and many other twenty-first century businesses are like this.

InfoQ: What’s different about the modern economy - why is “digital” such a big thing, or is it just another buzzword?

Kelly: I've grown up in the digital world - first computer was a Z80 based ZX81 with 1Kb when I was 12. For a long time I just ignored the word "digital" - those analogue computers never got very far so the word added nothing.

But for people who never had that experience digital is a thing. People who didn't grow up with Facebook or even MySpace. For such people "IT" was awkward interfaces, slow terminals, Word crashing and corporate IT. To these people the digital world is completely different - iPhones, Twitter, GPS, sensors, and all the other things cheap CPU cycles make possible.

Yes it is all technology but it is completely different. Consider Amazon, looked at one way it is the digital version of a Sears or Littlewood catalogue, but technology alters the customer experience so much that it is completely different.

InfoQ: You make a strong point about teams as the sources of value and the focus of work - why teams and not individual contributors? What’s special about the team focus?

Kelly: It’s the scaling question. There are times when individuals can make a massive difference. Usually at the point when technology changes - think of the teenager who got rich writing Commodore-64 games, or the early iPhone apps.

On the whole our systems are too big for individuals to take on now. Plus, there are more technologies and therefore more skills required. Knowing C is not longer enough, you need JavaScript, CSS, Jenkins, pipelines, plus skill for design, testing, stakeholder liaison and analysis. Then there are the non-technical skills: marketing, sales, support and so on.

Even when an individual can write a large valuable system working as a team can produce the system sooner, unlock the value sooner. Once you start factoring cost-of-delay into your thinking so much changes.

InfoQ: Why NoProjects - haven’t many organisations been very successful using the projects model?

Kelly: Clearly technology development works: iPhones, self-driving cars, Netflix, but is that because of they are successfully run projects or despite the project model?

We regularly hear people say 40% or event 70% of all IT projects fail. I don't know the real number is but when so many fail one has to ask: is it the model that is failing?

Of the ones that succeed I would suggest many do despite the project model not because of it. They succeed because someone breaks the rules, doesn't follow the process or games the objectives.

Then there are the cost that don't get counted, the technical debt, bugs, unpaid overtime, value lost from late delivery - many of those successes don't look quite so successful.

That is yesterday. We live in a different age. We need a model that works more often than it fails.

InfoQ: You make a point about diseconomies of scale - this seems to be counter-intuitive are there some explanations and examples?

Kelly: When you buy milk you expect that buying two 500ml carton to cost more than one 1 litre carton. And you expect a 2 litre carton to cost less than two 1 litre cartons. That is economies of scale, that is Henry Ford making a million identical blank model-Ts.

Economies of scale are true for milk and many other places. But not software development. When you are developing software diseconomies of scale prevail, to be efficient you have to get good at lots of small.

Fixing a bug in a 1,000 line program is far easier (and faster) than fixing a bug in a 10,000 line, which is still easier than a 100,000 program. Bigger programs are more difficult to maintain - hence the micro-services movement.

Smaller teams out perform large teams, a team of 4 people will deliver more per person than a team of 14, let alone a team of 40. As teams get bigger communication costs go up, it is harder to create a shared vision or for an individual to comprehend the whole.

Testing is another case: when companies roll-up fixes into big releases because they want to exploit economies of scale. But testing a release with 100 changes is an order of magnitude more difficult than testing 100 releases with one change each.

Image you do one release a year with 100 changes, imagine one defect slips through which blue-screens the user machine. Users see 100% of releases fail. Imagine instead you do 100 release with one change each. One defect slips through and blue screens the machine. Users now see 1% of releases fail, 99% succeed.

Now imagine you are the programmer tasked with fixing that defect. Which scenario would you prefer?

Again, add in cost-of-delay and you find the financial benefits of 100 small releases cause return on investment to rocket.

One last thing. Once you have your release, once you are "done" everything changes. Once software is in operation it exhibits the greatest economies scale known.

InfoQ:How does this approach impact the hard areas of financing and governance?

Kelly: Similar story. We need to optimise for small. Stop handing out large sums of money and start giving small sums of money with regular review.

Basically its the Silicon Valley Venture Capital model. Sure the VC has downsides but the basic funding model works. Teams get a little bit of money, if they can demonstrate progress - working software, validated market - they get more. Its a portfolio model where you make lots of small bets and weed out the underperformers.

Again, you need to get good at lots of small, sot the portfolio process needs to happen regularly and efficiently.

InfoQ: In Project Myopia you contrast continuous delivery with project based approaches - can the two co-exist?

Kelly: Yes, plenty of places fudge it today but it is an awkward marriage and I have to ask: why? whats the benefit over doing one or the other?

The groups apply continuous delivery within the project model but this eventually creates tensions. When you think about it the two are incompatible. Continuous is continuous, teams are finding work to do and delivering something. Projects are temporary structures created to do something specific which is known in advance.

Using both makes it hard to understand what is going on and to have clear goals. That itself is a problem, particularly when if comes to governance: applying project success criteria to a continuous team is illogical.

InfoQ: Why “myopia” - what is causing organisations to be short sighted about their ways of working?

Kelly: When companies run projects they tell themselves that the work will end. They ignore the fact that technology and change doesn't stop. New opportunities are arising, as people use the technology they find new ways of using it, see new uses and opportunities for technology.

Saying "No, we finished the project" leaves value on the table.

Saying "OK, we'll start a new projects" acknowledges the value but brings in a project set-up costs, losses time (remember cost of delay) and assumes you know what is needed in advance.

InfoQ:We’ve had agile development practices and thinking for nearly 20 years - how come it’s taken so long for these ideas to become prevalent, and are they?

Kelly: These ideas have always been lurking in agile but even while agile has been growing so too has the project management movement. Many of us in agile have shied away from challenging that, if only because it might be career threatening.

The thing that is driving these changes now is the shift to digital business. Technology is driving growth and locking it into the project model neuters the potential.

InfoQ: This makes sense for new companies like AirBnB and Uber but what about existing companies which have been delivering projects for years?

Kelly: Legacy business with corporate IT departments see the world as projects which will "be finished." Can you imagine AirBnB announcing their system is finished? - the moment a digital business declares itself "done" they give up on leadership and invite competitors.

Legacy business can't keep on this short sightedness, the extra costs it brings and the ambiguity in goals. Businesses which do are leaving themselves open to challenge. A digital business cannot treat technology development as something that happens separately from the business - preferably cheaply and far away - and is done one day.

Look at the legacy banks. For all their talk they treat technology creation as second class. That have been cutting corners, running up technical debt, taking advantage of employees and declaring success for 30 years or more. Now they find that their technology and business systems can't complete with the FinTechs.

The practices and process which brought success before - heavy project management, extensive analysis, outsourcing, offshoring, big batch working - are now a hinderance. Superficially digital looks like old style IT (Java becomes JavaScript, tests get automated, etc.) but it is completely different.

Here in the UK we have a bank called TSB. Six months ago TSB did a big-bank system update. Even while the bank was Tweeting pictures of celebration customers were complaining they couldn't access their accounts or pay bills. Six months later they still haven't resolved all the problems. TSB may claim to be Agile but they clearly aren't and the managers clearly don't know how to manage technology. Such banks are dead, they just don't know it yet.

Banks are right to say they are "technology companies that happen to make money by offering banking services." But they have run up hideous technical debts and Reverse Conway's Law limit what they can do. A few will succeed, most will fail but they all have to try.

The banks are sitting ducks for FinTechs. I suspect many of the senior people know this which is why so many are taking sidebars by investing in FinTechs rivals.

About the Book Author

Allan Kelly is Agile Improvement specialist, he helps companies large and small enhance their agile processes and improve their digital products. Past clients include: Virgin Atlantic, Qualcomm, The Bank of England, Reed Elsevier and many small innovative companies you've never heard of. He invented Value Poker, Time-Value Profiles and Retrospective Dialogue Sheets. He is the author of "Dear Customer, the truth about IT" and books including "Project Myopia", "Continuous Digital", "Xanpan" and "Business Patterns for Software Developers". His blog is at https://www.allankellyassociates.co.uk/blog/ and on twitter he is @allankellynet.