My most recent books are the Leader's Guide to Radical Management (2010), The Leader's Guide to Storytelling (2nd ed, 2011) and The Secret Language of Leadership (2007). I consult with organizations around the world on leadership, innovation, management and business narrative. At the World Bank, I held many management positions, including director of knowledge management (1996-2000). I am currently a director of the Scrum Alliance, an Amazon Affiliate and a fellow of the Lean Software Society. You can follow me on Twitter at @stevedenning. My website is at www.stevedenning.com.

The Best-Kept Management Secret On The Planet: Agile

All truth passes through three stages. First, it is ridiculed. Second, it is violently opposed. Third, it is accepted as being self-evident.

–Arthur Schopenhauer

In 1714, the British government offered a prize for a method of determining longitude at sea, with an award of £20,000 (three million pounds in today’s terms). John Harrison, a Yorkshire carpenter, worked on the project for several decades and eventually in 1761, came up with a design that proved accurate on a long voyage to Jamaica. The scientific establishment refused to believe that a Yorkshire carpenter could possibly have solved the problem that had stumped the best scientific minds. In 1773, when he was 80 years old, Harrison received a monetary award in the amount of £8,750 from Parliament for his achievements, but he never received the actual prize. A Yorkshire carpenter was the wrong person to have solved the problem.

In 1865, Gregor Mendel, an unknown professor read a paper at two meetings of the Natural History Society of Brünn in Moravia, giving the results of studies in which he had cultivated and tested some 29,000 pea plants. His study presented a solution to a problem that had stumped the finest scientific minds. The paper was ignored by the international scientific community for the succeeding 35 years until it was eventually realized that Mendel had indeed come up with the solution. His work later became known as Mendel’s Laws of Inheritance and he was hailed as the father of modern genetics, but his work was ignored by the scientific community. A researcher on peas in Moravia was the wrong person to have solved the problem.

In 1981, Barry Marshall, a pathologist in Perth, Australia, came up with an odd idea: stomach ulcers are caused by the presence of spiral bacteria. His idea was at odds with the thinking of the international scientific community. It was ridiculed by the establishment scientists and doctors. How could bacteria possibly live in the acidic environment of the stomach? In 1984, frustrated by the widespread disrespect for his ideas, Marshall actually drank a Petri dish of liquid containing the spiral bacteria, expecting to develop, perhaps years later, an ulcer. He was surprised when, only three days later, he developed ulcer symptoms. It took until 2005 before he was awarded the Nobel Prize for Medicine. An obscure pathologist from Perth had been the wrong person to have solved the problem.

Over and over again, we see that when a bold new idea challenges an entire way of thinking of an international community, if the idea comes from an unexpected source, it can languish in obscurity for decades, even though the solution to a problem that needs to be solved is staring the experts in the face.

The best-kept secret in management today: Agile

Something similar seems to be happening in management which for decades has struggled to solve a fundamental conundrum: how do you get disciplined execution along with continuous innovation? Promising efforts to improve one dimension always seem to cause losses on the other one. Disciplined execution crushes innovation, and innovation by its nature is undisciplined. The problem has seemed insoluble.

Yet, just over a decade ago, a set of major management breakthroughs occurred. These breakthroughs enabled software development teams to systematically achieve both disciplined execution and continuous innovation, something that was impossible to accomplish with traditional management methods. Over the last decade, these management practices, under various labels such as Agile, Scrum, Kanban and Lean, have been field-tested and proven in thousands of organizations around the world. My own recent work distills, builds on and extends these principles, practices and values so that any organization can now achieve to apply the elusive combination of disciplined execution and continuous innovation.

Why software?

Unfortunately, these management discoveries were not made by “the right people”: academics in business schools or high-paid managers in big corporations. The discoveries were made by the people that, in prospect, you would think are the least likely people to have solved a management problem: geeks. Software developers were known to be antipathetic to both managers and management. Badly dressed, unkempt, even sometimes unwashed, speaking about issues that managers could hardly grasp, these employees were the most problematic of a big organization’s employees. How could they possibly have solved a problem that had stumped the finest management minds on the planet for decades?

In retrospect, it’s not hard to see why they solved the problem. In the 1990s, huge sums of money were being lost because the work of software development was always late, over budget, and plagued by quality problems. Clients were upset, and firms lost money. The developers were seen as culprits and were punished. They worked harder and harder. They labored evenings and weekends. They got divorced. It made no difference. The software was still late, over budget, and full of bugs. They were fired, but their replacements did no better. The standard prescriptions of management didn’t work with software development. Something different had to be found.

Yet the management world remains generally in denial about the discoveries of Agile.

You can scan the pages of Harvard Business Review and find scarcely even an oblique reference to the solution that Agile offers to one of the fundamental management problems of our times.

Or take a look at a standard management textbook, by Chuck Williams’ MGMT. There is not a single reference to Agile, Scrum or Kanban and only one obscure reference to Lean.

The only general management book that I have discovered to mention Agile in any significant way is Mark Addleson’s Beyond Management: Taking Charge at Work. Addleson writes, “By looking for practices that are good for knowledge-work, which break the stranglehold of management on how to organize work, we can learn a lot from from a mini-revolution in software development known as ‘agile methods.’… Agile methods offer a way out of this impasse, with a far reaching, even radical solution to building good software.”

Beyond Management is only one of the very few general management books to recognizes the obvious truth that the solution to an intractable management challenge has come from an unexpected source.

Post Your Comment

Post Your Reply

Forbes writers have the ability to call out member comments they find particularly interesting. Called-out comments are highlighted across the Forbes network. You'll be notified if your comment is called out.

Comments

Hi Steve Have look here plenty info maybe too much! http://bit.ly/qlSUvM In particular the Q&A might help. In terms of methodology it really is “agile” working direct with users who are encouraged to contribute to ideas how to work more effectively both in build and subsequently, as such it becomes their process. The challenge becomes knowing what you want and this relies on your skill sets but with this new approach you no longer need to have a detailed final spec which speeds up deliver and helps keep users engaged. Yes your final comment is the innovator’s dilemma. Domination is bad for innovation and society in general – but that is another subject!

My company has been building and testing a software development platform that uses flowcharts as real code, making the development process easy, not source-code centric and usable by more people than just programmers.

Once the business logic is ‘programmed’, and the user interface painted, the code is generated, compiled and deployed. Easy, transparent, and stable. We extended it in such a way that it became a collaborative platform.

Initial comments from programmers was very negative. After investigation, their major complaint was that the code could be understood even by the project manager…

Things look brighter now, since we could prove that an average project took considerable less time, and 50% less money. That last argument is always a winner.

All this, simply by breaking some barriers and applying the principles of agile development.

One last thing : It took us 15 years. In some way, I can relate to the historic examples.

I have for the last few years shifted from being a biomedical researcher to managing large multi-partner biomedical research projects. Whenever you say “project management” to a biomedical researcher they break out in hives and begin to shake. They fear that project management will take away their creativity, and science is very creative. You almost never end up finding or doing what you set out to do.

Your phrasing made it completely clear for me: “Disciplined execution crushes innovation, and innovation by its nature is undisciplined. ” Just by feeling my way through what works I have settled upon something that resembles Agile. I call it Open Innovation Project Management. I have as a result had some run ins with dyed in wool project managers.

I have now linked up with some software developers to address the painful inefficiency that is inherent to research proposal preparation. Sure enough a kan ban board features prominently. You can see some screenshots on our landing page www.grantsnap.com.

When we show this to researchers their eyes light up. I would be curious to hear your and other experience with taking Agile beyond software development.

That’s what my book on radical management is about: distilling the substance of the principles of Agile and applying them to management in general. Glad to hear that you have been doing the same thing in biomedical research with similar results. Would love to hear more.

When you work on a project that involves up to 40 different institutions or companies, it becomes clear that there are different challenges from what you see in typical project management. One of those is that you are competing with individuals’ demands and their institution/company hierarchy so that their ability to commit time to the project waxes and wanes. So, you need to have flexible resourcing: who is going to work on what next and who is going to solve what issue? They also tend to be complex projects formed by people with a lot of ideas. Hence, what you do seems to be in constant flux addressing new issues, following new opportunities and figuring out what you need to do to accomplish what you set out to do. Typically it is impossible to anticipate or even scope all the complexities that need to be addressed.

The result is that you need the flexibility that is embodied in Agile, but yet there needs to be structure as well. That, I think it really the trick – balancing between flexibility and structure, innovation and delivery.

I don’t think many project managers will agree with you when you say that “Agile is the best kept management secret on the planet”. Agile is heavily criticized for many reasons, and some project managers claim that Agile’s existence is merely to support the consultancy business of its creators.

You can read this post we have published a long time ago on Agile Limitations ( http://www.pmhut.com/limitations-of-agile-software-development ) and see whether Agile is really that panacea or not.

I agree that you can find people in some software circles who are negative about Agile, just as you could find people who were negative about John Harrison, Gregor Mendel or Barry Marshall.

Some of this stems from mediocre Agile implementations. Some of it stems from hostility to departing from the traditional command-and-control model of management that is still pervasive in large organizations.

Agile isn’t a panacea. It isn’t even very clear and the Manifesto itself needs to be upgraded: http://www.forbes.com/sites/stevedenning/2011/08/12/agility-is-not-enough-beyond-the-manifesto/

However the best implementations of Scrum and Kanban do show the way forward in resolving the fundamental dilemma of reconciling disciplined execution and continuous innovation in management generally.

My article is merely pointing out that management is in error in relation to Agile when it maintains that it has nothing to learn.

I would suggest the biggest obstacle to agile-type development is offshoring. IT outsourcers have great difficulty sending developer jobs overseas when the developers are embedded in the business. The waterfall approach, defined by many processes, makes offshoring much more feasible.