Software Development

Monday, 9 February 2015

I have heard it said, and have had the feeling myself, that Agile (Scrum in particular) is a bit like Communism. The follow-on from this is that Scrum is doomed to failure and/or that people won't accept it (presumably on ideological grounds). Let's explore the validity of this argument.

What is Communism?

First we need to decide what Communism is. I have always been a little vague on this but here are two simple definitions.

The workers control the means of production.

Classless society where you give what you can and you get what you need.

I will only discuss the first definition to say that it is not particularly relevant. First, the means of production in software development is the computer hardware and the brains of the developers. I don't think it's a big deal who owns the computers, and I don't think anyone would argue that the company you work for owns your brain. Of course, owning some of the company you work for (eg, stock options) may be a motivator but probably does not make much difference to the use of Scrum.

Please note that this article is in no way politically motivated or an attempt to promote Communism or any other political system.

I think the 2nd definition is more useful to our discussion. So the question is: Is there any similarity between a Scrum team and Communism as in the 2nd definition? Let's look at the definition in detail by splitting it into three parts...

Classless society

"all team members are equal"

A scrum team is self-organizing. There are no bosses (within the team) to tell team members what to do. Of course, the PO determines the product backlog which gives direction to the team, but how the team accomplishes their goals is completely up to them.

Note that you shouldn't make the mistake of thinking that "self-organizing" means the team can do what they want - that would instead be anarchism. The team must work within the bounds set for them by senior management, the product backlog and the rules of Scrum. It is an advantage of Scrum that self-organizing teams usually organize themselves more efficiently and flexibly than an individual manager can.

Contribution (give what you can)

Just because all team members have equal rights does not imply whatsoever that they have equal abilities. Obviously team members will vary in what they can contribute depending on experience, knowledge and aptitude.

"everyone contributes according to their ability"

An advantage of Scrum is that team members, after time working together, grow to know of their colleagues' abilities and weaknesses far better than any outsider. This is one of the reasons that self-organization works so well. This, plus being empowered, means that the abilities of individuals are made far better use of. Further, the sense of camaraderie encourages the team to work towards the same end.

This is how Communism is intended to work but in practice "Communist" states fail because there is no incentive to work to your best ability. Scrum only works because in a small team everyone knows who is not pulling their weight and there is a team spirit and "peer pressure" to do your best.

Rewards (get what you need)

In Communism (in theory) there are no privileged individuals. Nobody has special benefits. Everybody is treated the same. Similarly, in Scrum the whole team is rewarded, regardless of the contribution of different individuals.

"everyone is rewarded equally"

This is another important advantage of Scrum. While rewarding on an individual basis encourages competition and hard work, it also promotes rivalry and virtually destroys collaboration.

A common argument against Scrum is that those in the team who make a greater contribution will be aggrieved because they get no better treatment than the worst performers. In my experience this rarely happens unless there are one or two team members not pulling their weight (in which case the team has the right to vote them out of the team). Generally, team members are committed to the team and not aggrieved when they know that everybody is doing their best.

Ringelmann Effect

So if Scrum is like Communism, the next logical conclusion is that it is doomed to failure like numerous Communist states. Wrong! There is a fundamental difference - a Scrum team is much smaller. In my opinion Communism (and Scrum) can work well for a small number of people. The ideal size is probably around five. Ten is stretching it. It can't work at all with a team size of 20 and there is no hope at all with a population of millions.

This is all due to the Ringelmann Effect, which has been known, and extensively studied for more than a century. The Ringelmann Effect essentially says that the larger the group performing a task the less productive each group member becomes.

The interesting thing is that when the size of the group exceeds about eight the

* The actual "plummet level" varies depending on the type of project. For a complex project it may be just 6 people. For a well-understood project it may be 10 or even 12.

My rule of thumb is to follow the Scrum guideline of a maximum team size of 9, but preferably around 5.

productivity, per person, plummets*. Studies have shown that a team of 14 people will produce less than a team of seven - that is total output not per capita output. Many would expect that by doubling the number of people in the team, the output, if not doubled, would at least increase, but it actually falls!

What is the cause of the Ringelmann Effect?

After much study it is generally recognized that there are two principal causes: social loafing and coordination overhead.

Social Loafing

Social loafing is the name for the phenomenon where people exert less effort to achieve a goal when working in a group than when working alone. Further, as the group size increases the average (per capita) effort decreases. The interesting thing is that this is often unconscious - the individuals may not even be aware that they are not trying as hard!

Social loafing is the main effect that Ringelmann discovered in his original experiments early last century. He found that, given the task of pulling a rope, the more people that are pulling the less was the average force generated. Importantly, when the size of the group reaches 7-8 then the average effort falls quickly.

What is the relevance of social loafing to this discussion?

First, I believe that social loafing is the principle reason for the failure of (large scale) Communism. However, social loafing in a Scrum team is reduced since team size is restricted to 9, and is typically 5 or 6. Moreover, social loafing is well managed by other Scrum practices such as the daily standup and is often counteracted by such things as team spirit.

One aspect of Ringelmann's experiments that we shopuld consider is they placed people in groups at random. I believe that if the groups had been given an incentive and were allowed to generate any sort of team spirit then the results may have been significantly different. For example, instead of simply pulling a rope with a group of strangers perhaps the groups could have been organized into "tug of war" teams that pulled against each other.

The benefit of "team spirit" is born out by the fact that in sporting events (swimming, running, etc) an individual will often perform at a higher level in a relay event than in an individual event (see Hüffmeier et al), though I suspect this depends on the specific individual and team involved.

Another important effect, which counteracts social loafing, is peer pressure. Peer pressure is often portrayed negatively but can have beneficial effects. The visibility of the daily standup (see Why It's Good in Scrum Standup) means that the team quickly becomes aware of any members who are not pulling their weight. I will mention (again) that for this to be effective the team must be of a small size (9 or less) otherwise the team will not be aware of what everyone is doing and individuals can fade into the background.

Coordination

Coordination refers to getting everyone working towards the same goal and simply communicating effectively with other team members when necessary. As the goals of the team, and the means to achieve them, become more complex, coordination rapidly becomes important. Once social loafing is under control (as discussed above) coordination become the more significant effect affecting the team's performance.

The problem of coordination affects any group endeavor so it should affect a Scrum team just as it would affect a Communist state. Again there is the obvious difference that a Scrum team has no more than nine team members so -- coordination is easier the smaller the team.

I will also mention in passing that some of the techniques of Scrum such as dividing a project into many small iterations avoid many of the problems of coordination. A proper discussion of this will have to wait for a later time.

Summary

All team members are equal, within the team. Team members contribute according to their abilities. The whole team is rewarded according to their achievements.

So is Scrum is doomed to the same fate as Communism? Not at all!

As was discovered by Ringelmann, social loafing can greatly affect performance of a team. This is a major problem for Communism and for Scrum. However, social loafing does not really kick in until the team size reaches eight or more, which is why Scrum teams should not be larger than 8 or 9 members.

Scrum practices also help. The visibility in the daily standup (again, this depends on the team being of small size) provides peer pressure which counteracts social loafing. Further, team spirit is enhanced by having the team work towards a common goal (ie, finishing the sprint backlog) which also overcomes loafing.

Coordination is another problem that is particularly important in software development due to the complexity of the work. Keeping the team small and adopting Agile practices can minimize this too.

It is generally agreed that Communism failed due to a lack of incentives -- people don't excel when they are given no reason to. This is often taken to mean that people are inherently greedy. But not all (or even most?) people are motivated by money/material possessions. A great motivator in a small team can be helping your colleagues and doing your part for the success of the team.

My conclusion is that Scrum is like Communism but on a small scale. Communism (and Scrum) can work well for a small group of people. It's just that it does not scale well.

Saturday, 31 January 2015

Lean Software Development ostensibly tries to apply many of the principles of Lean Manufacturing to software development. I first remember reading about Lean Manufacturing, and the Japanese Quality Assurance movement, more than 20 years ago - before I had even heard of Scrum and well before the Agile Manifesto. In fact when I first read about XP (in '98 or '99) I remember thinking that a lot of ideas came from the quality assurance movement and even Lean.

However, in the last few years I have seen a lot of people saying or assuming that Lean Software Development is a "variety of Agile", like XP, Scrum, etc are. For example, I recently saw an advertisement for a course at a reasonably well respected university on SCRUM vs LEAN: Agile Methodologies Analysed and Compared.

Let's be clear on this, Lean is not a variety of Agile.

“ Lean is not
a variety
of Agile ”

Brief History

Lean manufacturing grew up as part of the Japanese Quality Assurance Movement around the middle of the last century. This movement was a major contributing factor in Japan's rise in a few decades to be the world's manufacturing powerhouse. The main focus of Lean Manufacturing is eliminating waste in its many forms.

The success of Japan's QA movement meant the ideas spread to other countries and many different industries. The ideas of QA and Lean Manufacturing found their way into SQA (software quality assurance) around the early 1990's. A great deal of them also found their way into Agile (perhaps via XP) in the late 1990's, but Agile is more than just a rehashing of the ideas of SQA and Lean.

A fundamental principle of QA/Lean is the concept of "Right First Time", which is trying to do things properly by eliminating mistakes. Agile outright rejected this idea, not in all things, but in the most crucial aspect of the software development process - how the software is specified and designed.

Agile is based on evolving the design based on responsive feedback. The assumption is that waste is unavoidable and that an emphasis on eliminating waste is counter-productive.

Lean Software Development

About 10 years ago Mary and Tom Poppendieck wrote a book called Lean Software Development: An Agile Toolkit, which apparently started a Lean movement "within Agile". I think I first heard of this book a few years ago but I have only just started reading it. I haven't had time to read all of it yet, but what I have read I agree with absolutely. It explains many of the principles and practices of Agile really well.

The only problem I have with the book is that it tries to apply "Lean Manufacturing" to justify Agile practices. The implication is that the core idea behind Agile is the lean philosophy of eliminating waste. This I find entirely fallacious.

The critical flaw in the premise of the book is demonstrated by this direct quote:

“Just as manufacturing processes are continuously improved by production workers, so must a software system be continuously improved by developers.”

- see Tool 19: Refactoring on page 137

The problem here is that it is confusing two different things: the manufacturing process and the product being manufactured.

A core principle of Lean Manufacturing and QA in general is to attempt to continually improve the process by which a product is created. But the actual product is not normally modified only the way in which it is made. For example, a watch manufacturer does not sell a watch (version 1.0) with only an hour hand, with the promise that later upgrades would add features such as a minute hand, increased accuracy, etc.

Applying this principle from Lean Manufacturing to software development means continually improving the way the software is developed. That is, Lean does not say to continuously refine the software (the product), only the process.

Rework is a QA term meaning work that has to be fixed or repeated because it was not done correctly the first time.

Hence the primary application of Lean ideas to software would be to improve that part of the process which far and away causes the most mistakes and rework - poorly understood, poorly communicated and continually changing requirements. Lean Manufacturing says to eliminate mistakes in the development process. In other words we must look at how to remove mistakes in the requirements.

By applying Lean Manufacturing principles to software development we must try harder to get the requirements right the first time. This is intended to avoid the enormous waste caused by repeatedly changing the design and the code. But this idea of BDUF is now generally regarded as futile.

BDUF?

BDUF stands for Big Design Up Front and is the name given by Agilists for the way that software is normally developed using waterfall methodology. Decades of experience and studies have shown that, no matter how much effort is put into the initial design, there will be a disparity between what users need and what the software actually does.I discussed this recently in Agile Design.

Agile Software Development

Agile takes the opposite approach to that of Lean Manufacturing. The assumption is that you can never get the requirements right the first time. There will always be mistakes.

The Agile approach is to start with the very basics of what you know, creating a minimal working initial version and then evolve the design based on feedback.

Software Development vs Manufacturing

Lean tries to take ideas from a manufacturing process and apply it to the software development process. However, these processes are very different - eg:

Manufacturing is about making many identical copies of a product. Every software product is different.

Manufacturing involves a repetitive process. Although there is some repetition in software development many things are unique to a project.

Eliminating even a small waste in manufacturing can make a large cost saving. The cost of eliminating waste in the software development process often outweighs the saving.

It should not be assumed that principles beneficially applied to manufacturing are similarly beneficial to the software development process.

Software Development vs Engineering

OK, software development is not like manufacturing, but isn't it like the design as opposed to the manufacture of products?

On the surface, this argument seems much more reasonable than the one above. A software developer is just another type of engineer; just as a civil engineer designs bridges or an aeronautical engineer designs planes, so a software engineer designs programs.

This is getting a bit off track but it seems relevant to clarify this here. Creating software is not like creating physical objects. There are important reasons when designing physical objects to get a whole lot of things right up front. For example, you don't want to be almost finished as skyscraper when you realise the foundations are inadequate. Software does not suffer (as much) from these sort of limitations and so can be easily evolved as required (subject to use of Agile practices such as Unit Tests). I discussed this at length in the Software is Different section of Agile Design.

The thing to remember: software developers are not engineers. The idea of "software engineering" has bugged me for many years and I will probably write about it in this blog soon.

Similarities

I should be clear that the QA (and Lean) share a lot in common with Agile. I touched on this in SQA and Agile Methodologies. Some examples are:

emphasis on JIT and deciding as late as possible

customer focus

automation of repetitive tasks

retrospectives and continuous improvement

importance of visibility of the process

quality is everyone's responsibility

empowering workers to make decisions

emphasis on training and learning

Despite that I do not believe some of the QA ideas typified by slogans such a "Right First Time" and "Quality Is Free" are always applicable. I might discuss this further in the future.

Conclusion

The first principle of Lean Manufacturing is to eliminate waste. This has been shown to work well for manufacturing. For a repetitive process designed to mass produce identical products, eliminating even the smallest waste can produce large savings.

However, this principle of Lean Manufacturing cannot be applied to the software development process. Every software project is different. The main type of "waste" that traditionally (and still commonly) occurs in software development is due to rework (of the design and the code) due to the requirements. To eliminate this waste the Lean philosophy says to improve the process - in other words try harder to get the requirements right from the start.

Agile takes the opposite tack. We start with the minimum requirements needed to get something working and evolve the design based on responsive feedback. In effect, Agile methods (especially Unit Tests) provide a means to deal with the "waste" in a far simpler and less costly manner than trying to eliminate it. This waste in unavoidable and trying to eliminate it is counter-productive.

This is not to say that Lean does not have anything to offer the software industry. In fact most of the principles of Lean are also shared by Agile. Even the philosophy of eliminating waste can be usefully applied to many areas of software development. But at their heart Lean and Agile are diametrically opposed.

Monday, 12 January 2015

Last week I talked about production line mentality. Now it's time to consider vendor mentality.

How are they related? In essence:

production line mentality - I just do what the boss/job spec. says

vendor mentality - I just provide what the customer asks for

Note that I did not invent the term vendor mentality (though I did make up the term production line mentality).It is apparently a well-known effect to which I was first introduced by a colleague - thankyou Rixt.

Vendor Mentality in Developers

In the software industry many developers and contractors suffer from vendor mentality. It is based on this simplistic approach: the client says what they want, the "software vendor" creates the software and the client pays for it. This works fine for (simple) manufactured goods. Creating software is usually much more involved than simply exchanging money for a product.

Here are some reasons:

the customer rarely knows exactly what they want

when the customer knows what they want, it may not be what they need

a customer will often specify what they see as the solution without really defining the problem or their objective

different customer representatives will have different understandings of requirements

what the customer wants will change as they understand their real objectives

what the customer wants will change due to external factors

what the customer wants will evolve as they try the software

what the customer wants can often change for no apparent reason

Vendor Mentality in Clients

Clients and customers can also suffer from vendor mentality - "just give me what I want". However, in my experience clients are far more amenable to feedback, negotiation and looking at the problem from a different perspective than developers believe. I have already discussed this problem on the customer side (see Customer Management) - which details how and why you need to educate your client if they have a tendency towards vendor mentality.

The Developer/Client Relationship

To overcome this problem both sides need to work together to establish a partnership rather than consider it a simple buyer/seller relationship.

developers need to ask questions rather than just take requests at face value

developers need to understand the business of the customer

developers often need to consider the customer's customer (see The Customer's Customer in Customer Management)

Mistakes happen. A great indicator of the health of the relationship between a software supplier and their client is what happens when things go awry. In an unhealthy relationship a supplier will try to cover up their mistakes, or (if that is not possible) excuses will fly or attempts made to shift the blame elsewhere. On the other hand, mistakes by the client are often not pointed out and the price is grudgingly born by the supplier.

This situation is commonly attributed to a difficult customer. Indeed, the customer's representatives may make unreasonable demands, but this is usually due a lack of understanding on their part. First, they may not appreciate the difficulties of the development process. More often the problem arises due to a customer who does not really know what they want. In this case you need to clarify, for everybody's sake, exactly what they are trying to achieve.

However, rather than the customer it is more often the developers, and particularly managers, that suffers from vendor mentality. They will insist on staff bending over backwards to accommodate the client, and do anything to prevent the company (particularly themselves) from looking bad. In such a blaming culture mistakes are punished which results in a large amount of effort spent in trying to deflect blame and in trying to avoid getting caught up in other people's problems.

Better than adopting a vendor mentality (and trying to cover up mistakes or deflect blame) is to be open and honest with your customers. They (even large companies) are almost always less demanding and overbearing than their suppliers believe, especially when they fully understand the situation. It may be up to you, as the developer, to help them to fully understand the situation as already discussed. But if a customer cannot accept what you tell them and insist then you may be better off without them.

A major problem is that many developers do not have the skills and personality to effectively interact with customers. They may be afraid to question the client and point out mistakes in their thinking. This is exacerbated in a culture of fear which does not encourage risk taking.

Sometimes it takes courage to do what is right but in the long-term the customer will thank you. Here are some examples of what you as a developer can do:

never agree to unrealistic deadlines

never agree to take on too many projects at the same time

never take a request at face value - ask more questions

if what the customer is asking for makes no sense then try to ascertain their objectives (see below)

simply say no if it is not in the interests of the customer (or the customer's customer)

don't be afraid to tell the client of their mistakes

admit your own mistakes

5 Whys

OK, so how do you handle a client that asks for a software modification that makes no sense? Often a client will state exactly how they think the software should work, making many assumptions. Developers need to look beyond the request and ascertain the client's objectives. A particularly good technique I have found to arrive at the root cause of a problem is the 5 Whys technique. (Thankyou Inbar for introducing me to this technique!)

The 5 Whys technique simply says to ask "Why" to the initial request then to the subsequent answers until you can get no further. You can typically ask "Why?" up to five times (possibly more or less) before you get to the root cause of the problem.

Consider this example (where C stands for client or customer and D stands for developer):

C: We want to ask and record why the operator has chosen the recovery operation.D: Why?C: Because operators are selecting the operation when it is inappropriate.D: Why?C: Because they do not understand its purpose.D: Why?C: Because they have not been trained on its use.D: Why?C: We don't have time to give all operators training before they use the software.D: Why?C: Because training is expensive and we only have courses twice a year.D: I understand. An alternative would be to move the "recovery operation" to the supervisor menu so only experienced users can select it. This assumes that all supervisors have been trained and understand its purpose.C: Yes, all supervisors have undergone training. That sounds like a great idea!

Despite its usefulness there are things to watch out for. If you use the technique on a
different client representative or group (or even the same representative at a different

The Wrong Approach :)

C: I would like you to launch off a surface and into the air by using the muscles in your legs and feet.D: What vertical dimensionality do you require?

time) you may get a completely different result. In my experience, it is quite surprising that a different set of answers, taking a completely different tack, often come to the same conclusion.

However, occasionally you can follow a different path and come to a completely different root cause. Of course, this is actually a good thing as it allows you to find different approaches to a problem, or even completely different problems.

Often asking the 5 Whys can reveal flaws in the client's understanding. The previous example may have instead gone like this:

C: We want to ask and record why the operator has chosen a recovery operation.D: Why?C: Because operators are selecting the operation when it is inappropriate.D: Why?C: Because it is too easy to select the operation.D: Why?C: Because it is presented as the first option on the sign-in menu.D: Why?C: Because that is the obvious place to put it.D: Why?C: We don't want people to sign in and use the system in a bad state.D: Why?C: That causes data corruption whence we have to restore the previous day's database.D: That is not the purpose of the "recovery". How often do you have to restore the database?C: It has recently been happening almost daily - on average once every 4 months per site.D: We need to look into the reason for this!

Conclusion

As a developer you need to work with your customer(s) to find out what they are trying to achieve. Ask questions, take an interest in their business, and try to see things from their point of view (or even their customer's point of view - see The Customer's Customer in Customer Management).

Don't let the customer dictate the solution to a problem. Hopefully, you are much more familiar with different techniques and technologies, as well as any technical limitations. On the other hand, you may need to understand more about the customer's business in order to choose the best solution.

Conversely, the customer needs to better understand the development process (see What Every Customer Should Know in Customer Management). Some customers may need persuading to work with you in a partnership, but you will probably be surprised how amenable most customers are to developing a mutually beneficial relationship with their suppliers.

Above all, be frank with the customer, even at the risk of offending them (for example, by pointing out their mistakes due to lack of understanding). Of course, this should be handled tactfully, but it is in everyone's best interests to ensure the customer understands the truth.

Equally, you should not hide your own mistakes. This will help to build a relationship based on trust and mutual respect. All parties should be honest and open to work together to come up with the most satisfactory solution.

Wednesday, 31 December 2014

At the start of this month I mentioned, in passing, what I called Production Line Mentality. This is the propensity for individuals working in a large organization to concentrate solely on what they have been told to do, ie, their "job specification". This behavior is promulgated by a multi-layered, controlling management structure.

Two people pointed out that what I was describing already had a name - Functional Silo Syndrome or simply Silo Syndrome - a term ascribed to Phil Ensor. Though Production Line Mentality and Silo Syndrome are very similar I don't believe they are exactly the same as I discuss later.

Since the discussion of Production Line Mentality created so much interest I thought I should elucidate and mention a few things that I did not have room for last time. Later on I will talk about how the Agile approach addresses it.

Production Line Deficiencies

Some say the production line is the most important invention of the industrial revolution. However, it has well-documented problems such as being boring, demoralizing, demeaning and, if the truth be known, tends to generate product of low quality. Luckily, in manufacturing the traditional production line has almost vanished in the last few decades due to studies (eg, see next) demonstrating better approaches. Unfortunately, in the software industry elements of the production line remain.

To show the problem, and the solution, I will first mention a study of a manufacturing production line. This occurred at a Swedish washing machine maker in the 1980's. The company switched one production line to a system where different teams assembled the machines from start to finish.

Originally with a production line of about 20 people (six machines per day), they reorganized into four teams of five people. (Of course, there were costs involved in re-tooling, training etc.) Each worker was given the skills and knowledge to assemble a major component or even the whole machine.

In brief, they found that productivity quickly increased to about 7-8 machines per day (and even more over time). Morale improved and less supervision was needed as individuals were motivated to get the work done and even suggest improvements that boosted their own efficiency. Importantly, the product had a marked improvement in quality. There were many other documented advantages such as less conflict and much greater flexibility when someone was absent.

Production Line Mentality in Software

OK, so the production line has problems, but what has that got to do with software development? Actually, quite a lot.

First, many developers are quite happy to potter along on their own problem with little or no interaction with others. By focusing on a small narrow task they lose sight of the bigger picture. To be productive all team members need to communicate and understand the organization's objectives.

Second, traditional software development methodologies encourage this behavior The Waterfall model divides the development process into a series of phases (much like a production line). The work is divided into specialized areas: business analysts, system analysts, designers, coders, various flavors of testers, technical writers, etc.

Finally, many large IT organizations are placed under a large amount of pressure by senior management. This often leads to a blaming culture and a controlling management style which is probably the largest contributing factor, as I discuss below.

Silo Syndrome

Although the Silo Syndrome has essentially the same causes and effects as the production line mentality it is usually described in terms of the behavior of departments or "groups" within a larger organization rather than individuals. In the interests of self-preservation a group becomes protectionist of their knowledge and uncooperative with other parts of the organization.

The original article on Functional Silo Syndrome by Phil Ensor beautifully explains this (see Functional Silo Syndrome). The following discussion of causes and effects of production line mentality generally also apply to the silo syndrome.

How Does It Happen?

The behavior starts as a response to an autocratic, controlling management style. As a defense mechanism the individual (or group) withdraws from any activity that involves risk. Some also restrict access to information, since knowledge is power. Of course, this is detrimental to the organization as a whole by stifling creativity, knowledge sharing, and collaboration.

There are many reasons for this poor management style. It could simply be due to an individual manager. More often it is due to the culture of the organization -- coming from the top. Autocratic managers are generally attracted to an organization with such a culture (but they can pop-up anywhere).

I can think of a few different scenarios:

Egotism

A common problem is a technically gifted individual who has been promoted to a position of power and attempts to micromanage all subordinates. Often they believe they know better than anyone working for them how to do the work. They only have people working for them because they don't have enough time to do everything themselves.

Their enormous ego leads them to believe it, but they generally do not know better than those actually doing the work. Even if it were true it is still demoralizing and demeaning to be micromanaged in this way.

Insecurity

A related problem is a manager that is insecure in their own ability. Surprisingly this may occur in concert with the above since the manager may project confidence but are worried that eventually others will see through the façade. By controlling information they can more easily cover up their mistakes or find someone to blame.

Self-preservation

One of the most basic instincts is for self-preservation. If a manager gives up control then their role may be seen to be less important or even redundant. By embedding themselves in the work processes they make themselves an essential cog of the machine. Also, as we saw above, an insecure manager tightly controls information to hide their own ineptitude.

This is perhaps the biggest impediment to addressing the problem. In a top-down, authoritarian structure, many layers of management are required to monitor and control everyone. If the people who actually do the work are empowered to make decisions then many of the layers can be reduced or even eliminated. Unsurprisingly, there is a huge amount of resistance to this sort of change.

Transference

If a culture of fear emanates from the top then it will permeate the entire organization. There is no advantage, and significant disadvantage and difficulty, in treating your staff well if you are treated badly yourself. In other words a manager cannot empower their subordinates if they are not themselves empowered.

Megalomania

It is rare but some managers simply enjoy wielding power to the point that it is almost sadistic. Luckily, I have only ever encountered one such manager who would deliberately inconvenience or even humiliate those that worked for him.

Ignorance

Finally, there are those managers who don't know any better. Either, they are inexperienced and are following the lead of others or this is the way they have always worked and are not open to any other options.

How Does It Work?

In most organizations, production line mentality results in a large hierarchical management structure. Each manager is given strict control of a group which is split into sub-groups each with their own manager, and so on. This hierarchy can contain many layers until you get down to the people at the bottom, who actually do the work.

Information flows vertically. If you need information from someone in another section this is done through your supervisor. Information is distributed by a manager on a "need to know" basis.

Rewards are doled out by management based on the perceived value of each individual. Inevitably this fosters competition and rivalry, which kills any sort of collaborative spirit. Individuals expend effort making themselves look good (and even try to make their co-workers look bad) to their boss, rather than focusing on the goals of the organization.

When something goes wrong it's easy to point the finger at the person responsible. However, the root cause of the problem is usually that the person who made the mistake was not given enough information or assistance to do their job properly.

Negative reinforcement (ie, punishment for mistakes) is the primary motivator, rather than positive reinforcement. This leads to an aversion to risk where nobody wants to take responsibility for anything beyond their small area of influence.

Indicators

Many people working in this sort of environment are not even aware of it or just accept it. This is often because they have have never worked for a more enlightened company. Here are some possible clues based on the work environment:

there are more chiefs than Indians - ie more managers than workers

monetary rewards are given based on individual performance reviews

each position has a detailed written job description

you often have to sign legal documents such as non-disclosure agreements

requirements documents must be signed off by several people

developers are not allowed to talk to customer representatives

Here are some clues that your boss is the problem:

you don't really know what your boss does

you can never find your boss or arrange a meeting

you are not given enough information to do your work properly

you have to ask for a (deserved) pay rise

you cannot send emails to the client without clearing it with your boss

your boss threatens to fire you when you express any sort of dissent

you get into trouble for swapping tasks with a co-worker

when you book a meeting with your boss she postpones or is late

if your boss books a meeting with you and you are late she hits the roof

all emails you send must be CC'd to your boss

you cover up a problem rather than tell your boss

you discover there was a meeting (on a subject crucial to your work) to which you were not invited

you are asked to work overtime when there is no reason to do so

your boss keeps reminding you about the poor job market

your boss never admits they made a mistake

your boss keeps reminding you of your own past mistakes

Here are a few general clues about an organization with a poor culture:

you have never met anyone who uses the software that you write

everyone knows about a problem that needs fixing but nobody is willing to tackle it

there is often conflict in the workplace

many problems occur - the same problems happen over and over

people in the organization have stress-related health problems

you are regularly given large design documents about projects you have never heard of

two developers discover that they have both fixed the same bug

If you are a boss, here are some clues that you might be the problem:

you get the feeling that you don't know the full story about what is happening

the people who work for you show no initiative

the people who work for you are incompetent

the people who work for you are risk-averse

you don't know the name of everyone who works for you

there are people that you have not talked to in over a month

you have an "open door" policy but nobody takes advantage of it

the client asks for a simple change to which you give an estimate, but are shocked when the team cannot deliver

Consequences

By now I hope you are convinced of the detrimental effects of production line mentality (and silo syndrome). To reiterate it makes for an organization with little collaboration and creativity to solve problems. Insular thinking results in inefficiencies such as poor processes, redundant processes and bad decision making.

The effect on morale results in poor employee engagement, absenteeism and sub-optimal productivity. Moreover, because managers build themselves into the system they become a single point of failure which affects flexibility and resilience.

Prevention

So how do we prevent this from happening? The simplest way is to follow an Agile methodology. For example, in Scrum the team is self-organizing, so resist the urge to "help" them get organized. Another piece of good advice comes from XP (Extreme Programming) to "eliminate fear".

The most important point is for managers to empower their charges. Don't assume they will get it wrong. If they don't do it the way you would do it, it may actually be a better way. If you trust people they will usually honor that trust.

Scrum is great for counteracting "production line mentality". Allowing the team to self-organize promotes interaction (within the team) which results in better organization, processes and products. It is also good for morale due to team members having a feeling of making a contribution and working together toward a common goal. In particular, the daily standup allows everyone to know what everyone else is doing, with improved coordination and less mistakes.

But does Scrum inhibit collaboration between teams? The simple answer is - it shouldn't but it can.

When different groups do not collaborate this is known as the "silo syndrome" as we saw above. Sometimes Scrum teams have this problem due to a misunderstanding of the roles on the product owner and scrum master. Let's first look at the problems these roles are designed to overcome.

Product Owner

Even in (or especially in) an open, collaborative environment there can be a problem of conflicting and volatile requirements. There are often many people (users and other client representatives, managers, business analysts, etc) with input into how the software is to be modified. One manager might insist on all bugs being fixed as a high priority, another might want to sneak in their favorite feature or improve an existing feature -- you have probably seen this sort of thing yourself. Different people have different ideas and priorities; and these ideas and priorities tend to change often.

In Scrum, having a product owner and sprint backlog avoids this problem by having all requirements come through one person. This ensures that the developers are not pulled in many different directions at once. Further, because the sprint backlog is unchanging they know that things will not suddenly change, at least until the end of the sprint.

However, having a product owner does not in any way preclude the team from talking to people outside the team. In fact this sort of collaboration is encouraged. It simply means that all requirements come through the PO and are prioritized in the backlog.

Scrum Master

The scrum master is in many ways an interface between the team and higher-level management. The main task of the scrum master, outside the team, is to remove roadblocks. This involves them using their influence to obtain resources, help or information. Some scrum masters take it upon themselves to intercept interference from outside the team, such as a senior manager who wants to control the team. Their are always managers who think they can organize the team better than it can organize itself.

In summary, an over-zealous scrum master can turn a scrum team into a "silo".

Finally, a Scrum Team can be unfairly perceived as a silo. A team should be composed of all the persons required to accomplish their tasks from start to finish. This means that they may not require a lot of communication with people outside the team.

Conclusion

Production line mentality (and silo syndrome) begins with a culture where personal interest (or the interests of a group within the organization) is not aligned with the organization's best interests. Fear prevails and individuals are driven by self-protection (or protection of their department) rather than desire to do the best for the organization as a whole.

This culture is characterized by many layers of bureaucracy which is a classic example of the DIRE principle. Employees show no initiative and just wait to be told what to do. Risk-taking is discouraged since mistakes are punished.

The solution to this problem is to empower employees to take control of their own destiny. Encourage collaboration and discourage rivalry by rewarding the team and not individuals. Share the vision of the organization so that everyone is working towards the same end.

Generally, following the ideas behind Agile will encourage collaboration and work against the production line mentality (and silo syndrome).

[Note: I was also going to talk about the closely related problem of vendor mentality, but that will have to wait for next week.]

Monday, 1 December 2014

Back in May I wrote about how software design is done in Agile. I received some feedback which I promised to respond to in a future post (this one). Sorry for the delay.

Someone asked about the role of designers in Agile. They thought I was implying that specialist designers were not needed. Well, just to be clear, I will say it outright: No dedicated designers! All or most of the team should be involved in the design though typically some will have more input than others.

"No dedicated designers!"

When moving to Scrum people who just do "design" need to stop it. Designers traditionally are either technical (sometimes called architects) or business-oriented (sometimes called analyst/designers). I talk below about how these (and other) roles may change when moving to agile but in brief designers can move in two different directions - either to more development work or to more of a PO type role. Architects often become more hands-on - spending more time coding and reviewing. Analyst/designers can move back to development or if not from a technical background will need to move into a role involving mostly acceptance testing and customer interaction.

Another comment was: "If design, programming (and testing) are to be done together then should they all be done by the same person?" In essence: YES. But before I get into that I need to talk a little about roles in Agile teams.

Multi-skilled Teams

Everyone agrees that Agile development uses multi-skilled (or cross-functional) teams. The problem is nobody really agrees (or even pretends to know) exactly what that means. Does it mean each of the team members have the skills to do all, or at least many, of the required tasks? Or does it mean that altogether the team has the required skills but each specializes in their own skill-set?

In my experience most Agile "experts" use the latter definition. For example, I was recently involved in the implementation of Scrum in a large company, and one of the things that the Scrum consultants emphasized is that a Scrum team (ie, "pigs" not "chickens") is made up of members with a wide variety of skills. However, I get the strong impression that this is mainly to placate those employees who feel that they may be redundant in the new order of things.

In reality, in an organization suffering from production line mentality (see below) adoption of an Agile methodology will mean that there will be less need for some existing roles. Less BDUF (see my post on Agile Design) means that there is no need for dedicated designers as design is performed continuously by the developers. Also, developers take more responsibility for testing so there is less need for dedicated testers doing black-box testing (like regression tests). This means that developers have more responsibilities (rather than just coding), but this is a good thing as it promotes DIRE (see DIRE) but also makes the work more rewarding and interesting.

If you need less people for other roles, such as designers, does this mean you need more developers? The simple answer is no. In my experience the same developers can actually be more productive, so you may even need less developers too!

Here I should note that I am talking about using Scrum in a pure software development environment.

I know that Scrum has been used successfully in other environments and that is great. However, whatever the environment all team members must have enough shared context to understand what everyone else in the team is doing.

Scrum Team Composition

Ideally all members of a Scrum team should be developers. That is, team members (apart from the Product Owner and perhaps the Scrum Master) should have the ability to write code. To understand why I will first recap on the advantages of Scrum.

Scrum (and Agile in general) has many benefits such as promoting team-work, working towards a common goal, quick response to change, etc. This is achieved by good communication and empowering the team to make their own decisions. The daily standup is particularly important as it allows everyone to know what everyone else is doing.

When everyone in the team can (and does) understand what everyone else is doing this avoids issues like:

differences in understanding of what is to be done

poor division of work

duplicating work - such as two developers fixing the same bug

forces pulling in different directions and introducing conflicting changes

not realizing when someone needs help

not being able to have someone else take over a task due to illness

not being able to have the whole team focus on a critical problem

inability to find another team member who understands what you are doing well enough to perform a review

Unless the team members are all developers they cannot really understand what everyone else in the team is doing (and how, why, etc). This is essential for Scrum to work (see Why Scrum Fails).

Of course, different individuals will have different levels of experience and different aptitudes. Over time, team members become familiar with each other's abilities. It is easy and natural for team members to take each other's skills into account when allocating tasks during the sprint planning.

Production Line Mentality

The production line was a major invention in the history of industrialization, since even unskilled workers can become very good at a simple repetitive task. However, its drawbacks were quickly discovered. Workers suffer from frustration and alienation. Once a task is mastered boredom sets in and mistakes are made, even for simple things.

Quality and productivity are greatly improved when workers are given a variety of challenging tasks. Performing different jobs as part of a small team working towards the same goal can be a tremendous motivator.

Another problem with production line mentality is that it usually results in post-production testing - ie someone inspecting the product coming off the end of the line. This sort of "Black-box testing" is not a good way to add quality particularly for software (see Unit Tests - White Box Testing).

What I call "Production Line Mentality" is where everybody has a specific well-defined job specification. This is common in an organization with a top-down management style where a worker's purpose is to impress their boss rather than work with others. In other words it does not encourage people to work together to accomplish their team's goals.

It is caused by a controlling (rather than empowering) management style where each role is kept simple to be easily understood and hence controlled. However, it results in a large number of specialized positions and a multi-layered management structure to be able supervise everybody in the organization.

This discussion actually ties in nicely with some of my recent posts. First, multi-layered management is a classic example of the Layer Anti-Pattern which is a pattern of bad software design which I have found can be applied in other domains.

Moreover, these layers form an excellent example of the DIRE principle. First, the managers at the top of the hierarchy are typically completely out of touch with the people actually doing the real work in the organization as they are isolated from them by all the layers. Perhaps even worse is that it encourages the workers themselves to concentrate on their own specific task without regard for the goals of the broader organization.

In an Agile development environment the developers are empowered to do whatever they need to do (within their bounds) to finish their tasks by the end of the sprint. For a traditional controlling manager it may be hard to deal with an environment where you can't even understand what everyone is doing let alone control it.

A controlling manager can be a real problem for the Agile approach. On the other hand, having developers who are happy to (or used to) being controlled like this is another major obstacle for the adoption of Agile. Years of manipulation (and even threats) means that many are happy simply to be given work as outlined in their job spec.

Traditional Roles

Let's look at traditional roles and how they change in an Agile world.

Team Leader

Scrum books simply say that the team leader role is no longer necessary. The team self-manages so the need for a manager within the team ceases to exist.

I disagree with this. In fact most of the same books describe a role for the Scrum Master which is almost exactly what I have always thought a team leader should be. (Perhaps they like the different title to emphasize that many team leaders are using a bad style.)

The team leader is now called the Scrum Master. Many team leaders also write code. That's OK too - you can have a part time scrum master who also codes.

However, if you think a team leader/scrum master is there to tell the team members what to do, then forget it. One of the main advantages of Agile is that teams are self-organizing and self-managing (within the bounds set for them by the PO and higher management). Small self-organized teams generally work much more efficiently and harmoniously than "centrally-organized" teams.

Designer

There is no longer any room for analysts/designers who spend weeks or months understanding a problem and trying (unsuccessfully, of course) to come up with the perfect, well-documented design. In an Agile world analysis and design is performed "just in time" (see JIT). The developers design, code (and test) all at once. This is not to say that what is designed is left to the whim of the developers - in fact how the design evolves is more closely controlled through the customer representative (product owner).

Traditional designers can move toward a role with industry knowledge, in particular how the software will help the business (or even the customers of the business). Perhaps they will be the actual PO (product owner), though I prefer someone even closer to the customer than that. Probably they will become a "business analyst" assisting the PO. Depending on the software (and despite the Agile preference of JIT design) there may be a need to mock up designs slightly ahead of time to garner feedback from the PO and/or customer.

Alternatively, or also, they may also do a lot of Acceptance Testing (see below). Whatever they do, their main focus (as for any Scrum team member) should be on completing the sprint backlog before the end of the current sprint.

Architect

There are just two points I want to mention about software architects using quotes from the excellent book 97 Things Every Software Architect Should Know.

" [the architect] should not sit in an ivory tower dictating the way forward but should be on the ground working with the team "

- from 19. Architects must be hands on by John Davies

"As well as being prone to incidental complexity and inertia, designing large systems upfront means larger projects, which are more likely to fail, more likely to be un-testable, more likely to be fragile, more likely to have unneeded and unused parts, more likely to be expensive, and more likely to have a negative political dimension. ...

"Design the smallest system you can, help deliver it, and let it evolve towards the grand vision. Although this might feel like giving up control, or even shirking your responsibilities, ultimately your stakeholders will thank you for it."

- from 97. Great software is not built, it is grown by Bill de hOra.

In brief: architects work with the team to evolve the design, spending most of their time coding and reviewing.

Programmer

In an Agile environment programmers now do everything that is required of the team (analyze, design, code, test, etc). They do whatever it takes to complete the sprint tasks by the end of the sprint.

This is in stark contrast with some traditional environments where programmers have just been coders - they take a design from the analyst/designer and code it for "QA" to test. This is demeaning for programmers who are usually capable of a lot more. Even if you care not for the morale of developers giving the programmers free reign to think about the design and even experiment at little will produce a much better design than any analyst/designer (no matter how clever) can invent in their head.

Of course, this does not mean the programmers can do whatever they want. They are closely guided by the product owner who, after all, prioritizes the backlog effectively deciding on the sprint task(s) that are the focus for the team.

Tester

If waterfall development is disempowering to programmers it is much worse for testers. As compensation, testers are often given the impressive title of quality assurance experts, despite the fact that quality assurance has little to do with testing.

You may not believe this but using Agile development (especially Unit Tests) there is a greatly reduced need for dedicated testers. How great this reduction is depends to a large extent on how much testing the developers do already. At one extreme I recently worked in an environment where there were twice as many testers as developers! The software contractor had there own tester(s) who worked closely with the coders. Then the customer had several testers, plus an "offshore" test team to perform regression testing, as well as an automated testing team.

There are many different names for the types of testing (and testers) that are used in the software industry. In Agile there is just two general types of testing:

Unit Tests: automated tests written by the developers of the code to test individual (low and high level) modules

Acceptance Tests: performed by the PO, customers, or anyone who wants to try the software

Note that Unit Tests are always automated, but acceptance tests may be automated too. Often the developers will add a simple scripting system to make it easy for the PO and others to create test scripts.This is useful for testing the current changes. But don't go overboard and add lots of tests for existing functionality as they will most likely fail when changes are made (and even act as a disincentive to change). Leave the regression testing to the Unit Tests.

To understand this better consider the testing that an aircraft manufacturer does. First, individual parts are developed (or purchased). The developers of these parts (for example, spark plugs) will design, build and test them themselves and provide a hopefully reliable product. Some parts may be made from a combination of other lower-level parts, but it is still up to the developer of any part to thoroughly test it. Of course, larger, more complicated, parts may require more elaborate testing - testing of an aircraft engine may require a huge test rig. In the end it is always up to the maker of a part to ensure it performs to the requirements of the plane manufacturer. Testing of parts is the equivalent in software development of Unit Testing.

When the first plane (or a prototype) is finally built it is then up to a test pilot to perform various flights and manoeuvres to ensure that the plane performs as expected. This is the equivalent in software development of Acceptance Testing. This analogy is not perfect though, since Acceptance Testing of software should usually be performed much earlier than test flights.

If a Scrum team requires a dedicated tester at all then perhaps one or two members (at most) could be required for Acceptance Testing on behalf of the PO. They should spend much of their time understanding what the software is to do by discussing it with PO, business analyst, customer and any other stakeholders - something that an analyst/designer traditionally does. They might also attempt to automate much of their work using acceptance test scripts.

Whatever, they do their main focus should be producing working software at the end of the sprint. This means giving the users what they need as opposed to testing for conformance to any written specification.

Conclusion

If you are about to, or have been trying to, move toward an Agile development environment then you should ignore the advice given by many Agile coaches and experts that most roles will not change. There is no longer any need for software designers and less need for testers. Instead the whole team is involved in analysis, design, coding and testing - and this is all done "just in time" (see JIT). (Of course, if they are willing to adapt a role can be found for anyone -- it may be more challenging but will probably involves better use of their abilities.)

Most importantly, managers have less control of the team, which is now self-organizing. The major benefit is simply that a creative team can organize themselves much better than any manager. It also makes for a more rewarding and enjoyable working environment. The developers concentrate on creating good software and overcoming hurdles rather than making themselves look good for their annual performance review or trying to shift the blame when something goes wrong.

It may be hard to overcome the unthinking production line mentality (I do what the boss tells me to do) or vendor mentality (I do what the customer tells me to do). But please try!

Tuesday, 30 September 2014

You have probably heard of the "Agile Manifesto". This was designed to enunciate Agile's fundamental approach by those who coined the term "Agile" software development (see Agile Manifesto).

Favor individuals and interactions over processes and tools

Favor working software over comprehensive documentation

Favor customer collaboration over contract negotiation

Favor responding to change over following a plan

Elements, Principles, ???I'm never sure what to call the Agile Manifesto's 4 main statements. They can't be principles as there are also 12 Agile Principles (see Agile Principles).I have seen various names, such a guidelines, values or statements. I think Elements is the logical choice as it emphasizes their fundamental importance.

For a long time now I have felt there is one thing missing from the above. It is at least as important as a couple of them. I call this Agile's Fifth Element.

Favor simple design over reusability and generality

What is an Agile Element?

To see why we it should be added to the other four we need to decide how a statement qualifies to be an Agile Element. In my opinion an Agile Element needs to:

challenge, or at least de-emphasize, (previously accepted) best practice

be fundamental to the Agile approach

be important enough not to simply be considered an Agile Principle

There are many books on software project management typical of the Waterfall era (mainly written from the 1960's to the 1980's) that emphasize the importance of following strict procedures, having everything well-documented, and having a water-tight contract with the customer. Most of all they talk about the importance of planning a project down to the smallest detail.

I won't go into why these ideas don't work (again), though the authors make them look very good on paper. Luckily Agile came along and turned all of this on its head. The four elements of the Agile Manifesto clearly highlight what was (and often still is) wrong with what was considered the correct way to develop software.

How Does it Qualify as an Agile Element?

Let's look at how the proposed fifth element satisfies the above criteria by looking at each criterion in turn.

Does it challenge accepted best practice?

The big paradigm to come out of the 1980's was the emphasis on reusability. Many in the Agile world now think that this emphasis was a mistake as I previously explained in Reusability Futility.

In brief, experience has shown that too much software is designed to be flexible and reusable for no purpose. This leads to increased complexity to the detriment of other attributes such as understandability, verifiability, efficiency, etc. The worst effect is that software becomes harder to maintain and cannot adapt to change.OK, but why is it fundamental to the Agile approach?Keeping things simple is part of the agile approach. In fact the 10th Agile Principle says:

Simplicity -- the art of maximizing the amount of work not done -- is essential.

Simplicity is necessary in order to create small, frequent software releases - a major part of the Agile approach. Further, keeping things as simple as possible allows the software to more easily evolve, since unnecessary constraints interfere with continual refactoring -another fundamental feature of Agile development.

designing for flexibilityadds complexity

Of course, you may ask: can't you design software to be both simple and flexible? Sometimes you can find a design that is simpler and more flexible than any alternative but usually there is a trade-off, particularly as modules become larger. The overwhelming general experience is that designing for flexibility adds unnecessary complexity.

Is it really important enough to be the Fifth Element rather than just a principle?Yes, it is! It counters one of the biggest problems currently facing software development - the tendency of developers to over-design and add unnecessary code. There are lots of reasons this occurs such as:

temptation to add something now which is "sure to be needed later"

desire to use some new design pattern that one has recently been read about

If you're still not convinced I recommend reading Chapter 18 of the excellent book 97 Things Every Software Architect Should Know. It is written by Kevlin Henney and entitled Simplicity Before Generality, Use Before Reuse. It says exactly what I have been trying to get people to understand for many years.

Why Is It Missing?

You would have to ask the authors of the Agile Manifesto why they left it out. I suspect that its importance was simply not appreciated at the time.

I guess the first four elements highlight problems with best practice that have been promulgated since the 1960's. The proposed fifth element highlights the problem of something that only became considered best practice later (1980's) - that is designing software for reusability.

Another theory is that many of the creators of the Agile Manifesto were heavily involved in OO (Object-Oriented) technologies. To de-emphasize reusability would be anathema to them, since reusability is a fundamental part of OO design.

A Different Fifth Element Proposal

As I write this post I have discovered that our dear old friend Uncle Bob (Martin) has also proposed an additional element for the Agile Manifesto:

Favor craftsmanship over execution

In essence he is saying that many developers are only concerned with getting the code working without regard to the quality of the code they produce. I agree with Uncle Bob that this is a serious problem that affects much, if not most, software development.

But Uncle Bob's proposal, though eminently worthwhile, does not qualify as an Agile Element for the reasons mentioned above (see What is an Agile Element?).

First, it does not question accepted best practice since software quality experts have been emphasizing the importance of various quality attributes of software (such as maintainability) for decades. I also do not believe it is fundamental to the Agile approach, though many Agile practices, which could be considered as promoting craftmanship (such as pair programming), will improve code quality.

Summary

My proposal is for an additional element to be added to the existing four elements of the Agile Manifesto. I consider this new element to be a least as important as a couple of the big four - personally I would rank it third. I believe it to be fundamental to the Agile approach as it is practiced today. For example, many practitioners of XP (Extreme Programming) talk about the importance of YAGNI and simple design (see Simplicity is the Key). Similarly in Scrum, developers focus on solving current customer needs by concentrating solely on the most important backlog items.

... software is designedfor general purposeand, ironically,ends up being fitfor nothing at all!

The problem of unnecessary generality and reusability is largely unrecognized, but significant and growing. I believe it is mainly due to the training and mentality of developers, particularly software architects. Perhaps with the best intent, software is designed for general purpose and, ironically, ends up being fit for nothing at all!

Far better is to start with a simple design that can be easily modified (with the assistance of Unit Tests) into whatever the future requires. Without the burden of unnecessary complexity the software may be able to evolve forever or at least not become a "legacy" application almost immediately as most software tends to do.

So do you think this is a worthwhile proposal? I would be interested to hear your comments.

Sunday, 25 May 2014

There is one aspect of Agile which people have trouble coming to grips with. I talked a little about this recently when I mentioned JIT design (see JIT). You may have heard it called other names like evolutionary design, incremental design, emergent design, continuous design, etc. I will use the term Agile Design or JIT Design.

Some people are adamant that designing like this is fraught with danger or just doesn't work. However, there are many, many projects where it has worked better than anyone expected and certainly better than the Waterfall approach of Big Design Up Front (BDUF). For example, see Continuous Design by Jim Shore.

The opposition is simply due to not really understanding how design is done in Agile development. In particular, using JIT design without using other aspects of Agile methodologies such as regular feedback, collaboration, continuous integration, collective code ownership, etc, will doom it to failure. And as usual I will mention that Unit Tests are crucial (see Unit Tests).

How it works as part of an Agile methodology is hard to understand so I will use an analogy. Hopefully, this allows you to understand my point.

An Analogy

I will use the analogy of designing a house. I chose this analogy as you probably already have an idea on how that's done.

Houses are built from a plan. Often an architect has spent many weeks or months creating a design on paper. Even the worst houses are built from some sort of floor plan. (In fact, in my country, government regulations preclude building without a detailed plan that has been approved by the local council.) The point is that for any house, a lot of time is spent in design and planning before construction can even begin.

Over many years (even millenia) this has been refined to be the best way to build a house. In fact this is essentially the same way that all complex physical devices and structures are built - some sort of designer, architect, or engineer (or team thereof) creates a blueprint which is carefully perused, verified, and approved before the final product is actually built.

If you just went and bought some materials and started building a house do you think it would be anything other than a disaster? Of course not!

So why do I think that designing and building software should be any different? The simple fact is that you can't build a house or software in the conventional manner without a fairly detailed design. However, if you change the way you build the software things are different. This is why Agile software development is considered to be a methodology - it only works when you understand and embrace it, in toto.

Let's see how we might build a house in the Agile manner. Note that, in the discussion below, D stands for developer (the house creator) and C stands for client (the house user).

From this ...

D: What is your most basic requirement for the house?
C: I want something that stops me getting wet when it rains.
D: Try this. [A piece of corrugated iron on four posts.]
C: I keep bumping my head on it.
D: How's this? [The corner posts have been extended.]
C: Better. What about something to stop the wind.
D: Try this. [Four walls attached to the posts.]
C: It's less breezy but I can't get out.
D: I could add a door. ...

to this!

This continues until C has their dream house (or they can't afford to pay for any more changes).

Software is differentI hope you realize that I am not advocating this way of building houses! This would be a particularly slow and expensive way to do so. However, when building software this can be a better way (and I would argue almost always is). Building software is different for these reasons:

1. Modifying physical objects (like extending the above-mentioned corner posts) can be time-consuming and expensive. Software is far more malleable. To make something larger may be a simple matter of changing a constant and recompiling.

2. Continually changing the design of anything of any complexity quickly becomes a nightmare of managing and controlling the interactions. With software this is a little easier because you do not have to worry about physical interactions (such as the location of pipes and wiring). However, it is still important to decouple different parts of the software whenever there are no major drawbacks to doing so (see DIRE). Crucially, Unit Tests can be used to ensure nothing is broken as the software is continuously redesigned and refactored.

3. House design is well-understood and there are many standards and conventions that are well-known, proven and followed. Software design is far more variable. Moreover, the client does not have a very good idea of what they want. Even if they did the developers do not have a very good idea of how to give it to them until they try to build it.

4. For practical reasons design constraints for houses do not change much if at all. For example, the government is not going to one day say that all electrical wiring in all houses is to be changed immediately. (Even if there were an urgent safety issue this would be logistically impossible.) However, these sort of major changes are regularly thrown at software developers. BDUF cannot easily cope with change the way continuous design does.

The Argument For BDUF I have often read the blog of Joel Spolsky, whom I respect and admire greatly. However, many years ago I read a post which saddened and disappointed me. Part of Joel's blog is even quoted on the BDUF page on Wikipedia as an argument in favour of BDUF.

However, if you carefully read the full paragraph from which the quote is taken, Joel's example is actually an excellent argument for Agile Design. Here is the full paragraph:

"Many times, thinking things out in advance saved us serious development headaches later on. When I wrote the first draft of this spec, I had a more complicated flowchart that assumed that either party (helper or victim) could initiate the process. This became extremely complicated and convoluted for various reasons. As I worked through the screens that would be needed to allow either party to initiate the process, I realized that Aardvark would be just as useful, and radically simpler, if the helper was required to start the whole process. Making this change in the spec took an hour or two. If we had made this change in code, it would have added weeks to the schedule. I can’t tell you how strongly I believe in Big Design Up Front, which the proponents of Extreme Programming consider anathema. I have consistently saved time and made better products by using BDUF and I’m proud to use it, no matter what the XP fanatics claim. They’re just wrong on this point and I can’t be any clearer than that."

- Joel Spolsky, 2006

The first thing that hit me is the mention of the first draft of the spec with the "complicated flowchart". It is obvious that the first draft is already suffering from the problems commonly associated with BDUF. In particular, it is clearly making unnecessary assumptions.

Flowcharts

Does anyone use flowcharts anymore? Back in 1979 I was told by my lecturer to avoid flow charts (probably due to their association with the evil goto). Personally, I have found UML activity diagrams (and sequence diagrams) to be more useful.
But using Agile Design obviates the need for lots of diagrams.

Luckily (for Joel) he had the experience and insight to be able to identify a problem in the spec and fix it before it made it into the code. Apparently in his development environment code was hard to modify since a change to the spec that takes an hour or two would require weeks of code changes. (Note that this is another symptom of not using Agile techniques such as Unit Tests.)

Now imagine if the designer(s) of the software had less time, less experience or less insight than Joel. They may have gone with the first draft of the spec with the consequent later weeks of recoding that this entailed. This is plausible as I doubt many designers are as experienced and insightful as Joel. So rather than demonstrating why BDUF is better than Agile, Joel's example clearly shows its weakness.

Now consider the Agile approach, which is to first create a minimal working version of the software. In this initial version it might not have even been obvious who (helper or victim) could initiate the process. Users would try the software and provide feedback which is used to evolve it into what they need. In this way the conundrum of who should initiate the process would probably not even have arisen.

Another thing to note is how Joel simulated the use of the software in his own mind by "working through the screens". Not many people can do this effectively. It's always far more effective to have working software than try to imagine how the software would work. (Note that this is another advantage of Agile element 2: Favour working software over comprehensive documentation.)

Finally there is one thing I need to be absolutely clear on: thinking things out in advance is not the only way to avoid serious headaches later on. And most of the time it's not even the best way.

Problems of BDUF

The plain fact is that for more than 50 years BDUF has been tried and it consistently fails. Even for simple, well-understood projects, no matter how hard you try, it's impossible to produce anything approaching a good design without attempting to implement that design. I have seen this time and again. Trust me on this.

"What?" I hear you say. "You can't say that without an explanation." Well, I can if I want to, so there! Actually, I don't need to, since it has been discussed elsewhere. For example, see the Wikipedia page.

I have also touched on the subject many times in previous posts in this blog. For example, it is the principle cause of the Gas Factory Anti-pattern. More recently, I have talked about it in the series of posts on Unit Tests and specifically in the Cost of Rework section in JIT.

But just because you asked so nicely, here is a summary of recognized problems with creating a detailed specification up-front:

the customer often does not really know what they want, at least till they try it

it does not adapt to changing requirements

it tries to anticipate future changes, adding unnecessary complexity

it tries to document every known "fact" (of which many are usually incorrect)

it makes unnecessary assumptions about how the design will be implemented

it is very hard to foresee problems

many things (often major things) are overlooked

it takes too long to create - stakeholders become worried when there is little sign of progress

it separates design from coding, even though they are very closely related (see DIRE)

it results in a contorted implementation - it's often easier to change the code than fix the spec

the document is not updated when the inevitable changes are made

There are probably more, but those are the ones that occur to me now.However, now that I think about it there is one major drawback that I have not seen discussed. This is that developers don't really come to a full understanding of what the software is supposed to do until they actually try to implement it. It's only then that logical inconsistencies, omissions and other mistakes in the design become obvious. Further, limitations (unknown to the designer and often even to the developers) only become apparent when you attempt to write the code.

Possibly most significantly, only when the developers implement the design do better ways to do things occur to them. I have lost count of the times with BDUF when I have finally completed the code, only to then think of a much better solution (simpler, more efficient, more usable, more maintainable, etc).

Agile Design

The Agile approach is only to design what is needed for the next feature or iteration. This is usually the smallest possible addition to the system. The designer should only concentrate on the current iteration and never consider any possible future changes. This is a lot harder than it sounds. (I do concede that occasionally there are exceptions to this rule which I will discuss later.)

Since past iterations did not look to the future, a particular new feature may completely invalidate a large part of the existing design. Don't panic! This is normal. Traditionally such frequent refactoring would create chaos, since every iteration would have the distinct possibility of breaking existing functionality. Without complete regression testing after each iteration bugs would creep in and not be found till some time later

This is why Agile Designrequires use of Unit Tests. Unit Tests allow continual redesign and refactoring to be performed without fear of breaking existing functionality.The advantage of the Agile approach is that the closely related activities of design and coding are done consecutively and, to some extent, simultaneously. As I discussed previously in DIRE separating the design and coding leads to the major problems of Waterfall development.

The one huge benefit that I have found is that, after developing earlier features, the developers are very familiar with how the software works. This greatly improves the chances of a better design being developed for the next iteration. Mistakes and omissions are far less likely and a good (simple, DRY, etc) solution is much more likely.B2DB Syndrome

Every programmer experiences the B2DB (Back To Drawing Board) syndrome when they are first learning to program. This is where one tries to write some code but in doing so obtains a better understanding of how to do it. The consequence is one has to go back to the drawing board, often many times. (I discussed this in more detail in JIT Testing).

For many this is a very painful experience and they believe there must be a better way, especially (as is often the case) they have already been trained in another field (such as a branch of engineering) where it is essential to do up-front design.

I believe this is the real reason for much opposition to Agile Design - it feels like the bad old B2DB syndrome.It's true that there is a lot of refactoring going on in Agile Design, but this is actually an advantage. Of course, more experienced developers are going to have a better initial design, but even the best will see better ways to do things once they fully understand what they are trying to do. Also (as all the Agile experts emphasize) it makes the process much more amenable to changing requirements.Finally, at the risk of sounding like a broken record, I will mention that Unit Tests make the refactoring safer and easier. Rather than having the feel of dread of B2DB, you can actually learn to enjoy refactoring and the process of creating the best design you can find.

Summary

only make the simplest possible enhancement that will work

forget the large design document with lots of UML diagrams

use the code (especially Unit Tests) as the "documentation"

don't design for the future, concentrate on what is needed now

don't make the code flexible, just write for the specific task at hand

try to forget all the extra (irrelevant) information that has been given to you

don't be afraid to refactor existing code

use Unit Tests to allow changes to be made without fear of breaking anything