Featured in DevOps

Adin Scannell talks about gVisor - a container runtime that implements the Linux kernel API in userspace using Go. He talks about the architectural challenges associated with userspace kernels, the positive and negative experiences with Go as an implementation language, and finally, how to ensure API coverage and compatibility.

Big Ball of Mud, Still the Most Popular Software Design

Big Ball of Mud, is a code jungle which is haphazardly structured, sprawling, sloppy and connected by duct-tape. Over the years we have been introduced to various guidelines such as SOLID, GRASP and KISS amongst age old, high cohesion and low coupling to deal with this Mud. However, the situation still remains bleak and Big Ball of Mud still seems to be the most popular way to design and architect software.

Inappropriate Intimacy – client has to know about the internal structure of other classes to get to the thing

Indecent Exposure – beta, gamma, delta etc allow indecent exposure to get to a third object

According to Dave, it seemed that

Despite the wealth of information available on the subject, it seems that the vast majority of people who write software for a living are either
(a) completely unaware of any guidelines for sound software design, or
(b) largely misunderstand the guidelines.

Copy / paste metastasis (faulty code is reproduced in many places and spreads)

Interestingly, as per FJ, Yoder felt that many aspects of Agile directly lead to mud. These included,

Lack of upfront design

Late changes to the requirements

Late changes to the architecture

Piecemeal growth

According to Yoder, Agile can help not as a process but because of practices like continuous attention to technical excellence, retrospectives, face to face conversations and motivated individuals. One of the tools suggested by Yoder is simple refactoring and testing whenever the software is degrading. Hence, it seems that rather than the process helping with less mud, it would ultimately be responsible developers who would have to stand up and get noticed. Unless that happens, Agile or no-Agile Big Ball of Mud might be here to stay.

As Dave put it,

Despite the emergence of development methods that encourage and facilitate well-factored code, and the growth of the Software Craftsmanship movement, the Big Ball of Mud remains the most popular design for software, including greenfield development that has the full benefit of hindsight regarding the bad design approaches of the past.

Are the mentioned Agile practises actually Agile practises?

Your message is awaiting moderation. Thank you for participating in the discussion.

Lack of upfront design

Personally I think it upfront design is just as much a requirement with Agile practises as it is with any other practise. The way the design is specified and how the design evolves are different. In my opinion in Agile projects the design evolves during the duration of the project. What I do see is that projects, and especially (non-Agile) project managers, are really afraid to spend time in refactoring. I think an Agile team that is afraid to refactor code is doomed to fail. Because refactoring is fine tuning the code and thus the design.

Late changes to the requirements

This typically happens when coding starts too early and requirements have not properly been finalized yet. This again has to do with 'managers' not wanting a team to do nothing and just let them write code for the sake of being 'productive'.

A lot of development teams have to deal with empirical designs. This means we learn on the job what works best. Changes in requirements also means that customers are involved and they start seeing what could be improved. Is this a bad thing?

Late changes to the architecture

Upfront architecture is important, but change will happen. Personally I think it is best to wait as long as possible to make a descision because you have more information available at this time to make a proper descision. This implies late changes? An experienced architect should also know what parts of a system should be flexible enough to deal with change.

I dont think its Agile that is to blame for this but more that Agile is being blamed because projects fail to adopt.

It's a DSL

Define "most popular" !

Your message is awaiting moderation. Thank you for participating in the discussion.

"Most popular" ... that they have seen I guess. How can we draw rules and conclusions from that "fact" ?? How statically true is it ?? In California ? In the US ? in the world ??? project size ? team size ??

Anyway I like the conclusion: It is up to the developers to produce good code. How helpful ...

At least, the Agile method gives you a chance to manage changes in a proper way, instead of refusing or hiding them.

Completely agree!

Your message is awaiting moderation. Thank you for participating in the discussion.

I could never understand how a system of any reasonable size and complexity could be developed without a concerted and focused effort on upfront design or more broadly on what McConnell calls the “upstream activities”. (whan a waterfallish heresy!)

Creating sound architecture and design requires thinking at a different level than when doing the coding.

I understand that many details cannot and should not be specified before the coding starts and certain elements of the design emerge as the coding progresses. But hoping that the entire architecture and design will somehow magically emerge from daily mess produced but multiple developers is insane.

Furthermore, while beneficial in the short-run, Agile focus on customer-value of finished user-stories, produces long-term damage in form of not-refactored, un-maintainable and incoherent code.

Re: It's a DSL

Your message is awaiting moderation. Thank you for participating in the discussion.

Point taken that the code snippet looks like a DSL because of the dot notation, but that's only a surface resemblance. The code on which the example is based is not a DSL.

Re "agile" practices - IMHO sound software engineering practices are orthogonal to process/methodology. If projects (agile or otherwise) resulted in badly-factored code, etc., it's more likely due to the craftsmanship practiced by the development team than to the process/methodology they followed during the project. Granted, it's easier to blame a buzzword than to question one's own habits, but that's a non-technical issue.

Re: "get" and English. Luiz' comment reminds me of a time when my wife, whose native language is Spanish, asked me about the many meanings of "get". I started to write down all the idiomatic usages I could think of, and explained them to her. We stopped at around 100 examples. "Get" must be one of the most overused words in English.

Not Agile's Fault

Your message is awaiting moderation. Thank you for participating in the discussion.

Architecture is simply not at a premium with the average coder. This has nothing to do with Agile -- most guys I know that are into Agile are also into Design Patterns, etc. And don't forget that the SOLID principles constitute the first five chapters of Uncle Bob's Agile Software Development: Principles, Patterns, and Practices.

The problem is most coders are more focused on tactical problems rather than high level architecture. SOLID and GOF are too often derided as over-engineering or snake oil of some kind. A basic understanding of design tools like UML is not very common, in my experience, and in general are not seen as essential parts of a coder's toolkit. So as Joel and others dismiss or downplay the importance of SOLID, how is it Agile's fault again?

Quality Mindset

Your message is awaiting moderation. Thank you for participating in the discussion.

Most of the times, I see the developers who are writing badly crafted code today have worked/trained under/with the developers/team who have similar type of coding practicing for the several years.

The newly trained developers are mostly trained to know more technology related stuff rather than good code crafting practices.

If any developer today spends all of his life coding like BBoM and grows to upper positions (Sr.Dev, Lead, PM Tech-Manager, Architect etc.) will never allow people (new or juniors) working/spending time on the code-base to refactor and fine-tune it.

The industry should realize this and change their thought process rather than creating new names for the quality software craftmanship and processes and finally blame them.

Wont be solved by the Software Craftsmanship Movement??

Your message is awaiting moderation. Thank you for participating in the discussion.

If you work on a high performing team or attend IT conferences, chances are that you are a reasonably proficient developer who is interested in improving their knowledge of IT, and keeping up to date. There are many people who do not. They do not keep up to date.

The Software Craftsmanship movement are some of the top X% of developers. They have read a few books. Most of the problems I encounter are caused by developers who are the bottom Y% types. They have not read any books for a while, they have not stayed current, and they do not attend conferences. Its the bottom Y% who cause the problems... Unfortunately many of them have risen to senior positions because they have a track record of delivering what the business need now (by delivering spaghetti made from mud)

Theoretically Agile Teams do not do up-front design and architecture. The reality is that Agile Practitioners DO do up front design and architecture. They just don't write it all down in a nicely bound document (We give thanks for white boards and digital cameras)

Adopting Agile methods is not an excuse to ignore common sense.

It would be great if the Software Craftsmanship Movement could address the real problems and come up with an approach for the bottom Y% of developers. They who generate the mud. :-)

Re: Wont be solved by the Software Craftsmanship Movement??

Your message is awaiting moderation. Thank you for participating in the discussion.

The problem is that in most organizations a 'Big Ball of Mud' delivers the greatest *recognizable* business value. The business values are greatest possible speed at the least possible **short term** cost. Software quality simply has no real business value placed on it except by some IT folks that have to live with it (but have no say). When it starts to cut into executive performance bonuses (IOW becomes a visible cost) then we'll see change. Until then, any good development practices, be they Agile or not, will simply be subverted.

Re: Wont be solved by the Software Craftsmanship Movement??

Your message is awaiting moderation. Thank you for participating in the discussion.

Are the people interested in software craftsmanship trying to solve the "problem" of the Y% types? I wonder if it's possible to pursue craftsmanship with the same approach as one would use to make the Y% produce better results. Two different topics, maybe?

According to Toyota Product Development Process

Your message is awaiting moderation. Thank you for participating in the discussion.

According to Toyota Product Development Process we should "Front load the process while maximum flexibility exists". I think this is exactly why a god up front architecture is necessary in any project. I'm not advocating big up front design but only to think ahead of things that are going to be costly to change (for instance the framework) and to prepare counter measures in case of troubles (for instance a high risk module should be isolated as much as possible and a plan b should be determined in advance).

Late changes to the requirements

Your message is awaiting moderation. Thank you for participating in the discussion.

... many aspects of Agile directly lead to mud. These included ... Late changes to the requirements... Late changes to the architecture

Some 90% of our company's projects do imply late changes to the requirements. It's not boiled down to the question of Agile-or-not-Agile, it's something we just cannot control. As that Real World thing changes, our clients have to change their processes. Late changes to the architecture are also inevitable, because our systems are so huge they cannot fit into a single person's head. Does it mean all of our code is written in a Big Ball of Mud style and we can't do anything about it? I hope not, but even if this is the case, we have no choice but to continue growing our Big Ball.

Piecemeal growth

Your message is awaiting moderation. Thank you for participating in the discussion.

From my experience, the biggest source of mud is piecemeal growth without a defined architecture or goals in mind.

In every place I've worked, the business never comes to you asking for (metaphorically speaking) an entire house to be built. If they did, you could make blueprints and architect the thing properly.

Instead, the requirement is more like "I want place to sit down". So you clear off a patch of ground and give them a lawn chair. Then they say "I want a bathroom...and hurry!". Well, without the infrastructure for plumbing, you just drop a port-a-potty next to the lawn chair. Then they want protection from the elements, so you make a big tent to cover everything. Then they want a place to do laundry, and management decides that laundry is an outsource-able feature and points the business to the local laundromat. And so on.

Eventually you have an entire house...feature-wise....but it looks like no house you'd ever want. It's a big disconnected mess of mis-matched implementations, and there is no way to go back and put a proper foundation for a house underneath all of this stuff without disrupting everything terribly.

And this is the most common Software Design methodology I've seen in my 20 years of experience over many different companies.

Re: Piecemeal growth

Your message is awaiting moderation. Thank you for participating in the discussion.

Robert,

Nice spot on analogy! But I think that what you and I want doesn't matter. It is just the nature of the beast. To quote the paper(www.laputan.org/mud/mud.html#BigBallOfMud): "Shantytowns fulfill an immediate, local need for [s/housing/computing] by bringing available resources to bear on the problem. Loftier architectural goals are a luxury that has to wait." So how many slums and shantytowns have you ever seen rehabilitated? Exactly. Analogous intransigent motivations and forces make change equally difficult in business. It is simply a reflection of our values.

Re: Wont be solved by the Software Craftsmanship Movement??

Your message is awaiting moderation. Thank you for participating in the discussion.

Re: Wont be solved by the Software Craftsmanship Movement??

No, it won't. It is just a movement, not a silver bullet. The software craftmanship movement is just that, a movement. People who adopt its practices have been doing craftmanship before the movement even had a name.

Big-ball-of-mudders, on the other hand, have been also doing the same, writing piles of turds, despite us knowing the values of architecture, design, clear ROI, maintainability, and desired code qualities at the code.

Take any random developer and asks him about cyclomatic complexity, LCOM, or the mere basics of Böhm/Jacopini's theorem or what do modular, procedural and object-oriented programming mean, and you'll likely see him flunk. How could someone (other than a natural genius) be expected to develop non-ball-of-muds out of large systems without knowing the most basic premises that make solid code?

And although I fully embrace the tenets of the software craftmanship movement, it is just a starting point. For as long as we stick to the "craftmanship" part and never really take this as an engineering discipline, we are going to be stuck in this sh*t. Craftmanship is necessary but not sufficient.

If you work on a high performing team or attend IT conferences, chances are that you are a reasonably proficient developer who is interested in improving their knowledge of IT, and keeping up to date.

They are the minority.

There are many people who do not. They do not keep up to date.

These are the majority, and sadly so.

The Software Craftsmanship movement are some of the top X% of developers.

It has been like that always, even before the software craftmanship movement got a name for it. It doesn't matter if a group of think-alike people concentrate the top X% of developers. The rest does not, and that's all it takes to create seas of crap.

What lacks is not craftmanship (because we have known how to write good code for the last 3-4 decades.) What we need is engineering accountability. Real engineers would lose their license (or even land in civil court) were they to work like the average programmer.

Not that I'm arguing for professional software licensing (would be nice but we are not there professionally yet). But we need engineering discipline, accountability and ethics. Craftmanship is just pedal-to-the-metal application of engineering principles. True software craftmen are first and foremost software engineers. They have always been, and they are simply codifying their engineering craftmanship into a manifesto.

But without widespread engineering discipline, enforcement and examination, the masses of clueless (and many times unethical) coders that our colleges irresponsibly throw at the industry will always be there, pooping all over.

They have read a few books.

It takes more than that, and I would actually argue that you don't even need that. What you need is a solid CS education that imbues a sense of engineering discipline, hopefully with exposure to real-world software engineering practices... and an aggressive selection process - not everyone should get into a discipline, any discipline, just because. Unfortunately, we do not have that.

Most of the problems I encounter are caused by developers who are the bottom Y% types. They have not read any books for a while, they have not stayed current, and they do not attend conferences. Its the bottom Y% who cause the problems...

And it will remain so, independently of whether the software craftmanship movement exists or not because the problem lies somewhere else - in the lack of an engineering discipline. Craftmanship != engineering.

Unfortunately many of them have risen to senior positions because they have a track record of delivering what the business need now (by delivering spaghetti made from mud)

Here you have to be realistic. People get paid to get things done and solve the problems on the spot. There is the other side of the coin (which I've seen painfully) of people engineering these perfect solutions (on paper) and never deliver. When push come to shove, a big pile of mud is more important and relevant if it gets the job done than a pi in the sky.

The bottom line requires us to balance that fact with our ability to not only be craftmen, but engineers.

Theoretically Agile Teams do not do up-front design and architecture.

Not true. Agile theory calls for the minimal amount of design and architecture required for a system, delayed to the last responsible moment. Depending on the system, that last responsible moment might be just right up front, and the minimal amount of design and architecture might very monumental.

People who misunderstand agile theory don't get this and think they can get away with a paper napkin and a pen to jot ideas and start coding, even if the system (and its encapsulating business) is critical, complex and massive. And that's not agility; it is stupidity.

The reality is that Agile Practitioners DO do up front design and architecture.

Unfortunately, most people who claim to do agile fail to grasp that. They think they are craftmen, but they are not. They are hacks who latch to the latest silver bullet buzzwords as a clutch for their lack of ethics and discipline.

They just don't write it all down in a nicely bound document (We give thanks for white boards and digital cameras)

Not true either. It flies in the face when it comes to MDD to begin with. You can also be bound to produce documents if they are part of the deliverable or if you are subject to audits (think HIPAA or SOX). You might be mandated by the customer, law or contract. If work on a contract for the DoD, you are mandated to have a sizeable amount of documents following an iterative/spiral model. If you are sufficiently versed in agile, you negotiate a project plan where the different iterations are run in an agile manner with an agile production of deliverables (software, test cases, and formal documents.)

Agility is not the reduction of documentation for the blind sake of it. It is about the procedural efficient production of deliverables, usually software, but also whole systems and/or documentation. If your deliverables are not required to have large sets of formal documents, then you don't create them. But if they are, then you define an agile way to produce them.

Adopting Agile methods is not an excuse to ignore common sense.

Yeah, but people don't adopt agile methods. They bastardize the notion because they don't work as craftmen, much less engineers.

It would be great if the Software Craftsmanship Movement could address the real problems and come up with an approach for the bottom Y% of developers. They who generate the mud. :-)

It won't. The problem is more basal and fundamental. You need a widespread, enforceable engineering approach and ethics to building software and on which craftmanship can operate.

In fact, the whole focusing on craftmanship without any mention engineering is deeply troubling and flawed. We are not artisans building decorative items.

We are building complex, very expensive systems with actual monetary impact to real people and businesses. Our successes or failures make and break companies and contracts and can easily put people in or out of the jobs. We build infrastructure.

As such, when it comes to building infrastructure and equipment, electrical, mechanical, chemical, civil engineers are held to high ethical and professional standards, with not room for quackery and hackery, craftmanship just being their day-to-day application of their engineering disciplines.

Being that we build infrastructure that makes or break businesses, we should held ourselves to similar standards instead of just stopping at craftmanship thinking that it is the end by and onto itself.

Re: Are the mentioned Agile practises actually Agile practises?

Your message is awaiting moderation. Thank you for participating in the discussion.

> I think an Agile team that is afraid to refactor code is doomed to fail.A team that is afraid to refactor code (and/or doesn't have the test coverage to allow them to do that) is not an Agile team by any definition I know.Agile stresses learning, frequent feedback, which kind of implies that the feedback is acted upon - oftentimes, this requires refactoring.