Lessons Learned : Software Dev As a Process

Requirements are not code. Thus, requirements are boring. That’s how real devs feel about requirements. We devs did not choose a life of Business Analysis because we like creating code, not gathering requirements.

I’m a developer too and since most of you who will be reading this will be developers here are three promises for this chapter:

It will be as short as possible

I will provide new material in a new way -- this is not rehashed info about requirements

I will provide you with a specific process that you can use on all of your projects.

The Requirements Dream Quickly Becomes a Nightmare

As devs we would love it if a BA (Business Analyst) would go out, investigate the manual process and return to us with a list of requirements we could use to build the software.

That’s the dream.

Then, if we had any questions about those requirements we would just go to the BA and ask him for the exact details so we can move on. Unfortunately, this is rarely how it works in the real world.

Often, the process starts with the BA (or any other role) coming to the dev and saying, “Hey, the system needs to store customer information in the database, so go ahead and make it do that.”

Does that sound ridiculous, as if it would never happen in real life? Well, I took that example from a real life situation when a BA came to me while I was working at a large mortgage bank.

I simply asked him. “Do you have any use cases for how the data will be used?”

“Uh, no. We don’t do that here,” he said.

Right. As if a good and necessary practice such as creating use cases has to be mandated from above. It does not.

Since we devs are going to be expected to build the right thing we have to somehow convince those around us to do the work that will give us the best chance of creating the correct system.

It goes back to our second tenet of Software Development:

Everything the program is supposed to do must be known by the software developer.

I continued the conversation with the BA.

“Well, do you have a list of data elements that describe the customer information?”

“What do you mean? Just put customer information in the database,” he said.

I took a deep breath trying to slow myself down because I felt so frustrated.

“For example, what elements would make up a customer info record in the database? Would it be just first name and last name?”

“That’s what I need from you. Gather up those details and send them to me. And, you see the other problem right?”

“Other problem?”

“Well, how is the data going to get into the system? Where will it be entered?”

“You’re going to create a new form in the application,” he said.

“Now we are getting somewhere. Did you remember that when you arrived you just said that we just needed a table in the database? Now there’s an entry form. Will the user be able to edit a customer info record once it’s in the database or just create a new one? Will the user be able to delete customer records? Will different users have different rights such as read, or add and delete?’

At this point the BA looked annoyed. “Whoa, whoa. Hold on. I just wanted to get this new customer info thing set up. Are you trying to be difficult?”

That is the reality of how poorly requirements are often created in the real world of IT.

Creating Software Is Easy

Every developer can create software.

However, creating software which creates a solution to the proposed problem is far more difficult. That's because the proposed problem is often not defined very well, if at all. Different roles end up having different ideas about what the completed product will really do.

Capturing Requirements

However, creating software which creates a solution to the proposed problem is quite a bit more difficult. That's because the proposed problem is often not defined very well, if at all. Different roles end up having different ideas about what the completed product will really do.

The problem is how to capture the things which describe the system in way that allows everyone on the team to actually know what we (the devs) believe the system will do.

I’ve worked in IT for twenty-five years, most of them as a Software Developer. I’ve noticed that most beginning Software Developers (and many experienced ones) think their job is to produce code. However, in reality, one of the most valuable things you might do as a developer is to warn against the creation of code.

There’s not much guidance for Software Developers on where their roles begin and end, but growth means that your role must broaden (into a Senior Software Developer, Software Architect or Solution Designer). However, the lines between roles are blurry making it difficult for companies and individuals to determine where any particular Software Developer stands.

Purpose Of This Book

Because the roles are blurry there is often a lot of disagreement about what Software Developers do and how they do it. Is there any process to software development? Some companies lean toward the belief that software development is an organic thing and developers are artists who need creative freedom. Just start hacking away and see what we get. Other companies believe developers are assembly line workers who should sit in cubicles and write the code they are told to write. The truth is probably somewhere in the middle.

A Flexible Process Which Provides Guidance

What we need is a flexible process which can provide guidance while not crushing creativity. This process can provide a loose working structure that can be used by individual developers even if no one else in the company is adhering to a particular methodology. By the time you finish reading this introduction you’ll have a better understanding of the process I propose. I believe (and hope) you’ll see the great benefit this loose structure can bring to the software development process.

Even if you decide not to continue further into the book, I hope you’ll find this introduction an interesting discussion that brings a new angle to software development as a process.
To start our discussion, let’s go way back to the basics.

What Do Software Developers Do?

I believe we need a simple, single line definition of what software developers do. This definition will serve as a foundation we will continually return to in this book to determine where we are at.

First Tenet of Software Development

You may think that is too simple to cover all software. However, if you consider any software, even a game, from an abstract viewpoint, you’ll find that it is really just a manual process that has been turned into an automated process.

Pac-Man As A Manual Process

Take a look at Pac-Man video game. Originally, Pac-Man could’ve been a board game where players roll dice to move Pac-Man and the ghosts. The player could decide the direction for Pac-Man and you could spin a spinner to get the direction for each ghost movement. As you move the Pac-Man along the path on the board you would simply tally up how many dots he eats with each roll of the die. Next, you’d roll the dice for each ghost and spin the spinner to indicate movement direction. Each turn the ghosts could possibly land where the Pac-Man token is and destroy him.

Now that we know those things, it would be far easier to write the video game. The video game simply automates this process. The code is “rolling the dice” and “spinning the spinner” to move the ghosts behind the scenes. It’s also updating the location of the ghosts each time making it seem as if they are moving. You see? It all really is just automation. However, you’ve learned another important axiom related to software development.

Second Tenet of Software Development

Everything the program is supposed to do must be known by the software developer.

Just as the designer of the board game has to know exactly how the rules work so she can create the game, so too the software developer must know what the program will do so she can create the code.

Immediately, A Problem Arises

Here’s the problem. A lot of what we just talked about in the description of the Pac-Man board game seems obvious. It’s the nature of how human brains work.

“Of course you have to roll dice,” someone says. “How else would you know how far to move Pac-Man and the ghosts each time?” How things should be done may seem obvious to one person on the team. Those requirements that seem so obvious may not even be discussed much. The problem occurs later when the obvious assumptions are actually different in numerous team members’ minds.

Without Detailed Definition, The Developer Does It His Way

Maybe the Pac-Man board game doesn’t use any dice. Maybe the ghosts move X units every time. The point is that without the actual explanation and description the game builder (analogous to software developer) may do it however he decides. However, his way may be completely wrong.

If the designer and the builder are two different people, the builder’s decision may be entirely wrong. Furthermore, unless the design element is described properly it may force the developer (builder) to make a decision that she believes is correct. For example suppose the description for movement is something like the following:

Each turn each ghost will move a random number of units in a random direction.

No maximum movement has been defined so now any ghost may move up to infinite units in a direction.
So now, the game is created and Pac-Man moves then a ghost moves infinite units towards Pac-Man and destroys him every time.

That’s ridiculous. I know. But let’s circle back to the original definition of what Software Developers do:

Automate the Wrong Process

Now we are seeing that if the manual processes are not described properly or are not described at all, the software developer (builder) can automate the wrong process.

That’s why an important part of Crafting Software is :
Communication.

Two Core Challenges of Communication

Creating a common understanding

Capturing the appropriate details

Creating Common Understanding

Why do we have meetings? It's so we can discuss things. Why do we discuss things?

Three Reasons For Discussing Things

The following three things are probably basic drivers for the most meetings.

Obtain a baseline of understanding : Determine what others think or know about the subject

Get to a common understanding about the subject under discussion

Decide on a plan of action - what will we do after the meeting?

Meeting Failure

The problem that often occurs is that a team can generate thousands of words and hundreds of ideas in a short meeting. Often, a stakeholder may take a dev team into a meeting to explain what she wants from the product. Numerous ideas are discussed and people are nodding their heads and the meeting feels like a success. However, no one can actually be sure that they all have the same understanding of what the stakeholder wants. It is likely that they do not have a common understanding.

Were the Salient Details Captured?

Also, consider how most teams capture the details of a meeting. Maybe someone jots down some notes. The stakeholder may pause and ask, "Are the devs getting this so they can create the functionality?"

Very sharp devs who have a lot of experience can probably wing it and get fairly close to some percentage of the desired functionality. The problem is that there are so many details. How do you capture it all?

Other devs are completely disinterested in this entire process.

Ugh! Communication

Designer / creative types (Software Developers, portrait painters, song writers, etc) don’t normally want to talk a lot. That’s because they are busy making stuff. These devs know that talking about the details and gathering the requirements is Business Analyst work.

At times software developers require extremely specific details and creating and gathering this kind of detail can be:

Difficult to get from business users

Boring (from a Dev's viewpoint)

Extremely difficult to track

Easily forgotten

Did I mention boring? :)

For all of these reasons (and more) Software Developers try to ignore these things. After all, software developers want to be about the business of writing code. However, writing the wrong code isn’t helpful, even though it can feel like progress. "Hey, I wrote some code that does some stuff."

You Can’t Wait Until You Have Everything

The counterpoint is that you cannot wait until you have everything before you start coding, because you will never have everything.

Again, let’s circle back to my second tenet of software development:

Everything the program is supposed to do must be known by the software developer.

As a software developer, you cannot just start typing code. Well, you can. And, unfortunately, many software developers (and dev groups) do. But it is generally a terrible idea. I mean if you literally know nothing about the project and you’re already typing code then you have little chance of success. You must have at least the germ of an idea of what you are attempting to create.

Remember, software devs must write code which creates software that :

Fulfills stakeholders' expectations

But, as we said you will never actually know everything the software will do before you begin to build. Isn’t this all just a paradox? I admit that it does feel that way, but there is actually a way out of this swamp.

The Team Needs a Process

Everything I’ve explained so far is why you must have:

A process (method)

Structured yet flexible design which allows you to change things as you go

Those two little items look so short on the page but there are entire books within each of them.
Many of you with more experience in the IT industry may have bristled when you read the word method. Please notice that the word is not capitalized and it is not intended to invoke any defined Methodology (ISO 9000, Six Sigma, Agile Scrum).

One way to identify an amateur organization of any sort, be they accountants, lawyers, craft shops, or software developers, is a lack of process.

~Jack Ganssle, Embedded Systems Design (September, 2003)

I’m talking about a much more basic meaning of the word method.

Dictionary.com defines method, perfectly or our purposes:

“a procedure, technique, or way of doing something, especially in accordance with a definite plan”

For our purposes, in Crafting Software, we will alter that definition slightly to show the tight focus we’ll have in this book:

A way of developing a system (software) guided by an appropriate plan.

A Process That Solves Both Problems

Earlier I mentioned two problems:

Creating a common understanding

Capturing appropriate details

Crafting Software guides you into a process that will help you resolve both of these challenges by showing you effective ways to capture the appropriate details using specific artifacts and user stories. Chapter 1 will make all of this clear.

There are three things you can do to instantly make your communication more effective:

use a common language - every role must use the same terms to talk about the desired business solution ( no tech jargon)

focus on nouns (the things you want in the system)

focus on verbs (the functionality you want from the system)

As we move through Crafting Software we will discuss these in far more detail.

Hit Your Target

If your devs do not have the same understanding that the stakeholder has for the product, you will never hit your target. If you have not captured the appropriate details in a manner that builds a common understanding of the expected result, you will never hit your target.

Again, that is all based upon the Second Tenet of Software Development:

Everything the program is supposed to do must be known by the software developer.

Crafting Software helps create a process that creates guidance, like a map.

A map is a good analogy for the type of plan we are talking about in Crafting Software.

Software Development Map

Crafting Software will provide a map which will allow you to create an appropriate plan for developing your specific system.

Isn’t a Map Too Limiting?

I believe the the word map is quite appropriate in our context. Consider a geographic map. It provides a set of specific details but it does not tell you where to go. Nor does the map tell you how to get to your destination. Instead, it lays out the current situation as a simpler abstraction of the reality that exists.

The map user decides upon the destination and the path to the destination.

The map provides guidelines but it does not enforce a specific route. The map may indicate routes that are simpler (paved roads) or it may indicate areas that are impassable by automobile (mountain cliffs). However, if a map user decides that he will take a parachute and jump off the mountain cliffs and parachute into the town below, the map does not forbid it.

That’s exactly how Crafting Software will work. It will provide you with guideposts along the way that will show you the easiest way -- the smoothest route for most travelers. However, there will be some paths that you and your team may decide not to take. There are many times you just want to take the shortcut and it makes sense. But, you’ll consciously know you took the shortcut and be able to explain why you did so.

Crafting Software Will Provide the Map

Crafting Software will provide the larger map as guidance. However, each reader will hash out her own and exact route, deciding on the exact artifacts she will use to describe how the software will be created and which decisions were made along the way.

There are portions of the map that every software development team will use.

Examples of those items are things such as:

database schemas

class diagrams

requirements documents (made up of user stories)

Please don’t get too caught up on specific types of artifacts at this time because if you do you may think I’m just talking about doing a lot of documentation and that I’m just disguising some other Methodology and it’s all the same and why bother.

Unnecessary Documentation Kills Creativity

That’s not what I’m talking about at all and I believe you’ll agree by the time you complete this introductory chapter. I am a software developer first and I hate unnecessary documentation too. It kills creativity.

What Do Software Developers Really Need?

There are a few things that every software developer needs.

A basic starting point - what are the basics of what we are supposed to build?

A way forward - how should the code be created?

A way to continue - A basic process for the daily work the software developer will do

A way backward - what did we actually do to create the software?

Examples of things you might need to know as you look backward:

Configuration - so you can deploy

OOP Design -- so you can add enhancements, or more easily determine why a bug is occurring

DB Schema -- so you know where data is stored, for examination

Etc.

Crafting Software will focus on fulfilling these to guide the software developer into always knowing where she is in the overall System Development Life Cycle and what she should be focusing on.

Now, let’s close the loop on the other item on our two item list (structured yet flexible design) and talk about what it means to us in Crafting Software.

Structured Yet Flexible Design

This special item described here as structured yet flexible design provides the way forward for software
developers.

Software Design : Large & Growing Body of Knowledge

The ability to create a structured yet flexible design pushes a developer to learn a specific set of concepts which comes from a large and growing body of knowledge.

Crafting Software will have a strong focus on learning OOP principles and design patterns and more, but always with a focus of why we might choose one design over another. We will always focus on applying designs as a way forward in creating working software. We will only move forward with designs after we understand why we’ve chosen a specific way forward while understanding the trade-offs we’ve chosen.

The things we will learn about in Crafting Software will help you as you create software systems of every kind.

Here’s a list of some of the things we will cover (there will be far more in the completed book):

VCS (Version Control System) is non-negotiable for professional devs
It’s not overhead, good VCS is a developer’s best friend

Enterprise Patterns

Reusable system structures which help you solve Enterprise challenges

Integrating solutions - One of the main challenges is that data is stored in disparate formats and yet the Enterprise needs to share certain data elements with various systems

Deliverables / Milestones

Working software (Minimum Viable Product (MVP))

continuous delivery and the necessity of tooling (build & deploy)

Appropriate UML Docs

You will have to explain things to other people in various roles. This is how you choose the appropriate document to show the person.

Highest level of abstraction first, drill down into detail only as necessary - speeds up communication, makes meetings faster!

You must use the language of the business domain. You must get everyone to use the same language (ubiquitous language) so that everyone can talk about things the same way.

These UML docs will help create our Software Development Map

Best software for creating UML - Enterprise Architect -- non-negotiable if you’re a professional dev

Also provides nice code reverse engineering and code generating

Every UML document explained with a why -- why you should produce the document

A large portion of developers do not know why they create these documents. After you read Crafting Software you will be able to produce and explain every UML document covered and defend your reasons for not creating specific documents should you decide not to.

How will I show you all of those things I’ve just listed? Will Crafting Software simply provide dry descriptions of what you would do if you were developing software? No.

Crafting Software : Design and Code Real Apps

As you read through Crafting Software we will be designing and writing actual code which I will provide on github. All of these applications will be linked in some way. In other words, they will share data and be available via desktop apps and also accessible via web and/or Android and iPhone/iPad apps.
This will allow us to see the software dev map in action and also see how certain components (OOP structures) can be reused across language and platform boundaries very easily. That’s the power of process and great OOP.

Now, let’s wrap this introduction up.

Was This Some Kind of Agile Trick?

Earlier I told you that this book is not based upon some giant Methodology. However, I have found after a lot of reading about best practices and numerous methodologies I am aligned in all of my foundational process philosophy with the original principles of the Agile Manifesto. Before you balk and think I’m trying to force Agile down your throat, please read the twelve simple principles from the Agile Manifesto: http://agilemanifesto.org/principles.html^

They are truly fantastic because they are simple. They are short. They are not wrapped up in some five hundred page book. The reason I mention them is because Crafting Software is focused heavily on one specific principle from that manifesto. It’s the one that all Software Developers should care about the most.

Crafting Software: The Focus Is Always On Working Software

The Crafting Software method will always focus on what helps us to deliver working software.

That’s one of the twelve original Agile Manifesto Principles:

Working software is the primary measure of progress.

As we move through the book, we will always focus on the MVP (Minimum Viable Product) and working software. I’m hoping that will help make Crafting Software a lively read since readers will always have their focus on developing a working software system.

We've covered a lot here because a Enterprise Software Development touches a lot of areas. I've attempted to create discussion that will make you think a bit differently about process and hopefully that will make you want to read more. If so, let's jump into the first chapter where we'll begin at the beginning: requirements. I promise it'll be a fast chapter and it'll make you think about requirements in a different way.