Sunday, March 24, 2013

As
Benjamin Franklin pointed out there are two parts to organization. The
first is that absolute everything needs to fit somewhere. With software
this really translates to having a solid 'reason' for every little bit
in the system, be it config files, methods, data etc. and a reason for
its location. It all needs its place, and often for that it also needs
some level of categorization. "It doesn't matter" is synonymous with
"it's not organized".

This
includes names, conventions, even coding styles. Everything. Each tiny
piece of work, each little change should all have its place. It should
all have a reason for being where it is, as it is.

The
second half of the quote is just as important. What's the point of
having an organizational scheme if its not being used. As new things are
added, they need to be added into their place. Sometimes it’s clear
where they belong, but often times it hasn't been explicitly documented.
Assuming that lack of documentation equates to lack of organization
(and as such it is a free for all) is a common failing amongst
inexperienced coders. Things can be organized with having an
accompanying manual.

Failing
to keep a development project organized is the beginning of the end.
Disorganization is probably the worst aspect of technical debt because
it is a direct path into various forms of spaghetti. And spaghetti is a
quagmire for time, either to fix bugs or to extend out the
functionality.

Part
of software development culture over the last couple of decades has
been a strong desire for 'freedom'. Freedom is great, but used
improperly it just becomes an excuse for creating disorganization. For
instance, utilizing the full freedom to create a new screen in a much
fancier way than the existing ones is really just breaking the
organization of what is already there. It's like helping someone dry the
dishes in their kitchen, but then insisting on placing the clean stuff
in any other location than where it actually belongs. It isn't really
helpful is it?

It's
true that in some regards a well-organized existing development project
is a boring one. If the work isn't exceeding the existing
organizational bounds then it can really just fit in like all the other
pieces. But success in development doesn't come from making any one
small piece of the system great, it's an all or nothing deal. The system
is only as good as its crappiest screen or stupidest functionality.
Thus changes to enhance it or its organization can't be selective. The
whole thing needs to be fixed or it’s actually just making it worse.

If
you're evaluating an existing development project, disorganization is
easily the best indicator of the future. A small project can get away
with disorganization, but anything larger absolutely needs organization to
survive. If it is lacking or failing, then the whole effort is in deep
trouble.

Sunday, March 3, 2013

There
are two distinct sides to software development. Since software is a
solution to someone’s problem -- a tool to help them in some way -- it
lies in a specific business or ‘domain’. Its purpose is tied directly to
helping automate, track and organize very specific information. But the
software itself is constructed using parts like computers, networks,
libraries, programming languages, etc. so it is composed of and limited
by ‘technology’. Both the domain and the technology are necessary
ingredients in being able to help users, rather than make their issues
worse.If
a software development project goes wrong by mostly ignoring the domain
issues we usually say that it was built in an ‘Ivory Tower’. It’s a
nice metaphor to describe how the developers may have completed the
technical side, but they were too far up and away from the domain
problems. The software is useless because the creators didn’t bother to
dig into the details.But
what about the opposite problem? What if the developers pay close
attention to the domain, but fail entirely to properly handle the
technical issues? Since
that is fundamentally the opposite of an Ivory Tower we can flip the
terminology. An ‘Ebony Dungeon’ project then is one that delves deep
into the heart of the domain, but so deeply that it ignores the
technology side. We often see this in in-house projects where the
business or domain experts exert too much influence over the process,
techniques and construction of the software. Most
domains have to tie themselves closely to some form of revenue stream.
Those ties mean that they need to react quickly to changes. That sets up
a culture of just focusing heavily on the immediate short-term, trying
to be as malleable as possible. That works for business, but a large
software development project is actually a very slow moving ‘ship’. It
slowly goes from version to version, plodding along for years. While the
business may need to wobble back and forth as its demands change, the
only efficient path for development is to steer as straight a course as
possible. There are a lot of moving parts in software and to get them
all working properly they need to be organized; put into their proper
places. A constantly shifting direction undoes any of this organization,
which wastes time and causes severe problems.A
large system full of useful domain functionality isn’t actually very
useful if it is technologically unstable. If it crashes frequently or is
prone to serious bugs because of mass redundancies or if its
performance is dismal, all of that existing functionality is
unaccessible. A smaller more stable system would be far more effective
at helping the users. The features available should work properly, be
complete and be organized.A
very clear symptom that a project is trapped in an Ebony Dungeon is
that most of the decisions keep getting punted back to the domain
experts. “We’ll have to ask the business what they want”. If the project
is balanced then at least 50% of the effort is related to the technical
side. That includes using industrial strength components and
algorithms, keeping the code base clean and insuring that the operations
and installations side are built up to an automated level as well.
Technical debt is unavoidable, but it needs to be controlled and managed
with the same importance as any other aspect of the project.In
areas like GUIs, industry conventions should trump individual's
preferences, so that the screens don’t become a sea of eclectic
behaviors and that the functionality isn’t randomly distributed
throughout the screens. Failure to properly organize the functionality
at the interface level will cause a failure to use the functionality at
the user level. Proper organization of a huge number of features is an
extremely difficult problem that takes decades to master. A domain
expert may understand the functionality requirements, but organization
is just as, if not more, critical.Being
trapped in a dungeon exerts a lot of pressure on the programmers to
create code as fast as possible. This usually manifests itself as a
significant amount of “copy and paste” programming. Old code is copied
over and then hastily modified with a small number of differences. We’ve
known that this style of development is extremely bad for decades, but
it is still commonly used to satisfy time pressure. Programmers need
time to understand and refactor their work if its going to be extended
properly. A rushed job is a sloppy job.An
Ivory Tower system misleads its backers because the real problems don’t
become apparent until the users start working with the system. An Ebony
Dungeon system also misleads its backers because it starts off fast and
agile but slams into a wall when the work is to be extended. What
appeared to be a big success in the early days ends up dying a premature
death, usually costing way more than it should. The
software industry has swung rather heavily towards Ebony Dungeons
lately. It’s easy to do because domain experts over-simplify the real
work involved to build a system then they get mislead by the rapid
progress. Without an experienced development crew it becomes easy to
miss all of the symptoms of a project in deep trouble. Most that are
dying get caught up in tunnel vision, thinking that just a few more
features will turn the direction around and save the project. But “just a
few more...” is actually the problem.The
best way out of a dungeon is to properly partition the system
requirements. Everyone talks about ‘user requirements’ but they are only
a small subset of what’s needed for a successful system:

user requirements

management requirements

technical operations/release requirements

support/debugging requirements

development/programmer requirements

If
you factor in all of the different requirements necessary to build and
run the system, you see that ‘features’ are only one aspect of the work.
If you ignore the other four (or more) categories than obviously there
will be serious problems with the system. If all of the requirements
come from the domain experts, since they don’t know about or understand
the other issues they won’t place any importance on getting them done. The
real art in building systems is to not go too high or too low, but
rather to build on solid ground that is accessible to everyone. Towers
and dungeons are equally bad.