Thursday, December 22, 2011

[
Author’s Note: I really don’t know what to say. It was just one of
those days. If you don’t feel like reading a broad rambling on loosely
connected ideas, please don’t read this! ]A
good place to start -- as always -- is with a few definitions. For this
particular discussion it works better if I create them loosely and use
somewhat non-conventional meanings. I’ll try to explain why that helps
as I go along. The
first thing I want to define is a ‘system’. For this discussion it is a
set of rules that act on ‘stuff’, where I intend stuff to be nearly as
vague as possible. Stuff could mean static objects, or it could mean
dynamic processes, or anything in between. Systems
can be broken down into two loose categories: formal and informal. A
formal system rigidly constrains the stuff, so that it stays within the
space occupied by the rules. That is, stuff is in the formal system when
it obeys the rules, and it is invalid or outside the system otherwise.
Thus the rules are ‘formal’ and essentially unbreakable.Informal
systems, on the other hand, obey the rules most of the time but there
are plenty of things that can occur within the system that are outside
of the rules. That is, an informal system loosely defines what most of
the rules are, what the boundaries are, but there are always exceptions
and somehow by some ‘stickiness’ these still fall within the system. To
put these two definitions in more candid terms: a formal system is
black and white and an informal one is just shades of grey. Both are
systems and have what amounts to deterministic behavior, but one only
approximately follows its rules, while the other is bound by them.So,
why these definitions? Because we can apply them to many things within
our world. The rigid abstract formalisms of mathematics are obviously
formal systems. The rules are the axioms, the definitions, the lemmas,
the theorems, etc. that are built up on top of the stuff which is the
underlying well-defined mathematical objects. There are many different
branches in mathematics and all of them have a plethora of underlying
formal systems. Formal systems can also be built on top of other formal
systems, by creating relationships between them. So
a very simple formal system in mathematics is a set of operators like:
plus, minus, multiply and divide, that works on objects like: the set of
all whole (positive) numbers. What is interesting about this system is
that the subtract operator, when used on a small number and a much
larger one, will produce a new number that is no longer within the whole
numbers. That is 3 - 10 = -7, which is a negative number. There
are a number of ways to view this, but again I’ll go a little
unconventional and say that the above formal system has an ‘exit’ point
into a larger formal system. That is, if I had chosen integers, which
include the negatives then there would be no problems with the subtract
operator. Thus we can consider the first formal system as being an
incomplete subset of the second, with respect to subtraction. We can
also consider it to be an incomplete subset of one based on the
continuum of real numbers, with respect to division. This gives us a
large set of structural relationships between the various systems. These
distinctions may seem a little weird, but they will come in useful
later. In
contrast to a formal system, an informal one is not so well-defined.
There are rules and there is stuff, but there is also a great deal of
flexibility built in. That is, there are no real ‘exit points’. If
something does not fit within the system, somehow it still seems to
remain there. The types of informal systems I am thinking of include all
different sizes of groups of people. Companies, clubs, governments and
all other organizations are informal systems. They have rules which bind
them together, but do change from time to time. Other examples are
markets, schools, disciplines, professions, soft sciences, etc. You
may note that all of the informal systems I’ve listed so far are based
around people. That’s more of a coincidence rather than a deliberate
intent, for there are plenty of systems out there that we have tried to
rigorously formalize but our underlying knowledge is incomplete, or the
basis of the system is just too chaotic. Evolution, physics and weather
are good examples. These too are informal systems, but our underlying
comprehension of the rules is gradually approaching formalization. That
is, rather than an exit point, outliers get feed back into the informal
system to enhance our understanding of it as it approaches, but never
quite reaches, formalization.And
this brings up a very important point. At the bottom of everything, as
far a s we know, are particles. And although we don’t know all of the
rules that define their behavior, we are pretty sure that the rule set
is static. Thus there is one underlying formal system that quite
possibly binds everything. That is, nothing can happen unless it is a
physical possibility. On top of this we get ever increasing
‘epiphenomenon’; larger and larger collection of particles that at their
collective levels behave within a set of what are essentially
meta-rules. So we get galaxies, planets, stars, etc. And built on these
we get even more: water, earth, plants, people, etc. And on these we
build up even higher to stuff like buildings, farms, etc. And some
epiphenomenon that are less physical, but still bind together stuff like
organizations, companies, countries, etc. So
it would seem that for all of our informal systems, they exists within
the bounds of lower and lower systems that eventually fall back to one
rather low-level formal system. What
is rather odd though, is that we are also aware of the multitude of
mathematical formal systems out there, and these seem to exist without
any ties to those based on the many layers of epiphenomenon ones. We do
however, use them to explain the others. That is, physics is a set of
formal systems that are remarkably accurate in explaining the physical
world around us. So in that very sense, mathematics is the
meta-formality that binds together the epiphenomenon with some degree of
precision. Thus it has its roots in our physical world, even if it just
appears as an abstraction based on our intellectual thinking
capabilities. Any
real argument about the how tangible the formal systems of mathematics
are has to take in consideration computers. For the first time, our
species has manager to instantiate one of our previously abstract-only
formal systems -- Turing machines and/or lambda calculus -- into a
physical reality. Well almost, since computers are still bound by finite
resources and by the occasional hardware failure. Regardless, it is
still a tremendous accomplishment and their existence has been rapidly
transforming our societies ever since. One might argue that how physics
models the world is similar, but we need to consider that that link
between math and reality depends on a person utilizing the models in
their head, in order to act on the knowledge. Thus the formal systems
that underpin physics stay internal to our minds. Computers, on the
other hand, do their computations entirely independently of people, and
thus are a concrete physical manifestations of a formal system. A very
different ballgame. So
why are these definitions of formal and informal systems useful? One of
the main tasks of software developers is to construct useful formal
systems that solve problems for people working in informal ones. That
is, any software we write must be formal in its construction, but its
usage lies in one or many informal systems. If
you are writing a system to track how a company deals with its clients,
the company and the behavior of the clients is entirely informal. Most
things happen according to plan, but lots of exceptions occur and the
nature of the relationship changes as the market expectations change.
But those rules need to be formalized, so they can be coded, so the
computer can execute a deterministic series of instructions with each
interaction between the client and the company. From
this angle, you can see the problem right away. The informality of the
base systems leads to difficulties in finding a formal one that both
precisely matches them and remains matching for any prolonged period of
time. There
are ways around this. The underlying formality of the computer still
allows for it to be dynamic with respect to its resources. The resources
may be finite, but these days memory and disk are so large that their
finiteness isn’t a significant factor. So, instead of coding a large
number of finite static rules to contain the system, the designers find
dynamic ones that bend and shape as the underlying informal systems move
about. One of the great failures of software development these days is
to ignore this reality most of the time, and rather just blame the
inherent mismatch on contrived issues like ‘scope creep’. To
simplify this discussion somewhat, we can combine any underlying
informal systems together and take a subset. This somewhat monstrous
informal system is the object that we need to mirror with a formal
software one. As one could easily guess, this type of combination can
easily be contradictory or incomplete. And that is an all too frequent
reality when building software systems.While
the relationship between software systems and the informal ones that
the software is trying to solve is interesting, these days I am more
interested in the general behavioral characteristics of informal systems
by themselves. One thing we can do is use the size of an informal
system as the metric of the underlying complexity. It isn’t the only way
to view complexity, but it does prove useful in understanding how and
where the complexity grows.As
an example, we can look at law. Since we’re generalizing, we won’t
distinguish between civil and criminal law, nor tie the discussion to
any specific country or international organization. Law is basically a
set of rules that the people of a society have agreed to follow. There
are often a lot of rules, and generally through government action they
are always increasing. The laws define which actions are right or wrong
but that is really a black and white view, where the world is rather
grey. So along with the laws are the precedents from legal judgements
that together often define how the law is interpreted and applied. Both
of these can also rely on a vast number of legal definitions, that do
not necessarily match the more common definitions used by industries or
people. To add to this, some laws or interpretations of them have never
been fully tested in the courts, so there is some ambiguity as to
whether or not they are valid. Thus
the informal system that binds the laws of a country consists of the
laws themselves, the history of precedence and some unknown guesses or
assumptions about what is, and what is not, legal. And this is
constantly being added to with new laws, or new interpretations of the
laws. Thus it is perpetually getting larger all of the time. At
least from the outside there doesn’t seem to be any real mechanisms to
reduce the size of the system. I’m sure there are some instances of
people replacing laws with simpler ones, or laws just getting ignored,
but in general this does not seem to be the trend. The system just gets
larger every year, there are more lawyers and judges, and the complexity
get worse. New interpretations that form precedence get added, but the
old ones hang around and can always be dredged up again. The
irony is that at some point the sheer size of the system prevents
people from being aware of the full scope of it. That seems to
contradict its usefulness in being there as a common set of rules that
everyone in a society should obey. Why have rules if most people are not
aware of them, or they are not properly enforced? In
a formal system, one could likely find an equivalent system, but with a
less complex set of rules and then reduce it to it. In an informal one,
while that option may exist, it doesn’t seem to be the path chosen. The
system gets larger, it get more convoluted, and any earlier
discrepancies instead of getting fixed just get built upon for the later
works. An
insider’s perspective on a informal system is generally predicated on
familiarity. They understand the rules, or at least a subset of them,
and they accept that this is the way it was, and the way it will be. An
outsider’s perspective, particularly if they need to dig down into the
depths and see all of the ugly warts and bumps that are being ignored by
others, is quite different. They can see the twistedness of the
informal system for what it is. The gradual accumulation of the rational
and irrational, over a long period of time. For
software developers, if they are building systems to handle very
specific domain problems, they are the outsiders. That is, they come at
it from a distant perspective, and they get to evaluate the rules based
not on history growth, but rather a rational logical foundation. They
have no choice, they must map the informal system onto a very rigid and
precise formal one. Any impedance mismatches between the two systems is
both obvious and problematic. That
does give developers a rather unique point of view. We’re not looking
at informal systems like law from the vantage point of acceptance. We’re
looking at it from the vantage point of complexity, and mapping it to
something simpler. Software systems, almost by definition, are going to
be simpler and less complex than the informal systems they are
mirroring. That’s a product of the sheer amount of resources to build
them and keep them running, but also because they generally focus on
just a small number of problems contained within the informal system,
not the full system itself. From
all of my experience digging to many different informal systems there
are a couple of things which really worry me. One is that virtually all
of the informal systems I’ve seen contain some type of irrationality.
There is always at least one thing there that is just not the product of
rational thinking. It just gets built up at the cross roads between
other parts of the system. Nothing that would even be constructed
deliberately. It’s easy to see these because any domain expert
explanations are contrived, messy, hidden or seem to change. That
usually points to some underlying irrational component.The
other thing that worries me is that there is always a way for the
system to grow, but almost never a way for it to shrink. One can see the
consequences of this all over our societies. Everything gets more
complicated and more convoluted. People try to justify the growth, but
these often are just hollow excuses. One
might consider computerization a simplification method, and sometimes
it is. It is not infrequent that implementing a new computer systems
means changing the process to some degree to fit the new limits of the
code. However development of systems often goes on for decades, and
gradually the complexity heads back to its former glory. Computers
in general make it easier to manage complex informal systems. They can
do a lot of the grunt labour for people, and they can do it fairly
accurately, if they were correctly instructed to do so. We can see this
shift as well. Part of the transformation of computers has been the
ability to manage informal systems of such complexity that they would
have easily collapsed without the aide of a computer. Overall, however,
I’m not sure that this is a good thing.A
while back I read in a blog somewhere, that once a complex informal
system gets large enough, the only remaining option is for it to
collapse. That is, it falls apart suddenly and dramatically. History
seems to confirm this view, as we have seen the rise and fall of many a
great civilization. At some point the system can no longer increase, it
can’t remain as it is and there is no ‘release valve’ for complexity.
There are no other options. It basically implodes on itself. That
notion, if true, does not bode well for our current societies. We can
see that our complexity has dramatically increased over the last half
century, and we can see examples of systematic failure showing up at
ever increasing rates, but we seem to have no viable paths of fixing
what is wrong. We should be concerned.If
there is some way out of our historic fate, my sense is that we would
need to rely on both computers and aggressive simplification to get
there. Computers, because they can show us alternatives to our current
informal systems, and help us model and control the side-effects of
changing what are essentially chaotic systems of n-dimensional
variables. And with that initial knowledge, we can start the slow
gradual changes needed to simplify our informal systems. The changes
have to be slow, just because in general people don’t adapt well and
their fragility generates another form of complexity. No doubt it is
easier said than achieved, but given the alternative of imploding it
doesn’t seem that bad.There
is lots more I could say about formal and informal systems. Defined
this way, they provide a general underpinning for a great deal of what
happens in our lives. And along with the way we model things internally
they help to organize the way we see the epiphenomenon around us.
Perhaps, someday I’ll get more of a chance to delve deeper into this
area, although all things considered, keeping up with modern life’s
requirements is probably too time-consuming and complex these days for
me to get another chance :-)

Monday, December 19, 2011

The
last thing I ever wanted to be -- twenty-six years ago -- was an
engineer. When I started university I had heard that 70% of all
engineers hate their jobs and that they were often bored at work.
Boredom sounded horrible. What
I wanted, was to master the black art of coding. To follow in the foot
steps of those early magicians I had seen in the computer magazines. The
ones that crafted weird cryptic concoctions to do really neat things to
their machines. I wanted to be a hacker.It’s
funny how time and experience temper one’s thoughts; how your actions
change once you really understand the consequences; how much you grow
once you see the larger picture. Development
shops span the range between the cowboys, who are just hacking at the
code furiously, and the bureaucrats, that are so ground down in
organizational paperwork that little actually gets accomplished. In
my younger days the cowboys where fun. It’s a dynamic environment.
Things are happening. And they are happening fast. Well, OK, there are a
lot of needless problems cropping up, but hey, that just adds to the
excitement, doesn’t it? You could leave work feeling like you got
something accomplished.Well,
that lasts for a while. Until you start to get caught in the same
hollow accomplishments, over and over again. That is, you’ve fixed the
same bug before, wrote the same code before or stuck a band aid on the
same gaping wound before. Doing it once or twice is fun. Doing it
repetitively, while giving up your life to endless overtime, starts to
become depressing after a while. Then suddenly one day you find yourself
looking down at the growing pile of band-aids before you and you start
thinking “isn’t this all just a waste of my time and my abilities?”Early
on I moved away from the cowboys. They were driving me nuts. I landed
deep in the heart of a reasonable engineering culture. Some people on
the outside might have confused that development process with a classic
waterfall one, but it wasn’t really. Although there was a significant
design process up front, the last of the details were always worked out
in the various iterations. It was actually a healthy mix. Some thinking,
some paperwork, then a hard run at the code. Some testing, a release
and then back to thinking again. Later
when I set up my own shop, it naturally started as chaos. Startups are
sensitive to cash-flow problems so sometimes the only way to deal with
that is an ugly short-term hack. But as time progressed -- every time I
got the chance -- I applied long-term fixes. These gradually paid off
and when I finally moved on, the place was humming along smoothly
releasing a steady stream of mostly high quality code and the occasional
patch within a day or two of finding a bug. Each development cycle was a
bit easier than its predecessor. Everything was tracked. We had managed
to built a big application. One that was far larger, and far more
sophisticated than you would have expected from such a tiny shop. I
learned a lot in those years.I
found that it isn’t all that complicated. If you boil it down again and
again, software development keeps coming back to its roots in
engineering. Not in the classic sense that I was afraid of when I was
younger -- the large, slow, boring, mindless process -- but rather in
the sense that as the work progresses, developers like myself needed to
be hacking less, guessing less, and understanding more. The project may
start with a lot of unknowns, but at some point it always gets down to
routine engineering. That is, we should be doing the ‘right’ things that
work because we know that they work, and fixing the little problems --
permanently -- before they become big serious ones.My
earlier fears were misplaced. Engineering itself is not boring. I’m not
even sure where I heard that statistic about unhappiness. Engineering,
particularly when it comes to something like software, is simply knowing
what works and applying it correctly so that the project is successful.
It isn’t hack and hope, it isn’t wild guesses, and it isn’t a spin job
to cover up the fact that the system is horribly broken. No, quite
simply, it is the path to dreaming up cool solutions and then
implementing them as cool solutions.Yes,
there are boring times. There is a lot of digging to see what is
already out there. No sense reinventing the wheel especially if it’s
already decades old. Note that that doesn’t mean you can’t recode the
wheel -- making a better one -- but rather that you don’t try to rethink
the wheel, that work is already done and accessible.There
is sometimes paperwork. But if one focuses on documentation that’s
useful, rather than on the documentation a management consultant would
want, then it’s not so bad either. It helps to refine ideas and
highlight missing corner-cases long before they become too serious to
fix. A little design goes a long way and any organizational chaos, at
any level, is always going to grow into a serious problem someday. What
you’re missing, or not thinking about, often becomes really obvious when
you try to write it down. It also helps you to understand the
vocabulary of the problem and to empathize with your users (if you can’t
easily explain it in a document then the users aren’t going to be able
to understand it either).There
is a lot of thinking. And ironically, although programmers preach
heavily about their need for freedom, many choose to turn off their
minds when they code, resorting to as much brute force as possible. But
at the end of the day, code is just the by-product of thinking about the
solution, so it is only as good as the thinking that went into it. To
build sophisticated things you have to have a deep understanding of the
problem and you have to spend a lot of time working through it.
Thoughtless code is dangerous code because of it lacks intelligence. It
just causes stupid problems.And
of course, if you’ve really set a firm direction and you know where you
are going with the development then the rest of it is just hard work.
Little issues have to be sorted out, details need gathering, but the
main flow of the development can be worked out in advance. The project
may drift but either the long-term perspective still holds, or it is
rethought again to insure that it is still valid.I
remember reading decades ago in Scientific American about how the
pharmaceutical industry moved from art, to science and then on to
engineering. The pills we pop these days are highly likely to contain
exactly what they say they will. That wasn’t always the case. It took
them a long time to go from alchemy to engineering. The article said
that software also started as an art, and has some scientific basis, but
it will eventually follow the same path to engineering. I remember the
article because I figured I’d bail when that happened. Sounded boring.
But these days, as I am surrounded by more software with increasing
complexity and decreasing quality I realize that what I want most right
now is to get a whole lot more of the engineering mindset into the
software development industry. It is time, I think, that we started
taking our profession seriously. There
is still room out there for the hackers and the cowboys. Sometimes a
band-aid is the best choice, or the solution is so small it doesn’t
matter how much of a mess it is. For quick one-offs I think hacking
will always be common, but when I go out and rely on someone else’s
code, on their thinking and on their intelligence, I would like to know
that they weren’t just mindlessly flailing at the keyboard or that they
skipped their homework and just tossed it together. It makes a
difference. I don’t want to build on a house of cards. I don’t want to
find out later that I’m dependent on somebody who didn’t really ‘get it’
at the end of the day. I don’t want that degree of instability in what I
am doing. I can’t build great software on top of a mess; dealing with
other people’s mistakes eats through my resources like a knife cuts
butter. So
these days I am all in favor of us seriously moving into engineering.
Not doing so is a major obstacle that has prevented software from living
up to its true potential. I think we can do it, and do it in a way that
it is not boring and it isn’t just paperwork. We can still build
quickly, but we can do so from a place of understanding, not one of
panic. It’s about time that software development grew up and became a
real profession, one that is taken seriously. Even in our deplorable
state we are changing the world, so can you imagine what we could
achieve if our collective works didn’t suck?

Saturday, December 10, 2011

One
of the most enjoyable things about building software is that it
requires the developers to dig around in other domains. If you’re
writing financial software, you need to know how the industry works,
understand the terminology, and all of the complex details underneath.
If you build something for the printing industry, you need to grok their
equipment and business models. Everything you write depends on
knowledge from elsewhere.For
a naturally curious person like myself, this is a huge benefit.
Basically a license to try and learn as much about the world as I can.Every
domain I’ve ever worked in has had a fascinating array of history and
depth. You can’t actually solve any of their problems with software
until you really understand them; until you get down to what makes them
tick. Get to the roots. Get to the truth.Software
is very rigid and to automate any processes they need to be fully
understood. To collect valid data you need to understand its structure,
quality and volume. To make the interfaces usable you need to understand
the environmental issues and the people. If you don’t get to the truth,
the software either won’t work or it will cause more problems than it
fixes. If your goal is to actually solve something, the only place you
can do that is in reality. But
getting to the truth is a funny business. There are all sorts of people
out there that will stand in your way. Sometimes it’s because they have
something to hide, sometimes it because they don’t want to think about
it. Sometimes they just don’t care or figure it isn’t necessary. The
causes very. Lately
I’ve come to think of these people obstructing the work as ‘confused’.
They are confused about what’s happening below, or above, or why you’re
trying to find out, or even what value the software will bring to their
lives. The reasons don’t matter, they are just confused. Building
a system on confusion isn’t any better than building an apartment
building on a foundation make of foam. It’s obvious that it’s not going
to work, even if it looks promising for a moment. If you build on
fantasies or delusions or ignore hidden secrets, the result may run, but
it’ll quickly drift away from being useful. It doesn’t matter how much
data you collect if the data is scrambled or incorrect. And if you
ignore the people who ultimately use the work, then they won’t use it or
they’ll use it badly, causing massive support headaches. A forced, but
miserable user-base can disruptive. Software
has the power to transform, but only if the embedded intelligence
within it is correct, and that only occurs if the programmers were
exposed to the truth. No truth, then there is no intelligence, and so
the results are just burning through wasted resources. This is the
fundamental rule that binds the code to something useful in the real
world. We’ve known this for a long time, referring to issues like ‘ivory
towers’ when talking about programmers who write systems far away from
their users. But
this can also happen when the business or management players seek to
block access to the truth. And it can also come about as a failure to
properly complete reasonable ‘business analysis’, or because the domain
experts are confused or lack enough experience. There are a multitude of
different ways that confusion can enter into the process and obscure
the truth. Distort reality. And without enough truth there is no
understanding, which always prevents success.