A sage once gave a twist to an old adage - the best trick that chaos ever pulled was in making people believe that it wasn't real.... "See the pattern?" my friend Mark, a chaotician, waved his hands over the multi-colored fractal images on the screen, "the roll to the left and to the right?" My eyes had to adjust for a moment, it was a lot like reclining and imagining shapes in the clouds... "So the question is - does the pattern exist because chaos has some degree of order, or - is the appearance of order simply artificial because it's being manufactured within a structured system - the computer providing the hard walls to control chaos that would otherwise unravel into something meaningless?" In this session, two others present were new to the development game, Jeff and Steve, so I let them ask the questions. Mark continued, "So you've got software systems and you're curious as to how to reduce instability. The short answer is - reduce the opportunities for chaos to take a foothold." "I'd rather just control the chaos," Steve said naively, "or just eliminate it completely." Mark smiled, "That's the grand illusion. The essence of chaos is that it deceives us, or it wouldn't be chaos would it? The short answer - it cannot be controlled, at least not completely. It can be harnessed, like a horse is harnessed for work, but don't imagine for a moment that chaos ever goes away." I had hoped this would drive toward a focused session and not a lesson on philosophy. "I can make several predictions, knowing nothing about your software or systems, but knowing something about how chaos creeps in, and I can tell you where to shield yourself." "Go on." "Moving parts," he said, "it's that simple. Reduce the moving parts and reduce the chaos." "How many is too many?" Jeff asked naively. "One too many is too many," he grinned, "so ideally one moving part is the goal, but it's not realistic. If you're system is unstable, either there are too many moving parts, or not enough framework to control the ones you have. Capacity planning is more than just about disk and memory space - it's about building something that is within the physical and logistical limits of your environment." "Physical and logical- " Steve corrected "Not logical - logistical - the ability to track, monitor and administer the moving parts. Look, when you're at a swimming pool, is one lifeguard always enough? Of course not - it depends on how many people are in the pool. When you have kids in a classroom, is one teacher always enough? It depends on how many kids are present - and that is a logistical problem." "And it affects the effective capacity of the procedural environment," Jeff agreed. "Exactly - but the physical is obvious. Logistical is not. I think it's safe to say that all of the problems you are experiencing now are logistical. Here's an example," he pointed to the Ab Initio graphs displayed on three different screens. "All of these graphs run one after the other. They never run separately, and each one completes its mission in five minutes or less. That being the case, why is this divided into three, and not just one?" Steve's answer was instructive, "So more than one of us can work on the complete problem, " he swallowed, "if need be." "Aha," Mark replied, "you have what you think is a logistical priority with developers, being able to apply more of them - but it is overridden by the lack of communication between them. Instability will creep into this flow over time because chaos will see to it." "Will see to it, " Steve huffed, "you act like it has a mind of its own." "Well, it does, in a sense," Mark said, "chaos does have predictable patterns, but they are more in the way of guarding yourself from the effect of chaos more than trying to trace the chaos itself. Two of the patterns you must remember are the butterfly effect and the concept of attraction." Jeff and Steve sat forward, and I must admit Mark has a very intriguing teaching style. "Firstly, chaos has natural and artificial forms. We can manufacture all kinds of chaos. Yell fire in a crowded theater and you'll see what I mean," Mark began, "and in whatever form, it mercilessly attacks a complex system." "The butterfly effect - basically says that tiny changes in one part of a complex system can significantly affect other seemingly unrelated parts of the system. The concept of attractors means that once chaos sets in, it acts like a magnet and draws to it forms of chaos that would otherwise have passed without incident. If you have a sheet of glass with two tiny holes in it, then pour a gallon of water over the glass, will the water find the holes?" "Well, it's not looking for the holes, so how can it find the hole?" Jeff asked, disbelieving that inanimate matter can display animate behavior. "It will fall through the holes because it seeks a level," Steve said. "Don't miss this - the water found the holes and fell through. Even though the glass acted like a shield the water found the two holes and the majority of the water fell through them. This is both concepts rolled into one - the butterfly effect is the two tiny holes that allow all of the water through, and the attractor effect is that the water went through the holes and not another way." Mark continued, "In the inner cities, many people complain about the seeming "reign" of chaos, as evidenced by the crime rate - but the aspect of chaos - that is random crime and mayhem, is an attractor for more of the same - hence a "high crime" area becomes an increasingly higher crime area. Here's a thought - why is it that the areas highest in crime are often those associated with crime - like crack houses, liquor districts, or even police stations? It is because the presence of chaos attracts more chaos." "In the 9/11 attacks, " Mark took a more serious tone, "the terrorists depended directly upon the natural chaos in the airline passenger monitoring environment, and the natural chaos in the national security systems that had grown to rely heavily upon technology for sureveillance and monitoring. They used only cash to keep themselves "off the grid". So that even if some dramatically complex filtering mechanism for credit card systems had been in place, cash is king. For the systems that they knew about, one of the terrorists was able to fool it into thinking he was in another country even as he was boarding the doomed plane." "Each of these systems has natural logistical kinks in the pipelines and chinks in the armor. While nefarious characters can eventually find the holes and exploit them, natural chaos creates a funnel effect - an attractor - making them more easily found and exploitable." "So how does this apply to our problems?" Jeff wanted to know. "In data processing systems, "Mark explained, "the chaos arrives in the data itself - fortified by null, blank or erroneous values waiting to bring down your application. Other sources include the systems, their software, the CPUs and network support, the supporting ERPs and DBMSs and their supporting systems, and the list goes on and on - almost infinitely. "The total absolute count of moving parts is potentially infinite, or at least unknown - so harnessing it means a lot of detailed machine-facing diligence that over time weakens, becomes brittle, reveals hairline fractures that ultimately endanger the entire structure. Each time we add a new moving part to the flow, it creates a gaping hole for the chaos to hijack our innocent, well-meaning operation and turn perfectly good data into garbage." "The Ab Initio GDE and Co>Operating system work in tandem to tighten down and close practically all of the holes before we've even started thinking about the application problem." "So we're good," Jeff smiled, "then what's the problem?" "Exactly that," Mark replied, "assuming that you're in good shape, when you probably are 99% of the way to your goal." Steve and Jeff both grinned, "But the 1% is where the holes in the sheet of glass are." Steve and Jeff stopped smiling. "And that's why you can't afford to assume that you're all the way there. Chaos never sleeps and always seeks a level - the place where you spent the least amount of time, gave the least attention to - or thought would somehow heal itself given enough time. Another aspect of chaos you need to be aware of - it's in no rush. In fact, it's own patience can woo you into complacency. This is the fear of national security leaders now - that complacency will once again give rise to chaos." "The way you describe it, " Jeff interjected, " is as though chaos is enabled by humans." "Bingo," Mark confirmed, "because people are error prone. That's why, especially for mechanical, repeatable activities, it is always best to get the human out of the mix. The human will be a source of chaos, and when things go wrong, will often be an attractor of chaos as well. The assembly lines in Detroit were automated for a reason - quality control." "Here's another example of inviting chaos," Mark explained, "Breaking up these graphs into smaller granular graphs multiplies the total moving parts needed to solve the problem. The left-hand graph drops data to disk, which is picked up by the right-hand graph. Two graphs, their infrastructure, the CPUs, involving the spinning disk drives when we don't have to - so many moving parts - so many failure points. It's virtualy guaranteed that you will reduce your failures by orders of magnitude when you consolidate these operations - you'll reduce your total points of failure." "But also reduce our ability to attack the problem with multiple programmers," Jeff concluded. "Assuming that you need them," Mark asserted, "ever wonder that the reason you need so many developers is because you have so many developers?" "What?" "The presence of developers increases chaos - they are error-prone humans - so more developers means more need for developers, and people to manage developers - more humans in the mix. Fewer developers means a lesser need for developers.. One actually feeds the other." "I don't buy it," Steve said flatly. "Remember that gig we did at the bank?" Jeff reminded, then explained, "there was a team of fifteen developers who took eight months to get their stuff into production, but our team of two people had several times the functionality to produce and were in production within three months. " "Exactly my point," Mark agreed. "Now wait a minute," said Steve, "we need lots of developers for these problems - there's always too much work to do." "And you fan the flames by adding more developers, and increasing the chaos rather than decreasing it. Suddenly you are controlling developers and not harnessing business functionality." Mark went on to explain that because chaos will attract chaos - if the instability in our implementation refuses to stabilize, it means that chaos is being attracted to our application. In other words - the hairline fractures are no longer hairline - they are getting bigger, and the holes are inviting chaos to come and party. "It's not how smart we are," Mark said, "or how well we sling code - it's whether or not we respect the unspeakable horror of chaos - and embrace its presence as something to be harnessed and shielded rather than ignored, or even feared." "Whenever our data processing center has an issue with an application, especially around its initial promotion, the terms "putting out fires" or "firefighting" always pop into the conversation. The "fires" are invariably associated with some form of logistical chaos the developers never anticipated. Many of the fires are configuration problems - things the developers could have avoided - that have little to do with the business logic." Mark noted that the benefit of using Ab Initio is that the product developers, in building the product - have boiled countless years of experience into the software, and in so doing have refined countless points of potentially devastating chaos out of the mix. Mark said, "Now, quite unfortunately, when chaos reduces an Ab Initio application to rubble we cannot blame the usual suspects. We've either over-engineered something, created too many moving parts, or we've opened holes in the application that must be closed - and closed quickly. In many cases the problems are actually artificial- we create them ourselves by not following basic, time-tested rules." "Most of the time, an implementation isn't perceived to be in trouble until it starts to attract chaos, evidenced first by logistical instability. What are some signs that your Ab Initio implementation is attracting chaos? What are some things we can do to keep chaos at bay?" Mark gave us the following simplified guideline, but is by no means exhaustive: Attractors: "XYZ will not change" said one of our developers, wanting to gold-plate some application constants into a graph-level sandbox. If you ever hear a developer wanting to institutionalize something that "will not change", be aware - this is the structure that will incur a hairline fracture first - because the assumption of "no change" means the implementation is brittle - and crackable - at the outset. It also betrays something else - that your developer has never taken an implementation into production before. Declaring that an application quantity will not change is like saying grass will never grow. It's the assertion of a simple and inexperienced mind. Battle-hardened developers, especially those exposed to adaptive approaches, understand that chaos is the rule, not the exception. "We have time to fix XYZ later," said one of our developers, while another one said in the same meeting, "We cannot fix that ZYX on this release,". Pay careful attention to "deferrals". Sometimes developers assume that some things that are critical to you - you will simply forget about. But chaos will not forget - it never sleeps. It is patient. Usually an adaptive approach gives us the freedom to defer some decisions. But putting hard controls on obvious instability is absolutely non-optional. Letting it remain will steal the energy of the project like an Olympic runner trying to sprint through a swimming pool. It will slow down the developers, the testers, and otherwise make one look less than stellar. Keeping chaos at bay:Eliminate or harness all sources of errors, starting with humans. Simple things like eliminating positional parameters (using keywords only) and leveraging the internals of the graph to perform everything, including initialization of an application environment. Eliminate wrapper scripts and clever external 'special' processes. Make it simple for testers, administrators and operators. Throwing it over the wall - and making chaos control their problem - will only come back on you in the end. Think functionally and logistically. In short, eliminate the holes that attract chaos and otherwise invite the users to fall into them. Standardize the simple things. Legend has it that Einstein had a closet full of dark slacks and white shirts, all the same. It kept him from having to waste time each morning with wardrobe decisions. While that sounds strange, chaos starts in the simple things, like a hairline fracture - the more complex we make things - the more we artificially attract chaos to things that should never be an issue. In the end, when it looks like we don't have control over the simplest things, the entire architecture will be held in doubt. Is this fair? Perhaps not. It's just reality. Simplicity seems to be anathema to the developer who wants to make things look harder than they actually are. Not to say that Ab Initio problem-solving isn't hard work, but why not focus efforts on the hard things without being derailed by the simple things? The foundation of a house is simply a monolithic slab of concrete - but lots of attention is paid to getting it right. Chaos cannot gain foothold in the midst of simplicity. Complexity is where chaos hides, waits and pounces at the worst possible moment. Don't innovate without a foundation. I see countless developers racing off to build some wild new innovation without ever having mastered the basics. Innovation rests on a solid foundation - so if the simple things aren't under utter, sovereign control - innovation will never gain traction. Follow enterprise rules. Whenever designing and ultimately building an enterprise solution, you are not an island. Find out what you can about the environment around you and make no assumptions.Assume that boredom is a good thing. The graph runs. The green counters flip over. The components go from green dots to blue dots. This causes innovators, architects and system "master" mechanics to decend into the madness of boredom. This is another "strange" attractor by itself - that the developer refuses to accept that the solution is simply that - a flow-based operation with input-processing-output. Nothing fancy, alluring or particularly exciting about the solution, or about testing it. The developer is tempted to innovate, change or spice the flow with something "else" - But this only attracts chaos. A leader at a company walked through a production testing area and witnessed three Ab Initio testers and one developer. All of them were reading books and magazines while the graphs chugged along on their processing missions. The company leader complained up the chain about this until it reached the ears of a seasoned data processing veteran. "Count your blessings," the veteran said, "when the processes are running, and your people don't have anything to do but wait for spontaneous combustion - and it never comes - this is a very good thing." In short - If they are bored enough to have the confidence to find something else to do instead of breathlessly monitor the process - you've succeeded - The developers of the graphs had removed the need for human interaction - and had eliminated a significant source of destabilizing chaos. Is this an exhaustive list? Of course not - but the spirit of the game is to be thinking outside the box, right? For Ab Initio development this means thinking beyond the application problem. It means thinking about how a data analyst will review our business logic and lineage. Is your flow too clever for them? Simplifying will remove a source of chaos - that is the lack of understanding or abilty to understand our "clever" implementation. Thinking logistically Is it too complex for a tester to execute, examine and validate? Is it too complex for an operator to monitor and potentially recover? For the people involved in operating your application, make it simple, boring and self-contained. This part isn't for free, but it's one of the most valuable things you can do for those about to experience the fruit of your labor.