Have We Got it all Backwards with Software Assembly?

I am as guilty of this as anyone else. Back in the 90s I was on a big project to standardize enterprise software. We wrote a few papers about it, and a chapter in a book. We often used the “Henry Ford” analogy, which relates to the impact standards for interchangable parts had on hard goods manufacturing.

In the original story (which the link directly above summarizes), the crucial quote for us was:”The key to mass production wasn’t the continuously moving assembly line, as many people believe, but rather the complete and consistent interchangeability of parts and the simplicity of attaching them to each other.”

But of course in the updated book, Toyota further changed the world from craft to mass production (i.e. Ford’s achievement) to lean production. In software however we are still struggling to achieve mass production, never mind lean production.

The application of the Ford analogy to software is that if you can standardize application programming APIs and communictions protocols, you can meet requirements for application portability and interoperability. If all applications could work on any operating system, and easily share data with all other applications, IT costs (which are still mainly labor) would be significantly reduced and mass production achievable.

The industry has seen many efforts in this direction fail, or only partially succeed. Today’s environment is better than the early 90s, but we still have incompatibilities across various editions of Java, enough differences among J2EE application server implementations to inhibit easy migration among them, and of course a signficant level of incompatibility between enterprise Java, Microsoft .NET, and IBM mainframe environments. Applications that want to leverage the best of breed across these enironments typically have to do a lot of craft, i.e. hand coding.

Seven years ago I remember thinking Web services and XML might finally solve the problem, but perhaps because of the way the specifications were implemented (basically adapting to existing technologies) in the end only a partial soluton was achieved. Yes, interoperability is improved compared to what it had been, but it still requires too much hand coding.

Even though I’ve been working towards the “Henry Ford” analogy for more than a decade, recent exposure to inversion of control concepts (e.g. Spring and Guice) and OSGi makes me think the mass production analogy is backwards for software after all.

The Ford analogy has played out in software typically by positioning the assembly problem as the easy part of the job and creating resuable services for assembly as the hard part of the job. I can’t tell you how many times I’ve heard business process modeling and orchestration tools pitched at “business analysts” only to discover the proper use of the tool requires someone who can actually code.

The easy part should be developing the reusable services. The hard part should be their composition and assembly.

Corporations around the world are squeezing IT budgets, which means looking to reduce labor costs. Many are turning to outsourcing to China and India, and others are looking to hire college graduates in place of highly paid (and more highly skilled) coders.

But almost by definition the Ford analogy can’t work. You cannot really get lower skilled, untrained developers to tackle sophisticated problems such as component reuse. They can create simple objects incorporating business logic, and to use one description, the plainer the old Java object (POJO) the better.

What we need are not simple tools for business analysts to compose services into flows. We need sophisticated tools for architects and designers to import POJOs and plain old anything else, check them for conformance to the plan, and fix them up for deployment. What’s the right analogy here? Farming?

15 responses to “Have We Got it all Backwards with Software Assembly?”

The big problem here is that you’re comparing software development to product manufacturing.
There’s two stages when creating a product:
* product development,
* product manufacturing.
Product development is an iterative process which creates the final design. This design will be tested a number of times before product development is finished. Before ford or toyata start manufacturing a car they’ve already driven a number of prototypes to validate and improve their design.
Once the manufacturing starts every detail of the product is known. Everything.
In IT we think we can design products in ms-word and uml. Then some people will review the design and decide if it will work or not. Manufacturing starts without actually knowing all the details. Without actually having tested the design.
I think we need to think of the a software product as the result of product development. Product manufacturing merely consist of copying and installing the code.
Mass production is not relevant to software development. Neither is lean manufacturing.
You need to look at the product development process of ford or toyota. There are actually a couple of interesting books about lean product development, for example:
* The Toyota Product Development System: Integrating People, Process And Technology
* Product Development for the Lean Enterprise: Why Toyota’s System Is Four Times More Productive and How You Can Implement It

Andrej,
Thanks a lot for the comment. The analogy for software that seems to work best for me is the difference between craft production and mass production.
In software today, as it used to be for automobiles, every application is created essentially by hand.
We can’t assemble applications from reusable parts, but we should be able to, don’t you think?
Another way to put it is that software is still too hard for people to use, and it costs too much to develop applications.
I do not think software projects are very predictable. Even with a known set of inputs, the output is unknown, and the time it will take to complete is a huge guess at best.
But perhaps this is what you’re saying.
I am suggesting we can learn something from other industries about scientific production methods.
For the car industry analogy I was looking to highlight the importance of standards, which we still don’t have in enterprise software.
Eric

One important difference between the “creation of reusable services” part and the “service assembly” part is that by nature the first one needs to have solutions that are pretty widely shared industry-wide (whether through standards or not). If we all expose our reusable services in different ways, they aren’t reusable. The service assembly question on the other hand doesn’t have to be solved in the same way by everyone (there are benefits to portability, but it is not required). Your post seems to imply that the industry at large should be focusing on a common solution to service assembly, which I don’t think is the case. Or do I read you incorrectly? And if this isn’t you point, then what is it? Lots of people are actively working on the “service assembly” problem, in open source and in middleware vendors, I don’t think you can say that this problem is being ignored.

There’s this beautiful tv-show i watch every week: Grand Designs (http://www.channel4.com/4homes/ontv/grand-designs/)
It’s about people building their special home. No mass production.
It’s really funny to watch, because it’s rare that anything ever goes according to plan. It’s just like IT.
One show, after just 5 minutes, they were already 30.000 pounds and 5 weeks over budget, because their foundation choice turned out to be unfit for the soil.
Another show, the skeleton of the house was build, the future owners walk into it, and said: oh, if it looks like this, then we want a glass roof.
In another shows the future owner was also the architect, and he practiced a sort of just-in-time architecture, driving all the builders mad.
I think what is happening in IT is not really uncommon for custom-made, one-off projects. We’re always trying to compare IT to mass production, but when you look at other one-time projects, IT projects are not that unique.
It’s seems like every building project, be it a railway, a road, a big building, or a bridge, is always finished late and over budget.
It think it’s just hard for humans to predict projects and design things just on paper. The best (intelligent) design is evolutionary design. We’re best when we make small changes, and incrementally improve them.
Look at anything around you: computers, cars, television, chips, etc. Was the first design ever the final design? No, cars have been improving for over 100 years, televisions for over 50 years, computers for over 50 years.
We don’t get anything right the first time. So we need to iterate, test and improve.
But to get back to your point, i think we constantly use a lot of reusable components: application servers, esbs, bpel engines, logging frameworks, orm frameworks, ui frameworks, etc. (most of my projects contain at least 25 different apache components).
There’s a lot of reuse.

William,
No, that’s right, I think the industry needs to change the way it thinks about service assembly.
If you look at the Web services model, it’s all about developing individual services using a standard interface that can be assembled using BPEL or SCA or similar.
This still puts the burden on the developer of designing for resuse or suitability for assembly.
That’s the part I think is backward.
If you look at SCA the assumption is that the developer will indicate in metadata how the service is to be consumed, what policies it supports, and whether or not it is distributed. This requires the developer to understand all this complexity.
Eric

Andrej,
In construction, to use the standards analogy, they have standards for all the components, the size of the lumber, measurements of the bricks, size of windows, measurements of the shingles, tiles, doors, light fixtures, electric wiring, plumbing etc.
Sure you can build everything by hand but why?
At one time no one had such standards. I remember visiting Chartes Cathedral a few years ago, and the guide mentioned how lucky the builders were, to have had the same guy for the whole project, so they could use his foot for the standard measure.
And yes, the situation is a lot better than before – there are more standard components than ever. But at the application level it still seems like whatever coding is done on top of the app server has to be redone if the application is moved, or if the application on the app server has to talk to an application on the mainframe, or on .NET, it takes a lot of hand coding because there are no real standards for it. Ok, with Web services there are some but not complete, and not enforced.
Eric

“What’s the right analogy here? Farming?”
Molecular biology. Take a look at how those guys work. You won’t catch them outsourcing to unskilled people. If they outsource it’s for cheaper labour, but the people are just as skilled as elsewhere. They have to be in order to understand the intricate interactions that occur between atoms, molecular components and the finished molecules with the rest of the environment in which it’s meant to behave. Sound familiar?

Eric, thank you for bringing this topic up as it is one often neglected in our industry, what with S-this and W-that, not to mention R-the-other:-)
You write at the end:
“What we need are not simple tools for business analysts to compose services into flows. We need sophisticated tools for architects and designers to import POJOs and plain old anything else, check them for conformance to the plan, and fix them up for deployment.”
I would agree with the above if you were to change the “…need are not simple…” into “…need are simple…”, and keep the rest. I cannot see architects and designers (unless you mean people with business knowledge) being in a position to understand which POJO goes where in the flow of things, as it were. Over and above what you have written, I’d like to see more “tools” which are capable of analysing existing code (it’s going to be lots of COBOL, FORTRAN, etc.) and expose it as POJOs or other easily manipulated entity. Perhaps POJOs are a little low-level for all this? How about tools capable of presenting POJOs and other “routines” as “services”? These can then be “dragged” and “dropped” into any order, adhering to the rules imposed by the business requirements and rules.
Cheers, John

Hi John,
I guess I am thinking that composing services into flows can’t be made simple enough for business analysts, but creating the objects and services can.
With the new inversion of control/dependency injection mechanisms, it seems like the complexity of distributed systems can finally be abstracted away from the “average” developer.
I agree that the tools at the next level up from plain old programming (in whatever language) shouldn’t be exempt from the responsibility to be simple.
But I think the complexity of assembly needs to be recognized better than it is today, since the concept of simple tools at the assembly level puts too much burden of dealing with complexity on the “plain” developer.
Perhaps an architect is the wrong role to pick. Maybe a new role, like an assembler or deployer is needed, someone who understands how to construct the architecture. Like a builder or something.
Eric

Great analogy: architect and builder! Whereby perhaps builder is a bit too generic? However, one does not want to start a discussion about plumbers, electricians, tile layers and so forth. A “deployer”, one might need several with various skills, just as a builder tends to have people with various skills, is a good way of describing what is required here.
Cheers, John

“What’s the right analogy here? Farming?”
A closer analogy would be the movie (film) industry – once a movie has been created mass production (making prints is easy) – same with software – making a copy for distribution is easy. Movies live and die by their image, people involved, marketing, fan devotion, etc and you can never really tell if its going to be successful or not – same with software. The analogy breaks down however after release – there is no other industry except software where major continual enhancements are made to “improve” the product – buildings, bridges, cars, planes, etc are never “re-worked” to the degree software is – to carry twice as many passengers, use 1/10 of the fuel, re-purposed for something completely new and so on – perhaps the way government legislation / regulation is continually “adjusted” is similar(?). Software is a unique activity an needs to be treated at such.

Hi John,
In the builder (or perhaps contractor) analogy I think the plumber, electician, carpenter etc. would be domain specific languages, and the builder or software assembler would know how to put the various special purpose language modules together.
Eric

Eric Newcomer has an entry up asking Have We Got it all Backwards with Software Assembly? Basically, Eric comes to the conclusion thatThe easy part should be developing the reusable services. The hard part should be their composition and assembly.Mysel…

Since I posted the initial entry questioning the validity of the Henry Ford analogy for improving software productivity through interface standardization, there’s been some good posts by Hal and Richard, and some good feedback to the Sys Con site that…