IT And Architecture: Inside-Out Perspectives

The software industry is in disarray, costs are escalating, and quality is diminishing. Promises made by Cloud Computing are still far from materializing on any significant scale: the panel discussion at the recent Batler Group’s symposium [“Business Process Management and Service Oriented Architecture”] admitted that the real cost saving’s available are in Public Cloud only, but it is not transparent enough to be considered by medium-to-large businesses as an IT cost driving solution. There are now many such catch-phrases within the industry, but do they address the real problem? What is the real problem?

The industry is throwing methods and tools at the perceived lack of productivity in software development of less skilled developers hired massively nowadays, creating a large learning curve and a fragmented environment. What is the real catch? Aren’t systems simply tools to assist the business in accomplishing what it must do? Or are the systems more than tools? Do we only need to choose the right tool and the right method to achieve software nirvana? Why is it that software development seems to be so hard?

Two IT professionals - Bruce Laidlaw and Michael Poulin - each with more than 30 years of experience compared their conclusions about the past and present of IT, focusing on what we are doing to make progress. Bruce has been exposed to just about all methods and techniques in the industry through involvement at every level, on projects ranging from a few thousand dollars to projects consuming over $300M, and initiatives which have consumed over $2B and counting. His experience covers the small mom and pop operation, private industry, governments at UN, national, provincial and local levels, and a whole rainbow of industry sectors. Michael has worked mostly in large software development houses and the IT departments of several major financial institutions on both sides of the Atlantic. He concentrates on bridging between the business and technology at the cross-functional and enterprise levels and contributes to the work of a few global Standards Bodies. His background includes many years of mathematics algorithm development, advanced software design and testing, distributed computing, service orientation and development governance.

Bruce and Michael have boiled down their experiences within a few topics:

state of the SW industry

business needs

architects as a business-technology glue

how we do things in IT

architecture as an agility instrument of delivery.

They discuss these topics from their own perspectives with the objective of uncovering what we as an industry need to know and to do to move IT forward.

The State of the SW Industry

To know how to move Software (SW) Industry, we need to know where we are and where we have been.

Bruce

Well designed and well-built software has been implemented throughout time since the beginning of the SW Industry regardless of methods and tools. Poorly written software has also existed throughout that time and continues to be developed today. Unfortunately, most software systems developed from the early days until today fall into the poorly designed and poorly written category. And to be honest, some software developed today falls into the poor category as well.

To be objective, one has to stand back and ask the question: Is the enterprise better off today with the systems it is buying or building than it was 30 years ago? I believe overall that answer is a resounding no. Here are characteristics of the software industry as I see them today:

High Cost: Projects that would have cost tens of thousands of dollars years ago now cost millions. Those projects, even in today’s dollars, would still be a fraction of the cost paid today.

Low Quality: Software quality is certainly no better today than it was years ago despite ‘advances’ in testing and ‘methodologies’ aimed at correcting this situation.

Endurance: There is a myth that software systems last only 5-6 years before they have to be replaced, while most enterprises are still operating on systems that are 20-30 years old.

Technology: There is a theme and attitude within the industry that technologies have advanced well beyond where they were 30 years ago. With respect to hardware, no real argument, machines are smaller and faster; however in software, absolutely not. In fact, it could be shown that productivity has declined in general. Some innovative approaches to business have mitigated some of that by spreading the work around, but software is still pretty much an art, and the tools, though perhaps different, are just tools.

Michael

For the last decades SW advanced in all dimensions: from nuclear power plants to socializing customer forums that affect merchandising. For instance, did we have programs for planning a landscape for our backyard gardens 30 years ago? I did not see many of them and the reasons for this was in our opinion about the role of SW technology – such things as gardening were not serious enough to consider spending expensive computer resources on them. Nowadays, SW is everywhere and does so many different things that any one example would not be informative enough.

Nevertheless, for example, comparing apples to apples, modeling in the financial industry, the level of administrative overhead which has grown on the top of the SW program has increased dramatically and the same has happened with respect to cost. Some people attribute these to the architectural/design and management infrastructures that today are built to handle all kinds of SW programs. However, a majority of these infrastructures are assumed to increase quality of SW products. Why, in this case, do we see an opposite picture in this area? I believe that there may be a lot of different explanations and I do not pretend to cover any significant part of them. I will only point to some aspects that jump out at me:

The scale and complexity of regular SW projects have outgrown the skills of the average SW developer. For instance, people still approach SW projects with the same two ideas as many years ago: a) we have to deliver something that works; b) we deliver the best that we can. However, ‘something that works’ and ‘the best we can’ are not the things that the business needs in too many cases. The IT projects require a much more professional contribution to succeed then before in the form of either architectural work or analytical work within the Business that can fill the gap between needs and skills.

Proponents of simplicity oversimplified solutions, referring to resource constraints as the driving factor. Any further change resulted in the primitive, haphazard, spaghetti-code that coupled everything with everything and contributed into the rocketing cost for following projects.

An attempt, related to massive production of programmers, to organize their work in a manufacturing/assembly-line style. However, even simple programming is a much more complex task than an assembly-line operation and people resisted this approach. This statement seems going contrary to popular Lean development and management method in IT only on the grounds that it has come from the manufacturing in Japan. In reality, adoption of Lean to SW has significantly modified the initial lean manufacturing concepts and benefits from applying these principles still need to be proven.

SW development methods have shifted towards satisfying the interests of individual stakeholders instead of meeting the corporate needs. In many cases, Agile Methodology was used to calm down disappointed stakeholders while it also allowed the business to substitute business requirements with aspirations.

Modern Tools for writing code have contributed to the reduction of SW quality to some degree. Good tools cost good money and, if you are not a perfectionist, you can justify spending this money to your CFO only by demonstrating the failures that happened before. At the same time, cheaper tools are more primitive and encourage programmers to write simple code only. Simple code is not always the right code (though the right code is usually simple) but it is suitable for delivering ‘something that works’. That is, such tools promote a mindset of low quality development into the “standard approach”.

Overseas outsourcing has dropped SW quality returning us a decade back. Being blinded by questionable “economics” of overseas outsourcing, business and IT decision-makers moved IT into an immature technology culture with different understanding of the business values and IT’s role in the company (cultural differences in the business environment.) Though the situation improves slowly, local developers still have to ‘refine’ deliverables from the overseas contributing a lot to the already increased cost of outsourcing.

“We will fix tomorrow what we have screwed-up today.” This regressive project management philosophy has found a “theoretical” excuse in some development methodologies that insisted on implementing only given requirements without concerns about tomorrow’s changes. The business allowed itself to be fooled by the ‘lower costs’ of such projects and had to pay a lot (often much more) for the ownership of the project results.

Understanding the Business of Your Own Enterprise

If you work in an Information Technology department of any organisation of any size, there are two questions you have to ask yourself periodically: “what am I doing?” and “why am I doing what I’m doing?” If your answer is something like “I am doing this for my paycheck and I will do whatever it takes to get paid”, nothing is wrong with that and no questions asked. Though it is desirable, not everybody has to be an enthusiast. However, if you are working for an employer and your answer is “I want to produce the most elegant solution or the neatest code”, we have to talk about it because this motivation usually creates waste. The ideal approach would be like this: “I want to produce code that is meaningful to the business and enables the business to grow without having to rewrite this code”. If the people developing your system do not understand your business, and make no effort to do so, then you are going to either fail to implement, or your business will be crippled by the system that is implemented.

Bruce

Who understands your business the best?

In most cases, the persons that understand the overall role and vision of the business the best are senior managers. That vision is not always represented in the lower levels of the organization. The lower levels will have a much better handle on the processes of the business, how the business is organized to accomplish its vision right now, but the lower echelons don’t always understand why a process is the way it is, or when it should change. This is more the case for large corporations than for smaller firms but is a persistent challenge when attempting to understand a business.

Some methodologies begin by having business analysts look at the ‘current system’, whether that is a manual process or an automated one, or some hybrid of the two. It is true that a business analyst can gather some information about your business from looking at the current systems, but more likely than not, what they will learn is how your business works today, which may have little to do with how it should be working. This activity of mapping the current system has its place once we know what the new system will look like and need to plan a transition, but for analysis and requirements purposes it is hugely flawed and can waste a lot of time and money, and set the project on the wrong path right from the start.

Who is this business analyst that will capture this understanding of your business? What is their training? What skill set do they require to do the job effectively? Do they need to be an expert in your industry sector in order to do the job well?

In the ‘olden’ days, a new recruit into IT went through a series of progressive steps, one such example follows:

Junior programmer

Programmer

Senior programmer

Programmer analyst

Analyst programmer

Analyst

Business analyst

It was understood that a new recruit might have the technical skills in place: some favorite language de jour or database technology, etc., but that they would be ‘junior’ in their ability to apply those skills effectively. Part of the training that allowed one to go up through the ranks was to begin to see patterns and to understand how businesses work. What is a general ledger? What is an inventory system? How do those two systems interact? Through this mentored growth, the computer skills were honed, but so too was business acumen and knowledge. By the time someone reached the stage of the business analyst, they understood fully both how generic business works, and how technology and systems come along side to enable that business.

Yet, there were no “good ‘ol days” in software. There were many of the same problems then as there are today. Not every company had the foresight or patience to manage this growth process within their IT staff and all of the normal human foibles got in the way of the right people getting promoted into the right jobs.

However, when it worked, it worked very well.

Today, those seven levels identified above are two, and they are not related to one another as levels, they are different disciplines and there is no difference between ‘developers’ today, whether they have 1 years’ experience or 20. Similarly with ‘business analysts’, you either are one, or you are not, regardless of experience. The downside of this is that very often junior people are controlling the outcome of very expensive projects.

It is also true that experience, though a necessary component of competent architects and analysts is not a sufficient condition in itself. Some people have 20 years of 1 years’ experience it seems. (It is my opinion that a competent architect takes on the role of the business analyst during initial project phases, however the industry has taken to creating ‘business analysts’ outside of the architecture group, and without the experience and depth acquired by the architect. For this reason, I will refer to the ‘competent business analyst’ as a distinct position, but assume competent means they are essentially an architect as well.)

So, what are the hallmarks of a good business analyst or architect? The following expresses my two cents:

they have worked on projects across a broad spectrum of industries

they have ‘been there and done that’ in regard to building and successfully implementing systems

they are conceptual thinkers, able to quickly see the similarities and patterns from one business to another

they are quick learners and can communicate clearly what your business is about in a very short time; perhaps better than any of your staff might

they ‘get it’ when you express what your business is about, and what it needs.

The last point begs a question: who knows what your business needs? Is the customer, the business user or owner, the one who should be articulating system requirements? Are they trained as systems architects or analysts? No. The business user can best describe what the business is about and what it needs with respect to information to do what it has to do. They can even have a voice as to how they see things working, but that is where the input needs to change from the direction to a suggestion. It is (should be) the business analyst/ architect that can best ‘systematize’ the business requirements for the business user.

Many methodologies today break down on this point, and many systems fail as a result. I have heard time and again how the system’s staff complained that the requirements weren’t properly articulated by the customer. Sorry folks, it’s not the customer’s job to develop the requirements, that’s your job, my analyst friend. The analyst must have the skills to listen to the business, and from those conversations, uncover the true requirements. In the vacuum of solid analysis, and the trust relationship that is built with management when an analyst ‘gets it’, the business will force its view of things. Unfortunately, it does so to its peril more often than not. They get what they asked for, but not what the need.

Michael

The role of the business was undermined or ignored by IT for several years. IT people constructed a belief that they know better what and how to do things for the business. Even in centralized IT, developers did not see further than their projects, i.e. what their colleagues did, and some SW development methods promoted this focus on delivering something. When the bridge between the Business and Technology had almost crashed, IT started talking about agility with the Business having a very little understanding of the subject.

I learned about this problem by working hard over the last 10 years with service-oriented solutions. I watched how many companies failed with their SOA initiatives despite being told that SOA magically delivered agility. I know that one of the major reasons for these failures is not only that IT misunderstood SOA, but that IT tried working with the process-centric Business when offering service-centric solutions. In other words, IT reacted to how the business worked instead of reacting to how the business should work.

In many industries, IT has converted from a tool-developer into the solution-developer, which includes business solution development. However, to produce a right solution, IT has to know correct business tasks or business needs. Dealing primarily with the low-level operational business teams, IT is locked into the problems (in the majority of the cases) of this level of the business while more often than not this level reflects business requirements of the past. The truth is that the low-level operational business processes are nothing more than the business’ realization of the functionality and, over time, these processes become detached from the current needs and strategic corporate goals / objectives.

To be agile with Business, IT has to know where the Business is going and where it plans to go in the future, and adjust SW development appropriately. Adoption of business changes in IT takes much longer than in Business because of hidden and undocumented dependencies in the code and significant sharing of the program libraries between unrelated applications, i.e. one change affects many things that must be re-balanced and stabilized before becoming available to the business users again. This means that IT has to be even more knowledgeable about anticipated changes in the Business world than the middle- and low-level business operational units. The economic turbulence of the last decade has created conditions where IT could not ‘grow’ business-savvy SW specialists. This means, IT cannot rely on naturally absorbed business knowledge in the technical teams anymore; IT has to have a dedicated stuff that is obliged to work with Business. This stuff comprises Architects and Business/System Analysts.

So, IT Architects and Business/System Analysts have to work closely with middle-to-senior business personnel to learn intentions and directions, on one hand, and provide guidance on existing and new technical capabilities on the other. This work cannot be done on a project-by-project basis; it has to be done either as parts of the full-time duties, or in the scope of global programs. In other words, IT has to have an organisation of Architects and Business/System Analysts across current and future projects. In situations of outsourcing or frequent lay-offs, it is the very Architects and Business/System Analysts, not managers, who constitute the IT treasure for the enterprise. If you cut the business knowledge carriers in IT, you cut the IT brain. Then you have to work with brainless IT...

Following this line of logic, let me walk one step more toward the business. It is obvious that the middle-to-senior business people do not have time (or desire) to help IT with its problems. An understanding of this fact resulted in the emerging of the institution of Business Architects within the business. Together with Business Analysts, who actually work with management on implementing tactical and strategic business tasks, Business Architects deal with top-level business planning identifying business needs, constructing business architectures, and formulating business tasks for implementation. Business Architects also interact with IT management and Technical Architects. This means that Business Architects and Business Analysts within the business are the people who construct core Business Requirements (BR). These requirements are finally split between business units and IT.

From my experience, and the practice of my colleagues, I know several cases where the BR were extracted from, and dedicated to, the subjective needs of different low-level business processes, and these BR “drowned” IT by exhausting all resources. I see only one mitigating solution for this problem – all BR that are planned to be handed over to IT should go through some form of business validation against the Business Strategic and Divisional/Departmental Plans and the validators should be the Business Architects and Business Analysts within the business. I have witnessed this approach – and this works very well, and when IT pushes for business validation of the BRs, respect for IT grows.

The Role of Architects in Modern Enterprises

Over the last few years, it has become a bone tone agreeing on semantics before getting into debates. So, for this discussion we both agree that Architecture (software, business, technology, hardware, etc.) is defined (not described) in accordance with the IEEE 1471 standard. We can only add to the IEEE 1471 that the elements comprising the architecture

must be fundamental, self-sufficient and coherent

can exist independently from others (i.e. cannot be derivatives from others)

should provide the “views of” (inside-out) rather than by the “views on” (outside-in) the architectural elements.

The last statement has to be explained. Particularly, any view on the thing, regardless who is looking at it or where from (what the viewpoint is), always carries a very high risk of misunderstanding what the thing is in reality because a) it is subjective, 2) it does not know the boundaries of the thing and can consider a part of environment as a part of the thing. In a contrast, a view of the thing is based in the actual knowledge of what the thing is and can produce consistent views for any interests of external stakeholders.

While a notion of Architecture is relatively clear, we still cannot say the same about the role of Architect. There is no standard industry definition of what the characteristics are for an Architect, and there is a definite partition between the two primary camps of opinion. One says that an architect is a super-technician that understands intimately some or all of the current technologies: such an architect is called a .Net Architect, or a JEE Architect, and so on.

The other camp suggests that an architect is less technical, and is more concerned with ‘systems science’ and its methods, which have nothing to do with automation per se. This discussion presumes an architect to be of the latter kind, but one that has also a depth of technical expertise as a result of years of experience. By the 'less technical', mentioned above, we mean that the Architect is the person who knows why, where and when, e.g., messaging has to be used in the solution; whether it is a vendor-developed messaging or home-made messaging is immaterial from the architectural perspectives (though it is very important to the implementation and management viewpoints). This architect will not be the leading-edge technical expert, but will be able to communicate between the business and technical worlds competently. In this context, the super-technician is referred to as a Software Engineer or a Tech Lead rather than an Architect.

Bruce

Titles are thrown around with too much ease today so I hesitate to suggest that the best business analyst is a systems architect, but in a world where those people have learned their craft through experience and skill, the architect is the one with that experience and insight usually. In that case, the role of a business analyst is played by the architect prior to the architecture being crafted. In either case, the resource doing the business analysis should meet the specifications listed already.

When the business requirements are properly articulated, it should be clear to anyone within the business who reads the requirements, just what is being requested. The requirements do not state the design, though the requirements can highlight some directions for consideration within the design. For example: if the vision of management is for a customer self-service portal, then it is appropriate that the requirements say so. However, to suggest that a customer number be 20 alpha-numeric characters with a breakdown of meaning is not a business requirement, it is someone’s idea of how customers can be tracked and is best left to those experienced in these things.

The enterprise that understands the need for a coherent strategy within IT to meet the needs of the business, will invest in a business architecture consisting of a business activity model and a business information model. These models are NOT related to automation, but rather capture the essence of the business with respect to what it has to do, and what information it requires to do it. If these models exist, then all systems projects will have a firm foundation and place within the overall strategy.

In lieu of this kind of strategy, and lacking these models, a systems project is forced to do the best it can within the scope of its time and budget. With staff that has an enterprise vision, this approach can work, resulting in the enterprise business architecture incrementally. Unfortunately, this rarely happens. The best one can hope for in this environment is that the redundant parts of systems and standards don’t conflict too much with one another, or cause expensive data integration processes to be put into place to ‘fix’ the disjoint data and processes.

In conclusion, your business requirements are best understood when competent business analysts and architects work closely with those within your enterprise that understand the vision and direction of the business. Through this cooperation, a coherent and useful requirement document can be expressed that will serve your business, not only for immediate system projects, but for business requirements that are planned into the future.

Michael

The discussions about the position and role of Architects generate a lot of heat nowadays. Some enterprise managers still do not accept the project role of Architect over the Business Analyst. At the same time, the institution of an Enterprise Architecture and related Architects conquers traditional business areas more and more.

Technical Architects have brought a structural reasoning and formal logic into the business operations built on very personalized human interactions and relationships. This may be good for some business activities but not for all. Mechanical application of automation and formalization certainly works for tangible Value Chain methodology but people are not mechanisms. Neglecting intangible, interaction-oriented values of Value Networks makes business and technical innovations more difficult to implement and, thus, slower and more costly.

However, close cooperation with Business Architects can benefit not only IT Architects but also the enterprise itself. Such cooperation is not very productive over the border between Business and IT and requires a new organisational form. Both Business and Technical Enterprise architects have responsibilities covering an entire enterprise, which means that the only organisational form suitable to the task is a cross-functional and cross-departmental team where Business and Technical Enterprise architects work together. The authority of this team is equal or above the authority of separate business and IT departments and oversees enterprise level programs and projects.

The Enterprise Architect is at the head of the hierarchy of architectural function in the organization that cascades throughout all divisions and departments (in Business and Technology) down to each development project. Architects define what should be done at each level of organisation in accordance with the corporate Business Strategic Plan. The corporate management, in turn, concerns itself with how the architectural decisions and solution should be implemented. This means that when a project is set-up by a Project Manager, the related Architect verifies that the project goals and means are set and interpreted properly.

Technical Business/System Analysts contribute into the BR by finding and documenting requirement details and dependencies. Technical Business/System Analysts are not the “staff that have an enterprise vision” but rather focus on particular project needs. What other projects do and whether they overlap (in implementation) is usually beyond the scope. That is, relying on the BR, gathered only by the technical Business/System Analysts; IT has a high risk of successfully moving in the wrong direction because significant dependencies may be easily bypassed. Unfortunately, we have seen this occur too frequently.

I strongly believe that all business ideas, proposals, formulated needs and tasks that are intended for IT implementation, support and integration have to be verified and validated inside the business before being introduced to IT. Business Architects together with Business Analysts within the business have to construct a requirement control pipeline at the ‘entrance’ into technology. Technology Architects transform BR into architectural solutions and pass them to the Technical Business/System Analysts for enriching with details.

So, the pipeline may work like this:

A business need is formulated by the business requesters.

Business Architects and Analysts review the need, challenge it against Strategic Business Plans and corporate goals.

Business Architects and Analysts analyse the impact of realization of this need on the surrounding business execution context and vice-versa.

If collectively approved, the need is introduced to the IT Architects who consider IT capabilities with regard to this need and introduce several high level solutions back to business.

If IT solutions impact the business need somehow (e.g., optimizes it to some extent), a final agreement has to be reached via an iterative process.

Agreed needs are formulated as the core BR and passed to IT.

IT Architects and System Analysts work on the details and produce business/technical requirements for the low-level design and implementation in IT.

This is a universal business requirements construction process that can be applied in any industry and enterprise. This process is quite suitable for Business and IT governance and related quality controls. I think that potentially lost revenue from improperly implemented IT products highlight the worth of the efforts and time spent on defining real business needs in any marketing rush.

A Process for Building “good”/”bad” System

Bruce

Assuming that the ‘right people’ have been found and the requirements have been developed as described above; are we guaranteed to have a better system? Well, yes and no.

If the requirements are well developed, then we have a much better chance of getting a ‘good’ system than if they were not well developed, so from that perspective things are looking better. Unfortunately, there is still a significant opportunity to fail.

Solid technical leadership by someone who has successfully completed a similar project.

Interestingly enough, it does not matter what the ‘process’ was – in other words, the style of development was not an issue. Where then did this failure of management commitment become evident?

It seems that when a project begins, management classically has little patience for analyses. An expression ‘analysis paralysis’ gets bandied about, and time lines for analysis get sacrificed on the altar of ‘producing results’, as though a design was an absolute waste of time. I have worked with very strong technical teams that have ultimately failed to deliver a successful project, simply because management did not let them do their work. Unfortunately, the IT industry has created this situation as a result of using junior staff to perform tasks well above their experience level. In response, many methods have been conceived that get around the problem by simply not doing the analysis. Problem fixed! Well, sort of.

In the first few paragraphs of this discussion, it was stated that the method employed is not the deciding factor in whether or not a system project is successful. Holding to that concept, the methods: Extreme Programming, Agile, Rational Unified Process, and so on will not in themselves make things right. What these methods do is try to accommodate the already broken process, making the best out of a bad situation: and to their credit, they do that to some extent. Unfortunately, if the business requirement is not communicated well through a competent design, what will the developers develop? These methods put the business person with the developer in hopes that the requirements will unfold as they work together to build the system. That is fine if the business persons are exceptional in their understanding of the enterprise and its vision, but not so fine if that is not the case. There must be a vision of what the overall system is to become. That is not to say that the entire vision must be fully articulated prior to any work being done – this vision can be incrementally grown as long as the keeper of that vision (the architect) is competent and knows where s/he is going with the design.

The question is, what makes a system ‘good’ or ‘bad’ for the business? Who judges this, and is it really a black and white judgment?

In reality, any system no matter how poorly conceived, generally accomplishes at least a part of what it was intended to do. It may require more money and time than originally planned to get there, but most systems do get there in some fashion. Unfortunately, a poorly conceived system has a very little capability of making sure the business can grow and will not be able to meet its needs over time.

A ‘good’ system is going to be defined here as a system that meets the business requirements, allows the business to change over time with agility, and is seen as a true enabler of the business by its users.

Conversely, a ‘bad’ system is defined as one that may accomplish some or all of what the business needs to do today to some degree, but does so in such a way as to hamper the ability of the business to change how it does things tomorrow.

Michael

My interpretation of a ‘good’ technical system is tied to the Execution Context (business and technical). I have noticed that in a prosperous time, business changes in the external business environment happen less frequently than in downtime. Depending on this, business sets different expectations and requirements for the same technical system. For example, at the end on the 90s, in the USA, system scalability and performance were the major objectives; nowadays system flexibility/extensibility and security take the top priority.

Business flexibility now demands that technical solutions would be flexible enough to adopt business changes with minimal investments in system alteration, and with minimal investments into the alteration of surrounding systems, and all together should be done for minimal time (this is how I define system ‘flexibility’). This is in a sharp contrast with the old days ‘pragmatic’ practice, which preaches that given requirements must be delivered ’in stone’, with no ‘hooks’ for modifications and replacements like there will be no changes literally coming tomorrow.

The natural process of the implementation of business needs in IT is usually broken and it is not necessary the fault of IT. IT has constructed and inherited a lot of old legacy systems with multiple patches and ‘spaghetti’ code. If the business wants to move forward quickly, it has to fix the IT assets making them flexible. This requires time and investments that are usually absent. Cloud Computing and outsourcing will not help much in this because the old legacy systems in IT carry the corporate treasure – knowledge/logic and data, which is not easy to outsource.

IT generates one agile method after another just to build good systems. Majority of agile methods are based on assumption that business cannot and will not work differently than it does today; these methods do not dare to challenge the way how business works even if this way is contra-productive for the enterprise. This leads to the trend where many “agile practitioners advocate simple design where there is very little or no emphasis placed on upfront architecture and design”. Indeed, the development team is accountable for the delivery of a piece of software in accordance with the Business Requirements; the business value of this software does not concern the team: “If the Business requested this they know what and why they do”.

It is not a rare situation where agile teams promise to add an error handling mechanism, escalations, notifications/alerts, compensation logic, transactional integrity, security and operational stability later on, in the following “time-boxes”. However, all these technical things are IT concerns; they are attributes of Quality of Service (QoS) that the business expects from technology right away, not next time. The business logic is straightforward: “if the software works already [well, does it really?], why wait longer and invest more into the ‘final’ release? Instead, since IT works that well, let them do another task.” This is how agile teams set the traps for their own business and then suffer from overwhelming work of fixing ‘old’ problems when ‘simplifying’ the new ones.

Let us look, for example, at how Lean method would work in the SO ecosystem where a Lean Team builds a SOA Service. Based on amount of attention put on the Lean principle “Eliminate Waste”, it is the first task for the Team. By the way, eliminating waste in the software development is much more challenging task than in manufacturing because there are many uncertainties in any Business Requirements and architectural vision does not necessary reflects in them. So, if we hate and neglect up-front architectural design, how can we justify what is the waste and what is not?

In SOA, any service may be used by any consumer while the one, who initially requested the service and whose requirements were used to construct the service, may be forgotten a while ago. SOA Service must serve unknown users or, in Lean terminology, customers. Every customer can find its own value in the service and this value may be very different from the one targeted by the development team. Does it mean that the manufacturing definition of waste does not work in service-oriented development? Or, maybe, it does not work in any SW development without prior strong architectural design...

Another Lean principle – “Delay Commitment” – conceptually contradicts the principle of “Eliminate Waste”. We might delay commitment, i.e. delay completion of product implementation, because we expect some business changes in our tasks. If we anticipate changes but do not know what they are, how can we identify waste - one thing may be the waste while another one may be very useful feature for the change to-be. I think that the service-oriented formula – “design for changes” - works much better than delayed commitments.

Two other principles – “Build Integrity In” and “See the Whole” – may question the principle “Empower the Team”. The Lean Team is supposed to focus on particular task, which makes the ‘whole’ difficult to observe. And it is even more difficult to build integrity with other elements of IT that are not in the scope of the project. Both these principles belong to the cross-team architectural function rather than to the development team; it is the Architecture that is in position to define the waste.

Finally, it is interesting to note that the principle “Empower the Team” (versus empower the management) is added into the IT Lean method while the original principle ‘Stop if something goes wrong’ is excluded. Does this mean that only ‘sunny days’ in development and execution are anticipated? Without the stopping principle, empowered team and the Lean project manager do not have a mechanism to retreat, i.e. they continue delivering whatever available, including the waste. Thus, there may be only one conclusion out of two choices: either the IT Lean method is still immature in its foundation or it is a wrong method, again.

Can we do better?

Bruce

So, in this discussion we are saying:

the method isn’t a deciding factor in success

the business needs a business architecture

the business should specify need, the architect/analyst should specify systemization of requirements

management needs to endorse a method and support it consistently

the measure of a good system is found in its ability to enable the business.

How do we get there? The harsh reality throughout the history of the IT industry is that you cannot always find the best people, and have to make do with less experienced or less capable staff. Is there any hope in that reality of producing good systems? I believe there is.

If there is a good business architecture that is well articulated, then there is a solid basis for moving forward with requirements. If there is a solid application architecture built upon that business architecture, then the specification of projects can be done in a controlled and progressive manner.

If there are instituted sound architectural practices and guidelines, and an expressed enterprise vision that is supported by senior management, the rest falls into place without as much reliance on the experience and skill of the project teams simply because there is a solid framework already defined within which they work. This is not a perfect approach, but that is the reality we are faced with.

Is it possible to provide those ‘necessary’ pieces without taking years of time?

If the right people can be found to model the business, it can be modeled fairly quickly. Regardless of the business (government, manufacturing, military, etc.), a team of two modelers, working together, to capture the activity and information models, can gather those models quite quickly. In a medium to a large environment, a six month project should suffice to accommodate the need.

The derivation of the application architecture beyond that could require another 2-3 months depending on the coherence of the senior management vision and direction. The time frames involved are subjective and very much dependent on the culture of the organization and the degree to which the organization is buried in legacy applications.

The recognition of the things discussed above can lead to better and more consistent development projects. But there is another aspect of this that can free the business from decades of poor implementations and redundant systems.

The application architecture spoken of above includes an inventory of all of the applications currently in use within the enterprise, and an assessment of how those applications assist or hinder the real needs of the business. Taking a more business requirement focus when looking at these systems provides opportunities to consolidate based on function and on data. Reducing the number of applications in the portfolio can not only reduce operational costs, but provide significant benefits through more reliable and timely data, and a much more responsive development environment to meet the challenges of business change.

Michael

My answer to the question of this topic is ‘Yes, IT can do betterbut not alone’. IT has to involve the business in:

All decisions made with regard to technical solutions.

Capability reviews toward business strategies.

Joining development and maintenance funding.

Evaluation of produced technical systems.

Such involvement has to make the business responsible for its needs and preferences.

When technical architects propose a few alternative solutions to be implemented, the clients and stakeholders have to know that if they choose the cheapest but not forward-driving solution, they will be held responsible for this decision, no IT. I have seen so many situations where the cheapest solution in development became extremely expensive in maintenance. How can one balance these two? It is easy – the business and IT have to target the cost of ownership of the technical solution. Targeting the project/development cost alone has to become the absolutely exceptional cases.

When IT reviews its technical capabilities, it is not a rare finding that technical platforms are old if not obsolete and may be not even supported by the vendors any more. Can such platforms properly contribute into the business strategic goals and objectives? The answer is, probably not. That is, IT has to say in open that available systems do not support strategic business needs demonstrating this with the “cost of ownership” of existing solutions whose maintenance consumes the major part of the IT budget leaving very little for strategic development.

When a new solution, product or system is developed, it has to be reviewed by future business consumers. Traditional User Acceptance Testing usually does not check deeper than the business functionality. In contrast, the business review has to challenge IT products in all of the exceptional or non-standard business situations, for example the case where scalability has to be demonstrated in the real environment rather than in a dedicated test system. If the system passes the review, it will live well; if it does not, it is possible that some business requirements might be refined. In any case, IT together with the business, has to re-set the business expectations and identify improvement directions in the collaborative and proactive manner rather than fighting with reports about bugs in production.

How can IT involve the business more? Probably, I’ll be not that original if I say: a) support and promote an idea of Business Architects in Business, b) support and promote the idea of cross business-technology Enterprise Architecture, instituted by mixing Business and Technical Architects, c) setup a periodic exchange of Business Analysts between the business and technical teams working in the same business domains, d) create a Director-level structure in IT that would be working directly with business management as a pipeline for all business requirements intended for IT, and e) create a cross-project analytical group with responsibilities to analyse low-to-middle level business operational processes and come up with optimisation proposals based on technical capabilities, i.e. make IT proactive.

I see one more aspect of potential IT improvement in Technology Governance. I really mean Technology Governance that covers all Architecture, Development, Data/Information and Management governance areas. This governance is about not only what-is-what and which technologies may be used in the development and testing. This governance is about architectural and design controls over development, development and testing tools and methodologies, project management methods and collaboration over existing and ‘work-in-progress’ systems, products, and solutions. The governing policies have to cover IT involvement in the end-to-end business development, address solution ownership objectives and responsibilities defined in the SLA between IT and Business. Governance may not be substituted by management: governance says ‘what’ and ‘how’ things should be done while management enforces governing policies. If governing policies were be preserved by IT management, there would be a chance to streamline agility to the business via steering the entire IT function from the single authoritative governing position.

I completely agree with both of you, except for the parts where Agile methodology is criticized. Agile at a moment concentrates on one task, but when planning for the task the whole picture is to be kept in mind.

A wonderful article that should be read by all the people involved in IT. It is very important for organizations which are involved in outsourcing, no matter which side of the table they sit on.

It is not to criticise Agile per se ... the more important point is to note that the method has little influence on the outcome of the system development effort. Agile is primarily a development method ... and in that environment models what early software houses did in most cases, and is based in proven approaches. When applied to analysis ... not so much.

I tend to agree with the issues of outsourcing of development. The costs saved are spent either managing the process and outsourcing agency (and it's very hard to work with them in an Agile way,) or fixing their work after it has been finished. For every 6 hours of development, we have to spend 1 hour communicating. In the end, it is only a good resource to use when our local developers are too busy.We've had good results with crowd-sourcing some work such as web design, but we wouldn't use that for doing the development work.

I agree too. People are key. But how to take advantage of collective knowledge? How to transfer knowledge from the experts to the mass of new/average developers in a systematic way? My answer is software (code) reuse.

Reuse-based software engineering is supported by automation and model-driven development methodologies. However model-driven approaches like UML and MDA largely failed due to their complexity. We need new model-driven approaches that, coupled with automation, will be able to increase productivity and reduce errors.

My research has been centered on these goals. ABSE is a pragmatic generative model-driven approach with reuse in mind. You can know some more at the www.abse.info site. There is also an experimental IDE that you can try, at www.atomweaver.com. Some other people are also active on this field like Metacase, the Eclipse Foundation, and Intentional Software.

Some analysts say that model-driven development, domain-specific modeling, and intentional programming will be the next hits (or hype?) in the coming years. What we certainly need, is to redefine the software development landscape. And as Bruce and Michael say on this article, people are concentrating too much on implementation details / rushing for next delivery / playing with shiny new dev toys, and not worrying enough with the big picture.

Reuse is certainly a significant way to save costs, and SOA as an architecture helps to clarify things. Unfortunately, no matter how good the tools, if the proper objects are not understood and articulated, then these really good tools will create really bad services / APIs / Classes / Objects ... whatever the implementation.

One of the major points in this article is about how one ensures that the correct 'services' are able to be identified and deployed, regardless the tools or methods used to build them. It is my opinion that there is NO method or tool that can replace solid business knowledge and sound analysis of 'what the business has to do' and 'what information' the business needs to accomplish what they need to do. That kind of knowledge is present in those that have the experience and broad industry understanding that assists in developing proper templates / best practices for how business works.

"It is my opinion that there is NO method or tool that can replace solid business knowledge and sound analysis of 'what the business has to do' and 'what information' the business needs to accomplish what they need to do."

Agreed. At the end of the day, it all comes down to the decisions people made. And people are the ones who make decisions, not tools...

these days it's worse than in the past, because the so-called 'business' culture hates technologists

the devaluation of software development skill is seen in the outsourcing trend, where all that counts is cost

one of the worst trends is to try to have nontechnical people manage development; this is a consequence of the distrust of 'techies'

yes, there are a lot of 'programmers' who lack the skill, wisdom and depth of truly experienced developers, who are the victims of rampant age discrimination

look at the typical job posting where they ask for 3 years experience for a 'senior' programmer, absurd!

however fashionble, agile is nonsense; there is and never will be a substitute for good analysis of requirements, good design and good architecture

these things take time and are hard and there are no shortcuts

delivering 'business value' in 'time-boxes' is a delusion. any old request by marketing or some 'stakeholder' does not constitute a requirement, it is our jobs as developers to analyse, question, challenge, refine and understand before we rush to implement

the reason methodology doesn't work is that mangement never feels contstrained by it; they want to change their minds without changing the schedule, and think that working more hours is the answer

until the proper respect is accorded to real senior technical talent, things will keep getting worse

The essence is in truely understanding the business ... not necessarily as it is structured (for that can change on a whim) but in terms of what it must do to function as a business in whatever sector it is based. Some things are the same regardless of the business (G/L, A/R, A/P, Payroll, Inventory, etc.) and can be templated and built as software services that any company could use. Unfortunately, if one looks at the plethora of such systems in existance, and the degree to which they differ, one will see the full range of different design skills evident in the designs, and just how integrated (not in a good way) many of those systems are to other job functions.

G/L for example should not know anything about A/R, A/P or G/L. Only those other (outer) systems should know about the need for G/L and how to propose G/L transactions from their points of view.

This is the kind of knowledge one accumulates over time as one works on many different systems for many different customers.

Personally, I don't think one can be an Architect without putting in the time across a spectrum of business sectors and projects.

Hi there - I must say that I really appreciated reading this article and have been sharing it around my workplace. What we knew or suspected, you guys have articulated it precisely. I would like to add a comment to the notion of business analysis and specifically what I have observed to be fundamental barriers to improving software development or business improvement programs. I hope my points come across despite them appearing as generalisations.

1. The quantity and quality of business analysts in the market place seemingly tips towards not enough of the high callibre candidates and too many of those willing to put their hands up to try for various reasons. The constant demand for business analysts to perform duties or fulfill activities ranging from supposed requirements gathering through to specification writing and to selling ideas to business stakeholders means that many employers would rather have someone in the role than not at all. They gamble to pay the price afterwards or consider trusting someones claims on face value. Having 10 resources on a project that don't add value is worse than having none. Resources burn money and time, not to mention the lost opportunity. I constantly hear lamenting from solution architects, project managers and testers on the subject of business analysts and what they don't do.

2. The turf that many happy and productive business analysts walked upon is being encroached upon by project managers who may have the tendency to be the business analyst from engaging with business, determining context and scope, expressing requirements as solutions. Solution or technical architects who seek to engage business more, define processes and events, systems to business interfaces, expressing technical solutions as business oriented concepts. The user centred designer tends to blend into analysis and systems solutions canvasing user interfaces, roles, processes and business needs as 'usability' themes. In my particular organisation (and program) I studied all the project documentation and saw tremendous overlaps that I suspect contributes to this encroachment. Firstly the documentation of other ICT roles demand a level of content similar to the business analysts work at the time or before hand. Second, the contract vendor market place is encouraging workers to compete against each other to move away from pure back office technical dealings to in front of the face business value adders. The problem is that projects don't operate as effective teams or produce effective outcomes.

3. Training in business analysis generally doesn't exist. I don't apologise for this statement and many business analysts I know complain about this situation. Many training courses stretch for a period of days where the trainer or facilitator spends as much time looking at the clock for home time. They would typically read out powerpoint notes prepared by someone else and give out exercises that go over basic systems context mapping or step through a simulation of trying to elicit requirements. At the end of the course or the following week, typically people cannot remember what they have learned other than feeling a sense of frustration that they used up their training allowance and didn't get 'business analysis mastery' or whatever. There are a number of training providers that see this as a major cash cow. Can they change? I don't think so. The better thinkers and doers are happily working and some subjects are esoteric or require years of experience that cannot be compressed into a week or less. It takes a special mind to be able to convert this thinking into reusable patterns that can be remembered, applied to both engaging, thinking, and doing. Lastly, alot of business analysis training tends to lean on the IT side of systems analysis. To get the other soft business skills such as, interviewing, critically thinking, problem solving, researching, communicating orally, writing etc, is something already done alot better by other disciplines. I'm often amazed at how policy people, accountants and lawyers do better business analysis naturally without having to resort to using inappropriate IT business modelling tools. Its frankly embarrasing and creates bad rapport with business who often develop designs and strategies outside of the ICT division and employ consultants or vendors with their workarounds. This is not the golden rule, but happens often enough.

4. I won't harp on business analysts outside of business - you guys expressed the point exactly as I had thought. My experience has been that BA's rarely are given the opportunity to be with business for a range of reasons: - control on the resource and paid by the ICT project or program, managers have the mindset that their sole job is to fill out templates to give to developers or help PMs push through their documents past the governance processes, or they need to sit with the IT lads, because PMs or other IT managers wouldn't want the business to be marauded with IT people asking them questions. Its madness!

5. Leadership and management...what can I say? If we have no vision as to what the enterprise should be, how our people should be used, and what tools or support they require to do their jobs well, what is the use of coming to work each day? I have ideas on this matter which is too long to go into here, but will say that with aging workforces in developed countries reaching or into retirement, the younger generation are not interested in traditional workplaces, training is deficient to get people effective sooner, and people who have good conceptual-social skills are been snatched by other thinking professions - then the IT woes in your opening paragraph will get worser! Outsourcing will not fix it, skilled migration tends to not source the best resources from foreign countries and lastly, adoption of 'cookie-cutter' methodologies create complex web of problems later. Leadership in IT has a big responsibility in this - and promotion to these positions in part tend to be not based on merit or achievement, but on the age old dynamics!

I'm hopeful that things can improve. Many of my colleagues are tired, unmotivated and not sick of IT, but sick of not doing IT properly! I'm hopeful that training can improve and create greater capability where it is needed - I've demonstrated to my own people that with a direct approach, a novel hands on method of teaching, you can achieve notable differenecs in confidence, thinking and perspective.

And that is my last comment, from the subject of the first post - 'perspective'. Business analysts have to gain perspective and know what this means and how to obtain it.

Can't disagree with anything you have said. I would offer this however. Many business approach modeling (if at all) on a project-by-project basis, and so business analysts end up covering the same ground from different perspectives (different business units, different job processes.) If instead, a real business model (capability required with needed data) was done, all subsequent business analysis would be completing aspects of that one model. The end result, I think, would be to reduce the demand on the number of business analysts, reduce the overlap / redundancy often found across projects, and allow for the proper definition of 'services' however one might choose to implement them.

So ... I don't think we are lost, there is a road ahead, but those of us that have been there and done that, need to be used to mentor and direct these eager newbies so that they too can feel the rush of being able to do things the right way. It can be an exciting field once again if we allow it to be.

I enjoyed reading your article, and will encourage others to do the same, both the reading and the enjoying.

At two points I like to react on your text. The first results from my view on organizational life [1]. There are some things based on good theory, combined with solid research findings. Things that are nevertheless taboo. Try telling your management team that shared goals are a nice concept, but quite imaginary. Chances are you're seen as swearing in church. What this boils down to is that there is no such thing as "the business". Not at any given moment, certainly not over longer stretches of time. There are more examples like these, like that cooperation is natural - I wouldn't count on it. The point is that you have to be critical: take the time to make your own model of reality. Listen well to the business, but take into account that we are building our own models of reality. Make sure your own model of reality knows its way with change - do not make it resistant to change though! Only then you can go with the flow without getting disorientated. An old example that often works well: pay attention to the data, it's less volatile than objectives and processes.

The second one has to do with the word architecture. In my opinion the undisciplined use of that word results in it - the word, not the good things that it might point to - having far more con's than pro's [2].

[1] Further reading? ;-) It's rooted by e.g. Weick (The social psychology of organizing), Lindblom (The science of muddling through) and Rorty (Contingency, Irony, and Solidarity). I owe much to these writers, for they were able to take away a lot of my prejudices in all kinds of areas.

I think 'the business' exists ... but as an abstraction behind the organisation. The task of architecture, I think, is to implement 'the business' in the face of all the rest you have identified.

What you highlight is the need for experience that can understand 'the business' in the face of siloed organisational structures, in-fighting, empires, and such.

As to the data, agreed. And by way of example: any two data 'architects' can create a 3rd normal form model from the same information base, and those models can look very dissimilar (see any number of 'industry models'.) Take two very experienced data architects, and they will take the 3rd normal form model and apply abstractions and generalizations (experience) and end up with surprisingly similar models. They do this because they have a great depth of experience across many industry sectors and have learned the templates of 'the business'.

This is why we argue in the paper for more of an 'apprenticeship' model for the development of 'architects'. You can't be one by simply saying so.

I take this chance to thank all people posted comments to this article. I am touched by how many of our colleagues think and feel in line with us.

@ Wil

I've never met a point of view like yours: "there is no such thing as "the business"". I am not sure that your critical point - "make your own model of reality. Listen well to the business, but take into account that we are building our own models of reality" - is a consequence of the 'absence' of the business. What is the 'reality' in this case if not the business?

To m understanding, every person creates own model of reality - this is how and why people live together: the models overlap. If I'd ignore business, I were lost the criteria of doing what I do (except for the pleasure). However, our world is not ideal and we have to pay bills...

Actually, I agree with your last statement: "Only then you can go with the flow without getting disorientated". We have to have our own view and stick with it but this view better be in context of other views. This context is the business. And this is the key - the context is not programming or SW because they are you sticks, the context is what for your programming is done. So, we are on the same ground at the end of the day.

Great high level insights on business and architecture.
by
Marcel Sorger

I think this is one of the most insightful articles I come across in years. The way the roles of architect and analyst (and to some extend project leader) have developed does thigh in a lot of loose ends.In my mind analyst are concerned with the users, architect concerned with the developers and project leaders concerned with the business. This article gives a depth to these roles by defining a business, system or technology focused version of them. Although I have only seen people in enterprises be appointed to such a specific role, the concerns of these roles do exist on a much smaller scale. The maturity of the business determines how much these concerns are heard. Most of the times the smaller the company, the less they want to hear about architecture and just use the architecture that spontaneously erupts.Another great thing is that there is finally someone standing up for the big picture. Especially in agile project some up front thinking about the architecture should be done. Delivering in iterations doesn’t exclude having a plan, just as much as being agile doesn’t exclude having an overall vision.What does strike me is that SOA without a proper thought about the enterprise architecture does very little, as I would expect that some proper architecture and design would naturally flow from the effort. The impact on enterprises as I see in practice as a bare minimum, less maintenance as a service on a server is easier to maintain then a library on many PC’s, services replacing stored procedures for business logic so it can be testes and in a language more suited to the task, force a strict separation between front and back that should enable reuse and flexibility. On the other hand the design of services, as was with components, is key to reuse and flexibility, security of services is a bigger issue than with libraries and versioning of services can be quite complex.

Re: Great high level insights on business and architecture.
by
Bruce Laidlaw

Marcel, thank you for your kind words.

1) Architect vs Senior Software Engineersimilar to the building market ... I view an Architect as being very much involved with the business, and as being the one to best articulate both the business requirements and the structure of the solution. I view the Senior Software Engineer (SSE) as being the best to work with the architect in terms of the technologies to be employed. The SSE is in this context the one that has years of development experience (as would the architect) but is remaining current in the latest technologies (certifications in .Net or JEE, ORACLE, etc.)

2) On the SOA point ... simply moving existing bits of software intot eh SOA space 'can' be beneficial but the real power of SOA comes from recognizing what those bits shoudl be? What are the core practices (not processes) the business must employ to gets the job done? Choosing the wrong services won't help anybody ... choosing the right services (and this is where experience comes in) can allow a business to move ahead at a rapid pace without being hindered by IT.

Thanks for the great article. One damaging thing I have noticed in architects is the Ivory Tower Thinking or wasting resources to cater hypothesized requirements; Agile was a response to that but it went to the other extreme i.e. being shortsighted or not going beyond spoken requirements. Anyway I got your point about the shortcoming of methods.

Interesting to note the glaring conflict between the rise of the various Agile sects, which typically demand frequent communication between an empowered business expert and a skilled pro-active developer, and increasing efforts to de-skill software development to fit a crude Taylorist production-line model, where much of the work is performed by cheaper semi-skilled (often offshore) labour, allegedly guided by a much smaller number of "skilled" architects/analysts/designers.

The separation of analyst/architect and designer roles away from purely technical development roles, and the de-valuing of practical technical experience across the whole software development life-cycle, mean that even those "skilled" roles may be filled by inexperienced staff without the appropriate skills to actually do the job. You don't need to be a technical expert to be a business analyst, for example, but you need to know enough to be able ask the right questions of your business experts, especially in larger/multi-site projects where developers have little or no direct access to the business themselves.

The overwhelming dominance of the OO and specifically Java monoculture, and the increasing tendency for analysts/architects/designers to understand little or nothing about other technologies (such as databases), further damages the prospects of delivering appropriate technological solutions to business requirements. If you think a database is just a big spreadsheet, for example, how can you advise your clients properly on their options, or choose the best technologies to implement any solution?

In recent years, my own experience is that the combination of all these competing trends means you can all too often end up with the worst of all approaches: coding monkeys led by donkeys.

Thanks Chris ... I like your analysis with respect to the dual tensions between the assumption that we can put the best people on the project vs the trend to outsource and apply the cheapest resources. Good observation.