10 SOA Commandments

A note to our readers: As per your request we have developed a set of features that allow you to reduce the noise, while not losing sight of anything that is important. Get email and web notifications by choosing the topics you are interested in.

Service Oriented Architecture (SOA) is an approach to organizing information processing. It is capable of lowering the costs of information systems interoperability, by greatly reducing the number of aspects of these systems that have to be brought into agreement with each other for them to work together. If it can be successfully applied on a large scale, the systems landscape will differ from that of today in much the same way as today’s freight industry differs from that of the pre-container era. However, it is currently being used in ways which result in additional overheads but fail to deliver on these interoperability advantages. Paradigms which are appropriate to database era are being applied to SOA, resulting in counterproductive, often stupid and sometimes downright dangerous designs. These paradigms must be replaced by patterns of thought and behavior which ensure that SOA leads to simpler interfaces, functionally better IT solutions and more manageable projects. This can be achieved by adhering to ten basic commandments.

Introduction

The potential impact of SOA

Service Oriented Architecture (SOA) is an approach to organizing information processing. The approach consists of describing all interactions in terms of services, in which a requestor asks an agent for something to be done, and the agent ensures that it gets done and delivers a response to the requestor. This way of thinking can be applied at a business level, in order to describe interactions between organisations, at a functional level, to describe how the activities of which business processes are comprised interact, and at the level of information systems, in order to describe how systems and parts of systems interact. At each of these levels the principle is the same: how the agent does what needs to be done is no concern of the requestor, not even whether it is done completely automatically, completely manually or something in between. Nor is the requestor concerned whether the agent delegates part or even all of the work to others. All the requestor needs to agree on with this agent is how the request and the response should be formulated, and what the effect of the service is.

SOA is widely touted as a paradigm with enormous potential to reduce the costs of systems development, testing and maintenance. In particular, it promises to lower the costs of getting information systems components to work together, by greatly reducing the number of factors that have to be brought into agreement with each other. Using SOA, differences in things such as computing platforms and data formats are much less of a barrier to communication between systems than with earlier paradigms. That makes cooperation on a larger scale possible, because it minimises the hindrances imposed by the need to get systems designers to agree with each other and, for that matter, to get systems configurators to agree with each other. If the promise can be realised, the consequences will be revolutionary. Like the car changed the shape of urban regions, the shipping container revolutionised the freight industry, and the lowering of transaction costs enabled modern free market economies to develop, SOA will enable new patterns of cooperation. When SOA dominates the way we apply IT, the systems landscape shall differ from that of today, as a city designed around cars differs from a city designed around trains. For those of us whose thinking is dominated by current technologies, it is difficult to imagine how big a difference SOA can make. But the flexibility advantage that SOA offers is like the advantage of cars over trains: whilst trains can be made to go as fast as cars, they can never support door to door transportation in the ways cars can. It is simply not practical to put a train station at the end of every driveway, or even rails in every street.

To reap the benefits of the new paradigm, we must make use of the new possibilities it offers. Unfortunately, most of the current hype around SOA has little concern for these possibilities. The major discussion seems to be concerned with establishing how SOA enables individual information systems to be developed more rapidly. However, that is not where SOA adds most value. In fact, it’s debatable whether SOA is really an improvement on previous approaches in which separate functions cooperate by sharing a common pool of data, normally implemented in a database. Using SOA to build an individual, isolated information system is rather like using shipping containers to move goods around your manufacturing plant: certainly, it imposes order and structure on your internal logistics, but the containers get in the way more than they help. SOA results in better interoperability between information systems, rather like shipping containers enable interoperability between carriers. That’s an important advantage, because the lead time between requirements finalisation and the information system becoming operational is often largely determined by the interoperability. Getting an information system to work together with the systems in its operational environment normally costs more time and energy than building the system as such.

The focus on the use of SOA within, rather between information systems is a symptom of a wider problem: because SOA is seen as a new way of doing what we have always done, we are unable to reap its benefits. SOA concepts and technologies are being used with current systems development paradigms. These paradigms have been developed in the database era, and incorporate the limitations of database technology. Applying SOA with these limitations will result in additional overheads, but fail to deliver on the additional benefits. But these “databased” paradigms are so pervasive and pernicious that we are often not aware of how much they affect our thinking. They are so ingrained that we regard them to be just plain common sense. Unfortunately, they are generally counterproductive, often stupid and sometimes downright dangerous. They lead to solutions that combine the disadvantages of the databased era with the down side of SOA, whilst failing to deliver on the advantages that SOA has to offer.

What needs to change

The SOA paradigm has its own common sense rules, and these are often completely different to those of databased paradigms. There are ten basic ones. The first five are concerned with making things simpler than the databased paradigms dictate – simpler in the sense of sticking to the essentials. When we make things simpler in this way, there is a dramatic improvement in the probability that different solutions to the same problem will be interoperable. The following four are concerned with making IT solutions better than their databased equivalents, by avoiding the blinkers that lead databased thinkers to develop ineffective solutions. The last is concerned with making IT more manageable, and in particular with organising systems development so as to reduce project complexity and risks. SOA makes this possible – but also necessary – because it allows more functionality to be delivered as infrastructure.

Making it simpler

The rationale

In a trapeze act, effective cooperation is based on each trapeze artist being in complete agreement as to where the other is going to be at each point in time. Some trapeze artists are so confident of knowing this that they regularly perform blindfolded. They can find each other because they have ensured that at any given time they can each have only one possible location.

The successful application of SOA in order to maximise interoperability is very much like a trapeze act. Interoperability demands that the millions of possible solutions as to how parties communicate have been reduced to just one, on which both parties can rely. That doesn’t have to mean that there is something wrong with all the other solutions: whether we drive on the left or on the right is arbitrary, but it is important that we all drive on the same side of the road.

When just one party performs a service for just one other party, it makes little difference as to which solution the parties agree, as long as they agree. Idiosyncrasies from one side or the other can determine the solution, without making the total communication effort significantly greater. After all, these idiosyncrasies will need to be handled somehow, whatever the solution. But if there are many parties requesting the service or many parties performing the service, the picture changes. Then it is important that the communication solution is reduced to the essentials, in such a way that each party has to handle its own idiosyncrasies and no other party is confronted with them.

It is illustrative to compare information communication with transplantation surgery. For a successful transplantation of an organ from one person to another, the transplanted organ must match the recipient on a wide range of factors, the vast majority of which have nothing to do with the biological function of the organ. In other words, the transplanted organ must have the same idiosyncrasies as the recipient. As a result, our organs cannot be transplanted as if they were Lego blocks. Currently, information communication is very similar. For one information system to perform a service for the other, they must agree on a wide range of things. They must use the same vocabulary (metadata), the same set of functions which can be invoked by the one and executed by the other, the same expectations for the data content of the questions and answers for each of these functions, the same coding systems, the same technical communications protocols, the same addressing scheme in order to channel messages to and fro, compatible notions of the tempo in which a reaction is expected, compatible techniques for ensuring that messages don’t get lost, compatible authentication mechanisms in order to ensure that they are talking to each other rather than to imposters, compatible encryption techniques and key management in order to ensure that messages cannot be listened to or intercepted, and so on. In order to facilitate interoperability, we must ensure that parties that start standardising on these things independently of each other arrive at compatible standards. This is possible only when some very rigorous rules are followed, so that interfaces are reduced to the essentials. This way there is no room left over for idiosyncrasies.

The rigorous rules are all concerned with making service interfaces simpler. The less we need to specify, the smaller the room for disagreement.

1. Don’t know what you don’t need to

What you don’t need to know can’t hurt you

The essence of the SOA paradigm is that it enables maximal cooperation with a minimum of agreement between the cooperating parties or systems. That is an enormous advantage, because anything you don’t need to know doesn’t need to be tested or maintained either. What you don’t need to know can’t hurt you. Given that 40 percent of the systems development costs go into testing and up to 80 percent of the lifecycle costs of an information system are incurred in the maintenance phase, anything that the SOA paradigm liberates you from having to know represents money you can save.

Metadata

The most important thing you don’t need to know is the structure, meaning and allowed values – the metadata – of any data which is not used by logic that is built into your system for selection, sequencing or performing calculations. You don’t need to know this, because SOA technologies make the metadata available wherever the data is. Your system can interpret this metadata on the fly, so that you don’t need to build the knowledge of that metadata into your system if all you want to do is capture, present or pass on the corresponding data. With a sufficiently sophisticated presentation facility it is even possible for users to perform all sorts of ad hoc selections and calculations using only the metadata that has been provided together with the data, rather than inbuilt metadata.

By interpreting the metadata that is provided along with the data, rather than building the metadata into the system, your system does not need to be changed when the metadata changes. Only the source system needs to be changed. Think of the savings in development, testing and maintenance if you can apply this principle! Remember that making a change in two systems rather than one is, on average, four times as complex, because of all the coordination involved.

For many consumer-facing systems, presentation and ad hoc selection covers most of their functionality. Such systems require inbuilt metadata for only the most basic customer data. That does not include current and past orders, customer contacts, photos, correspondence and whatever other data may be available for display, all of which can be presented in a manner which does not require any knowledge as to the nature of this data to be built in to the system.

Technology

Many of the things you don’t need to know are technology-related. With SOA, you don’t need to know whether the system with which you are interfacing is implemented using a Software-as-a service or not, where the computer on which it is implemented is located, what type of computer it is or which operating system it is running on, how the firewalls are configured, which database management system is being used, or which transaction management system is available. Other things that you don’t need to know concern the internals of the systems with which yours communicates. In particular, you don’t need to know anything about the metadata which it uses for its internal data storage, because any transformations that the other system may require in order to conform to the XSD are its problem, not yours.

Having said that, there remains a wide choice of technology-related standards on which parties that communicate using SOA must come to agreement. In particular, there are so many that standards concerning web services that SOA practitioners refer to them collectively as WS-* hell (with the * indicating that there are many possible labels that can replace it). In part, these standards have arisen precisely because the SOA community has not been content to not know what it didn’t need to know; this white paper presents guidelines to reduce the impact of the problems that gave rise to such standards. When these guidelines are followed, SOA requires substantially less prior agreement than other approaches.

Designing stable interfaces

If you want to reap the benefits of SOA, not knowing what you don’t need to know must become your second nature. Apply it everywhere! For example, when designing a service to order goods, remember that the service requestor needs to know whether the goods will be available when he wants them, but does not need to know the current stock level. And if your application invokes a security service in order to determine whether a requested action is authorised, don’t build in any more knowledge of the workings of the service into your system than is necessary. For example, if the security service uses the credentials supplied with the input to the application, all it has to do is pass them on! As far as you are concerned, they are just a single data item which is encapsulated in your incoming message. Don’t even check whether the credentials are well-formed XML. If, for reasons known only to them, the security boys have chosen to deviate from standard SOA practice or to encrypt the credentials, it is their problem, not yours. If they change anything related to the credentials, your application should not need to be changed or adjusted in any way. Anything you don’t need to know can’t hurt you. Unless, of course, you are so stubborn as to want to know it anyway, in which case perhaps everybody is better off if you don’t waste your time on SOA.

Not knowing what you don’t need to know is harder than you might think. If you develop information retrieval services specifically for a particular information system with which yours communicates, you are already on the wrong track, because you have incorporated some knowledge of that other system into yours. Any change in those requirements will necessitate a change in both systems. In general, a better approach is to expose your system’s data using a limited set of retrieval services, which, when used in combination, cover all the information retrieval requirements of attached systems. For example, a product database may be exposed by separate services: a simple one with just the code, description, units and list price of the product, a service which exposes all the financial aspects of the product, and a service which exposes all the logistical aspects of the product. Many systems will be satisfied by the simple service, and most will need some but not all of the data of either the financial or the logistical service, and some will need to use both together, but there will be no system that needs a special interface on top of these. This way of working is known as the McDonalds approach: the customer assembles his own meal from standard products. Supporting this approach is no hardship, because you need such a set of services anyway in order to support customer-facing applications. You can even support very specific information requirements in this way, because data that is not needed can be filtered out before consumption. If you don’t like the gherkin in your big Mac, remove it! The idea is that providing too much information is much less of a problem than providing too little, for the recipient’s systems can easily be programmed to ignore information he doesn’t need, but are stuck if anything is missing.

Not knowing what you don’t need to know also results in profound simplifications of the information exchanges which are required in order to support business processes. In the SOA paradigm, when you ask another agent to do something for you, that is all that you do. You don’t provide the agent with additional information which may be useful or even necessary for the agent to carry out the task. You ask for the dish, and it is the task of the cook to ensure that the ingredients which go into it are available. You ask what you want, and shut up. The agent may in turn request some or all of the information from you using an information retrieval service, but it’s his problem to decide what information to retrieve, when he wants to retrieve it and where he wants to retrieve it from. You don’t need to know about it, and still less do you need to incorporate that knowledge into your system. This way, changes on his side will seldom require a change on yours. For example, nothing needs to change on your side if he decides to discontinue maintaining his own copy of your data.

It is of course true that not knowing what you don’t need to know results in inefficiencies. Some operations that can be carried out in a single exchange will now result in multiple steps. The McDonalds approach often results in several services being required where one would have been sufficient, and leaving the other side to retrieve the information that it requires almost always results in more messages than strictly necessary. There will be situations in which it makes good business sense to optimise these communication patterns. There will also be many occasions in which you will want to optimise the user interface, if only because current presentation facilities are not very good at providing users with attractive interfaces. But before you optimise, consider what flexibility you are losing. And never, ever optimise anything for which the functional requirements have yet to be stabilised.

2. Don’t know what you can’t know yet

Premature specifications freezes

One of the real bugbears of databased paradigms is that it requires you to fix data structures in concrete before you know enough to be able to do that properly. That’s because they ignore a fact of life: users don’t know what they want until they see what they don’t want.

It works this way: once you have finalised a data structure design, any subsequent changes are going to require messy database conversions, in addition to changes in each system which accesses the database. All of these changes must be coordinated, which is hard enough if all use of the database is limited to a single system, but gets much harder when there are many systems, particularly if some of them are managed by parties not under your control. In fact, making these changes is already a problem in the systems development phase. In consequence, database designs are frozen early on in the systems development phase, and data analysts bend over backwards to get the design right.

The problem here is that data analysts have an impossible job. They have to finalise a design before the users can understand it, let alone appreciate how it will work out in practice. It is only long afterwards - once the system has been built - that users can get a feel for the system and assess how well it meets their needs. If there are any major problems with the data structure, it is far too late to fix them.

SOA prototyping

“How is SOA different? “, you might ask. When all is said and done, doesn’t SOA need to structure data just as much as databased paradigms? The short answer is that SOA works just as well when requested data is processed by a human agent as when it is handled by an automated system, and people can interpret data even when it is not optimally structured. For example, users can determine whether a letter is being sent to another country even when its address is presented as line 1, line 2, line 3 and line 4, whereas an information system would need at least the country to be an identifiable part of the data structure.

The long answer involves a discussion of SOA prototyping, which is carried out as follows:

Identify the metadata which should be built in to the system. Put it in a primary namespace, and store its data in a DBMS according to its structure in the conventional manner. For example, for a customer this metadata may consist of the customer-ID and the customer name. Because this metadata is built into the system, it is possible to build logic into the system in order to use these fields, for example to retrieve records using the customer-ID and to sort and select records on the basis of the customer name.

For each database record, put all the data that wasn’t included in the primary namespace into a single XML-string which is appended to the database record as a single field. For each such XML-string, make a secondary namespace, develop an XSD, and add a single data item for the string to the primary namespace. For the initial implementation of a customer record, this string may include metadata for address lines 1, 2, 3 and 4. The XSD of the customer record itself will then contain metadata for three fields: the customer-ID, the customer name and the XML-string containing the additional customer data. The XSD of the additional customer data will include metadata for each of the address lines.

Capture and present all data using XSD-based logic for each relevant hierarchy of database records, augmented by a single, standard validation service per XML string (including the string for the primary record). If at all possible, use a tool to generate interfaces using the XSD’s, rather than programming it yourself. This tool must unpack the secondary XML-strings using the metadata that they contain, and must capture new records using both the primary and the secondary XSD’s. The result is a working prototype which the user can use to assess the usefulness of the intended system.

Adapt and change the XSD’s and the validation services as necessary until the users are happy with the system. In the case of the customer record, a new XSD for the string may contain metadata for street address, zip-code, state and country. Data stored using the old XSD will still display correctly, so there is no immediate need to convert it. New records will be captured using the new XSD.

Consider migrating data to conventional database structures and to the primary namespace once both the users and you are convinced that the metadata is stable.

Of course, SOA prototyping is not always necessary. Apply it when the requirements are subject to so much uncertainty that it is more economical to prototype first and stabilise later. But remember that it is much easier to do than databased prototyping. In the databased era, prototyping was an option, but in the SOA era, it’s a way of life. Don’t fix designs in concrete until you know they are right; it’s as simple as that.

3. Don’t do more than asked

Product-driven service decomposition

One of the main strengths of SOA is that it is a business concept which is translated into technology, and not, as is the case for the databased world, a technological concept trying to catch up with the business. In SOA, each service must clearly add value, not just in some abstract sense, but specifically to those who invoke it. Everything that happens does so because a service requestor asked for it to happen. By not implementing anything within a service that the service requestor did not explicitly ask for, services can be limited to their essential functionality. If all participants in an SOA initiative do that, the result is an enormous increase in interoperability.

The highest level services in SOA are the business transactions: a customer places an order for goods or services, which – unless the order is rejected by the vendor – results in their delivery. In the SOA paradigm, everything that happens between order acceptance and order delivery can and should be implemented as services too. For each intermediate product or state that is required in order to deliver on the order, the vendor requests somebody or some organisational unit – not necessarily within the vendor’s own organisation – to deliver it. Each of these intermediate services in turn may themselves be decomposed into services, and so on, right down to the level of elementary units of value addition.

The hierarchical service decomposition is an essential part of the SOA paradigm, because it enables the delivery of services to be managed by means of service level agreements. It ensures that there is somebody responsible for each service, and that service consumers know what they are getting.

Common intermediate products and common services

Service decomposition is all about products. Processes only come into it when the subservices of which a service is composed cannot just be carried out all at once. For example, if the service determineCustomerOrderAcceptability requires that the subservice determineCustomer­Creditstatus is carried out after the subservice determineOrderValue, then the implementation of the service involves a process. Each such process is concerned with only one service. If you find you have a process which cannot be limited to a single service in this way, you have probably forgotten to model the initial customer request as a service.

It is possible that the same intermediate product – and hence the same service – may be required for different higher level products. For example, business products that require substantially different processes to be delivered will normally differ hardly at all when it comes to the intermediate product in which the customer pays – in order to extract money from the customer we need an amount, a date and a statement of what gives us title to ask for the money, regardless of what that was. In such cases, the normal line is that if there is a common intermediate product, it should be implemented by a single service that can be invoked from multiple higher level services.

Only the result of such a service matters – not the starting point – because collecting the information that is necessary to deliver the service is part of the service, not of the environment from which it is requested. Note that the decision whether to have a single service is a business decision; it has nothing whatsoever to do with information technology. If it is practical and useful to deliver the same intermediate product both now and in the relevant future, there should be a single service. But if there are business forces at work which now or in the relevant future will result in structurally different intermediate products, we are better off with separate services for each set of requirements. The bias is, however, towards single services. In SOA, things are the same until they are different, whereas in databased approaches, things are different until they are the same.

Common services with multiple behaviours

What, you may ask, is the point of making something one service if it requires two drastically different types of behaviour? Aren’t we busy with the same “One size fits all” restrictiveness that has bedevilled the databased world? For example, if we sell two types of product, one of which employs fixed prices whilst the other calculates them afresh according to some complicated formula, why not have two billing services, each tailor-made for specific types of product?

These are the right questions, asked at the wrong place. They are the right questions, because any design approach which cannot adequately handle the variation that occurs within the problem domain is doomed to failure. But they are asked at the wrong place, because the flexibility to adapt solutions according to the variation in the problem domain should be built into the services, rather than be built around them. In the case of the billing service, it should determine which of the two billing algorithms should be employed, each time it is invoked. That way, if a third billing algorithm were to be introduced, only the billing service needs to be modified. Note that this does not mean that the billing service has to be designed as some sort of “all purpose” machine; it could equally well invoke separate subservices for each algorithm. This choice – the choice between a multipurpose solution and a framework into which different components can be slotted – is a choice that can be made per service, because it does not need to be known by the service requestor. In SOA, this choice will more often than is currently the case result in framework solutions, because the perceived need for multipurpose solutions is in large part the result of the failure to separate service requests from the information which is needed to carry them out. The days of one-size-fits-all multipurpose solutions with so many parameters that it requires highly qualified specialists to implement them are numbered.

Sticking to the essentials

The process driven approach also enables us to distinguish between the things we do on behalf of the service requestor and the things we do on our own account. The things we do on behalf of the service requestor should be carried out as part of the service, and the rest should not. By making this distinction, we can keep the service as simple as possible, in order to be able to replace it later without having to change all sorts of other things as well. For example, we may manufacture products in order to satisfy an external service request, but maintaining the bookkeeping system is something we do for our own needs, not those of the requestor. If we were to develop a service in which a customer order is translated into both the manufacturing activities and the bookkeeping, we would need to amend the service whenever we implemented a new bookkeeping system. That may not sound so bad, but when we consider all the things we do on our own account it is. Bookkeeping, logging, stocking data warehouses with up-to-date data, maintaining employee productivity data: all of these things and more are typically carried out using systems that differ from one organisation to the next and from one time to the next, so that incorporating knowledge of them within a business service reduces interoperability and increases the difficulty of replacing the service with an implementation via another system. We can avoid this problem by generating notifications whenever something that is important to these functions occurs, and letting them retrieve the information they need to process the event using common services.

Another category of business activities that should not be carried out as part of a service consists of those activities which you will not reverse if the service request itself were to be retracted. Typically, this includes activities such as ordering supplies because the customer order resulted in stock levels under the replenishment level, registering a new customer and updating the information of an existing customer. These activities form separate steps in the process, which should be executed using separate services.

Of course, this way of thinking can be bolted on to databased paradigms. But it is not inherent to them, as it is with SOA. In consequence, many SOA implementations betray the databased thinking which inspired them by identifying services in a bottom-up fashion, rather than the SOA top-down way. In a bottom-up approach, a service that was initially developed for one problem can be re-used for another problem as well, thanks to the fact that the person who designed it gave serious thought as to how to make it more generally applicable. In SOA, the use of a single service in multiple contexts is the result of deliberate design, not of intuition, and it is designed for all of those contexts from day one. To talk about re-using a service which delivers a common intermediate product is like saying that you re-use your front door if you use it to you enter your house regardless of whether it is to access the living room, the kitchen or the bathroom. Can you imagine a kitchen designer saying: “Isn’t it wonderful? The guys who designed the exterior entry to the living room just happened to make something which I can use to provide exterior entry to the kitchen!”? Anybody who talks about “re-use” of, say, a billing service just hasn’t got it. Mind you, it’s not wrong, just strange and not very illuminating.

Services follow the business

One consequence of the SOA way of thinking is that SOA results in services that are expressed in terms of their propositional meaning, not their mechanical implementation. For example, a service to add a customer record is databased, whereas a service to register a new customer is SOA, even when they do exactly the same thing. The label we give to a service is important because it tells us something about who is asking for this to happen and why he wants it. In this particular case, the SOA top down approach leads to the conclusion that a business process needs a valid customer registration, which should be accomplished by amending an existing registration if there is one, rather than automatically creating a new one. In SOA, this is the responsibility of the customer registration service, whereas a databased approach pushes this responsibility on to the service requestor. Similarly, a SOA registerCustomer service will itself decide what the customer-ID is going to be, whereas a databased service may just as well allow the service requestor to decide this.

Convergence to a single solution

In general, SOA top-down thinking results in many design decisions having only one valid option, whilst a databased thinker will see that option as just one of the possible choices. That’s an important SOA advantage, given that SOA is oriented toward interoperability, and interoperability demands that we all drive on the same side of the road rather than negotiate this with each car we meet. Those who ignore this – for example by maintaining that web services are just one of the many ways to implement SOA across systems boundaries – have about as much chance of durable success as a chess player who thinks only one move ahead. True, there will often be simpler ways than web services to connect system A to system B, but what are you going to do in order to make the same data available to the call centre or to an output management facility? What are you going to do in order to ensure that the data warehouse is informed of the events which you transmit from one system to the other, or to inform your customers of such events as soon as they happen? In the SOA world, data and events must be made available to many systems, and web services are the most effective way of ensuring that this can be done with low investments and low maintenance costs.

One domain in which this difference is evident is EDI. Classical EDI is aimed at determining which information may need to be communicated between organisations, and defines a vocabulary for that information. That is not the same as defining a particular information exchange. For example, you can use the same EDI-message to place an order, to query its progress and to amend it. Two organisations who want to use these specifications to communicate must sit down and agree exactly how they are going to use this vocabulary. SOA separates these concerns: vocabulary is handled using namespaces, and these namespaces may be used in multiple services, each of which has a specific purpose.

Another domain in which the SOA top-down approach leads to more specific conclusions is the question as to what constitutes the boundaries of an orchestrated process. Typically, this is a question which leads to endless debates. For example, is the delivery of a confirmation message to the customer using an output management service something that should be orchestrated as part of a customer process, and if so, should it be a fire-and-forget affair or should the output management service report the successful completion of the action? In SOA terms, all of these things are part of the purpose of the customer, and should therefore be orchestrated. Other actions, such as updating the data warehouse to take account of the new customer order or updating the general ledger, are clearly not part of that purpose and should not be included in the orchestration of the process, even if they are carried out by the same party.

4. Don’t do your own chores

Generic functions

A business service should have inbuilt logic only for those functions which are specific to the service. It should delegate all other functions. That way, the service can itself be kept as simple as possible. That makes it more easy to design and test, and, if need be, to replace. It is a matter of elementary mathematics: the greater the number of factors on which a new software components must match, the less likely it is that there is some common of the shelf software (COTS) that fits the bill, and the more expensive such a solution is likely to be if it happens to be available. If you can delegate the non-specific functions using a standard service, that reduces the number of factors on which a match is necessary.

The primary tasks which the service can delegate are chores: tasks which are concerned with the housekeeping rather than the actual business functions. These chores are generic by nature, in other words the way in which they are done is not specific to the business context of the services which they support.

Generic user interfaces

It may surprise you to learn that the biggest chore that an information system should not do itself is managing the user experience. This is generic functionality, and should be handled as much as possible using a standard tool. The user experience includes the work lists from which he may select work items to carry out, the facilitation of work item execution – for example by launching custom user interfaces, if we still have any – and signing off completed work items. It includes selection of transactions which the user may perform, the presentation of input screens – normally generated from XSD’s – and their validation using the standard validation services corresponding to the transaction. It includes maintaining the current user context so that he doesn’t need to re-enter the current customer, product, case, process instance or whatever but can use these values by default and override them when necessary. It includes the presentation of all documents which are relevant in the current user context. It includes the presentation of alerts on which the user may need to react.

All of this can be implemented using a tool that requires no inbuilt knowledge of the business. It is in general much better for the user to have a uniform environment in which all of this is covered than to optimise a user interface for a specific type of action. Where there is a clear business case to deviate from this principle, remember at least this: don’t optimise the user interface before you have stabilised it.

Typical generic functions

Other chores include, but are not limited to, the following:

Security: Chores that are concerned with establishing the identity and access rights of service requestors.

Notification: Chores that are concerned with identifying which recipients should be informed of a business event. This includes maintaining the subscriptions on which this is based.

Output management: Chores that are concerned with communicating information offline, rather than as a service response. This is typically the case when a customer request must be formally confirmed, for example as an email that confirms the web purchase that you just made via your browser. It is also necessary for unsolicited messages, such as monthly bills. Output management must determine via which channel the information must be sent, and which address should be used. It should transform the message into a format which the recipient can take in, send the message, and add it to the document archive. Output management includes the maintenance of the templates which are used to transform data into user-intelligible messages and of the addresses and channel preferences of possible recipients.

Data transformation: Chores that are concerned with translating data from one format to another, bundling separate services into one service — in McDonalds terms, happy meals — and de-bundling, slicing service requests into separate requests for different populations and de-slicing the responses, queuing and de-queuing, or protocol conversion.

Process orchestration: Chores that are concerned with orchestrating a process, so as to ensure that the services of which it is comprised are executed in the appropriate sequence and only when relevant, that alerts are issued when deadlines threaten to be exceeded, and that interrupts caused by ancillary messages or elapsed deadlines result in new process paths being taken.

Dossier management: Chores that are concerned with maintaining and accessing dossiers of related information. These may be virtual dossiers, in the sense that the information which is presented to the user when he asks for a dossier may be retrieved using queries. For content which is extracted from databases this is regarded as normal, but there is no particular reason not to apply the same approach to documents. In some cases this should be augmented with facilities to assign documents to dossiers specifically.

Record management: Chores that are concerned with maintaining information that is not allowed to be changed.

Bottom-up delineation

The services that implement these chores do not form part of the business service hierarchy. They cannot be designed them in a top-down manner, because there is no “top”. For such services a bottom-up approach aimed at achieving maximum re-use is more appropriate. This allows the best service implementations to succeed and inferior ones to be removed from the stage.

With databased approaches it is seldom practical to replace an inferior generic function with a better one, because that requires changes in all applications which make use of the function. With SOA, on the other hand, such a replacement is simple, provided that the rule for not knowing what you don’t need to know has been applied, including the corollaries that a service request should contain no more information than is necessary to specify the request and that the service itself should take the initiative to ask for more information when necessary. Take, for example, an authorisation service which is called by an application in order to determine whether a particular user should be allowed to perform some function on the data of some customer – for example: “Is our employee Donald Jones entitled to access financial data regarding the Acme Widgets company?”. A simple version of the service might be capable of handling only situations in which these questions can be answered using a table of employees versus functions. A slightly more complicated version might recognise that Donald Jones is a member of one or more groups and use the group authorisations in addition to the individual ones. Taking things a step further, the authorisation service may use the credentials of the user in order to differentiate between employees and customers, and allow the latter access to only their own data. A sophisticated version might use a standardised service call to the Business Process Management System or a case management system, asking whether Donald Jones has been assigned any responsibilities in our dealings with Acme Widgets. Preferably, the authorisation service will log the requests and the answers, but simple ones won’t. The organisation can switch from one version of the service to another without any change to the applications which invoke it.

It is also possible to design generic services to automatically configure themselves for the situation in which they must operate. An authorisation service could, for example, check whether there is a service that will tell it about the responsibilities that employees have been given for particular customers, and decide to use only individual and group access rights if such a service is not available. In this way services can be made re-usable across many organisations.

5. Don’t bore yourself with testing

Why SOA is easier to test

One of the perceived disadvantages of SOA is the supposed difficulty of testing. This perception is entirely inappropriate, for many reasons.

Firstly, the use of metadata in SOA makes it possible to prevent errors from being incorporated into the system. It is possible to validate the system at the metadata level, for example to ensure that all data which is required for processing is collected and checked before it needs to be used. This can be done across the breadth of a business process. Don’t test the system when you can validate the design instead.

Secondly, almost all the testing, and all the systems integration testing, can be done automatically once a test base has been established. There are, however, some prerequisites to this. There must be a rigorous separation between presentation and processing. Fortunately, this is a natural way of working with SOA. For all input, there must be an XSD. Using this XSD, test records are made. In the same way, test records are made for all expected output. Where processes do not produce all the outputs which are needed to prove that the system is working correctly, the test script must include queries which produce additional output for this purpose. When the test is run, the input records are presented to the system one by one, and then the actual output is automatically compared with the expected output. This results in a list of exceptions, each of which must be accounted for. The test can be run as often as needed. Naturally, the results may depend on the data which has accumulated in the database, so this needs to be compensated for. Also, the system must not exhibit time-dependent behaviour. Rather than waiting, say, a week for a time-based trigger to go off, the system must be capable of reacting to events which can be set off at time intervals which are more appropriate to an automatic testing sequence. The user interfaces should be tested on a one-by-one basis, but should never be used in the integration tests.

Thirdly, SOA designs tend to result in more robust systems: there is simply less that can go wrong. SOA reduces the number of factors on which information systems must agree in order to cooperate, so there is less scope for a design error which results in a disagreement on some essential factor. And if things do go wrong, they can be detected before they can do any damage. With SOA, before a message is processed, it is validated in order to determine whether it is well-formed and conforms to the relevant XSD.

Realistic testing

Lastly, the rigorous separation between test and production environments that is characteristic of the databased era is no longer necessary, and is sometimes not appropriate. This is possible because we are not really testing systems anymore, but the passage and processing of messages. SOA offers a triply safe, effective manner of distinguishing between test and production messages. Each message contains version numbers of the message itself and of each namespace it incorporates, excepting those which are encapsulated. And each message contains a tag which indicates whether it is intended as test or production. All that is required is a SOA gateway just inside the firewall which carries out the following processing on each incoming message:

The message is checked in order to determine whether it corresponds to a known version of a known XSD (excepting the encapsulated messages).

The message is checked to determine whether it is valid, using our copy of the relevant XSD.

If the message is intended for production, we check whether the version numbers have been approved for production use. Only if this is the case is the message passed on to the production systems. All other “production” messages are rejected.

If the message is intended for test, it may be passed on to the designated test versions of systems. In special cases, where the message is used only for data retrieval, this may be a production system.

Only after a message has been adequately tested is the register of production versions and XSD’s updated.

This manner of working is not only triply safe, but also enables the routing of messages to be tested in realistic manner. It also greatly reduces the need for reconfiguration in order to make the transition from test to production. Because this reconfiguration is by its very nature untestable, it is a frequent source of error. Release managers compensate for this by releasing new software only in the middle of the night or at weekends; so that if something goes wrong the previous situation can be restored before anybody much has encountered a problem. But that’s no way of working if the change will affect other organisations. SOA release management is so much easier!

It is time that the prevailing wisdom about SOA testing is revised. SOA is a way to minimise the need for testing and the effort to set tests up. It enables crucial tests to be done more automatically, with better result.

Make it better

Rationale

SOA enables the development and deployment of information systems that support a far richer user experience than is generally achieved using databased approaches. The systems encompass more forms of information, have a broader repertoire of behaviour, achieve a higher level of uniformity and consistency in their behaviour and are more reliable, as measured from the points of view both of the customers and of those people within the organisation who are concerned with compliance. But achieving these benefits requires us to say goodbye to databased practices.

6. Always keep your promises

Why databases don’t always keep promises

The act of accepting a service request is by definition a promise to the requestor that the service request will be executed. This execution is by definition a process, involving at least one but often very many steps.

Databased thinking and processes don’t sit well together. By their very nature, databases are islands. And islands promote parochial thinking: anything outside the island can’t be that important. This is illustrated most vividly by the databased concept of a transaction: a unit of work, which takes the database from one consistent state to another. In some special cases this concept may be extended to multiple databases, using 2-phase commit technology, but that is the limit. It is entirely foreign to the databased view of the world that logical consistency might need to be maintained across a business process, rather than just a moment in time, and across all places where information is changed, including not just databases but also process management systems, messages and the human agents that send and receive them.

The SOA transaction concept

What’s foreign to the databased world sits naturally with SOA. A business transaction — and therefore the process by means of which it is implemented – is a service. To understand how well SOA supports the logical consistency requirements, it is important to understand what a business transaction entails. A business transaction consists of the following elements:

The vendor provides the customer with information on which the customer bases his purchase decision. Typically, the information is concerned with the properties of the goods and services on offer, the terms under which they may be acquired — including, of course, the prices – and their availability. In legal terms, the vendor represents this information to be true.

The customer places a purchase order with the vendor, based on the representations made by the vendor.

The vendor verifies that the information on which the purchase decision was based is still applicable, and if so, commits to the order. If there has been a change – perhaps because the goods or services are no longer available, perhaps because the prices have increased, perhaps because the specifications of the goods or services have changed – then some processing may be required in order to decide what to do: commit to the order anyway, negotiate amendments to the order, or perhaps just cancel it.

The vendor and the customer each perform their side of the purchase agreement.

How SOA maintains consistency

SOA maintains the logical consistency of a business transaction in a number of ways. Firstly, all communication which implies a change to a previous situation can be done using a protocol that guarantees secure message delivery. Whenever a commitment is made by either the vendor or the customer, the act of making the commitment involves such a change. In this way, it is not possible for customer and vendor to have different perceptions as to the current state of the transaction.

Secondly, the logical consistency of databases and the record of process progress in the process management system can be maintained using a 2-phase commit protocol. Logical consistency across multiple databases is maintained by a sequence of such 2-phase commits. First database A is synchronised with the process management system, then this system synchronises with database B, and so on.

Thirdly, any changes in the vendor’s reality between the time that representations were made to the customer and the placing of the order are dealt with using an optimistic concurrency control mechanism. This way of working is natural to SOA: the process of determining whether there has been a relevant change can be completely automated. And because SOA makes it feasible to access data at the moment it is used to make a decision, it is only rarely that the optimism is unfounded and manual intervention is necessary.

Lastly, an abort of the business transaction itself – for example that the customer withdraws the order, turns out not to be able to pay, or dies – can be handled relatively easily using SOA. Because the documents which have been used or produced in the context of the business transaction are clearly identifiable, it is possible to determine which countervailing messages are required in order to correct the perceptions of customers and vendors. Because it is clear which database updates are directly related to the transaction, it is clear which changes in databases need to be rolled back using compensating transactions. Given that, aborts occur relatively infrequently, it is normally not cost-effective to completely automate these actions, but your design must allow for them.

Note that the scope of a business transaction is limited to that which is done directly in order to handle a service request. In SOA, the rule is to orchestrate your own processes and provide event notification to others. If the service is aborted, it may be necessary to notify them of the change. What they do with this information is entirely up to them.

Note also that creating the content of a customer’s service request is, strictly speaking, something that precedes the process, rather than part of it. It’s not really your task: in principle it should be enough to supply your customer with an XSD and a validation service, and let him decide whether he is going to key data in or generate it from his own information systems in some way. For consumers this is not – yet! – A feasible approach, but it remains valid that collecting data and processing it are two separate things, performed in separate environments using separate tools. For the same data collection there may be many implementations, depending on the wishes of the customers and the channels that the use to communicate with you, but there should only be one service to process it.

7. Don’t limit yourself to models

Databases are models, SOA is more

A model of a domain is a representation of it which is easier to work with than the domain itself. The vast majority of databases are models. They represent some administrative, physical or social domain in such a manner that questions concerning this domain can be answered by a query to the database, and the need to act in the domain is signalled from within the database. It is, for example, easier to look in a database to find a customer’s address than to follow him home and note the address of the house he enters. And it is easier to count records in a database than persons in a city or products in a warehouse. The functions that need to be supported are in principle sufficient to determine the data structure of the database. However, because it is not normally possible to specify all such functions in advance, we use data modelling techniques to analyse the objects of interest within the domain, including their relationships and the items of information which tell us what we may need to know about them. For example, if the database is concerned with customers, there will typically be a database record for each customer, containing the information concerning the customer that the users of the database are interested in knowing. Because the data model is based on more fundamental considerations than just some particular uses of the database, basing the data structure on this model results in a database that is easier to adapt to unforeseen requirements. The database is semantically structured; in other words, it is structured according to the meanings of the things it represents.

Databased technology is very uncomfortable with any data structure that does not implement a data model. It simply can’t do very much with descriptions, documents, photos and the like. SOA, on the other hand, is just as good with documents and dossiers as it is with data that is structured according to a data model. SOA allows for a single design which handles both semantically structured information and documents. However, because databased systems dominate our thought patterns, we think it only natural to maintain their limitations when applying SOA. But in the SOA world there is no particular reason to apply such restrictions, and every reason not to.

The benefits of going beyond models

The primary reason to combine semantically structured data with such things as hyperlinks, documents, photos and sound fragments is that it creates a richer user experience. For consumers, this is not an option but an imperative. For the knowledge workers you employ, it makes them more effective. It is only for the staff performing routine administration that it gets in the way. But the more you apply SOA, the less you will need them. Their task of keying in data from forms can be outsourced to anywhere: all you need to do is scan each form and send the image to an agent who uses essentially the same web form as the customers to key it in. To the extent that at least the genuinely routine tasks of these employees can be automated.

A second major reason to not limit yourself to databased data is that a synthesis of databased and document-based data is by far the easiest way to maintain the records and audit trails which are necessary for compliancy purposes. Databases are by their very nature unsuited to such ends. Because a database is typically built to act as a model of some administrative reality, it should therefore be easy to change it when that administrative reality changes. Database management systems are designed to facilitate such changes. When they are used to maintain records which are not allowed to be changed – bookkeeping entries, for example – the designers have to build all sorts of defences into the system to prevent malicious manipulation of the records. Even then, no judge in his right mind would trust a database. The records and audit trails which are needed for compliance purposes must be entrusted to record management applications, and linked to the databased systems using SOA.

With SOA, connecting semantically structured data to other forms of information representation is easy. This makes it ideal for content management and content presentation. A database can contain hyperlinks to other information. For example, we can store the URL of the input document that gave rise to a transaction with the database record of the transaction. When the transaction record is displayed, the link is presented along with it, and may be activated by the user via a separate service. It is no more difficult to make a service to access a document from a document archive than it is to make one to access data from a database. Unless, of course, your document archive is not SOA-capable, in which case one of your first actions must be to replace it.

But it works the other way around too. Not only is the semantically structured data enriched with the links to records, but the records are enriched with links to the semantically structured data. When you store a record in a document archive, the semantically structured data that is associated with it will normally contain all the information which is necessary to index it. There is little to no need to index the records manually. Better yet, the index can be progressively enriched by updating it automatically, every time that the transaction is progressed in the database.

Integrating databased and document-based data has the further advantage that it is easy to support multiple versions of data. That doesn’t make sense in the database world, because a database is a model, and to be useful as a model it must give at least a best guess in answer to any question about the administrative reality of which it is a model. A document, however, is a statement made by a specific party at a specific time in a specific context, and it is quite possible – and often useful to know – that conflicting statements exist.

Culture shock

Note that databased thinkers are not the only people who will have trouble with the SOA unification of the worlds of documents and of databases. People whose thinking has been formed by the current generation of document and record management systems have an even bigger problem. Suddenly, archived documents are part of business processes, not things which come into existence after process completion. And indexing is not something done manually, once per document, but something automatic and ongoing. Nor is the index part of the archive anymore: an archive only contains documents, and the indexes are maintained separately. And it is therefore not necessary to limit yourself to one archive, or even to only your own archive. These changes are so profound that it is hard to imagine that anybody who was suited to the old world could also be suited for the new one.

8. Don’t use models as photos

Material and formal history

The biggest mistake of my career was to design a database in which all service requests could be supplied with a time stamp, so that if it was filled, the service could reply according to the data contents that the database had at the moment indicated, instead of the current contents. It was enormously expensive, slowed the database down to a crawl, and served no useful purpose. When users wanted to see how a particular result in the past had been calculated, they invariably looked in the document archives, because the archives always contained the result that had been communicated to the customer. And that’s all they wanted to know.

Note that it is not the use of databases to register past situations that is the problem. Databases are models of some part of the real world, and the state of that part of the real world at some period in the past – its material history – may very well be relevant to the purpose of the model. But a database takes itself too seriously if it wants to recreate its own state at some point in the past, in other words to register its formal history.

How SOA supports formal history

The SOA approach offers a far more effective solution to the problem of recreating formal history, not just of a single database but of all the information which an organisation uses and creates in its administrative processes. The SOA separation between the collection of the data required to update the organisation’s information and the actual execution of the update allows that data to be archived in a record management application. Once SOA has been applied, this can be done automatically, with almost no additional effort in order to incorporate a new data set. In this way, all incoming messages and all updates made by your own employees are captured.

The situation for outgoing offline messages is, if anything, even simpler. The SOA aversion to mixing chores and added value for customers leads quite naturally to the use of an output management service to ensure that offline messages reach the intended recipients via an appropriate channel, at an address which they really use and in a form which they can adequately take in. The output management service can take care of recording the outgoing information automatically.

There is, however, one problem. The ease with which organisations can disseminate information using SOA tends to result in a substantial increase in the number of service responses to with external parties. These service responses typically imply a commitment of the organisation to their propositional content. The organisation could get into legal problems if they are incorrect. The standard SOA means of handling this involves the following guidelines.

· Implement business transactions in conformance to the SOA transaction concept.

· Include disclaimers in the service level agreements for all information retrieval services.

· Don’t access data until you need to use it. That way it will be as up-to-date as possible.

· Don’t lie. If you always tell the truth, you will encounter fewer claims that your information was incorrect.

· If the use of disclaimers cannot prevent you from having to respond to claims that your information was incorrect, log each service request and response, using the standard organisation-wide service.

This manner of working is much simpler to organise than the databased solutions, because it does not require additional logic per message type, but only a call to the logging service, in which the entire incoming or outgoing message is treated as a single data item.

9. Don’t trust systems

SOA hygiene

If your body applied the same rules of defence against external agents as most web sites, you would be dead in a day. We live in a hostile world, where just about the only way to avoid your information systems being attacked is to make them inaccessible. In this world, SOA is a marvellously hygienic concept. In principle, the only messages we have to let through our firewalls are robustly enveloped XML-messages. And they can be passed through the firewall in such a way that they are checked for well-formedness, XSD-conformance and the validity of their authorisations before any further processing. The potential for them to commit mayhem is very small. Buffer overflows can explode only in bomb-proof areas, and SQL injections are nothing more than curses in a language to which we are entirely deaf. Corrupt messages are recognised as such before they can do any harm.

It is important to note that the strength of this approach is based on the authentication and verification of messages, not of systems. That is necessary, because encapsulated XML is often passed on by systems that bear no responsibility for their content, so that checking that a message has come from some authenticated system doesn’t guarantee much. It is more robust, because a successful hack of one information system or computer does not open the gates to hacks of others.

SOA security contrasts markedly with the systems-based security of the databased era. With systems-based security, it is the task of each system to avoid buffer overflows, for each message type and each data item. There are thousands of points where it could go wrong. The chances of a weak spot are therefore considerable. A SOA gatekeeper, on the other hand, does all of this at a single point. Systems-based access security tends to result in authorisation controls being built in to systems, which not only makes it almost impossible to keep track of them, but also enormously expensive to introduce structural changes, such as the addition of users who are external to the organisation.

A further advantage of using a SOA gatekeeper is that it can ensure that all outgoing messages are appropriately encrypted and signed, so that nobody can eavesdrop on them or change them en-route. In principle, most of the WS-*hell can be handled entirely by the SOA gatekeeper.

Beyond systems-based authorisations

Systems-based security also promotes mindsets which are much more determined by systems-based thinking than most of us would care to acknowledge. Because systems are structured around functions and data structures, authorisations are expressed in these terms. For example, an employee is either authorised to change product prices or he is not, based on his role within the organisation. For back office employees that need not be such a problem, but for front office employees and customers this is inappropriate. For them, the primary determinant of what they should be permitted to see and do is the slice across all systems and databases of the information concerning the customer at hand. A brief scan of information security literature is sufficient to establish that information security consultancies haven’t got this point quite yet: for every reference to slice-based access security there are thousands to role-based access security. On the positive side, there are emerging concepts which go a long way toward implementing the kind of access security that makes full use of the advantages that SOA has to offer: claims based access control, federated identities and de-perimeterisation.

Make it manageable

Rationale

To get maximal benefits from SOA, it is insufficient to merely adopt new rules of behaviour. The way the organisation thinks about its self and its IT needs to change too. Otherwise the current organisation will revert to the old behaviour, not because it wants to but because it sees that as the only way to maintain control over its own agenda. To counter this, you must start with the way projects are organised.

10. Don’t leave infrastructure until last

Why worry about infrastructure?

SOA is all about separating infrastructure from business, not just in the functional sense, but also in the project management sense: a project manager should be able to concentrate on the business added value of the services that he is responsible for developing, without having to worry about the infrastructure. It is no less absurd that a business project manager should have to organise a new authorisation service as part of his project, than that he should have to install a new telephone exchange. To burden the project manager with such things results in an increase in the number of things that he has to coordinate, which results in turn in a disproportionate increase in the complexity of his task. Even letting him wait until some other project manager delivers the infrastructure is something that should be avoided where at all possible.

The above remarks apply equally well to the hardware infrastructure as to the infrastructural services. The computing power and network capacity should always exceed the requirements, so that project managers who implement new services shouldn’t have to worry whether the infrastructure can support a thousand service requests more per day than originally envisaged. Given that software development is typically an order of magnitude more expensive than IT infrastructure, the last thing we want is that systems development projects get bogged down by bureaucratic processes concerned with ensuring the availability of IT infrastructure.

Infrastructure: how SOA is different

The good news is that the SOA approach of saying what you want and then shutting up results in simple, stable interfaces to infrastructural services. That is why it is not necessary to implement the world’s most advanced infrastructural services before you can start any business projects. The complexities of these services lie not in the interfaces which are used to invoke them, but in the information which they request from other sources – or maintain themselves – in order to produce an optimal result. It is therefore perfectly feasible to start off with rudimentary infrastructural services, and to make these available for test and production use in plenty of time for them to be used by business projects.

The bad news is that SOA results in much more functionality being classified as infrastructure. In particular, each chore results in a generic service that must be implemented as part of the infrastructure instead of being incorporated into each of the business functions for which it is required. That is not so bad by itself, but it does result in a new problem: how to organise things so that, for each part of the infrastructure, there is somebody with both the mandate and the resources to ensure that it is available on time with the appropriate functionality and capacity. An organisation that is not geared to addressing this problem is not geared to implementing SOA – it’s as simple as that.

Where do we go from here?

Although the ingredients of SOA have been around for a while, SOA is radically new. Like the shipping container, it is more than just a new way to do the same as we have always done, because it enables us to cooperate on a new scale. And it is not primarily a new technology, but a new way of thinking. It is so different to the current way of thinking that you should not be embarrassed if you have to read through this article several times before it starts to sink in. It may take a while before your natural tendency to think that the suggestions it makes are impractical or unnecessary or both subsides sufficiently for you to be able to take stock of it all.

To be fair, there will be situations in which the SOA solution is not directly practical for you. For example, you may acquire all your systems as packages from vendors who have no particular interest in applying SOA in the manner described in this white paper. In that case, treat the suggestions in this paper as a compass: use them to decide whether the choices you make are leading you in the right direction.

Because SOA is so radically new, not all the concepts, tools and standards which we need in order to make full use of it are available just yet. Current gatekeepers do not do everything we should expect of a SOA gatekeeper, and current user interface facilities have difficulties with encapsulated XML. Worse, as this white paper has argued, there is currently much in the SOA world that is confusing, unnecessary or wrong. Current applications of SOA concepts and technologies do not get anywhere near achieving the level of interoperability that it is capable of. Too often we are confronted with interoperability problems that are due to somebody else having chosen to implement a slightly different subset of the many WS-* protocols that are on offer. The millions of possible solutions as to how parties communicate have been reduced to a few handfuls, but that is still far too many.

But we have made a good start. The rest we can work out along the way. That, too, is SOA. We don’t have to know what we can’t know yet in order to make progress. An enormous potential to make things simpler, more reliable, more predictable, and functionally richer awaits us. It is time to start.

Enough said. OK, I'll say a bit more :-) My personal observation of SOA efforts in a large company: If a SOA-project does not have the management power to overcome system-fiefdoms that maintain that they can do whatever they please, it's not going to work. SOA is a company-global effort.

The idea that a your service must have a single web service (I assume this means SOAP) interface in order to be maintainable is just nonsense. This idea conflates the data with it's wire representation. There is nothing wrong with supporting multiple equivalent data structures in a single service. This is an important (but often overlooked) principle of REST.

The service should be built with an internal canonical format and transformed to and from the external formats. Of course supporting another format has a cost associated with it. The question is whether it's more expensive to support that format in the service or make the consumer work with an already supported format. It's really that simple. If you are going to lose millions in business by not supporting EDI, are you really going to tell your boss that you won't do it because the maintenance will be too high? It's really not that hard to do and you should assume you will need to.

I worked at a very large wholesaling company on B2B software. Before I started working there, a "one-interface" policy had been decided upon. This was going to save money. The B2B system was then designed to support a single B2B standard This was before SOAP became prominent and then turned into nothing more than an envelope. The result? Few of the customers wanted to use this format and we had no choice but to accept their requirements. You don't tell (e.g) WalMart what they will do, they tell you what you will do. SOAP became the technology du jour and we had done everything in this dying format and made no allowances for different structures. We put all our eggs in one basket. We ended up with a ton of redundant services because IT management was unable to admit this approach had been a mistake.

You won’t find me disagreeing with the need for management commitment and overcoming system-fiefdoms. However, for anything like plug-and-play interoperability across organisations that is not enough. If we merely replace system-fiefdoms with organisation-fiefdoms, getting services which have been built by different organisations to interoperate will remain transplantation surgery. The clout of managers typically doesn’t extend any further than the boundaries of their own organisation. That’s not enough for the challenges that I address in my assignments: getting the various departments and layers of the government of the Netherlands to function toward citizens and companies as though they were a single organisation. After having co-founded the Netherlands Governmental Reference Architecture and having co-authored the Municipal Reference Architecture, I have reached the limits of what you can achieve by means of central dictates. The way forward, in my opinion, is to create an IT culture in which people working for different organisations will automatically develop compatible solutions for similar problem domains. This culture can be achieved when we learn to understand what service oriented architectures really entail. That’s what the 10 SOA commandments are all about.

I couldn’t agree more with what you write: when the outside world dictates how you are going to communicate with it, it is wrong to stick to the principle that a business service should be exposed by a single web service, and right to build services according to an internal canonical format and transform it to the external formats as required. I go even further, in saying that there are many situations in which it is stupid to use web services. Using them to communicate within an information system is like using containers to move goods around your manufacturing plant. Web services, I believe, are the key to interoperability, and not to maintainability. Where we want interoperability, we need to ensure that solutions converge. So I do disagree with the subject of your reply. Convergence to a single solution is a great idea, starting with WalMart.

If you are saying that convergence to a single solution at the industry level is a good idea, then I agree with you for the most part. That's really not something that any individual company or architect can solve, however. One issue is that the current solution (WSDL and SOAP envelopes) is both bloated and inadequate and the commonly used tools often create more issues than they solve.

But the danger with this idea is making assumption to support only that one solution and that it is 'smart' to do so. In theory, it's great. You can concentrate on that one solution. But it's kind of like assuming you will never need any extra cash and investing everything in an illiquid asset. When something unforseen occurs, you could lose much more than you stand to gain by not being prepared.

The reality is that being prepared to support different kinds of formats and protocols isn't costly enough to make it worth the risk of not doing so. This doesn't mean you have to support them on day one. You only use your preferred solution until something else is needed. The point is making sure your architecture can support something else. In real terms this comes down to making sure you separate the business logic and in-memory data structures from the wire format. You can still optimize your approaches to fit your preferred solution.

Maybe this is obvious to you but I have to continually fight against the desire to make convenient (and completely unrealistic) assumptions. When you make the kind of statement you make in this article, it will be read by many as a recommendation to put on blinders to reality.

Is your profile up-to-date? Please take a moment to review and update.

Email Address

Note: If updating/changing your email, a validation request will be sent

Company name:

Keep current company name

Update Company name to:

Company role:

Keep current company role

Update company role to:

Company size:

Keep current company Size

Update company size to:

Country/Zone:

Keep current country/zone

Update country/zone to:

State/Province/Region:

Keep current state/province/region

Update state/province/region to:

Subscribe to our newsletter?

Subscribe to our architect newsletter?

Subscribe to our industry email notices?

You will be sent an email to validate the new email address. This pop-up will close itself in a few moments.

We notice you're using an ad blocker

We understand why you use ad blockers. However to keep InfoQ free we need your support. InfoQ will not provide your data to third parties without individual opt-in consent. We only work with advertisers relevant to our readers. Please consider whitelisting us.