Agile: The SOA Hangover Cure

You're at your desk, staring at the line of SOA books. Your eyes wander over the boxes of SOA tools your CIO let you buy. Wow, how much money is sitting on that shelf?

Sure, you got those wretched acquisitions to cough up their customer data via a SOAP service, in one format no less. Not only that, it's even being used! By who again? Oh yeah, that cute girl from Marketing. Geez, you never would have thought this SOA stuff would get you a date- or at least a lunch with her.

What a pity many IT organizations do not get past the point of installing some tools and deploying a few services. Quite frankly, it's that rare firm that actually is getting solid and consistent benefit from their Strategic Enterprise SOA Effort.

Why is that?

I would like to, if I may, take you on a journey where you will suspend disbelief. That journey is AgileSOA. Like SOA is not buzzword enough, right?

Just to be clear, we are going to talk about the convergence of Agile software development practices and building applications as a collection of services. As you will see, the two mesh together nicely, as if they were made for each other. A little Agile gets your first services going. And those services will allow you to apply some more Agile, et cetera. The word that comes to mind is "symbiotic", but "most excellent" and "lovely" apply as well.

In most cases, a firm's SOA effort is a more hype-overloaded EAI effort requiring more standards, more process and more Big Upfront Design.

Well, let me tell you my friend, this BUD's not for you.

What goes wrong here is that enterprise architects apply age-old engineering practices to something that defies description, planning and upfront design. It's the full-scale confluence of business and technology. Such uncharted territory needs to be approached with the most agile of methodologies. After all, do you understand all the business processes? Do you know exactly which pieces of functionality can be separated out, allow for an unambiguous interface definition and can operate stateless and without knowledge of their invokers and therefore will be best suited to become services? DO YOU?

No, you don't. And it's about time you simply admit it.

Engineers are brainwashed with the doctrine that they have to have all the answers, at all times. Not knowing means discredentialization, i.e. getting a wedgy in front of the whole class. Saying the wrong thing is not as bad as admitting to ignorance.

Scientist however know what they don't know and accept that to be the case. In their quest for knowledge, they devise experiments to either prove or disprove their assumptions. A failed experiment is not one that proves their thesis wrong; it's the one that is inconclusive. The lack of knowledge about the outcome and correctness of approach does not preclude the scientific team to apply the utmost rigor in planning, executing and recording the experiment. The findings from each experiment are used in each subsequent iteration, until either an answer is found or funding runs out.

Similarly, the Agile methodology advocates Doing over Pondering. With SOA, as with most software design, there are so many unknowns that most people who endeavor it don't even know which questions to ask. Let alone having any answers.

So I ask you: how can you design, plan and implement what you do not know? You can't. So you might as well design around just what is in front of you and plan for half of what you know to be invalid tomorrow.

You can't learn what it's like to make, own and interconnect services from a book, you have to DO it. Only when those messages start flowing and trigger events will you understand. Only when you invoke the same service from two entirely different pieces of code will you understand the power of what you have created.

Do you need to have your message headers standardized to do this? Do you have to have your governance set up? Do you need to have an SOA infrastructure up and running?

No, and you know it.

But just sending wild XML without schemas over JMS sounds so.., so... impulsive! Yes, and isn't it beautiful?

So, now that I have you convinced that there is an alternative way to approach SOA, let's look at how you can actually get true SOA off the ground, using Agile/XP practices.

First, select a team of 4, 6 or 8 developers. The more varied their skill set, the better. Reserve a room for half an hour at the beginning of every day. This will be your AgileSOA project room where you will discuss lessons learned, designs and practices and where you will plan your next steps. Get an easel with a flipchart, so you can carry your artifacts in and out at will.

Select a mildly complex process flow from your company's core business. To name some examples, try implementing a simple loan application. A basic stock trading system is a good candidate as well. Remember, you don't have to solve the entire business problem, just make your solution do something that appears useful to your business types.

Chart out the simplest path through the flow, initially leaving out all exceptions, errors and decision points (the so-called Happy Path). Now go and implement the first pieces, pairing developers with the largest difference in skills and experience. This may seem unexciting, but I challenge you to build that first consumer of MSMQ or JMS messages, get a few running and have a producer successfully send some XML data to them. (That means ALL of them, not just one)

So what if this first step is trivial and mundane? Did you write unit tests for the producer and consumer, so that you can verify that the data comes across intact? Did your pairs develop together, without walking back to their own desks? No? Thought so.

Figure 1: Most basic process flow services

Figure 2: Add validation service

Next, introduce a few more Happy Path services. Try to mix synchronous, asynchronous and message-based request-reply. Be adventurous! And by all means, write your tests first. After all, it's much easier to write and refactor code when you have unit tests in place.

Figure 3: Expand validation

Now that you have your Happy Path running it is time to introduce an exception path. This may require severing the ties between one or more producer-consumer pairs, and that's OK. In fact, it's great, because now you're experiencing how easy it is to change the behavior of your "application" while leaving most of it intact and running! At this time, you may want to add some metrics gathering pieces. Good data to collect are service response and latency times, throughput, uptime, message counts, etc.

Figure 4: Add validation exception

As you continue to refactor the services and the flow to accommodate the remaining or new requirements to finalize the business workflow, you will notice that it gets easier at every pass. You will also notice a chance in your team. They have now morphed into a cohesive blob of knowledge and understanding, almost moving and thinking as one. One would say they're "gelling"...

In and of its own, this exercise may not impress your CIO. But when you tell her/him the time within which your team completed it, plus how many times you actually changed the design, she'll think you have had a few too many cups... or she'll ask you to share the dope, dude.

...and that's exactly what you're going to do!

Now that you and your posse have set a precedent, you can move forward and undertake an actual implementation of new functionality (or a rewrite of existing stuff). You now have the experience and confidence to simply begin and work in close cooperation with the "client" to build what they have in mind. You no longer scowl, frown and sigh when they proffer the so-many-eth change. You oblige with delight, because your unit-tested code is rock-solid, your system scales nicely, your service-oriented approach allows you to deploy and redeploy pieces ad infinitum and you can do so with a running system.

As your service population grows, the need for some infrastructure and standards will emerge. Now is the time to identify services and formats that are used by many SOA participants. These become the de-facto standards, based on what you actually use and derive value from. Your services will be reliable, scalable and easy to maintain. This will attract new business users who now see the previously doubted promise of service reuse gain new momentum. Coupled with the basics of SOA governance, they will feel empowered to demand from their development teams to use existing services rather than implement their own.

And so, your SOA has come to life at last.

Think this is fantasy? Just ask our clients. Or even better, why don't you try it for yourself.

I basically agree with the steps to get SOA project started in an organization. I think the important point is in the following clause:

As your service population grows, the need for some infrastructure and standards will emerge. Now is the time to identify services and formats that are used by many SOA participants.

When is the "right" time to start identifying common services and formats and how should one go about it? Who should decide when it is the right time? Should one just consider the currently needed infrastructure and standards and keep adjusting them as the project gets bigger?

I think that properly handling this transition phase is the key to making a sucessful SOA project or making a large bowl of spaghetti.

There's continued debate on the merits of the Rational Unified Process (RUP) vs. agile methods such as eXtreme Programming (XP) for SOA design. While it's possible to run SOA projects in an XP fashion, my concern that the lack of rigor can lead to an incomplete or inconsistent process...

Just a reminder, though: not all teams use RUP in the same heavy-weight way. RUP is a methodology toolkit which includes recommendations for implementing various "flavours" of process, including "Agile RUP".

Good point. I agree that RUP is more project specific, but I was speaking in terms of project specific deliverables. I have used variations of IBM's SOMA and the OASIS SOA Blueprint for enterprise service discovery. RUP and UML is the next level down.

From reading the article, I thought that it meant usingAgile to get a SOA project going and to switch over tosomething like RUP when the SOA project gets some backing from the company - read as getting a budget.

In the beginning, I think it's difficult to get time and support from different groups to identify common services unless SOA is implemented in a top-bottom approach. If the organization is already committed to SOA and have large budget to implement it, I think I would favour to use something like RUP from the beginning and get as many people involved.

Where and how to get started with SOA is another interesting topic. I am working with quite a few clients and I have found that most start bottom up and then move to top down after some initial sucess. When moveing to top down, enterprise service discovery, I have not found that RUP is a good appoach. RUP and UML are very good a capturing project based deliverables that can be reused as services are reused - the deliverables support services reuse.

But enterprise service discovery is more functional and process decomposition with a good mix of IT strategy. What you want is a services blueprint (what the services catalog end state will be), a prioritized roadmap to get there and what the services architecture and infrastructure will look like over time to support the roadmap.

As I said in the previous comment I use variations of IBM's SOMA and the OASIS SOA Blueprint to accomplish this along with some IT strategy allignment techniques. I will write a longer post on this topic.

Here is another post on where to get started - this is an interesting topic.

Is your profile up-to-date? Please take a moment to review and update.

Email Address

Note: If updating/changing your email, a validation request will be sent

Company name:

Keep current company name

Update Company name to:

Company role:

Keep current company role

Update company role to:

Company size:

Keep current company Size

Update company size to:

Country/Zone:

Keep current country/zone

Update country/zone to:

State/Province/Region:

Keep current state/province/region

Update state/province/region to:

Subscribe to our newsletter?

Subscribe to our architect newsletter?

Subscribe to our industry email notices?

You will be sent an email to validate the new email address. This pop-up will close itself in a few moments.

We notice you're using an ad blocker

We understand why you use ad blockers. However to keep InfoQ free we need your support. InfoQ will not provide your data to third parties without individual opt-in consent. We only work with advertisers relevant to our readers. Please consider whitelisting us.