I'm implementing some automatic functional tests in an environenment that has about 20 applications talking to eachother using queues and/or restful interfaces.

The problem I'm facing is that I added some test to one of the application, lets call it Payments, and they all looked very nice and green. One of the changes on the Payments application added an extra field in a messages that is send to another application, lets
call it CRM, and the CRM discarded the message, which meant that the applications started to become out of sync. The Payments application (as most other applications) send messages in a fire-and-forget way, which means that the producer never knows if a message is processed or discarded.

I'm wondering is how to test these dependencies between systems. One option (that I dislike from my gut, but no idea why) is to connect to the other system and verify that the record was updated. Most applications have a restful interface and also provide clients, so is not a lot of work. But I have the strong feeling that doing this will
make the test more brittle. One of the reasons why I don't like this, is that most of the applications talk to another 4-10 other applications, so it's not just checking another app... and since lots of things are asyncronous, it adds even more uncertainity to the test.

I don't understand. You want to test, and therefore presumably it is a requirement, that an order (or whatever) submitted from one application to another actually has the intended effect. But the messaging system is also incapable of distinguishing successful from failed messages? In that case the specification cannot be fulfilled in the first place. If it is required that an order is either processed or backed out, then it's completely correct for you to write tests that perform that check via REST - but then so should the application itself!
–
Kilian FothMar 2 '12 at 13:35

Kilian, the requirement is that if Application A sends a message to Application B, Applicatoion B should process that message sucessfuly. The problem is that Application A doesn't get any feedback from Application B, because the message is put in a queue (ActiveMQ).
–
AugustoMar 2 '12 at 15:40

It looks to me like having a unidirectional queue was a poor design choice. You should have a queue of confirmations travelling in the return direction.
–
James YoungmanMar 3 '12 at 9:44

2 Answers
2

I try to test as close to the fragility as possible. If the CRM is the component that isn't doing its job, that's where there is a hole in testing. It's the component that responded poorly to change. It's the piece that dropped the message without raising an alarm.

I feel like people turn to big integration testing when a way to test a single system is non-obvious. If you find that non-obvious simpler testing problem, you'll often discover a better way to design what you're doing.

The problem was that both applications were correct from the point of view of the developers. Each app is maintained by a team of 9 people (unfortunately one team is in a different location, just mentioning this because co-location usually makes thess problems not happen as often). And both projects have functional tests and were passing. The issue was that the functional tests of the Payments system didn't pick this error up.
–
AugustoMar 2 '12 at 15:32

1

I see what you're saying. Personally, I think I would avoid fire-and-forget with a payment system. It seems at first glance like two new tests are needed. First, that the payment system checks to see if errors were thrown in processing, and second, that the other system it is coordinating with always raises an alarm when it can't perform it's work. Then, you have two systems that you can individual test.
–
menacinglyMar 2 '12 at 15:39

It may be a good idea to add some logging capabilities (or a logging service) to your message passing system. When logging is enabled, each application/service sending a message should log this. Also incoming messages should be logged, and it should be logged when a message was successfully processed, or when it was discarded. Your testing framework should have easy access to all logs, and the logging service should be designed to be queried by tests programmatically.

Then, when you create a test of your Payment application, you don't have to connect to your CRM service (and the 4-10 other applications) to verify if the message send by Payment was received. You will only have to connect to your logging service and check the log.

Doc, the CRM (which discarded the message) runs in a different server, so the Payments app doesn't have access to the logs. I think that accessing the logs of an external application to test something is even a bit worse than using the api to validate something, as the APIs tend to be stable, but logging is not.
–
AugustoMar 2 '12 at 15:27

@Augusto: I was not suggesting to use your existing (perhaps local) log files - what I meant "design a better log service for your testing purposes". Your log service could be a restful service just like your other applications.
–
Doc BrownMar 2 '12 at 16:27