Further thoughts on Distributed Observers

I’ve been reviewing the details of the ActiveTracker implementation of the MVC design pattern. As I noted here, the first problem I ran into was that the observer pattern in ActiveTracker only worked within a single process. I’ve been thinking that I would like a business system that implemented observers in a distributed manner.

However, as I’ve thought about this more, I’ve realized that while the idea might sound good in theory, it can quickly become troublesome.
It’s easiest for me to illustrate the problems by giving examples from the business process with which I am most familiar, which is the OLTP of a wholesale distributor.

While salesperson A has a customer on the phone, they might be keying in catalog numbers to advise the customer of both the price and availability. The customer service form has lots of data available to the salesperson. The customer’s A/R balance might be visible as well as the inventory status of the items that have been entered so far. With a distributed observer pattern, I can envision the customer service form automatically staying up-to-date with all the information on the screen as it might be changed by other users in the system.

In one kind of implementation, a central manager would be used to handle registrations and distribution of notifications. If the system had high activity, this could be a lot of work for the central server. In addition to the potential high volume of messages, the server has to keep track of all the registrations across the system. For a system with many users and many objects being accessed, this information would almost certainly have to be persisted in a database to avoid outrageous RAM requirements. Another problem that would have to be addressed is observers that “disappear” without properly deregistering. I’m not even sure where I would begin with addressing that problem.

In this implementation, the form would have to register each object it uses (and cares about). The customer’s A/R balance, if it were its own object, would be registered (or the whole customer might be registered). If the item grid is showing the status of a dozen different items, each item object would need to be registered.

Suppose salesperson B now posts an order that includes a couple of the items from the grid. The sales order tells the central manager that it has changed the items, and notifications are sent out to the form used by salesperson A. At the same time, an accounting clerk posts the checks that were received that day. One of the checks is for the customer on the phone with salesperson A. Additionally, a warehouse clerk is posting the receipts from a truck that had been unloaded that morning. That truck had some of the items that were registered by both salespersons.

How does the form react to this? The display information isn’t too big a problem. The form can use a background thread to refresh the data. I can imagine that it would have to make several round trips since it is going to receive separate notifications for the customer and for the items. I then also worry about handling the user closing the form while the background thread is in the middle of refreshing the data. I’m sure this is a common scenario to be dealt with when multithreading.

I think I can deal with display-only data. It’s got some challenges, but nothing that should cause a poor user experience if properly designed. What about editing existing objects?

When we edit objects, there are two practical methods of dealing with concurrency in distributed systems — pessimistic locking and optimistic locking. I know there are others, such as last-write wins, but I think the other techniques are not practical for most business systems.

I hate the idea of locking records pessimistically. I’ve worked with systems that locked records. Some locking mechanisms have been so bad that you couldn’t even access the locked data in a read-only manner. I know that SQL Server supports this kind of mode, but I sure would not want to have to work with this kind of system in a distributed environment.

So, we are left with optimistic locking. How does that fit in with observers? Should the editing form register as an observer when it is doing the editing in a distributed system. On first blush, it seems like it might be a good idea because we can use it for notification when somebody else makes a change to the system. However, would this be a good user experience? Imagine that I’m going through a batch of customer records to assign NAICS codes. I have the customer registered. One of the properties on the customer is the A/R balance. As I’m entering the NAICS code, the accounting clerk generates invoices, which affects this customer’s A/R balance. A notification is sent that the customer has changed.

Should a window pop up right away to notify me that the customer information has changed? I’m not sure users would like that. I certainly wouldn’t like it in this scenario because I couldn’t care less about the A/R balance changing. I am simply updating NAICS codes.

If a popup isn’t viable, we could display unobtrusive messages that the information has changed. The status bar is a common place for this. Some systems have “tickers” that display this kind of information. My experience is that unobtrusive messages like this become invisible messages. They are easily ignored and will be.

We could wait until the user tries to post their changes. But if we’re going to wait, why register to receive notifications if we’re going to ignore them. Optimistic locking systems catch discrepancies already during the update of the data, so there really is no need to register. If the A/R balance were read-only information, then maybe this would be OK, but we don’t know which data changed unless we do a field-by-field examination.

Finally, I think another way to implement distributed notification is using a broadcast method. Update notifications are sent to everybody. Each client process in the system would track the objects it is observing and pass on the appropriate messages to the other object instances on the client. I think the windows message system works this way. There is a single message queue through which all messages are passed. Each window process is supposed to act only on the messages that are meant for it, although it is not uncommon to “subclass” a window by intercepting its message, taking action, and then passing the message along to the intended recipient.

This whole concept is too much for my little brain. I need to be able to break it down and pick a simple place to start and test it out. I’m starting to think the broadcast system would be relatively easy to implement, but we would need to broadcast judiciously.

Using our previous scenarios, the inventory quantity changes would send out a notification for every change to the quantity of an item. With a more judicious use of notifications, we could only send a notification when the inventory status made a transition to/from the “overcommitted” state. When an order is entered that causes more to be on order than we have on hand, then an alert should be raised. If I’m entering an order, I generally don’t care if the available quantity goes from 100 to 99. I do care if it goes below 0, though!

Critical events for the A/R balance could also be defined. Most obviously, one alert would when the customer crosses their credit limit.

Perhaps this is where I should start my investigation. It’s not an MVC pattern, but that may not be as feasible as I had hoped.

This has become a really long post, but I just thought of how the MVC pattern could be useful within a process. If you have a editing process that involves parent and child forms, it would be useful to have the forms be able to automatically update when changes on the other affect their display. The ActiveTracker sample project seems to be a good fit fo this scenario. I will have to take a deeper look at it.