I am in the process of designing an architecture for a plugin based application and I have some questions about how to handle database access and logging. The goal of my application is to allow a device my company is manufacturing to send messages over any number of third-party protocols and to any number of third party devices. Because of the nature of my application, I need to log every message that is sent via my system in reasonable detail.

Here's what I am thinking on the Database side: There will be a global message table which stores all of the implementation agnostic data we have, and then there will be a table (or set of tables) for each messaging plugin as defined by the plugin writer which references the Global Message table via a foreign key.

It is handling these other tables that I am having a hard time figuring out how to design safely. On the one hand, I don't particularly want to give the plugins full access to the database (I want them to have the fewest possible privileges), on the other hand I want the plugin developers to have full reign to log any data they need to tables that they define similarly to how plugins work in Wordpress. Additionally, I don't want logging to the Database to be optional at all: Good logging is crucial to my application.

Things I have considered:

Designing a logging infrastructure which allows a plugin to define (maybe in an XML file) the tables it needs in the DB and a mapping between those tables and objects that they will return, and using those mappings and table definitions to write to the DB the results from method calls.

Making a naming convention for Stored Procedures in the DB that each plugin must define when defining their tables and automatically calling those stored procedures after sending a message.

Allowing each plugin to implement a logging method which returns a String holding the proper SQL query.

Each of the above solutions makes me a little uncomfortable, but I'm not sure what other approaches might exist. Any Suggestions?

Implementation Details:
I am writing my application in Java, and I am using a hand-rolled plugin solution, so I have a pretty decent amount of flexibility. Since none of the actual code has been written (other than some proof-of concept code which is really just for reference), I can make fairly major changes to my plan without great cost.

4 Answers
4

Why does it have to be a database? Why not just have a generic "data sink" and write streams of data to it. Plugins are responsible for serializing and deserializing their own data, and you provide a simple read/write facility. They give you a reference to a message stream, then you write it and hand them back a GUID (or some other token) to identify the data.

Well, that's a great question. Since we will be doing heavy reporting on the data we would really like all of the Data to end up in a database, but if I need to make an abstraction layer between the plugin and the Database then I will. Do you have any specific suggestions on how to accomplish this?
–
Brendon DuganMar 5 '13 at 20:34

You can store all the messages as CLOBs in your database, indexed by UID. For reporting you might also require plug-in writers to implement an extraction API, so you can pass in a UID and get back either some specific piece of data (status, recipient/target, response code, etc.) or possibly a loosely-formatted dump (XML or JSON) of the raw data.
–
TMNMar 6 '13 at 15:03

How are you planning to implement the "sending" of the messages ? There are different approaches based on this answer. For example, are you whipping up your custom implementation of Apache James Server ?
You haven't mentioned if you are working on any available frameworks. But I think one other approach you can consider is to use AOP for this. You can weave your aspects around each sender function and whenever such a function is called logging statements can be prepared and documented in a db.

The actual implementation of the message sending is irrelevant to me. I am designing a plugin architecture so that I can loop through the loaded plugins, tell them to send a message, and then tell them to log the message. As for how each plugin sends the message, it may be an SMS message, it could be an email, it could be routing through a custom text-to-speech protocol and broadcast through a loudspeaker, or it could be sent to any other system that exists. It's a method of allowing each plugin to flexibly log messages that I am struggling with.
–
Brendon DuganMar 5 '13 at 16:06

What is the purpose of the messages? It sounds like they can be schema-less/unstructured. Hard-coding the implementation to the application database is probably going to come back to haunt you. If the logging is unstructured anyway, why not just do like/use log4j or similar?
One option is to just write a thin wrapper API around log4j. Presumably you want to maintain logging context such that you can retrieve all logs from all plugins related to a single action/user. If so, then you need a way to tie the different logs together.

Well, one of the main things we are concerned about is reporting. As a little bit of background, we are developing this product for use within a medical environment and we are hoping to collect a lot of data about response times and also what types of calls are made in several distinct situations. Because of this, we would really like to keep everything in a relational database to be able to do some SQL-Fu on the data, but if there's a way to abstract that out I would be willing to use something else. Could you give a more specific example of what you're describing?
–
Brendon DuganMar 5 '13 at 16:34

What you are describing is a situation in which you are designing APIs - between your application and the plug-ins. At this level you should focus on designing the interfaces and the data that needs to be exchanged without worrying about any specific persistence strategy. Once you nail that part, then part of the implementation in your application can handle the data any way it wants.

Since you are using Java, take a look at any high-level API you already use for the design cues - like Spring Data, Collections (e.g. Map, List) etc. They all hide the details behind Interfaces, Abstract classes and Concrete classes that implement the desired functionality.

At a very high level, I can think of the following (very simplistic approach, not knowing the exact details):

An Interface that allows a plugin to register itself to your
application. Say you call it PluginRegistry. Then the method in it
should be relevant for any plugin to register itself to the
application and establish the communication links (like message
listeners, callbacks etc). Your interface can dictate what method(s)
the plugin needs to implement.

Say another interface defines the message exchange. It could be a high-level Interface coupled with an Abstract class (for default implementation) and one or more Concrete classes for specific implementations. You can leave the implementation of concrete classes to the plugins.

Your abstract classes can hide the details of logging in the database. Since you have the flexibility to design how the message should look like (plug-in will inherit from and/or implement the necessary interface), the abstract class with the default implementation can write the necessary data to the database (I'd prefer a log file) without the plugin knowing too much about it. This way you do not need to expose this functionality to plugins and can later change your implementations as needed.

The reason I mentioned that a log file is preferred over a relational-db is that you will have flexibility in logging any way you want in a file. A NoSQL db would also be a good choice here as it wont force you into hard relationships between your data early on. Then write some scripts (e.g. in Python, Perl etc) to analyze the log files and gather the necessary data and put in the relational-db as needed for reporting. But your main raw data store will be log files. As your application matures, you will get a good idea of the data's structure, important elements and automate the analysis scripts. As you log more type of data, you will need to update the scripts as needed, but hey you wont be changing the db tables where and breaking your application.