In the kitchen

Ok, so maybe it’s time to start describing some of the new stuff cooking in the kitchen lab.

I’ve been exploring software designs to use as basis for JeeMon, that new switchboard-type application for physical computing devices. The idea is to treat the system as one big message-passing machine – a bit like Message-Oriented-Middleware (MOM), which has been around for ages, but without getting sucked into any heavy-weight conventions.

Messages can be passed around, queued, stored, duplicated, filtered, transmitted, returned, ignored, sorted, etc. After all, living organisms do nothing but send chemical messages around, so why not do the same for an infra-structure focused on physical computing (sensors, displays, actuators) and home automation?

If you’ve been following this weblog for a while, you’ll have noticed that almost all the output I generate in sketches is of the form “identifier arg1 arg2 …”. So for example, packets received by RF12demo look like:

OK 61 9 2 8 79 243 87 13 0 15 0

A barometric reading from the BMP085 sensor on the pressure plug may get reported as:

BMP 233 10019

And so on. One or more upper case letters, optionally followed by digits. Then the payload (which may also include floating point and string values, not just ints).

Another convention I’ve been sticking with is to report the name of the sketch on the serial port during startup:

[ookRelay]

Or an identifier plus the current configuration settings:

[RF12DEMO] W i23 g5 @ 868 MHz

These two conventions can be used for an object-oriented design. With a bit of preparation, the name of such sketches can be automatically associated with a class, and each line treated as a method call.

Here’s the skeleton of the first level of code I use for decoding RF12demo.pde output:

Or to put it differently: with JeeMon running, you can hook up a device (JeeLink, JeeNode, Arduino, etc) containing some sketch and the matching class will automatically be associated with that serial port, once the sketch has been identified. A new object is then created, and its methods will be called whenever the device sends new output (messages?) over the serial line.

Simple!

What I would like to do, is manage the sketch (C/C++) and the class (Tcl) together, perhaps as two files in a common development directory for that project. That way the interface between the two pieces of hardware essentially vanishes into the background. The point being that each class can then do all sorts of things, such as storing results in a database, sending it to another system over the network, updating web server pages, popping up a GUI window to show incoming data in real-time, etc.

This mechanism is very simple, even under the hood. This matters, because it has to work even when JeeMon is running on low-end embedded Linux hardware. But at the same time, such a MOM + OO design will allow considerable flexibility for abstract / high-end use later.

PS. If you’re familiar with Tcl, you might be surprised to see all this “oo” stuff in here. That’s because JeeMon uses Tcl 8.6 with built-in object system. Multiple inheritance, mixins, filters, dynamic class change support, delegation, prototypes / per-object methods, it’s all in there. I’m also using ensembles and there’s an interesting coroutine-based web server waiting in the wings (called wibble).

For the record: nothing ever gets added just to be buzz-word compliant. If a feature simplifies application-level concepts and leads to a cleaner implementation in JeeMon, it’ll be used, else I’ll pass. Life’s too short to jump on bandwagons.

Did you consider buzz-word compliant CouchDB or MongoDB to store all your data before making a 1 on 1 coupling between a sketch and a Tcl class? nodejs should be available on your linux hardware as well…

Documented-oriented storage might not be the best fit for WSN’s, where timelines and repetition define the main structure – which IMO make an inverted column-wise database more effective, at least on the “raw storage” level. I’ve been tracking CouchDB for quite some time, it’s very interesting – but it requires Erlang, which might not fit on a small embedded Linux machine (it all depends on how small, of course). Hadn’t seen MongoDB before, thanks.

The central raw data item I expect to have to deal with in JeeMon is the (time,paramid,value) triple.

The current plan is to use Metakit for compact storage of raw data and SQLite when querying becomes the dominant issue. Both of these are well-embedded in Tcl. Metakit is available down to the smallest platform on which JeeMon will run, SQLite can be included for at least all desktops and the more substantial Linux-based webserver setups.

Having said this – at the end of the day, it’s all a matter of layering: anyone can set up a more document-centric system and keep it up to date will all the data they need from the above setup.

Thanks also for pointing out NodeJS – but from what I can tell, at least the server side of that sort of functionality is built into Tcl (with async I/O and coroutines). I do intend to deeply integrate with JavaScript in the browser, and am hoping that jQuery will be able to fill in the rest (and REST, heh).

I have some non-JeeNode-based hardware which happen to be connected the the local network.
I would like to let these nodes report data to the JeeMon in a convenient fashion. I guess this could fit in well in your design, and having some demo code for such a setup would be a good starting point for us tcl non-experts :)

The same could then also be used for data collection services running on the same server, without the need for these to be integrated into JeeMon (and thus separating data collection from data aggregation and presentation).

More concretely, what I want to do in the latter case is to run misc. network monitoring software and make the results available as very simple statistics about activity and resource usage (e.g. bandwidth, local network activity, web stats, presence of certain no-always-on computers).

What is convenient really depends on the service in question and whether I have control over it or not.
My initial idea was that it would make sense to layer the components in a way that the lowest-level JeeMon component doesn’t know the difference between a radio device and a network device (i.e. using the same line-based text protocol).
From there we could make a standard network listener (e.g. a http GET) for pushing these lines to JeeMon (radio-protocol over IP).
For more complex services, a separate process or separate component in JeeMon could handle application-specific logic (polling network, pulling info from the filesystem, scrape app output) and convert it to the same text format as above.

When it comes to database modeling and data presentation, I also have some requests which may influence the above interfaces as well. I’m naturally free to do what I want here, but in case this makes sense to you, planning for such datatypes might make sense:
o Poll the availability of a resource. I want to log (e.g. every 5 minutes) if a certain IP is reachable or not and log this in the database.
o Count binary events and log to the database (in this case, how often a door is opened)

In both cases, I’ll somehow visualize both the current status and history.