Monday, January 13, 2014

A little less work post-Christmas on philam (the current name of my Flex 6700 client), so far.
Nevertheless I've tidied up the code a little, particularly in terms of how streams from the radio can be associated with stream handlers.

One of the the truly nice things about Haskell is how elegantly concurrency and parallelism are integrated. Indeed as a pure, functional language Haskell lacks so many of the features that cause real problems with concurrency in other languages (e.g. destructive assignment, shared state, uncontrolled side-effects). While Haskell has many tools for concurrency, even the basic explicit concurrency (i.e. spawning a thread) is simpler and less burdened with issues. Haskell threads are lightweight 'green' threads that are scheduled onto hardware threads by the Haskell runtime. That means you can create hundreds or thousands, even tens of thousands of threads without gumming up the works. Moreover, Haskell threads behave very nicely when they are doing IO, they can be terminated without any problems and tasks like IO can be timed-out by just wrapping a 'timeout' function around some action. All of this lends itself beautifully to asynchronous handlers of the kind needed to react to data stream packets and notifications.

Philam's comms module now has a "startInStreamHandler" with the following type:

This takes:
A radio connection, to identify which radio the stream will be from
A stream specification, to describe what kind of stream is required
A packet handler, a function to handle each packet of the stream
An optional port number, provided if a port other than the standard VRT port is to be used

If the stream could be created and the handler was successfully launched, then the function returns a "FlexInStream", which is the record that keeps details of the created stream (such as the stream ID used to identify it in the radio).

startInStreamHandler forks a new Haskell thread to handle incoming packets from either the standard port (on which multiple streams are multiplexed) or the 'private' port.
If the common VRT port is used, then startInStreamHandler will also create a queue onto which the shared standard port listener for the VRT port will demux and dispatch packets of the stream. Haskell's software transactional memory (STM) package has a TQueue data type that works very nicely for this, whose 'read' function blocks if there are no elements in the queue, providing the synchronisation needed to do work as soon as data (i.e. a new packet) is available.

Although it looks like a 'packet handler' would lack context and therefore be unable to do anything useful, Haskell's closures provide all the context you need for arbitrary context depending on purpose. The audio stream decoder featured in my prior post has now been factored into a PacketHandler, consisting of a literal packet handling function and a cleanup function:

The function returns an action (signified by the IO) which contains the twin aspects of handling a single packet at a time and also cleaning up (the audioPacketHandler' and audioCleanup respectively). Both of these functions are defined in the context of the outer audioPacketHandler scope, which runs a sequence of actions to set up context before returning the PacketHandler action. In this case, these actions initialise the audio engine with its synchonisation object for sending samples. This is part of the closure of both PacketHandler functions, allowing them to utilise the context and tear it down properly when finished in the cleanup function.

The queue handler code, called by startInStreamHandler in the case that the stream is to be delivered on the shared standard VRT port does the following:

The handler uses 'forkFinally' to launch a 'forever' action that will continue to get an item from the packet queue, whenever one becomes available, blocking otherwise. The packet handler is asked to deal with each packet when one arrives. An onClose function is defined which will terminate the stream if anything happens to it that causes this concurrent routine to terminate. This function calls terminateStream which does common termination actions, such as removing the stream record from the connection record, then gets the packet handler to clean itself up.

Besides this improvement to the queue handling, my attention has turned to the UI, per my comments in the last post.

I have again surveyed the Haskell GUI-binding landscape and on balance I have decided to stick with gtk, the one cross-platform toolkit I have used before. I had wondered about wxWidgets (and the wxHaskell binding), but the Haskell binding hasn't been updated for a while. Conversely, a gtk3 binding has literally just been uploaded to the Haskell package repository Hackage. Not only does that bring Haskell bang-up-to-date with the gtk framework, but there is also an additional "gtk3-mac-integration" package that will make the UI work well on a Mac by using the Quartz backend for gtk3.

So, having decided to stick with gtk I had to build the underlying gtk libraries on which the Haskell bindings are based. I had previously built gtk2, so the usual build adventure lay ahead on me.

Indeed, things were a bit awkward to get going and it took a few hours to get through all the gotchas.

In the end, this is what is required:

Get ready for building the Haskell binding:cabal install gtk2hs-buildtools

Install the gtk+3 base with homebrew:brew install gtk+3This will finish by printing a caveat about dbus, which must be followed now or later

Ensure you have libxml2 with the python bindings (I had it without, so...)brew reinstall libxml2 --with-pythonThis will finish by printing a caveat about setting the PYTHONPATH, which must be used

Ensure you have gtk-doc installedbrew install gtk-doc

In order to build the Haskell bindings, you must make config visible:export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

Unfortunately on Mavericks there are issues with clang, so gcc must be obtained:

brew tap homebrew/versions

brew install gcc48

Cairo can be tricky, so better try the Haskell binding for it first:cabal install cairo --with-gcc=gcc-4.8

OK, now we can compile the read of the GTK 3 binding:cabal install gtk3 --with-gcc=gcc-4.8

Now we'll need the latest glade UI builder. So, download the latest glade sources and unpack the tarball. Glade is tricky to build clean. You have to do the following just to get configured:

Friday, January 3, 2014

I now have several days of hacking under my belt and have made some nice progress in learning the ropes and getting the radio to use useful things.

As I intimated previously, I am really impressed with what Flex have built (...and indeed are continuing to build). The 6x00 is a unique radio and it has been very much fun indeed to get acquainted with the programmatic interface of the radio as an appliance over the ethernet as well as enjoying operating it with the SmartSDR Windows software.

My initial goals have been to get a basic grasp of the control functions and data streams over ethernet. Flex have a nicely designed set of control sentences that are constructed and sent to the radio once you are connected. Before connecting, you can discover radios on your LAN via a periodic broadcast that each radio makes once it is initialized and ready for client connections.

To facilitate the usual business of sending command and reconciling responses with those commands I have built a blocking command queue in Haskell. This handles boring stuff like allocating monotonically ascending message IDs and also simplifies internal addressing of radios so that any connected radio is henceforth addressed by serial number. Incoming traffic (which can be command responses as well as asynchronous status messages) are immediately delivered into different queues from which various handlers can subscribe to consume them. Calls to send commands to the radio are handled through a central dispatcher that blocks on the specific response becoming available in the responses queue. Haskell makes this sort of thing rather easy and completely thread-safe through the use of the STM monad, which implements software transitional memory. The STM library has a range of useful concurrency tools, amongst which is the TQueue (transactional queue) that I use for all the received message queuing.

Haskell also handles threading beautifully - and it's fast and very flexible. It runs its own scheduler that allocates code to be executed on available hardware threads. Even if you restrict Haskell to a single thread, it will still do a great job of cooperatively multitasking between concurrent routines in your program... even when they are doing IO.

Importantly, Haskell is also cross-platform (at least for most platforms people care about: Mac, Windows, Linux). The core of Haskell runs on many other processors and platforms (such as ARM/ Raspberry Pi) but of course that doesn't mean that all the libraries I need would work beyond the core desktop platforms.

Once the command dispatch mechanism was working I added some convenience wrappers around a number of the available command sentences, including:

An early objective of my experimentation was to get streaming audio to be played on my computer and in preparation for that I have had to survey the available cross-platform audio libraries. I've had experience with Apple's excellent CoreAudio before, but I've never tried to use a cross-platform library. It turns out there were several to choose from: PortAudio, FxModRaw, PulseAudio, OpenAL and a few others I didn't recognize. The main feature I needed was a 'simple' streaming PCM playback and for the future also audio-in. In practice things are a little messy with these libraries with several of them being reliant on versions of a binary dependency that I found difficult to compile on the Mac at least ("difficult" meaning having to hack build files to get things working). The OpenAL library, despite its age - or perhaps, because of it - was very easy to get set up. The only problem with OpenAL is that it isn't really a streaming engine per se. Rather, it provides a basic mechanism to queue buffers sequentially for playback. It is left as an exercise for the consumer to set up whatever higher-level sample management mechanism they may deem appropriate. Because of this, I had to set about writing a buffer management component that asynchronously accepts samples from a source and queues filled buffers to playback (retrieving used buffers once OpenAL has finished with them).

Once this little audio engine was completed I turned my attention to the VITA stream format. Unsurprisingly, there is no off-the-shelf decoder for VITA-49 standard streaming in the Haskell package repository 'hackage'. This has meant rolling my own from specifications that I have found online. Haskell has some great tools for marshalling binary data too. Specifically, the ByteString and Binary packages provide high-performance processing of data packets/streams. The Get and BitGet monads in the Binary package make building bitwise decoders very easy. As well as the VITA packet decoding, I have used a Get decoder to prepare the audio sample stream into an appropriate format for OpenAL (which currently only supports int16 samples). Here's the code:

Even if you are not familiar with Haskell, you can probably grok was is going on here.
A VRT (VITA-49) packet is decoded and if that succeeds then another decoder is run on its "vp_data" component. This decoder is mostly composed of the "decodeStereoPairsToMonoInt16" function, which lives in the Get monad as you can see from the "Get Int16" return type. That function only handles a single left-right pair of samples - by reading them from the byte stream as 32 bit words in network order and converting each to a Haskell Float. These two values are then averaged, scaled to the dynamic range of an Int16 and then returned as a single mono sample. To perform this treatment on all the bytes in the stream ("allPairs"), the monadic loop function 'whileM' is used to apply this decoder to the stream until it is "not isEmpty" (i.e. empty). The runGet function actually performs the Get action on the stream, returning the result (the stream of mono samples if all is well). Assuming the mono samples are indeed returned, then these are handed to the sound engine by filling an 'MVar' with "putMVar samples mono samples". MVars are another Haskell concurrency feature that implement a kind of threadsafe pigeon-hole where values can be 'put' and 'taken' by different threads with put blocking if the pigeon-hole is still full and take blocking if it is still empty. The buffer-filler of the audio engine uses this mechanism to asynchronously take any number of proffered samples and ready these for playback.

With the basics of control and audio working I will now turn my attention to some UI.
I will have a similar question as to which cross-platform UI library to use. In previous experiments I have looked at GTK and wxHaskell. Most of my actual dev time on Haskell GUI projects has been with GTK. This works OK on a Mac with X as a dependency... although in theory you can build a gtk library that works directly against Cocoa and therefore removes that issue. I have no idea whether GTK works nicely on Windows at this time. wxHaskell might be a better choice on this occasion, so I'll spend a little time trying to figure out where that library stands (the underlying wxWidgets have just been updated to v3).