CTF / VSM MedTech

Introduction

The CTF system is one of most widely used MEG systems and one of the first on which real-time MEG analysis has been done for the purpose of BCI (see Mellinger et al., NeuroImage 2007). There are 151 channel and 275 channel systems. The systems provide real-time access to the MEG data through shared memory.

The acquisition software runs on a linux computer. If prior to starting the acquisition software shared memory with the appropriate details is initialized, the acquisition software will write a copy of the data to that shared memory. The shared memory is split over 600 packets, each packet holding 28160 samples. With approximately 350 channels (MEG, EEG and status/trigger channels) in the typical MEG system, that amounts to approximately 80 samples per packet. So in total there are 600*80=48000 data samples stored in a shared memory buffer.

A specific application for the CTF real-time interface is to monitor and minimize movements of the subject's head during data acquisition. This makes use of the continuous head localization (CHL) channels and is described in detail here.

Interface with MATLAB and FieldTrip

A new tool was developed that combines the old AcqBuffer (see below) and ft_realtime_ctfproxy in one stand-alone application. This program is called acq2ft and resides in the realtime/acquisition/ctf directory. It operates by grabbing one packet (setup or data) at a time out of the shared memory, and more or less directly transferring it into a FieldTrip buffer that can be embedded in acq2ft itself, or on any other location on the network.

acq2ft decodes header information from the .res4 file pointed to by the setup collection packet, and thus knows by itself which channels contain triggers instead of relying on a 3rd application or Matlab script to write that information back to shared memory. On top of that, acq2ft overallocates the shared memory by 1000 samples and is prepared to operate successfully even if the proprietary Acq application writes too much data into any slot. Instead of preparing the shared memory segment for access by a third application, it streams the data and events (decoded from the trigger channels) to a local or remote FieldTrip buffer, as depicted by the following diagram:

You need to start acq2ft before starting Acq so that the shared memory interface can be detected and connected to by the latter.
You have the option of spawning a FieldTrip buffer server directly within acq2ft, using

acq2ft

to use the default port 1972, or using (note the minus)

acq2ft - port

to spawn at server at the given port. You can also tell acq2ft to stream the data to a buffer provided by another application (possibly on another machine), using

acq2ft hostname port

Please note that for this to work, you will need to start up the remote buffer beforeacq2ft.
Once acq2ft is happily up and running, you can start Acq from a different command line or using existing GUI tools.

Since streaming the data to a remote FieldTrip buffer might incur a delay due to network traffic, acq2ft employs an internal ring buffer
for up to 10 data packets and a simple (socket pair) mechanism to synchronize between copying data packets from the shared memory to the internal ring buffer, and streaming the data from the ring buffer to the FieldTrip buffer. This means that small delays should not interfere with the time-critical operation of clearing up slots in the shared memory segment, as long as the average throughput is high enough.

Downsampling, channel selection, applying gains

There is a recent extension to the acq2ft tool, called acq2ftx which has the ability to downsample the incoming CTF data, apply the correct sensor gains, and write out only selected channels. That program needs to be called like this:

acq2ftx hostname:port:flags:decimation:channels

where flags can be any combination of R, which enables writing the .res4 file into the FieldTrip buffer header, E which enables sending events as decoded from the trigger channels, and G which enables multiplying the samples by the correct gain values, and consequently writing out single precision floating point numbers instead of the default 32bit integers. Decimation needs to be a positive integer number, and channels is a comma-separated list of channel labels, or a star (*) for sending all channels. However, it is important to note that *no* lowpass filtering is applied before decimation, that is, you have to use the hardware filters (setup in Acq) to use this option.

acq2ftx -:1972:RE:1:*

Actually you can have multiple definitions and stream different parts of the data to different buffers. For example, the following call will spawn a local FieldTrip buffer on port 1972, which will receive all channels, the .res4 header, and events (but data is kept at 32 bit integers), and in addition stream out 4x downsampled and scaled head-localization channels to a buffer on the lab-meg001 computer (also port=1972):

Compilation

On the command line, change to the realtime/acquisition/ctf directory and type make. This will produce both acq2ft and acq2ftx, as well as some tools for testing and managing the shared memory. Note that you might need to compile the buffer library first.

In FieldTrip it is possible to use the fileio module to read from shared memory. Because the shared memory also has to be freed to ensure that the Acq software continues writing to it, a daemon process has to be running in the background. The daemon process is called “AcqBuffer” and is started from the command line. It constantly loops over the 600 packets in shared memory, and if there are less than 20 packets free, it memcpy's the “setup” packet (containing the name of the res4 file that has the full header details in it) to the next packet, thereby freeing the packet previously containing the setup. This procedure ensures that the content of the setup packet can always be read, even while it is being copied.

Network transparent interface to the MEG data (based on the solution above)

The ft_realtime_ctfproxy function (part of the realtime module in FieldTrip) reads the MEG data from shared memory and writes to a FieldTrip buffer. The FieldTrip buffer is a multi-threaded and network transparent buffer that allows data to be streamed to it, while at the same time allowing another MATLAB session on the same or another computer to read data from the buffer for analysis.

Subsequently in another Matlab session you can read from the FieldTrip buffer using the ft_read_header, ft_read_data and ft_read_event functions by specifying 'buffer://hostname:port' as the filename to the reading functions.

Besides maintaining the copy of the setup packet and ensuring that there are always some packets free to receive the new data from Acq, the AcqBuffer application also performs trigger detection on indicated channels and stores these triggers in a convenient representation. This speeds up the trigger detection in the read_event function in MATLAB considerably.

In MATLAB the full header details of the MEG data set are determined by first reading the packet containing the setup information (i.e. the name of the res4 file) and subsequently by reading the details from that res4 file using the standard reading function. The number of samples available in the dataset is updated to reflect the amount of data present in the buffer.

After you start

>> AcqBuffer

on the linux command line, and in another terminal

>> Acq

You can access the data in MATLAB like this

hdr = ft_read_header(filename), this returns a structure with the header information
event = ft_read_event(filename), this returns a structure with the event information, (i.e. the triggers)
dat = ft_read_data(filename, ...), this returns a 2-D or 3-D array with the data

where filename should be a string containing 'ctf_shm://', i.e. similar as a Universal Resource Identifier. In case you want to use the header information from another res4 file, you can specify the filename as 'ctf_shm://<example.res4>', i.e. including the full path and filename of the res4 header file.

The (inter-)operation of the three involved software components, all running on the same machine can be summarised with the following diagram.

Known problems with CTF real-time acquisition

shmget: Invalid argument

It seems that the default linux/redhat configuratino of the shared memory does not allow a sufficiently large memory block to be allocated. To change the setting in the operating system, you should do (as root user):

echo 1000000000 > /proc/sys/kernel/shmmax

There appears to be two ways of (re)defining the amount of shared memory in your system (tips thanks to Dave Glowacki of SSEC):

add this line to your /etc/rc.d/rc.local file:

echo shared_memory_size > /proc/sys/kernel/shmmax

(where shared_memory_size is the amount of shared memory you want to declare in bytes) and reboot.

a more permanent solution would be to change the value of SHMMAX in /usr/src/linux/include/asm/shmparam.h and rebuild your kernel.

Number of channels

There is a problem in the CTF acquisition software that sometimes causes the shared memory interface to fail. The diagnosis of the problem is that the Acq software runs and writes the data to the shared-memory buffer, where it is detected by AcqBuffer, and that after a certain random amount of time (around one minute) the AcqBuffer stops. The problem seems to be caused by a memory buffer overrun. The shared memory consists of 600 packets, each defined as

So in total each packet is 5*4+28160*4 bytes long, and there are 600 of those in shared memory. If the numChannels*numSamples of the previous block is slightly larger than 28160, it means that Acq is trying to write more data points into the “data” section of that packet than fits in, causing the data to flow over into the next packet. The first couple of integers in the next packet (indicating the Type and other details) are therefore messed up, and Aqc thinks that that packet is already filled. Then it stops writing to shared memory altogether.

I have tested this idea with a specially tweaked version of my AcqBuffer shared memory “maintenance” program and indeed see this happen for a data block that has 91*310=28210 samples in it, which is 50 more than the 28160 that would fit in. The next block is therefore corrupt.

Now understanding the problem, we can start thinking about a solution. Somehow in the CTF code there is an incorrect estimate of the number of samples that fits into a block. Probably we can play with the channel number to circumvent the problem. Given a certain channel number, Acq will have to determine how many samples fit into a single block. I suspect the bug in the Acq code to be something like
sampleNumber = round(28160/numChannels)
where it should be
sampleNumber = floor(28160/numChannels)
i.e. rounding off to the bottom. To solve this, we can look at

Update: This calculation seems not to be 100% correct. For example, 359 channels do NOT work. 360 seems to be okay.

This is the table (up to 500 channels) that lists the number of channels for which it will work. The test that I performed happened to be with 310 channels, which is not in the list. Since we can always add a few (unused) EEG channels to the acquisition, we can work around the bug in the CF Acq software.