Motor neural prostheses, also termed brain machine interfaces (BMIs), have the potential to help restore motor functionality for patients suffering from a wide range of neurological injuries and disorders. These systems convert the electrical activity measured from neurons into control signals, which can then be used to guide assistive technologies, such as prosthetic arms. A variety of designs have been attempted for these systems, but they share the basic form shown in Figure 1. The electrical activity of neurons can be acquired from a variety of electrodes. These signals are acquired using custom electronics. Neural “decoder” algorithms are then applied to neural activity to estimate movement commands. These commands can be used to attempt a wide variety of tasks, and performance can be measured with a number of common metrics.

Figure 1: Block Diagram of a Brain Machine Interface

BMIs rely on a large body of prior literature in basic motor neuroscience that related the firing rates of individual neurons to various movement parameters such as position, velocity, and force (Evarts, 1968, Georgopoulos et al., 1982). The first human experiment using an implantable device was completed in 1998 (Kennedy and Bakay, 1998). The first two primate experiments demonstrating real-time control of a computer cursor by an ensemble of neurons occurred in 2002 (Taylor et al., 2002, Serruya et al., 2002). Brain Machine Interfaces are the focus of a rapidly growing field. As one informal measure of this, in 2000, there were approximately 10-20 abstracts presented at the Society for Neuroscience’s annual meeting in “Brain Machine Interface” sessions; by 2006 that number had increased to 75, and by 2010 to 589.

Input Electrodes

Intracortical Electrodes

The highest degree of spatial specificity at which one can obtain neural firing rates is at the level of individual neurons. This can be accomplishing using high impedance (several hundred kOhms to a few MOhms) electrodes penetrating 1-2 mm into cortex. For chronic experiments (0.5 – 5 yrs in duration) several multi-electrode systems have been developed. The Utah array (Fig 2) was originally developed in 1996 (Nordhausen et al., 1996). It is a 10 x 10 array of silicon microelectrodes, which can be fabricated in a batch process using silicon wafers in a manner similar to microchips. The recording tips are coated in platinum, or platinum-iridium for stimulation. One hundred wires are wirebonded to the back of the array, and attach to a connector pedestal that is then attached to the skull with titanium screws. The entire device is coated in biocompatible parylene. This is the only many-channel intracortical device that has previously been used for chronic implantation in humans. In primates, these devices have been used to record up to 300 channels simultaneously using multiple arrays (Hatsopoulos, personal communication).

Another approach to recording from individual neurons for many months uses slowly inserted microwires, inserted individually or in small groups in various locations around the brain. While this requires a more lengthy surgery, and more complex connectorization, these systems have the advantage of being able to record from many brain areas simultaneously. The largest number of channels recorded from an awake behaving nonhuman primate using this method is 704 microwires (Nicolelis et al., 2003).

Field Potential Recording

The summed electrical activity from many neurons can be recorded through lower impedance electrodes in various locations, as shown in Fig 3. These can be placed within cortex (called LFP) just above the cortex (called subdural electrocorticography or ECoG), above the protective dura (epidural ECoG), above the skull (subdermal) or above the skin (called electroencephalography or EEG). Extracellular neural signals reduce in amplitude in proportion to of 1/distance^2. Electrodes placed further away from the source signals average together the activity from a progressively larger population of neurons, as shown in Figure N. However, this difficulty can be mitigated somewhat since neurons in a particular area sometimes share various features. Also, field potential electrodes may cover a larger portion of the brain than intracortical electrodes as in Fig 3. Finally, in the case of EEG, electrode placement does not require surgery. Field potential signals from ECoG might be more stable across time than intracortical signals, and similar decoders have been applied continuously for up to 5 months chronically (Chao et al., 2010). And while information content is initially substantially lower than intracortical signals, it may be possible to increase it through regular training. Local field potential from penetrating electrodes closer to the neurons has also been indicated as a source of movement information, with higher spatial specificity than ECoG (Scherberger et al, 2005, Bansal et al, 2011).

Arrays of field potential electrodes for EEG or implantable ECoG can be obtained commercially, since these technologies are used in epilepsy for locating seizure foci. These arrays can be augmented by adding smaller, higher impedance recording sites between the clinical electrodes, known as “micro-ECoG” electrodes. These have also been fabricated in very large arrays for nonhuman primate work (Rubehn et al., 2002). In people, microECoG have demonstrated higher information content than tradition macro electrodes (Kellis et al., 2010). Also, conformal electrodes on very thin films that can follow the shape of cortex have been fabricated (Kim, Viventi, et al., 2010), which may become important for future BMI studies.

Hardware

Several commercial systems exist for recording and processing many channels of neural data simultaneously (Blackrock Microsystems, Plexon, Tucker Davis, NeuroLynx, etc). These systems can amplify very small neural signals on the order of microvolts and filter the bands of interest. To record neural action potential waveforms in primates, data must be digitized at ~30 kilosamples per second. A much lower data rate (~1 ksps) is required for most field potentials or simple threshold crossings generated from single or summed action potential waveforms. Once digitized by this specialized hardware, most modern PCs can run typical neural decoding algorithms sufficiently fast for real-time use.

To put all of these components into an implantable system, similar to a pacemaker, nearly all of these signal processing steps need to happen in a fully implantable device. Therefore, considerable work has been done on custom integrated circuits for BMIs. Harrison and Charles (2003) developed a bioamplifier that has been used for many channel wireless neural recording in nonhuman primates. Two groups have developed systems that can be fully encapsulated, and have undergone preliminary testing in vivo (Borton et al., 2009, Harrison et al., 2009, Sharma et al., 2011). These systems are shown in Figures 4 and 5. Researchers at the University of Michigan have developed a system where the electrodes are integrated with electronics, and have assembled 1024 systems (Wise et al., 2004). Chips have incorporated a varying amount of data compression on chip (O'Driscoll et al., 2006), including spike sorting (Chae et al., 2009). While many systems have power consumption too high to be powered from an implantable battery, Sarpeshkar et al have fabricated amplifiers and analog to digital converters at a power cost less than 9 uW per 20 ksps channel (Sarpeshkar et al., 2008).

While the small area and low power consumption of available integrated circuits is sufficient to process hundreds of channels of neural data, open research questions still remain. For example, how does one encapsulate the circuitry such that it can remain in the body for many years? Electronics coated with 6 um parylene C have been demonstrated to function reliably for up to 276 days (Sharma et al., 2011). Commercial implantable systems rely on hermetically sealed feed-through connectors into a welded titanium casing, in which the electronics are housed. However, BMIs use hundreds of electrodes, and high-density miniature hermetic feedthroughs are not currently available.

Decoder Algorithms

Front End Processing

A certain amount of signal processing must be completed before any decoder algorithm can be applied. For example, any neural signals including action potentials must be processed to extract the individual spikes. This can be as simple as detecting events that pass a certain voltage threshold, or by placing voltage boundaries or “hoops” on the waveform (See Fig 6) to extract all those with a certain shape characteristic. More complex algorithms have also been developed to separate out the activity of individual neurons. For example, one can extract the first few principal components of the waveform shapes, and then use an automatic clustering algorithm to separate out individual neurons (Lewicki, 1999, Sahani, 2009), as shown in Fig 6. Field potential signals also require pre-processing, to filter the band(s) of interest. For example ~10-30 Hz for beta band, ~30-100 Hz for gamma band, or ~100-500 Hz for LFP signals associated with nearby spiking activity. Then, often the voltage signal is squared to obtain the power, and the log is taken to better approximate a normal distribution. However, one can also use a principle components analysis to extract broader cross-spectrum features (Miller et al, 2009). It is currently an open research question as to whether or not decoder performance can be improved by considering individual sub-bands within the higher gamma frequencies as sources of independent information.

Continuous Decoders

The first several real-time BMI experiments used linear algorithms that fall into a few categories. First, the “Population Vector” approach was based the work of Georgopoulos et al. (1982), who evaluated neural firing rates while an animal reached in various directions around a circle. These firing rates could be well fit with a cosine. Consequently, a cursor could be driven by a population of neurons, each having a preferred direction \(\theta\) in which it tended to push that cursor, modulated by it’s overall firing rate Taylor et al., 2002). Such an approach is described by the following equation for the firing rate of a single neuron (Schwartz et al., 2004).

\(
firingrate - b_0 = A cos \theta
\)

Other early real-time experiments used a simple linear mapping, a Wiener filter, to translate neural firing into position or velocity of the cursor (Serruya et al., 2002, Carmena et al., 2003). This is described in the following equations, in which each neural firing rate Y at multiple time lags is multiplied by a coefficient B for each kinematic parameter of interest in X, and then the contribution of the whole neural population is summed.

More recently, recursive Bayesian methods have demonstrated higher performance than previous methods (Wu et al., 2006, Kim et al., 2009, Gilja et al., 2010). These methods specify a full probabilistic model for two things. First, a trajectory model describes the how the arm moves over time. For example, it cannot move instantaneously, and the next likely position and velocity (in the X state vector below) can be calculated by simple physics. Second, an observation model describes how neural data Y linearly maps to that kinematic state X. These are probabilistic models that attempt to capture noise and natural variation. One common implementation is the Kalman filter, with the trajectory model and the observation model given below, where the scaling parameters A and H, and the noise parameters W and Q are fit to the training data.

\(
X_t = A X_{t-1} + W_t
\)

\(
Y_t = H X_t + V_t
\)

Discrete Classifiers

In addition to controlling motion, neural signals from motor areas can also be used to communicate, by selecting letters for example. This does not require continuous movement. Only the correct one of N choices must be determined. Many other prosthetic signals are also classifications by nature, for example myoelectric control signals for selecting the grasp type on a prosthetic hand are often treated as discrete selection signals. Many classification methods have been attempted in the BMI literature. Here, we describe one of the most common, Naïve Bayes, for estimating which of N radial targets a subject wants to acquire from cortical spiking data. If the average spike count associated with each target is modeled as a Poisson distribution with mean spike count \(\lambda \) for each neuron, then given a new set of firing rates, the most likely target can be estimated by calculating the posterior probabilities for each of the possible targets as in the equation below, and simply selecting the target with the greatest likelihood of being correct. Using such a method, Santhanam et al. (Santhanam et al., 2006) was able to obtain ~6 bits per second of information using neural activity from motor areas during a 1 of 8 selection task.

Adaptation and Learning

Performance could also be improved through subject learning. In 1969, Fetz demonstrated that nonhuman primates can learn to modulate the firing rates of small numbers of neurons in fairly arbitrary ways (Fetz, 1969, Moritz et al., 2008). Since then, several groups have shown the modification of neural activity in a way that improves BMI performance (Jarosiewicz, 2008, Ganguly and Carmena, 2009). In addition to plasticity at the neural level, adaptation in this context could also reflect a changing behavioral strategy, such as re-aiming. One group demonstrated steadily improving performance across 2 weeks when using a decoder whose coefficients had been shuffled across neurons compared to normal arm movements (Ganguly and Carmena, 2009). Learning is particularly important for field potentials, in which limited behavioral information exists initially, but could be encouraged through training. It is however important to note that performance has generally shown to improve and then plateau rather than improve indefinitely with learning. Further improvement might have been expected if neurons could individually modulate their activity to an arbitrary degree and future research could explain or circumvent the limitations on this process.

Emerging Methods

Other methods for improving the performance of BMIs are also ongoing. For example, incorporating sensory feedback through electrical (O'Doherty et al., 2010) or optical (Gilja et al., 2011) stimulation in sensory brain areas or moving the paralyzed arm with an exoskeleton (Suminski et al., 2010) could improve the subject’s ability to control the prosthetic device. Several approaches are also underway to address the oversimplified assumption of a linear relationship between neural activity and end point position or velocity. Non-linear machine learning techniques such as neural networks have been used (Aggarwal et al., 2008, Sussillo et al., 2009). To produce a non-linear model that avoids overfitting and generalizes to novel circumstances, there is also work pursuing force based “biomimetic” decoders based on muscle activation (Fagg et al., 2007, Pohlmeyer et al., 2007).

Performance Evaluation

Since there are a wide variety of signal sources and a varying amount of information available from those sources, a wide variety of tasks have been used to characterize BMI performance.

Discrete Selection

Slower, but non-invasive, signal sources like EEG are well suited to communication prostheses, and the most common tool used to evaluate performance is the P300 speller. This system makes use of an event related potential (ERP) that occurs when something out of the ordinary happens, i.e. the “oddball” signal (Donchin and Smith, 1970). This signal can be used to spell out words using a 6 x 6 grid of letters. The subject focuses on the letter they wish to choose while columns and rows are rapidly illuminated. When the desired letter is illuminated, the “oddball” signal can be detected using EEG in parietal areas. Performance can then be quantified by the number of correct letters/minute or bits per second. The highest reported data rate at SFN 2010 using a P300 speller was ~0.6 bps (Fry et al., 2010, Ryan et al., 2010).

Beyond single bit detection, when discretely selecting 1 of N possible targets, performance can be accurately measured using an intuitive bits per second based on the number of possible selections. Some systems that allow apparently free movement of a cursor in space are still most accurately evaluated as classification systems. I.e., if a system cannot stop and select various locations anywhere along one of the degrees of freedom, and information is only extracted at the extent of the workspace when the cursor hits some boundary, this is fundamentally a classification problem and should be evaluated in bits per second based on the number of possible targets.

Continuous Movement

For continuous controllers, performance evaluation can be more complicated, and a number of metrics are in common use. The most common metric is the correlation coefficient. This is the correlation between the decoded position of the hand or cursor and the position of the actual hand, or the presumed correct answer. This metric is in wide use for several reasons. It can be used with offline decoders (i.e., decoding data that was recorded previously). Also, it gives an intuitive value for a wide variety of tasks. There are however, some serious shortcomings. For example, a decoder can provide very little information about target position and still generate a high correlation coefficient by simply detecting movement, though this problem is mitigated in a center out task. Indeed, many offline decoders never hit the target. Also, choices of a smoothing window can raise or lower correlation coefficients. Correlation coefficients and offline metrics in general also do not linearly relate to online performance. For example, Cunningham et al. showed an example where optimal parameters determined offline were substantially different than those that provided the best performance online (Cunningham et al., 2011).

Success rate, in terms of the number of targets successfully acquired, has also been used to quantify BMI performance. However, since task parameters can vary between experiments, this is not a good metric for comparing across studies. This performance metric depends heavily on task parameters such as acceptance windows in space and time. Another simple metric is mean squared error compared to an ideal path. This can be very difficult to compare in absolute terms across slightly different tasks. Also, it presumes an underlying “correct” straight reach during tasks in which only target acquisition is trained and rewarded, not the specific kinematics of a reach. The full path must be instructed to obtain an accurate measure (Sadtler et al., 2011). If one assumes that the subject is always motivated to move closer to the target, a more appropriate metric could be average distance to target over the course of a trial (Cunningham et al., 2011).

Potentially the best method for quantitatively characterizing continuous performance is based on the ISO standard 9241 for estimating computer mouse performance in bits per second, though this is not directly comparable to classification bits per second. For comparison, the minimum necessary throughput for a computer input device is 2 bps, with a maximum difficulty of 5-6 bits. This standard relies heavily on Fitt’s law, in which the difficulty of an acquisition can be described in bits as in the equation below where D is the the distance traveled and W is the cross section of the target. For example, accurately landing on a 1 cm segment of an 8 cm line has a difficulty of 3.2 bits. If this can be accomplished in 1 second, the bit rate is therefore 3.2 bps. Perhaps the highest difficulty level attempted in the intracortical signal literature is 2 bits (Ganguly et al., 2009). The highest continuous bit rate obtained to date was 1.81 bps compared to 1.97 bps using the native arm on that particular task (Gilja et al., 2010). Care must be used when extending this metric to multiple dimensions, but for comparable tasks, the continuous bits per second measure can be robust to small changes in task design, such as the size of the workspace or the number of targets. However, for this method to be valid, a selection signal must be used to select the target, whether it’s an explicit “click” or a substantial dwell time. If this is not present, a fast random walk would generate the highest performance.

\(
\log2{(D/W+1)}
\)

In the long term, as more BMI research is done with paralyzed humans, the true standard of BMI performance is the difference in the quality of life of the users. Several scales have been proposed to evaluate a patient’s capability for “activities of daily living”(Lawton and Brody, 1969, Katz, 1983). These standards become more applicable as BMIs are used to accomplish useful physical tasks. For example, one experiment recently demonstrated that monkeys could feed themselves bits of food by controlling a prosthetic arm (Velliste et al., 2008). Initial experiments have also been attempted with paralyzed humans controlling a robotic arm to manipulate a cup (Liu et al., 2010).

Translational Efforts

EEG electrodes have been used for human BCI since the 60s. Early work with intracortical devices in humans was completed by Phil Kennedy in 1998 (Kennedy and Bakay, 1998). Electrodes were implanted in a patient who had lost the ability to communicate due to amyotrophic lateral sclerosis. The patient was able to voluntarily modulate the recorded signals. Also, starting in 2004, real-time BMI experiments have been done with many epilepsy patients with ECoG implants for seizure localization (Leuthardt et al., 2004).

In term of many-channel intracortical devices, in 2004, the FDA approved a clinical trial to test the safety of the “Utah” array during chronic implantation in human patients. This generated the first chronic ensemble recordings from single neurons in human beings. Subjects were able to control computer cursors at a level of performance similar to monkey experiments at that time (Hochberg et al., 2006, Truccolo et al., 2008). Proof of concept experiments also demonstrated control over a wheel chair, a TV remote control, and a simple prosthetic hand (Hochberg et al., 2006). Similar performance was maintained for up to 1000 days with the same implant (Simeral et al., 2011).

In 2011, a clinical trial for cortical control of prosthetic arms using Utah arrays in paralyzed people was approved by the FDA. This project was the first to use the "Innovation Pathway" for the streamlined approval of new medical devices.