Mary Lou Jepsen: Could future devices read images from our brains?Jepsen’s is the kind of talk that tests the limit of TED’s short-talk format — it’s full of technical information and new science, but by definition it can’t offer the same detail as a textbook or a series of lectures. A TED Talk is meant to spark interest in learning more, and this talk is a classic case. So we offered Jepsen a chance to respond to questions from audience members in writing, and have condensed those responses into the FAQ below.

Our new dig deeper sections allow us to link supporting material right on the talk page, and since we hope this talk will inspire you to learn much, much more, we’re including it on Jepsen’s talk page.

First, an email from Jepsen:

My central thesis is that combining increased temporal and spatial resolution in MRI techniques with increasingly powerful data correlation techniques will allow the derivation of interpreted meanings from neural signals. I observed, further, that the techniques that exist already allow some correlations. I made it clear in my talk, as the critics do as well, that the correlations possible to date are clumsy and inaccurate.

To go from these statements to then assert that my thesis is inherently wrong requires believing two things:

that progress on both fMRI resolution, BOLD and alternatives ceases or at least becomes inadequate , and

that progress on data correlative techniques ceases or becomes inadequate.

I asserted a specific path to 1; every one understands that progress on 2 is inevitable.

While the best of the critical comments observe that there are challenges ahead (and I agree) nobody in their — excuse the topical pun — right minds could possibly argue that both 1 and 2 are really going to hold. And so long as progress on both axes is inevitable, the outcome I postulated is a foregone conclusion. Consider my modest submission as simply one path toward the inevitable outcome.

It is clear that some of the respondents did not understand the physics of holography, moiré patterns, Larmor precession physics, and so on. Of course, TED is not a forum for debating these specifics; however, even if my proposed approach has unforeseen challenges, there will be other approaches, and the outcome will remain unchanged.

—Mary Lou

And now the questions:

Q: Jack Gallant’s work of playing YouTube videos is very interesting; however, the decoded movies are all but impossible to make sense of on their own without the actual video playing next to it.

MLJ: These are low-resolution image sequences and are already possible to decode; further, improvements in both brain-scan resolution and data-correlative techniques are inevitable.

Q: The decoded videos are made from taking a database of movies the subject has seen and overlaying (mixing) them ranked on their correlation with the blood-oxygen-level-dependent signal (BOLD). This will never give a clear picture.

MLJ: I believe that BOLD will evolve to deliver better, crisper signals, but agree that moving beyond BOLD to other biomarkers such as temperature, acidity/alkalinity (pH), calcium-sensitive agents, neuronal magnetic fields, or the Lorentz effect can give even higher resolution.

Q: Increasing the resolution of the decoded movies so that they can be readable has little to do with scanning techniques, but more with our ability to decode the neural signal. Gallant makes an impressive first step, but there is so much more that would need to be discovered before we can decode better. The current decoding techniques simply will not get much better at higher resolution scans. We need to understand how the brain works in much more detail.

MLJ: Of course I agree that Gallant’s work is but a first step, and that there is much work to be done. I asserted in my talk that we can understand the brain better by taking higher resolution scans of what is going on in the brain! The respondent wants to understand the brain from the bottom up, neuron by neuron, while I propose a top-down solution using big data and massively higher resolution scan systems. Both are legitimate approaches.

Q: The warning call for a thought-reading device towards the end of the talk ignored the experimental paradigms used in these imaging experiments. As every brain is different, so the person decoding the signals would have to have built up a database of a priori controls of known images or movies first. It seems unlikely that this could be done without a person’s willing participation in such an activity.

MLJ: I agree, and I expressly say this in my original talk. As each brain can be different, each person may have to make a personal scan system set of their own brain for this to be optimally useful.

Q: The BOLD signal that is picked up by fMRI is a slow and diffuse blood signal, so increasing the resolution of a blurry picture may not have a linear increase in decoding power.

MLJ: It’s not that the image is “blurry,” so much as the BOLD actually measures change in blood flow. The hemodynamic response takes a few seconds. The recording doesn’t have to be blurry, but it does have to take into account the underlying temporal nature of the data.

Q: Jepsen made a claim, in passing, of being able to see neurotransmitters in newer inventions, but I am skeptical without pilot data. I would defer this to an expert in the field, which Jepsen may be.

Q: There are ~60 trillion (that’s 60,000,000,000,000) synapses in the cortex alone, and the signals are changing at a microsecond scale. Being able to read and decode all of these signals would take major leaps in technology. This is 6e+19 bytes per second, just to hold the processed data, not the images.

MLJ: This is based on a bottom-up approach. I suggest using big data and high-resolution brain scan systems and comparisons to a database of reactions to create a better system. The 6e+19 reference is interesting but not relevant; one of the points I, and others, have made about Gallant’s work is that moderate-scale correlative techniques already appear to yield insights into the brain’s workings. More precision will yield further insights.

Q: In her talk, Jepsen presented a vision for how high-resolution monitoring of brain computations in action might be achieved. The first example was from recent NIST work on arrays of nanoscale ferromagnets. Although Jepsen indicated these are very promising for neuroscience applications, I don’t see any way that these nanomagnets could be deployed for brain imaging. Jepsen did not specify any physical mechanism by which the weak magnetic fields of the brain might perceptibly alter the much stronger field patterns of nanomagnets.

MLJ: I’m sorry that this respondent missed my comments on how to do this. The very nature of the proposal is to beat the standing waves created with nanomagnets with the FM radio frequency that excites the Larmor frequency of the electrons of the atoms under study in the brain. This creates both amplitude and phase information that exceeds what is possible with FM radio frequency data alone, and which is currently used to get the electrons to “confess” their state. This is fundamental information theory — with additional dimensions of amplitude and phase data we can get a 100-fold increase (or more) in information extraction from the system.

Q: I can think of no physical mechanism (this includes spin-torque phenomena) by which this might occur. If such a thing were possible, how would the nanomagnets be put into the brain?

MLJ: I invented one and explain it above. The magnets are not put into the brain — I propose we integrate them into a new MRI system.

Q: As the field profiles of nanomagnets decay over very short length scales (tens to hundreds of nanometers, depending on the nanomagnet cross-sectional diameter), they would have to be in intimate contact with the neurons they’d purportedly “sense.”

MLJ: This is the biggest leap — that we can increase the field profiles of the nanomagnets as they hit the stage. They don’t need to be the “carrier” frequency like the big magnet of the coil; they can be orders of magnitude smaller and still be detectable in the model I propose.

Q: Jepsen seems to suggest using an image from an electron holograph of some magnetic system or material.

MLJ: I did not and do not assert use of electron holograms; I’m saying standing magnetic field patterns (effectively with FM radio frequency standing waves) making the Larmor precession frequencies confess themselves at massively higher resolutions is or will be possible. This will not be restricted to using BOLD but will also be possible via other techniques such as arterial spin labeling (ASL), and with other materials, not just oxygen. Some of these techniques have been impractical to date, partly because of the low resolutions attainable through existing scan systems and partly because of the state-of-the-art in data processing. Also, rather than oxygenated blood flow, i.e. BOLD, there are other candidates that can be measured but with lesser magnetic moment, quantum spin numbers and physiologic concentration. Again, from an information theory perspective, more sophisticated data processing and correlation techniques will enable us to unearth and interpret more subtle signals.

Q: The relative phase of the beams can be altered because of the presence of a magnetic field, and this allows really impressive magnetic imaging of small objects.

MLJ: Yes, and we can beat this effectively with the FM radio frequency excitement of the electrons to extract far more information holographically.