Posted
by
Soulskill
on Wednesday July 08, 2009 @10:31AM
from the it's-going-south dept.

conspirator23 writes "Jon Hamilton of National Public Radio brings us a story about 'voodoo correlations' in fMRI studies that seek to learn more about emotional states, personality, and social cognition in the human brain. Many of us outside the scientific community have been treated to fascinating images of brain activity and corresponding explanations about how the images reveal which portions of the brain are engaged in certain kinds of thinking. But these images are not actual snapshots; they are visualizations of data generated by repeated scans during experiments. Flaws in the statistical methods used by researchers can result in false images with a variety of inaccuracies. Yet the images produced are so vivid and engaging that even other neuroscientists can be misled by them."

These sort of images are pretty familiar to me and I must admit I was never skeptical of research showing that you could classify brain patterns based on the object they were looking at or how they were feeling. I had thought this had gone so far as to be used to classify terrorists and used in trials (which is quite unnerving)! Well, it saddens me to say this but in a field where we normally take two steps forward today, we are taking one giant step back. The brain is such a complex thing to study conce

These sort of images are pretty familiar to me and I must admit I was never skeptical of research [...] it's a shame that one of the few tools used to determine the hows and whys of it is being called into question.

I don't think this 'uncertainty' is anything new. Computing a tomographic reconstruction is an ill-posed problem, you can do least squares, you can be a Bayesian, but in the end you have to introduce information or assumptions to fill out the "null space" of your measurements.

I think there's been a lack of understanding on the part of many folks in the medical community about just what kinds of assumptions go into making those pretty CT and MRI results. Treating spurious features in reconstructions due t

I work in the field, and the field is *all about* trying to figure out what is signal and what is noise. We have this massive multi-site study wrapping up involving scanning the same subjects at different sites all over the country, on multiple days for each site, in the same conditions in each scan and each site, and seeing what sort of artifacts show up. And they're significant -- even the scans on the same hardware at the same site on different days, but even moreso for different hardware. But that wa

A fundamental difference between MRI and CT is that computing the MRI image involves (at least in some cases) a 2-D FFT. While signal to noise ratio is an issue, the processing is not an ill-posed problem.

A fundamental difference between MRI and CT is that computing the MRI image involves (at least in some cases) a 2-D FFT. While signal to noise ratio is an issue, the processing is not an ill-posed problem.

Hmm. My group definitely calls the process of moving from k-space to image space "reconstruction". There are good reasons for doing more than a simple 2D FFT (most of which, I'll admit, are over my head).

The assumptions underlying the choice of forward model and measurement process that allow you to use the FFT approach turn an ill-posed problem into a well-posed one, true.

Making different assumptions leads to different reconstruction techniques. *Any* scan like this (CT, MRI, ultrasound, infrared, microwave) involves dealing with an ill-posed problem in some way. It may be dealt with before any of the researchers in question touch the machine or the subjects, but it was dealt with by someone at some poin

I've always wondered how useful these images really are. Perhaps to the trained eye they can reveal a lot about how a persons brain works but they have always struck me as being too abstract. We can point at a portion of the image and say that bit controls movement, for example, but if anything goes wrong we are stuck because at a fundamental level we don't understand how it controls movement. I suppose it's a bit like looking at a block diagram for a CPU and not understanding how each bit works.

It will be interesting to see how we achieve the next level of understanding of the brains functioning. I can't see that we will ever get there with MRI or electrode probes because, I think, they are simply too large to get a true understanding of what is going on. I suspect we will gain our understanding through modelling but I'm not sure I'll be around when we do.

To my understanding, it's not measuring 'differential oxygen uptake' like you explain. It's measuring an impulse response through the axon/neuron/dendrite, which is an electro-chemical signal derived from potassium and some other compound. Correct me if I'm wrong, but I don't think it's oxygen that gives neurons their charge.

The GP is correct... functional MRI measures blood oxygenation level dependent (BOLD) response - that is, the change in paramagnetism induced by oxygenated hemoglobin. Active neurons require more oxygen than inactive neurons, so oxygenated blood is delivered to them more rapidly (the hemodynamic response). This induces a local shift in magnetic permeability (from paramagnetic to diamagnetic) which can be picked up by the scanner.

Whether the BOLD signal truly correlates well with neural activity is still a matter of contention within the medical community.

Whether the BOLD signal truly correlates well with neural activity is still a matter of contention within the medical community

True, and we should mention that the time resolution on fMRI is on the order of a second or two. This suggests also some significant time walk or smearing in the signal. The point is that minimum brain response time is quite a bit faster than this (a few tens of ms), and this is smaller than the resolution of fMRI.

That being said, if you don't have brain pictures in your grant proposal, your chances of getting a cognitive science grant are greatly diminished. So everyone tries to find some way to use it, wh

Whoa. fMRI is just measuring "current flow" across coarsely-defined regions of the "CPU"? And not instantaneous measurements, but time-integrated ones? And with only a fairly crude map of CPU area->function, trying to deduce exactly what instructions and data the CPU is processing?

Don't get too wound up about that analogy - it's fairly weak. My point being that the brain is really the ultimate 'black box' and our attempts to divine what is going on up there (or down there depending on whether or not we're talking about politicians, but I digress) are fairly crude at present. Functional MRI is interesting - there is undoubtedly a correlation between oxygen uptake and brain function, but I don't think we really know how strong that correlation is or what it really signifies.

I suspect we will gain our understanding through modelling but I'm not sure I'll be around when we do.

I agree. I've always thought that one of Edelman's conscious artifacts, http://www.21stcentury.co.uk/robotics/nomad.asp [21stcentury.co.uk], would be the way in to a better understanding of the brain, but I haven't kept up with their progress. I'm still hoping they'll find some answers while I'm around.

It's easy to dismiss the results as noise, but any decent researcher will estimate her or his error bars, and show that the signal measured is, indeed, small or comparable to the error bars. The way the article is written, it just sounds like they just totally ignored the results because they don't like them or something. To be fair, the paper itself probably does a better job of defending its position, but I don't have time to understand all its details.

I think (based on the admittedly slim article) their criticism is based on many researchers not properly accounting for pseudoreplication [wikipedia.org], which tends to be a problem when you have tons and tons of data like in an fMRI scan, but it is only on a few individuals, or it is on one individual measured a couple different times. The way to properly treat this is with split-plot or mixed effects models, this is a bit harder to get right than just fitting a linear model and looking at error bars.

It's not the noise that's the problem. It's the control. Ideally you'd have a blank slate as a control, then do an activity, and measure the difference. The problem is, the brain is always doing something, so it's really hard to make a controlled experiment. It doesn't matter how tight your error bars are if you're not comparing your data to a valid, standard, control.

I will say, it would be easy to make wild claims about what areas of the brain "do" things just by looking at a scan and showing a pretty picture.

That said, consider these things:
While non-peer-reviewed publications often publish exciting results, the scientific community typically does not accept brain regions without corroboration from many different studies with different stimuli, often including monkey studies where real electrodes and not just low-res fMRI can be used
It is difficult to get the numbers of subject that would be considered standard in other studies for fMRI studies. First off you actually need subjects who will do the assigned task, then you need them to do it perfectly still, for anywhere between 20 minutes to several hours (usually in no more than 1-hour segments). So the likelihood that just one study could prove something is quite small.
In many (perhaps most) studies, all the subjects brains are averaged together for data analysis, there are several different ways of doing this, none of them particularly accurate. This again calls attention to the need for multiple studies

It's also important to actually know what you're looking at when you see pictures of "brain activity", usually you are looking at the averaged activity of many subjects, after it has been run through (most likely) some form of general linear model or event-related analysis. Both of these methods estimate and fit a hemodynamic response function (the pattern of brain response to a stimulus), and what you're actually looking at is the fit or perhaps t-values (roughly fit/std. deviation) for each voxel.

Also note, that for almost any study, I could pick some random brain areas that are "lighting up" and claim a response, but they would almost certainly be shot down with more subjects, another study, etc.

bottom line, responsible investigators can make good sense out of fMRI data, but doing one experiment and claiming you "found the love [or insert whatever emotion/though] center is irresponsible and should be correlated with other studies and hopefully monkey studies as well.

I think most people in the neuroscience community are aware of the limits of current fMRI approaches. The general linear model, which is used to compute blood flow, is rightly under considerable attack from a number of directions (it assumes, among other things, that all measured hemodynamic response is the result of changes in underlying neural activity, and there is now quite a bit of evidence that this is not the case). And the basic paradigm for most fMRI experiments, especially ones examining 'higher c

fMRI is one of many imaging techniques which continue to evolve and give us more and more amazing data about the brain. But its just data, and subject to the limitations of its tech. Any neuroscientist worth their salt knows the limitations of their technology. The problem desribed in the article is something people have been dicussing as a valid methodological criticism of some studies, not all fMRI data. The summary is misleading, basically its saying a tool used incorrectly results in bad data. Duh.

I've been lucky enough to work with MR and CT imaging researchers for a while now. One of the benefits of this job is that I've gotten to learn a lot about how these images are acquired and reconstructed. It's not quite as bad as making sausage, but it's a lot more involved than a "snapshot".

For CT, we acquire a bunch of 2D images through you from different angles, then do a lot of number crunching to generate a 3D volume. The problem is that you don't hold still while we're doing it. You can try; you can even hold your breath, but you can't "hold your heart". As your organs move between views, we get motion artifacts -- shape distortion, bright or dark areas, even "things" that aren't really there.

For MR, it's even worse. I can barely tread water in the physics of it, but we're effectively capturing a line at a time in 3D space. (We're actually acquiring data in "k-space", then running it through a Fourier transform to make it spatial.) Not only is it subject to motion artifacts, it's also subject to susceptibility artifacts (distortions because of the magnetic properties of certain materials), flow artifacts (blood moves through vessels between the time that we apply a magnetic pulse and the time that we read back emitted signals), and lots of other things.

fMRI is just adding yet another layer of aggregation and interpretation on top of all this. Sure, it's a "visualization of data generated by repeated scans", but so is every CT or MRI image.

3D imaging, especially MRI, is hideously complicated and indirect. It's almost inconceivable that it could yield results with any physical significance.

...and yet, it does. It's become so routine, so reliable, so well-understood and well-controlled, that doctors and researchers know they can rely on it as a matter of course. They still have to be aware of the errors and distortions that can arise, but that's true of every imaging or monitoring system, all the way down to the stethoscope and the fever thermometer.

3D imaging, especially MRI, is hideously complicated and indirect. It's almost inconceivable that it could yield results with any physical significance.

...and yet, it does. It's become so routine, so reliable, so well-understood and well-controlled, that doctors and researchers know they can rely on it as a matter of course. They still have to be aware of the errors and distortions that can arise, but that's true of every imaging or monitoring system, all the way down to the stethoscope and the fever thermometer.

Modelling the measurement is fundamental to physical science, even something as simple as measuring the temperature of air, and these are much more complex measurements to model.

...and yet, it does. It's become so routine, so reliable, so well-understood and well-controlled, that doctors and researchers know they can rely on it as a matter of course. They still have to be aware of the errors and distortions that can arise, but that's true of every imaging or monitoring system, all the way down to the stethoscope and the fever thermometer.

The problem with the activation maps is precisely that one is NOT looking at an image, so there's no way to fine tune the algorithms. Therefore, fMRI is NOT well understood in the way that CT or MRI are.

Consider that in imaging, you have the luxury of comparing the output of a brain scan to the known physical structure of the brain. Is there a hippocampus? No? Well then it didn't work, go back and fiddle until you can show me a hippocampus.

That's exactly what you have to do, and assume (usually safely) that every heart cycle is pretty much identical. We call it "cardiac gating". You can either do it "prospectively", which means triggering acquisition at a particular point in the cardiac cycle, or "retrospectively", where you acquire freely and then go back and pick out the acquisitions that happened at the cardiac phase you want.

For CT, we acquire a bunch of 2D images through you from different angles,

Wasn't that "a whole bunch of 1D images, which are then computed into 2D-slices, which are then assembled into 3D volume representations"?

Last time I checked, CTs acquired one-dimensional images. However, it's been a while, and I work in a different field of biomedical engineering, so I haven't really kept up with more recent developments.

Good catch. "We" (well, our actual researchers, not me) built a small-animal CT system that captures 2D images, rotating the animal instead of the gantry between views. That's very different from a clinical CT system.

However, those clinical scanners are getting wider. You can now get scanners that acquire 64 slices at a time -- instead of an N-pixel linear sensor, they use an N x 64-pixel sensor, and scan in a helical pattern. It's all about speed in the clinical arena.

Good catch. "We" (well, our actual researchers, not me) built a small-animal CT system that captures 2D images, rotating the animal instead of the gantry between views. That's very different from a clinical CT system.

Thinking about it - using a 2D-sensor is a logical evolution to get faster and better pictures. I mean, the very first generation didn't even use a linear sensor array, they used a single sensor that was moved in order to get the measurements at different points in space.

If little magnets were going to affect your brain, wouldn't anyone who'd had a brain scan end up a vegetable?

Of course it pays to sell idiots little magnets and claim all sorts of health benefits. Some may even benefit from a placebo effect. (It doesn't pay to try to sell them MRI machines...there are so few idiots THAT rich).

If little magnets were going to affect your brain, wouldn't anyone who'd had a brain scan end up a vegetable?

Of course it pays to sell idiots little magnets and claim all sorts of health benefits. Some may even benefit from a placebo effect. (It doesn't pay to try to sell them MRI machines...there are so few idiots THAT rich).

I think i'll remain skeptical unless more solid evidence turns up.

You're confusing sterngth with field density. An MRI is indeed huge assed, 0.5 to several tesla. But within any given cubic millimeter there's precious little power. In transcranial magnetic stimulation, there's modest power, but it's focused in a very small area.

These facts didn;t keep California from upholding a lawsuit by an "ex-psychic" against a hospital. She clails she lost her psychic ability to a MRI.

Well, sort of. The tesla is a measure of field intensity, and if you've got a 1.5T magnet, every cubic millimeter inside the bore experiences that same field intensity, and the same "energy density". (It's not like a small-bore 1.5T system has a "more concentrated" field than a wide-bore 1.5T system.) But it's a static field, and (to a first but very accurate approximation) static fields don't do much to living systems.

(As long as you hold still, that is. If you're in a very high-field magnet, you need

Disclaimer: I'm a neuroscientist/psychophysicist that studies vision. Traditionally, those that study vision or motor control have the lowest tolerance for squishiness. This is because what attracts us to the field is the fact that we can correlate human behavior with objective measurements such as joint angles, eye movements, and luminance.

fMRI, like all scientific tools comes with its caveats. First, as has been mentioned, it isn't measuring current at all, but rather oxygenation of blood. I disag

Most of my psychology colleagues have no idea what they're looking at in fMRI. They assume if it lights up, it's making something go. They may know full well that neural activation can be excitatory or inhibitory, but fail to make the connection and figure out that what's lighting up may be de-activation. Both the gas pedal and the brakes appear the same to fMRI and nobody can tell which is which with this technology alone.

Even fewer even bother to try to grasp the math behind the analysis technique, statis