Science's pitfalls: How a small bug in the computer programme can undermine years of research

Functional MRI (fMRI) is 25 years old, yet surprisingly its most common statistical methods have not been validated using real data. In a recent test, scientists used resting-state fMRI data from 499 healthy controls to conduct 3 million task group analyses. Using this null data with different experimental designs, they estimated the incidence of significant results. In theory, we should find 5% false positives (for a significance threshold of 5%), but instead the researchers found that the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%. These results question the validity of some 40,000 fMRI studies and may have a large impact on the interpretation of neuroimaging results.

There could be a very serious problem with the past 15 years of research into human brain activity, a bug in fMRI software could invalidate the results of some thousands of papers.
Functional magnetic resonance imaging (fMRI) is one of the best tools we have to measure brain activity, and if it’s flawed, it means all those conclusions about what our brains look like during things like exercise, gaming, love, and drug addiction are wrong.
The main problem here is in how scientists use fMRI scans to find sparks of activity in certain regions of the brain. During an experiment, a participant will be asked to perform a certain task, while a massive magnetic field pulsates through their body, picking up tiny changes in the blood flow of the brain.
These tiny changes can signal to scientists that certain regions of the brain have suddenly kicked into gear, such as the insular cortex region during gaming, which has been linked to 'higher' cognitive functions such as language processing, empathy, and compassion.
Getting high on mushrooms while connected to an fMRI machine has shown evidence of cross-brain activity - new and heightened connections across sections that wouldn’t normally communicate with each other.
It’s fascinating stuff, but the fact is that when scientists are interpreting data from an fMRI machine, they’re not looking at the actual brain. What they're looking at is an image of the brain divided into tiny 'voxels', then interpreted by a computer program.
So, software, rather than humans ... scans the voxels looking for clusters. When you see a claim that ‘Scientists know when you're about to move an arm: these images prove it,' they're interpreting what they're told by the statistical software.

Scientists have tested how good this software actually is. They tested the three most popular fMRI software packages for fMRI analysis - SPM, FSL, and AFNI - and while they shouldn't have found much difference across the groups, the software resulted in false-positive rates of up to 70 percent.
Not only did the team expect to see an average false positive rate of just 5 percent, it also suggests that some results were so inaccurate, they could be indicating brain activity where there was none.
These results question the validity of some 40,000 fMRI studies and may have a large impact on the interpretation of neuroimaging results.
The bugs the team identified has been in the system for the past 15 years, which explains why so many papers could now be affected.
The bug was corrected in May 2015, at the time the researchers started writing up their paper, but the fact that it remained undetected for over a decade shows just how easy it was for something like this to happen, Because researchers just haven't had reliable methods for validating fMRI results.
Since fMRI machines became available in the early '90s, neuroscientists and psychologists have been faced with a whole lot of challenges when it comes to validating their results.
One of the biggest obstacles has been the astronomical cost of using these machines - around US$600 per hour - which means studies have been limited to very small sample sizes of up to 30 or so participants, and very few organisations have the funds to run repeat experiments to see if they can replicate the results.
The other issue is that because software is the thing that's actually interpreting the data from the fMRI scans, your results are only as good as your computer, and programs used to validate the results have been prohibitively slow.
It could have taken a single computer maybe 10 or 15 years to run this analysis. But today, it’s possible to use a graphics card to lower the processing time "from 10 years to 20 days".
Now what happens to those 40,000 papers that had used the faulty software? The studies have to be done again!
Last year when researchers tried to replicate the results of 100 psychology studies, more than half of them failed, we're seeing more and more evidence that science is going through a bit of a 'replication crisis' right now, and it's time we addressed it.
Unfortunately, running someone else's experiment for the second, third, or fourth time isn't nearly as exciting as running your own experiment for the first time, but studies like this are showing us why we can no longer avoid it.

Field of science, wake up. Do science just in the way it should be done. Otherwise people will lose their trust in you!

Source: Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates
http://www.pnas.org/content/early/2016/06/27/1602413113