Neuroskeptic is a neuroscientist who takes a skeptical look at his own field and beyond at the Neuroskeptic blog.
Fraud is one of the most serious concerns in science today. Every case of fraud undermines confidence amongst researchers and the public, threatens the careers of collaborators and students of the fraudster (who are usually entirely innocent), and can represent millions of dollars in wasted funds. And although it remains rare, there is concern that the problem may be getting worse. But now some scientists are fighting back against fraud---using the methods of science itself. The basic idea is very simple. Real data collected by scientists in experiments and observations is noisy; there’s always random variation and measurement error, whether what’s being measured is the response of a cell to a particular gene, or the death rate in cancer patients on a new drug. When fraudsters decide to make up data, or to modify real data in a fraudulent way, they often create data which is just “too good”---with less variation than would be seen in reality. Using statistical methods, a number of researchers have successfully caught data fabrication by detecting data which is less random than real results. Most recently, Uri Simonsohn applied this approach to his own field, social psychology. He has two “hits” to his name, and more may be on the way. Simonsohn used a number of statistical methods but in essence they were all based on spotting too-good-to-be-true data. In the case of the Belgian marketing psychologist Dirk Smeesters, Simonsohn noticed that the results of one experiment conducted by Smeesters were suspiciously "good": They matched with his predictions almost perfectly. Using a technique called Monte Carlo simulation---widely used in economics, neuroscience and many other fields---he showed that the chance of this really happening was extremely low. Even if Smeesters' theory were correct, the data should have contained some noise, meaning that the results would only be approximately, not exactly, as predicted. What’s more, Smeester’s data were much neater than similar results published by other researchers using the same methods. After Simonsohn confronted him with this evidence, Smeesters at first claimed that he'd made an honest mistake, but he was eventually found guilty of fraud by a university committee, and several of his papers have since been retracted---stricken from the scientific record. Using similar methods, Simonsohn uncovered evidence of fraud in a second researcher, Lawrence Sanna, who worked in the same field of social psychology, but whose fraud was entirely unrelated to that of Smeesters. The full details of Simonsohn's investigations are available here. These cases have attracted a great deal of attention. They made Simonsohn famous as the "data detective" or the "scientific sleuth," and they also created the perception of a crisis in social psychology. Researchers are now debating just how big the problem of fraud is and how best to fight it. Simonsohn wasn't the first to use statistics to spot too-good-to-be-true data. Several months previously, in the field of anaesthesiology, the massive fraud of Dr Yoshitaka Fujii of Toho University, Tokyo, in Japan, was uncovered by similar methods. Over the course of his career, Fujii had published hundreds of papers, many of them about the drug granisetron, used to prevent nausea in patients after surgery. British researcher John Carlisle took 168 of Fujii’s clinical trials and observed that several key variables, such as numbers of side effects, were exactly the same in many of these trials. Assuming the data were real, you'd expect variation in these numbers, just by chance. Carlisle found extremely strong evidence that Fujii's results were too consistent to be real. And Carlisle’s was not the first statistical report alleging that Fujii’s data were too good to be true. Amazingly, exactly the same charges had been made twelve years previously, back in 2000, when a group of researchers wrote in a public Letter to the Editor of a scientific journal that the data from 47 of Fujii’s granisetron trials were “incredibly nice.” These researchers noted that the number of patients reporting the side-effect of headache was exactly the same in over a dozen of his papers. They calculated the chances of this happening, assuming the data were real, as less than 1 in 100 million. We now know that, indeed, they weren’t real: Fujii had made them up. But for various reasons, the 2000 allegations didn't lead to any serious investigation into Fujii's work, leaving him free to fake dozens more trials. As useful as these methods are, it’s important to remember that they can only suggest fraud, not prove it. For example, too-good-to-be-true data might be the result of an honest error, rather than intentional manipulation. In the cases described here, further inquiries revealed other hard evidence of foul play and, eventually, admissions in most cases. However, if the too-good-to-be-true approach becomes used more widely, it’s likely that eventually, someone will deny the allegations, and stand by their results. What will happen then remains to be seen, and it might lead to an ugly controversy. On the other hand, though, the case of Fujii shows that these methods have a vital role to play in keeping science clean. If the first warnings about Fujii’s “nice” data had been taken more seriously back in 2000, huge amounts of time, money and effort would not have been wasted. Fraud can happen in any field of science, in any institution, in any country. We need all the tools we can get in fighting back against it.
