Last month, I blogged about Jona Sassenhagen, the University of Marburg neuroscientist who decided to have a go at Unilaterally Raising the Scientific Standard. To recap, Sassenhagen chose to pre-register a neuroscience study on a public database, the DRKS. The methods he planned to use, the major data analysis steps, and the predicted results are all here; they've been up there for over two months, i.e. before the data was collected. This is unusual. Pre-registration is something I havelong argued should be mandatory, as I think it would guard against the bad practices that distort science, but currently it's obligatory only in the case of clinical trials, not 'basic' research. The project, Alignment of Late Positive ERP Components to Linguistic Deviations (“P600″), was designed to test whether the brain generates a distinct electrical response – the P600 - specifically in response to grammatical errors. Now, Jona reports that he’s completed the study, written it up, and submitted it for peer review (already!). He kindly agreed to answer some questions about the whole thing...
Overall, would you recommend pre-registration? Do you regret not doing it previously?
I'd recommend it for everything, even for purely exploratory analyses. I regret not having done it before. As much as possible, I'll register everything I'll be doing in the future.
I have two papers right now that are confirmatory, but I can't submit them because the paper we made the predictions in is getting rejected. Had I pre-registered, that would not have been so problematic.
However, I also think it's not enough for confirmatory studies to pre-register outcomes and the basic methods. For truly confirmatory studies, we need full pre-registration of the whole analysis pipeline, like Chris Chambers' proposal at Cortex.
Did you find that pre-registering meant the study was more effort or less effort than the normal approach?
The pre-registration itself was some effort to begin with. But besides that, it did not change the amount of work actually done on the study itself - it just changed the order. It meant more work up front, but less work afterwards.
It was substantially more work before; I made sure my scripts and my manuscript were all written up when the first participant rolled in. But we submitted a week after the last participant had been recorded, 3 weeks after recording had started.
The study took less time to run than usual because since I knew I couldn't cheat later, I decided on a really strong effect, and therefore, my power analysis told me I wouldn't need too many subjects. Also, I was quite enthusiastic about the whole adventure… I was doing a lot of night shifts!
Did you consult with your colleagues before pre-registering? If so, what did they think about it?
I started doing it by myself, but when I ran into problems during the pre-registration process (some of the terminology didn't make much sense to me), I would ask my co-workers, and they liked the idea, though they were not too enthusiastic. It's still some ways to go before I can convince my lab to follow me down that road.
The general perspective was that it wasn't a bad thing to do, but nobody really got the point at first. Later, my office mate got the point when I sent him your blog post.
My PI got the point afterwards, when we were finishing up the paper. At one point she asked, “This sounds a bit explorative to me, what if a reviewer doubts if this is what we truly went out to test?” And I reminded her that if a reviewer did decide to question us there, we could point to the pre-registration, where I had said that I'd test it. That's when she warmed up to the idea.
But whatever her personal feelings, she gave me her full support here, running the study on short notice and keeping up with the rapid pace of finishing up the manuscript after data collection had finished!
Did you feel that the published protocol was 'tying your hands' in data collection or analysis?
Well, the pre-registration procedure I went through is still quite vague. It's the standard clinical trial registration procedure, and it's definitely a lot better than nothing. But it's not too strict. I had to note down my main outcome and a very basic sketch of the methods. There was still considerable wiggle room...
With full pre-registration of methods, this would not have been possible this easily. Which means, I think, you will need to do more work up front, including running a few pilot test subjects and testing your full scripts. If you do a full pre-registration of methods, you will not be able to get by without running a few preliminary test subjects. I’m confident that it would be impossible to run anything but the most trivial confirmatory study without having some sample data available to prep up your scripts.
Also, we would most likely see a shift towards mostly investigating reliable, strong effects. Telling the world you expect to find X, and then having to report Y... that idea was quite stressful to me. On the other hand, you can't argue that the pre-registration forced me to go with the mainstream. On the contrary. I'm decisively going against the consensus here (which associates the P600 with high-level combinatorial processing). This is in part why I decided to pre-register; so everybody would be able to tell I didn't cheat to prove my (controversial) point.
Finally, Jona also asked me to forward a question to Neuroskeptic readers. He's willing to make the raw data and his analysis scripts publicly available. Is anyone able to recommend a platform that hosts such things?
