A new blog called neurorant has launched a broadside at preregistration, my current hobbyhorse. (This will be the last post on the topic for a while, honest.)
Neurorant's criticism is anonymous, unscholarly, snarky, and not published in a peer reviewed journal... but it also has some bad points:
Pre-registration of studies is meant to stop unconstrained post-hoc analyses that end up finding differences that are actually just noise in the data.
That's one of the motivations; but equally important, it will facilitate the publication of negative findings. Indeed, there seems to be a common misconception that registration is essentially about prevention, a kind of scientific contraceptive intended to eliminate bad results from the gene pool.
Pre-registration will act to discourage exploratory science... In fMRI, for example, the really interesting developments almost all use existing data... i.e. working on defining and understanding functional brain networks using data that has not been acquired for the purpose (e.g., generic resting state data). There’s no way to pre-register for this kind of research.
Let's consider the example of the 1000 Connectomes Project, (1000C) an existing dataset of the kind in question. Anyone can download this, and analyze it. Should those analyses be preregistered? My answer is: they already are. Or rather, they're not, and everyone knows it, so everything's above board, which is exactly how exploratory analyses would be under a registration regime. We all know that any 1000C finding might be a picked cherry from a rich fruit basket. The trouble with the current system is that planned and exploratory analyses are confused. All registration does, at its core, is make the distinction clear. Incidentally, I'm strongly in favor of fMRI data sharing.
It will slow science down for two reasons: i) there will end up being several rounds of review of the pre-registration phase so these will have to be written carefully, slowly. (If there isn’t any review of pre-registration then the whole thing can’t work).
This is another common misconception. Registration doesn't need to be linked to any kind of review. In clinical trial registration, the model for the whole thing, there isn't any. You just submit. Likewise, anyone could just register their non-clinical work today, unilaterally. Jona Sassenhagen has, along with various others e.g. the Bem replication experiments. No-one formally reviewed those notices of registration. The judge is the reader of the eventual results, who can compare them to the registered protocol and decide if they think the final cherries were subject to picking, or whether (most likely negative) results are going unreported. That said, I believe that uniting preregistration and peer review (as implemented e.g. like this) would be a great idea, and that it could actually save time overall. This, however, is registration
plus
reform of peer-reviewed publishing. Plain vanilla registration is a lot simpler.
ii) In fast moving fields, there will be analysis and theoretically developments throughout the planning/data acquisition phase that will mean that if you’re taking pre-registration seriously, you have to re-register and collect more data, not just repurpose what you have. This may also involve people collecting even more boring FMRI datasets that all show the same thing (this already happens by the way), which swallows precious research resources.
I think the concern here is that registration will mean that if people have some data (or are halfway through collecting it) and want to try a new, unregistered analysis on it, they'd have to register and run a wasteful new study, rather than just repurposing the existing data. If that's the worry, then it's largely groundless. The authors in this case could do it, and publish it, as an exploratory analysis... which after all is what it is. It's true that
if
registration were combined with peer review, our hypothetical researchers might have to convince the reviewers (and the journal) to let them bundle the new analysis into the same paper as the registered stuff. But if it doesn't get allowed, they could just publish it elsewhere. Whether or not to allow a large "Secondary Results" section of a paper is a matter for the journal; it's a publishing decision. Neurorant's third point is that like an AAA credit rating, preregistration could be exploited by fraudsters. Well, I'm sure that will happen - unfortunately, fraudsters exploit systems. It's what they do. They're doing it currently. I'm not aware of an increase in fraud in US clinical trials after they were all made preregistered by law, although that would be interesting to investigate. Finally (and I detect a mischievous grin peering from between these lines)
I don’t think dodgy results in some papers have that big an effect, or not enough to lead to any drastic, restrictive countermeasures. In fact, it’s possible that having dodgy papers is good for science as a whole, maybe it’s like stochastic resonance in motor systems, a bit of noise may help the field.
Well, fair enough. Maybe it depends on the field of science you inhabit; some are worse-hit than others. But ultimately, I think this is a question of how you feel about science today:
Personally, I'm not smiling.
