Fixing Science - Systems and Politics

There is increasing concern that the structure of modern science is flawed and that most published research findings may be false.

Commonly cited problems with how science works today include:

• Publication bias and the file drawer problem.

• "Result fishing", data dredging etc. - analyzing data in different ways to "get a finding"

• The privileging of "positive" results over "negative" ones.

I have previouslyargued that, to solve these, problems we need a way to ensure that scientists publicly announce which studies they are going to run, what methods they will use, and how they will analyze the data, before running their studies.

We already have such a registration system in place for clinical trials. It's a good system. It's not perfect but it's helped. I propose we extend it to all science. But how would that work in practice?

I'm not sure. So what follows is a series of ideas. These are intended to spark debate.

Here are some options for

systems:

1. There could be a central registry, free and open to the public, where protocols are pre-registered. Call this the

   'clinicaltrials.gov option'

   because we already have one for clinical trials. This registry could also serve as a repository of results and raw data, but it wouldn't have to.

2. Academic journals could require studies to be pre-registered in order to be considered for publication: you submit the Introduction and Methods, these are peer reviewed, and if accepted, the journal is bound to publish the results when they arrive; the authors for their part are bound to follow their protocol (secondary analyses could take place, but they would be explicitly flagged as such.) and submit the results.

3. Scientific funding bodies could make all successful scientific grant applications public via an open database. These applications already contain pre-specified methods, hypotheses, and statistical analyses, in most cases; part of this plan could be to make these more detailed.

4. Authors could have the individual responsibility to publicly announce their methods, hypotheses and plans before starting studies on their own websites.

How can we actually make this happen? That's a question of

politics:

1. Governments could introduce legislation to force this. This is the most extreme option. It is probably unviable, because it would place researchers in different jurisdictions under different rules. Science is a global enterprise, and we don't have a global legislature. (The USA did this for clinical trials, but for various reasons these are a special case and more 'international' than others.)

2. A consortium of major scientific journal editors could announce that they'll only publish research that complies with the system. Notably, this was how clinical trial registration started.

3. A consortium of major funding bodies could refuse to finance research that doesn't adhere to the system.

4. Individual scientists, journals, and funding bodies could unilaterally adopt the system. This would, at least at first, place these adopters at an objective disadvantage. However, by voluntarily accepting such a disadvantage, it might be hoped that such actors would gain acclaim as more trustworthy than non-adopters.

My own preference would be for System 1 via a combination of Politics 2 and 3. Yet any combination of these options would be better than the current system.

Some possible objections:

1. Pre-registration of all science would be impractical. What about pilot studies and 'tinkering'?

   - I'm only proposing that any research which might be published, should be publicly registered. This leaves anyone free to tinker away all they like - in private. We just need to be clear, from the outset, whether we're tinkering or doing 'proper' publishable research, a line which is currently very murky.

2. Many interesting results are unexpected. Post-hoc analysis or interpretation of data is important. - There's nothing wrong with post-hoc analysis or interpretation,so long as everyone knows it was post-hoc. The problem is when it is passed off as being a priori. Registration doesn't seem to have discouraged legitimate post-hoc analyses in the case of clinical trials: there are lots of excellent post-hoc analyses coming out, clearly labelled as such.

3. It would be unfair to scientists to make them 'tip off' their rivals about what they're working on in advance. It would penalize originality. - My gut instinct here is that this is not a big problem; everyone would be in the same boat so it would be a fair system. However, if this were felt to be a concern, there's an easy solution - just build in a delay to the publication of registered protocols. Put them in a 'sealed envelope' to be opened after a 12- or 24- month 'grace period', and that would give people a head start while ensuring that their original protocol was eventually revealed.

4. This wouldn't solve all of the other problems with science. - No, it wouldn't, and it's not intended to. However, I do feel that we'll struggle to make progress in other areas without something like this happening. The current system of post-results publication is not the only problem, but it is a large part of it.

On that note, here's a sketch of how I see this relates (or not) to some other issues in science today:

Replication - there's been much discussion of late around ensuring the replicability of results in certain fields e.g. neuroimagingstudies and psychologytoo. My view is that most published false (i.e. unreplicable) findings are a product of publication bias and positive result fishing. Solving those problems, as outlined here, would increase the replicability of science. It wouldn't be a panacea. There will always be dodgy results due to fraud, incompetence, and bad luck, but the current system too often rewards scientists for fiddling around until they get a positive one.

Careers - There is a widespread complaint that the current system of science is unsatisfactory. Our jobs, promotions, funding and tenure depend on our ability to generate high impact papers - which means, in effect, novel and interesting positive results. Pre-registration of science would change the game. Scientists would be judged on their ability to design and run interesting experiments, rather than on their ability to generate 'good papers'.

Open Access - The issue of free open access to scientific papers is an important one. It's a separate question to the one I've discussed here. But I see a spiritual overlap. In both cases, the fundamental question is who owns science? At the moment, scientists own their work until and unless they decide to publish parts of it. When they do, they sell it to a publisher, who sells it to the world. In my view, the world should be told about science, from the beginning.

*

Fundamentally, this will only happen if a critical mass of scientists want it to happen. It will not be easy, but whereas four years ago I was, deep down, skeptical that it would ever be possible, today I really think it might.

Already we're seeing signs of hope, from informal pre-registration to calls forpreregistration in particular fields in major journals. 10 years ago, the idea was being written off as impractical, and with the technology available at the time, it probably was. I do not think that is true today.

Change can happen. All it needs is will.