Psychology has lately been hit by high-profile fraud scandals, and broader concerns over questionable research practices. Now the Society for Personality and Social Psychology (SPSP) has released a statement on "Responsible Conduct", and a task force has produced a report.
This is a start, and the SPSP is to be commended for facing up these problems (which affect many other fields) relatively early. However, neither of their documents contains much meat in my view.
Point One on the task force report is that "Replication is the key to building our science" and they suggest a "web site for depositing replications and failures to replicate" - but don't mention that various enterprising researchers have already made one. Nor do they tip their hats to the Open Science Initiative addressing just this issue. This makes me worried that they're planning to reinvent the wheel.
More fundamentally I disagree that replication is key to psychology or any field. Our goal should be replicability. Failure to replicate findings is a symptom of problems with those original findings, rather than being a problem in and of itself. Good results replicate; we want better results to be published.
In other words, we should strike at the root cause of invalid research, namely, the perverse incentives towards publishing as many eye-catching positive results with p values below 0.05 as possible by any means necessary. P-value fishing, selective reporting, post-hoc "prior hypotheses" and other questionable practices are a large part of what make unreplicable results.
We should encourage replication, but it's no panacea.
An overemphasis on replication, without addressing the incentives, could actually harm science. It could lead to scientists spending all their time worrying about the political drama of who's replicating who and why, and which questionable practices they can use to replicate their friends' data - rather than actually doing science.
This is why we shouldn't be satisfied with any reform effort that puts replication before replicability. If you can fudge a result, you can fudge the data a replication. How to fight questionable practices is another question but I've proposed reforms that I think would work, namely pre-registration of hypotheses, methods, and statistical analyses. Others have their own ideas.
A lesson from clinical medicine here. Clinical trials of new drugs adopted pre-registration, but only after they tried replication and it didn't work. Pharmaceutical regulators have long required multiple demonstrations of drug efficacy. One trial was not enough. Sounds good - but the problem was that drug companies just did lots of trials and analyses, picked the positive ones, and used them.
So in summary: replication is important, and we don't do enough of it, but replication alone is not enough to fix psychology.
