In a long and interesting article over at Edge, social neuroscientist Matthew Lieberman discusses (amongst other things) the 'replication crisis' in his field. Much of what he says will be of interest to regular readers of this blog. Lieberman notes that there has been a lot of controversy over 'embodied cognition' and social priming research. For instance,
There are studies suggesting that washing your hands can affect your sense of being moral or immoral, and so on. These studies are very interesting. They're very counter-intuitive, which I think leads lots of people to wonder whether or not they're legitimate.
Lately there was a particular, well-publicized case of a non-replication of one of these counter-intuitive effects, and Lieberman discusses this, but I think the issue is a general one. Here's what Lieberman says (emphasis mine) about the effort to try and replicate these findings:
I do have some issues with the process of selecting who's going to do the replications — what their qualifications are for doing those things, have they done successful work in that area previously — because if they haven't shown that they can successfully get other priming effects, or other embodied cognition effects, how do I know that they can do this? I wouldn't go and try to do chemistry. I don't know anything about doing chemistry. There are issues like that.
This argument - which Lieberman is by no means alone in making - might be called the Harry Potter Theory of social psychology. On this model, some effects are real but are difficult to get to work in an experiment ('spells'). Some people ('wizards') have the knack of getting spells to work. Other researchers ('muggles') just can't do it. So if a muggle fails to cast a spell, that's not evidence against the spell working. What else would you expect? They're a muggle! Only if a wizard fails to replicate a spell, should we be worried about the reliability of that particular piece of magic. Accordingly, muggles should not even be trying to test whether any spells work. Wizards can safely ignore muggles.
Lieberman would probably object at this point that he's not saying that some researchers should be banned from the replication process. Rather, he might say, he is only emphasizing the fact that some scientists are more qualified than others for particular tasks. If so, fair enough, but all I'm saying is that there's something odd about the idea that ones qualifications should include a track record in finding positive results in the field in question. That seems to be putting the cart before the horse. I agree that replicators should have the necessary technical skills, but I question whether generating positive (as opposed to negative) results can be used as a proxy for being skilled. That would make sense if we assume that our basic psychological theory (e.g. of social priming) is valid, and therefore that at least some of our effects are real and replicable. If we grant that, then yes, we could assume that people who fail to find effects, must be doing it wrong. (If magic exists, then non-wizards are muggles.) But can we assume that? Isn't that, in fact, the issue under debate in many cases?
