Are some scientific results so unexpected that we should just reject them? This is something I've been wondering recently. It's one thing to disbelieve a study because there are problems with the methods used. But is it scientifically valid to judge a study by its results alone, even if you don't know of any methodological flaws?
I'll admit it - I do judge studies by their results. The most recent example of this was this study which I read about yesterday. Briefly, the study reports that priming men to think about gender (by asking them one question about their spouse's wages) causes a 24-point swing in favor of Donald Trump over Hilary Clinton. I just can't believe this result. I don't have any criticisms of the methods of the study, but I simply don't believe that such a subtle prime would have such an enormous effect. For comparison, since the Democratic National Convention, there has been roughly an 8-point national swing in favor of Clinton; this is widely regarded as a dramatic change. Yet according to the priming study, one single question can provoke a swing three times as large as that (in men). I don't find that plausible. Yet is my skepticism about this result justifiable? Isn't there a sense in which it's unscientific? If I'm free to decide that a result is just unbelievable, how am I any different from (say) a creationist who maintains that it's just too incredible that natural selection produced humans and other life? To put it another way, how can I call myself a scientist if I sometimes reject scientific evidence that conflicts with my intuitions? On the other hand, maybe my skepticism is perfectly reasonable. It's a fact of life that experiments sometimes go wrong, and give the wrong results. Often, it's the unexpected or impossible nature of the results that first alerts us to the problem. If I measure (say) my own weight and the scale says I weigh 1 kg, I would immediately know that my scale is broken or miscalibrated. Some results really are just wrong. In Bayesian terms, we would say that my prior probability of a 24-point priming swing is very low. If my prior is low, it is perfectly rational for me to remain unconvinced after seeing one study in favor of the enormous priming effect (it might take ten such studies to convince me). My concern, however, is whether I can justify having such a low prior? If I had to explain my a priori skepticism I'd have to fall back on vague, intuitive statements such as "small influences have small effects" and "you just never see swings that big" - which sounds an awful lot like hand-waving, and brings us back to the problem of creationism, amongst other things. Is there a solution? Is it possible to be purely scientific in our evaluation of scientific evidence, or will there always be an element of intuition?
