Data, Truth and Null Results

Have you heard of the idea that smiling actually makes you joyful? Perhaps you know of the experiment where researchers got people to hold a pen in their mouth, so they had to smile, and it made them find cartoons funnier.

If you're familiar with this idea, then you're familar with the work of German psychologist Fritz Strack, who carried out the famous pen-based grinning study, back in 1988. Now, Strack has just published a new piece, called

From Data to Truth in Psychological Science. A Personal Perspective.

Ironically, this reflection on recent smiling studies is not a happy one. Last year, a group of researchers published a registered replication report (RRR) - an attempt to directly reproduce the 1988 pen-in-the-mouth effect on cartoon appreciation. 17 different sets of researchers carried out identical studies in parallel around the world. The total sample size was 1,894 people. The results were firmly null: there was no evidence that the 'smiling' pen condition made cartoons seem funnier. In the new piece, Strack responds to this disappointing result. He says that he "volunteered" to help facilitate the RRR, submitting the experimental materials to make it possible, but he questions the meaning and importance of the null result. Following a rather opaque discussion, Strack seems to come close to suggesting that the truth of claims in science can be known even in the absence of statistical support:

[Scientists whose work fails to replicate] may arrive at the insight that there exists no direct route from data to truth. Instead, they may come to the conclusion that science is about arguments that should be based on empirical evidence whose validity, however, is not merely determined by probabilistic parameters. Although, power, effect size, significance level, etc. provide useful information, they deliver no immediate link to the truth or falsehood of a hypothesis. Instead, they must be critically evaluated (Popper, 1959), not only by statisticians but by scientists who are experts in the field.

The obvious problem here is that Strack himself used hypothesis-testing statistics in the original smiling-pen study. Presumably, if the "probabilistic parameters" of that original study had come out null (non-significant), he would have revised his views on the influence of facial muscle contraction on emotion, at least to some extent. In which case, perhaps he should revise his views now that (much) more evidence has arrived in the form of the RRR. The alternative is that Strack would not have revised his views if the original smiling-pen study had given a null result, although this would raise the question of what the purpose of it was (and would be rather ironic for someone who cites Popper.) However he might have hypothetically reacted to a null result 30 years ago, Strack today views null results as uninformative:

It has been proposed to preregister the procedures of a study with the editors to assure that the results will be published regardless of their results. Although, this seems impartial, such a publication may not be informative and journals run the risk of becoming mere archives instead of media of the debate. As a consequence, it is not surprising that most journal editors prefer positive outcomes that add something new to what is already known.

This argument, which reminds me of Jason Mitchell's infamous Emptiness of Failed Replications post, takes aim at study preregistration, an idea that I have long championed as a way to combat publication bias. So, as you'd predict, I strongly disagree with Strack here. In my view, a well-designed study is one that will be interesting and informative whatever the results are (assuming the study 'works' in a technical sense.) Sure, there are cases such as "does this pill cure cancer?" where a positive result would be more exciting than a negative answer. Yet the negative result would still be an informative contribution to the search for a cure for cancer. There is a nice quote about this which I will leave here (h/t @jones_the_chris):