July 2024 EI Featured Article

"Selective Reporting of Placebo Tests in Top Economics Journals"

by Anna Dreber, Magnus Johannesson, and Yifan Yang

Placebo tests provide incentives to underreport statistically significant tests, a form of reversed p-hacking. We test for such underreporting in 11 top economics journals between 2009 and 2021 based on a pre-registered analysis plan. If the null hypothesis is true in all tests, 2.5% of them should be significant at the 5% level with an effect in the same direction as the main test (and 5% in total). The actual fraction of statistically significant placebo tests with an effect in the same direction is 1.29% (95% CI [0.83, 1.63]), and the overall fraction of statistically significant placebo tests is 3.10% (95% CI [2.2, 4.0]).

First published: 25 March 2024 | https://doi.org/10.1111/ecin.13217

Members access the full article by first logging in here: https://members.weai.org/my-profile