Assessing Replicability Via Multi-lab Collaborations

Multi-lab reproducibility collaborations such as the Reproducibility Project: Psychology (2015), the Experimental Economics Replication Project (2016), and the Social Sciences Replication Project (2018) have raised concern about the the reproducibility of research findings in these fields. However, the definitions of replication used in these efforts are arguably a bit impoverished. Further, more subtle, nuanced, and circumspect assessments are possible in many multi-lab reproducibility collaborations.

In this talk, we provide such an assessment for a multi-lab reproducibility collaboration that examined a number of prominent effects from behavioral economics and social psychology. Our results show that the various outcomes examined show a perhaps surprising degree of variability across the various labs and participants that took part in the project. This portends a much lower degree of replicability than may have previously been thought. Our results also show high correlation across the various outcomes that brings to the fore a potential moderator. When this moderator is incorporated into the analysis, it is clear that it is strongly predictive of many of these outcomes and does indeed moderate effects. This suggests that the more subtle, nuanced, and circumspect assessments of replicability that we advocate also have the potential to spark theoretical developments.