Research synthesischallenges, reproducibility and future directions

Abstract

Research synthesis projects play an indispensable role in the scientific process as they bring order to the vast array of scientific evidence, organizing individual pieces of evidence into a coherent body of knowledge on a specific topic. Given this prominent role, the results and conclusions of research synthesis projects carry greater relevance and impact compared to those of individual studies. Therefore, it is essential to keep an eye on the research practices and credibility of research synthesis projects. In this dissertation, we delve into various aspects of research practices and the credibility of research synthesis projects. The first study (Chapter 2) focused on assessing the prevalence of transparency and reproducibility-related reporting practices in research synthesis projects. The second study (Chapter 3) focused on reproducibility of meta-analytic results reported on these projects. Lastly, the third study (Chapter 4) explored the statistical power of meta-analytic synthesis when assuming a random-effects model. All three studies were carried out using a random sample of 100 published research synthesis projects on effectiveness of clinical psychological interventions. In the first study, we comprehensively examined the entire process of a sample of research synthesis projects, from literature searching to synthesis methods. We found major issues concerning completely reproducible search procedures report, specification of the exact method to compute effect sizes, or choice of weighting factors and estimators. Additionally, data availability was also examined and found a lack of availability of the statistics used to compute the effect sizes, as well as a lack of interoperability of available data. In the second study, we delved deeper into the issue of data availability in research synthesis projects. Specifically, we assessed the reproducibility of the meta-analytic results reported in these works. Our findings indicated that current data sharing practices significantly hinder the reusability and retrieval of the data collected during a research synthesis project. This lack of accessible data emerged as one of the most significant threats to the reproducibility of meta-analytic results. The results from the third study have shown instances where random-effects meta-analytic averaging could be underpowered. Typically, meta-analytic averaging is extremely useful, providing more precise estimates and enabling more powerful statistical testing. Nevertheless, under specific circumstances, this type of synthesis may yield less informative results. While this highlights a limitation of meta-analytic averaging in some scenarios, it does not apply to research synthesis as a whole; rather, it is a specific outcome of the latter. To summarize, we emphasize the role of research synthesis as an organizer of the research space.