Statistical Methodology of Reliability Generalization Meta-Analysis

Abstract

Quantifying the psychological capacities, traits or attributes of individuals is a fundamental part of social and health sciences such as psychology. To ensure that this quantification is truly representative and useful, the assessment process must be precise and be carried out using well-constructed and contrasted measurement instruments. One of the minimum requirements to be met is that the measurement should be reliable, that is to say, that the different applications of the instrument should yield consistent. Reliability is a psychometric property that represents the degree of replicability of the scores of an instrument. Reliability is not inherent to the instrument, so in order to raise this value to the test itself it is necessary to apply other techniques and statistical methods that allow it. Thus, Vacha-Haase in 1998 developed the concept of the Reliability Generalization Meta-analysis (RG), making use of the meta-analytic methodology (the best tool for the synthesis of empirical evidence) to generalize the reliability results of multiple applications of a test. This technique is particularly important as individual results from primary studies often provide different and even contradictory conclusions. The integration of all experimental results within a single field of study provides a more comprehensive and realistic view of the true effect. An advantage of this methodology is its flexibility and the absence of a strict protocol. This implies that each researcher must make the appropriate statistical decisions depending on the data available and the generalizability of the results. At least three fundamental decisions should be made in An RG meta-analysis: the transformation of the coefficients, the statistical model used and the weighting method applied. The conventional approach applied in an RG meta-analysis does not consider the possible dependency relationships. When a test has a multidimensional structure with several subscales, all forming part of the same psychological construct, this dependency may arise. Traditionally, in order to avoid these dependency networks, separate meta-analyses were carried out for each subscale. This practice is not particularly suitable because by selecting only a part of the available data, the results obtained are less accurate and the statistical analyses are less powerful (Assink & Wibbelink, 2016; Van den Noortgate et al., 2013). An alternative way to model dependence in meta-analyses is the application of multilevel models. Although the nature of this methodology allows any analytical strategy to be employed, either in a traditional way or applying multilevel models, detailed reporting is essential, specifying each of the decisions that were taken to obtain these results. This dissertation presents three fundamental objectives: the first aim is found in the first study (Chapter 2) where the results obtained based on the different statistical decisions (coefficient transformation, statistical model and weighting method) are statistically compared. In this way, it will be determined whether different decisions can lead to different conclusions. The second study (Chapter 3) will assess the degree of reproducibility of these studies, as well as the level of transparency and reporting of key information for the reproducibility of the analyses. Finally, the third study (Chapter 4) will examine the differences between the results obtained by applying the conventional methodology of a GF meta-analysis, which avoids possible dependency relationships, and the methodology of the multilevel model, which integrates and models dependency.