El enfoque meta-analítico de generalización de la fiabilidad

  1. Sánchez Meca, Julio
  2. López Pina, José Antonio
Revista:
Acción psicológica

ISSN: 1578-908X

Año de publicación: 2008

Título del ejemplar: Innovaciones metodológicas en la evaluación psicológica : perspectivas de futuro

Volumen: 5

Número: 2

Páginas: 37-64

Tipo: Artículo

DOI: 10.5944/AP.5.2.457 DIALNET GOOGLE SCHOLAR

Otras publicaciones en: Acción psicológica

Resumen

Frases del tipo «la fiabilidad del test es 0.80» son incorrectas. Es más apropiado decir «la fiabilidad de las puntuaciones del test en una determinada aplicación del mismo es 0.80». El enfoque meta-analítico de generalización de la fiabilidad pretende demostrar que la fiabilidad es una propiedad empírica que varía de una aplicación a otra del test. Este nuevo enfoque meta-analítico está contribuyendo a concienciar a los investigadores sobre la importancia de aportar estimaciones de la fiabilidad con los propios datos y evitar inducciones de la fiabilidad. Se presentan las fases en las que se lleva a cabo un estudio de generalización de la fiabilidad: (a) formulación del problema, (b) búsqueda de los estudios, (c) codificación de los estudios, (d) análisis estadístico e interpretación y (e) publicación. Se presenta una visión actualizada de los problemas estadísticos de este enfoque: (a) transformar versus no transformar los coeficientes, (b) ponderar versus no ponderar los coeficientes, (c) cómo tratar la dependencia estadística entre los coeficientes y (d) cuál es el modelo estadístico más apropiado (efectos fijos, efectos aleatorios, efectos mixtos).

Referencias bibliográficas

  • Barnes, L. L. B., Harp, D. y Jung, W. S. (2002). Reliability generalization of scores on the Spielberger State-Trait Anxiety Inventory. Educational and Psychological Measurement, 62, 603-618.
  • Beretvas, S. N., Meyers, J. L. y Leite, W. L. (2002). A reliability generalization study of the Marlowe Crowne Social Desirability Scale. Educational and Psychological Measurement, 62, 570-589.
  • Beretvas, S. N. y Pastor, D. A. (2003). Using mixedeffects models in reliability generalization studies. Educational and Psychological Measurement, 63, 75-95.
  • Beretvas, S. N., Suizzo, M.-A., Durham, J. A. y Yarnell, L. M. (2008). A reliability generalization study of scores on Rotter’s and Nowicki-Strickland’s locus of control scales. Educational and Psychological Measurement, 68, 97-119.
  • Borenstein, M. J., Hedges, L. V., Higgins, J. C., & Rothstein, H. (2005). Comprehensive meta-analysis: A computer program for research synthesis (Vers. 2.2). Englewood, NJ: Biostat, Inc.
  • Botella, J. y Gambara, H. (2002). ¿Qué es el metaanálisis? Madrid: Biblioteca Nueva.
  • Botella, J. y Gambara, H. (2006). Doing and reporting a meta-analysis. International Journal of Clinical and Health Psychology, 6, 425-440.
  • Campbell, J. S., Pulos, S., Hogan, M. y Murry, F. (2005). Reliability generalization of the Psychopathy Checklist applied in youthful samples. Educational and Psychological Measurement, 65, 639-656.
  • Capraro, M. M., Capraro, R. M. y Henson, R. K. (2001). Measurement error of scores on the Mathematics Anxiety Rating Scale across studies. Educational and Psychological Measurement, 61, 373-386.
  • Caruso, J. C. (2000). Reliability generalization of the NEO personality scales. Educational and Psychological Measurement, 60, 236-254.
  • Conway, J. M., Jako, R. A. y Goodman, D. F. (1995). A meta-analysis of interrater and internal consistency reliability of selection interviews. Journal of Applied Psychology, 80, 565-579.
  • Cooper, H. M. (1998). Integrating research: A guide for literature reviews (2ª ed.). Thousand Oaks, CA: Sage.
  • Cooper, H. y Hedges, L. V. (Eds.) (1994). The handbook of research synthesis. Nueva York: Russell Sage Foundation.
  • Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98-104.
  • Crocker, L. y Algina, J. (1986). Introduction to classical and modern test theory. Nueva York: Holt, Rinehart, & Winston.
  • Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 15, 297- 334.
  • Dawis, R. V. (1987). Scale construction. Journal of Counseling Psychology, 34, 481-489.
  • DerSimonian, R. y Laird, N. (1986). Meta-analysis of clinical trials. Controlled Clinical Trials, 7, 177- 188.
  • Dimitrov, D. M. (2002). Reliability: Arguments for multiple perspectives and potential problems with generalization across studies. Educational and Psychological Measurement, 62, 783-801.
  • Feldt, L. S. y Brennan, R. L. (1989). Reliability. En R. L. Linn (Ed.), Educational measurement (3ª ed., pp. 105-146). Nueva York: American Council on Education and Macmillan.
  • Feldt, L. S. y Charter, R. A. (2006). Averaging internal consistency reliability coefficients. Educational and Psychological Measurement, 66, 215-227.
  • Glass, G. V., McGaw, B. y Smith, M. L. (1981). Metaanalysis in social research. Beverly Hills, CA: Sage.
  • Gronlund, N. E. y Linn, R. L. (1990). Measurement and evaluation in teaching (6ª ed.). Nueva York: Macmillan.
  • Gulliksen, H. (1987). Theory of mental tests. Nueva York: Wiley.
  • Hakstian, A. R. y Whalen, T. E. (1976). A k-sample significance test for independent alpha coefficients. Psychometrika, 41, 219-231.
  • Hall, S. M. y Brannick, M. T. (2002). Comparison of two random-effects methods of meta-analysis. Journal of Applied Psychology, 87, 377-389.
  • Harwell, M. (1997). An empirical study of Hedges’s homogeneity test. Psychological Methods, 2, 219- 231.
  • Hedges, L. V. (1994). Fixed effects models. En H. Cooper y L. V. Hedges (Eds.), The handbook of research synthesis (pp. 285-299). Nueva York: Russell Sage Foundation.
  • Hedges, L. V. y Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
  • Hedges, L. V. y Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486-504.
  • Heldref Foundation (1997). Guidelines for contributors. Journal of Experimental Education, 65, 95- 96.
  • Henson, R. K. (2001). Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34, 177-189.
  • Henson, R. K. y Thompson, B. (2002). Characterizing measurement error in scores across studies: Some recommendations for conducting «reliability generalization» studies. Measurement and Evaluation in Counseling and Development, 35, 113-127.
  • Higgins, J. P. T. y Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539-1558.
  • Howell, R. T. y Shields, A. L. (2008). The file drawer problem in reliability generalization: A strategy to compute a fail-safe N with reliability coefficients. Educational and Psychological Measurement, 68, 120-128.
  • Huedo-Medina, T., Sánchez-Meca, J., Marín-Martínez, F. y Botella, J. (2006). Assessing heterogeneity in meta-analysis: Q statistic or I2 index? Psychological Methods, 11, 193-206.
  • Hunter, J. E. y Schmidt, F. S. (1990). Methods of metaanalysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.
  • Hunter, J. E. y Schmidt, F. S. (2004). Methods of metaanalysis: Correcting error and bias in research findings (2ª ed.). Thousand Oaks, CA: Sage.
  • Hunter, J. E., Schmidt, F. S. y Jackson, G. B. (1982). Meta-analysis: Cumulating research findings across studies. Beverly Hills, CA: Sage.
  • Leach, L. F., Henson, R. K., Odom, L. R. y Cagle, L. S. (2006). A reliability generalization study of the Self-Description Questionnaire. Educational and Psychological Measurement, 66, 285-304.
  • Li, A. y Bagger, J. (2007). The Balanced Inventory of Desirable Responding (BIDR): A reliability generalization study. Educational and Psychological Measurement, 67, 525-544.
  • Lipsey, M. W. y Wilson, D. B. (2001). Practical metaanalysis. Thousand Oaks, CA: Sage.
  • López-Pina, J. A., Sánchez-Meca, J. y Rosa-Alcázar, A. I. (2008). The Hamilton Rating Scale for Depression: A reliability generalization study. Manuscrito en revisión, Universidad de Murcia.
  • Marín-Martínez, F., Sánchez-Meca, J., Huedo, T. y Fernández-Guzmán, I. (2007). Meta-análisis: ¿Dónde estamos y hacia dónde vamos? En A. Borges y P. Prieto (Eds.), Psicología y ciencias afines en los albores del siglo XXI (Homenaje al profesor Alfonso Sánchez Bruno) (pp. 87-102). Grupo Editorial Universitario.
  • Marín-Martínez, F. y Sánchez-Meca, J. (1998). Testing for dichotomous moderators in meta-analysis. Journal of Experimental Education, 67, 69-81.
  • Martín, J. L. R., Tobías, A. y Seoane, T. (Coords.) (2006). Revisiones Sistemáticas en las Ciencias de la Vida. Toledo: FISCAM.
  • Mason, C., Allam, R. y Brannick, M. T. (2007). How to meta-analyze coefficient-of-stability estimates: Some recommendations based on Monte Carlo studies. Educational and Psychological Measurement, 67, 765-783.
  • Miller, M. B. (1995). Coefficient alpha: A basic introduction from the perspectives of classical test theory and structural equation modeling. Structural Equation Modeling, 2, 255-273.
  • Onwuegbuzie, A. J. y Daniel, L. G. (2004). Reliability generalization: The importance of considering sample specificity, confidence intervals, and subgroup differences. Research in the Schools, 11, 60-71.
  • Parker, K. (1983). A meta-analysis of the reliability and validity of the Rorschach. Journal of Personality Assessment, 47, 227-231.
  • Parker, K. C. H., Hanson, R. K. y Hunsley, J. (1988). MMPI, Rorschach, and WAIS: A meta-analytic comparison of reliability, stability, and validity. Psychological Bulletin, 103, 367-373.
  • Pedhazur, E. J. y Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Erlbaum.
  • Petrosino, A., Boruch, R. F., Soydan, H., Duggan, L. y Sánchez-Meca, J. (2001). Meeting the challenges of evidence-based policy: The Campbell Collaboration. Annals of the American Academy of Political and Social Science, 578, 14-34.
  • Petticrew, M. y Roberts, H. (2006). Systematic Reviews in the Social Sciences: A practical guide. Malden, MA: Blackwell.
  • Raykov, T. (1997). Estimation of composite reliability for congeneric measures. Applied Psychological Measurement, 21, 173-184.
  • Rodríguez, M. C. y Maeda, Y. (2006). Meta-analysis of coefficient alpha. Psychological Methods, 11, 306- 322.
  • Rosenberg, M. S., Adams, D. C. y Gurevitch, J. (1999). MetaWin: Statistical software for meta-analysis with resampling tests (Vers. 2.0). Sunderland, MA: Sinauer Associates.
  • Rosenthal, R. (1991). Meta-analytic procedures for social research (2ª ed.). Newbury Park, CA: Sage.
  • Rosenthal, R. (1995). Writing meta-analytic reviews. Psychological Bulletin, 118, 183-192.
  • Rowley, G. L. (1976). The reliability of observational measures. American Educational Research Journal, 13, 51-59.
  • Salgado, J. F. y Moscoso, S. (1996). Meta-analysis of interrater reliability of job performance ratings in validity studies of personnel selection. Perceptual and Motor Skills, 83, 1195-1201.
  • Sánchez-Meca, J. (1999). Meta-análisis para la inves- tigación científica. En F. J. Sarabia (Coord.), Metodología para la investigación en marketing y dirección de empresa (pp. 173-201). Madrid: Pirámide.
  • Sánchez-Meca, J. (2003). La revisión del estado de la cuestión: El meta-análisis. En C. Camisón, M. J. Oltra y M. L. Flor (Eds.), Enfoques, problemas y métodos de investigación en economía y dirección de empresas (pp. 101-110). Castellón: ACEDE/Fundació Universitat Jaime I–Empresa.
  • Sánchez-Meca, J. y Ato, M. (1989). Meta-análisis: Una alternativa metodológica a las revisiones tradi- cionales de la investigación. En J. Arnau y H. Carpintero (Eds.), Tratado de psicología general I: Historia, teoría y método (pp. 617-669). Madrid: Alhambra.
  • Sánchez-Meca, J., Boruch, R. F., Petrosino, A. y RosaAlcázar, A. I. (2002). La Colaboración Campbell y la práctica basada en la evidencia. Papeles del Psicólogo, 83, 44-48.
  • Sánchez-Meca, J. y Marín-Martínez, F. (1997). Homogeneity tests in meta-analysis: A Monte Carlo comparison of statistical power and Type I error. Quality and Quantity, 31, 385-399.
  • Sánchez-Meca, J. y Marín-Martínez, F. (1998). Testing continuous moderators in meta-analysis: A comparison of procedures. British Journal of Mathematical & Statistical Psychology, 51, 311-326.
  • Sánchez-Meca, J. y Marín-Martínez, F. (2008). Confidence intervals for the overall effect size in random-effects meta-analysis. Psychological Methods, 13, 31-48.
  • Sánchez-Meca, J., Marín-Martínez, F. y Huedo, T. (2006). Modelo de efectos fijos versus modelo de efectos aleatorios. En J. L. R. Martín, A. Tobías y T. Seoane (Coords.), Revisiones Sistemáticas en Ciencias de la Vida (pp. 189-204). Toledo: FISCAM.
  • Sawilowsky, S. S. (2000a). Psychometrics versus datametrics: Comment on Vacha-Haase’s ‘Reliability generalization’ method and some EPM editorial policies. Educational and Psychological Measurement, 60, 157-173.
  • Sawilowsky, S. S. (2000b). Reliability: Rejoinder to Thompson and Vacha-Haase. Educational and Psychological Measurement, 60, 196-200.
  • Schmidt, F. S. y Hunter, J. S. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529-540.
  • Schulze, R. (2004). Meta-analysis: A comparison of approaches. Hogrefe & Huber Pub.
  • Shadish, W. R., Chacón-Moscoso, S. y Sánchez-Meca, J. (2005). Evidence-based decision making: Enhancing systematic reviews of program evaluation results in Europe. Evaluation, 11, 95-109.
  • Silver , N. y Dunlap, W. (1987). Averaging coefficients: Should Fisher’s z-transformation be used? Journal of Applied Psychology, 72, 3-9.
  • Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54, 837- 847.
  • Thompson, B. (Ed.) (2003). Score reliability: Contemporary thinking on reliability issues. Thousand Oaks, CA: Sage.
  • Thompson, B. y Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174- 195.
  • Traub, R. E. (1994). Reliability for the social sciences: Theory and applications (Vol. 3). Thousand Oaks, CA: Sage.
  • Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20.
  • Vacha-Haase, T., Henson, R. K. y Caruso, J. C. (2002). Reliability generalization: Moving toward improved understanding and use of score reliability. Educational and Psychological Measurement, 62, 562-569.
  • Vacha-Haase, T., Kogan, L. R. y Thompson, B. (2000). Sample compositions and variabilities in published studies versus those in test manuals. Educational and Psychological Measurement, 60, 509- 522.
  • Vacha-Haase, T. y Ness, C. (1999). Practices regarding reporting of reliability coefficients: A review of three journals. Journal of Experimental Education, 67, 335-342.
  • Viechtbauer, W. (2005). Bias and efficiency of meta- analytic variance estimators in the randomeffects model. Journal of Educational and Behavioral Statistics, 30, 261-293.
  • Viechtbauer, W. (2007). Confidence intervals for the amount of heterogeneity in meta-analysis. Statistics in Medicine, 26, 37-52.
  • Viswesvaran, C. y Ones, D. S. (2000). Measurement error in ‘big five factors’ personality assessment: Reliability generalization across studies and measures. Educational and Psychological Measurement, 60, 224-235.
  • Whittington, D. (1998). How well do researchers re- port their measures? An evaluation of measurement in published educational research. Educational and Psychological Measurement, 58, 21-37.
  • Wilkinson, L. & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journal: Guidelines and explanations. American Psychologist, 54, 594-604.
  • Yarnold, P. R. y Mueser, K. T. (1989). Meta-analyses of the reliability of Type A behaviour measures. British Journal of Medical Psychology, 62, 43-50.
  • Yin, P. y Fan, X. (2000). Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies. Educational and Psychological Measurement, 60, 201-223.
Fuente de los datos: Dialnet