Statistical control versus classification of study quality in meta-analysis

References

• References to studies in Appendix 1 are marked with an asterisk (*).
   Google Scholar
• *Adesope , O. O. , Lavin , T. , Thompson , T. and Ungerleider , C. 2010 . A systematic review and meta-analysis of the cognitive correlates of bilingualism . Review of Educational Research , 80 ( 2 ) : 207 – 245 . doi: 10.3102/0034654310368803
   Google Scholar
• Altman , D. G. , Schultz , K. F. , Moher , D. , Egger , M. , Davidoff , F. , Elbourne , D. , Gotzsche , P. C. and Lang , T. 2001 . The revised CONSORT statement for reporting randomized trials: Explanation and elaboration . Annals of Internal Medicine , 134 ( 8 ) : 663 – 694 . (doi:10.7326/0003-4819-134-8-200104170-00012)
   Google Scholar
• *Bangert-Drowns , R. , Wells-Parker , E. and Chevillard , I. 1997 . “ Assessing the methodological quality of research in narrative reviews and meta-analyses ” . In The science of prevention: Methodological advances from alcohol and substance abuse research , Edited by: Bryant , K. J. , Windle , M. and West , S. G. 405 – 429 . Washington , DC : American Psychological Association .
   Google Scholar
• *Bernard , R. M. , Abrami , P. C. , Borokhovski , E. , Wade , A. , Tamim , R. , Surkes , M. and Bethel , E. C. 2009 . A meta-analysis of three interaction treatments in distance education . Review of Educational Research , 79 ( 3 ) : 1243 – 1289 . doi: 10.3102/0034654309333844
   Google Scholar
• Bernard , R. M. , Abrami , P. C. , Lou , Y. , Borokhovski , E. , Wade , A. , Wozney , L. , Wallet , P. A. , Fiset , M. and Huang , B. 2004 . How does distance education compare to classroom instruction? A meta-analysis of the empirical literature . Review of Educational Research , 74 ( 3 ) : 379 – 439 . doi: 10.3102/00346543074003379
   Google Scholar
• Borenstein , M. , Hedges , L. V. , Higgins , J. and Rothstein , H. 2009 . Introduction to meta-analysis , Chippenham : Wiley .
   Google Scholar
• Borenstein , M. , Hedges , L. V. , Higgins , J. and Rothstein , H. 2010 . Comprehensive meta-analysis (Version 2.20). Englewood, NJ: Biostat
   Google Scholar
• *Bowers , P. N. , Kirby , J. R. and Deacon , S. H. 2010 . The effects of morphological instruction on literacy skills: A systematic review of the literature . Review of Educational Research , 80 ( 2 ) : 144 – 179 . doi: 10.3102/0034654309359353
   Google Scholar
• *Bowman , N. A. 2010 . College diversity experiences and cognitive development: A meta-analysis . Review of Educational Research , 4 ( 1 ) : 4 – 33 . doi: 10.3102/ 0034654309352495
   Google Scholar
• *Bowman , N. A. 2011 . Promoting participation in a diverse democracy: A meta-analysis of college diversity experiences and civic engagement . Review of Educational Research , 81 ( 1 ) : 29 – 68 . doi: 10.3102/0034654310383047
   Google Scholar
• Campbell , D. T. and Stanley , J. C. 1963 . Experimental and quasi-experimental designs , Chicago , IL : Rand McNally .
   Google Scholar
• Cohen , J. 1988 . Statistical power analysis for the behavioral sciences , 2 , Hillsdale , NJ : Lawrence Erlbaum .
   Google Scholar
• Cook, T.D. (2004). Randomized experiments in education: Why are they so rare? (Institute for Policy Research Working Paper, WP-02-19). Northwestern University, Evanston, IL.
   Google Scholar
• Cook , T. and Campbell , D. T. 1979 . Quasi-experimental design , Chicago , IL : Rand McNally .
   Google Scholar
• *Cooper , H. , Allen , A. B. , Patall , E. A. and Dent , A. L. 2010 . Effects of full-day kindergarten on academic achievement and social development . Review of Educational Research , 80 ( 1 ) : 34 – 70 . doi: 10.3102/0034654309359185
   Google Scholar
• *Credé , M. , Roch , S. G. and Kieszczynka , U. M. 2010 . Class attendance in college: A meta-analytic review of the relationship of class attendance with grades and student characteristics . Review of Educational Research , 80 ( 2 ) : 272 – 295 . doi: 10.3102/0034654310362998
   Google Scholar
• *Denson , N. 2009 . Do curricular and co-curricular diversity activities influence racial bias? A meta-analysis . Review of Educational Research , 79 ( 2 ) : 805 – 838 . doi: 10.3102/ 0034654309331551
   Google Scholar
• *Edmonds , M. S. , Vaughn , S. , Wexler , J. , Reutebuch , C. , Cable , A. , Tackett , K. K. and Schnakenberg , J. W. 2009 . A synthesis of reading interventions and effects on reading comprehension outcomes for older struggling readers . Review of Educational Research , 79 ( 1 ) : 262 – 300 . doi: 10.3102/0034654308325998
   Google Scholar
• Eysenck , H. J. 1978 . Comment: An exercise in mega-silliness . American Psychologist , 33 : 517 Retrieved from http://psycnet.apa.org/journals/amp/33/5/517a.pdf (doi:10.1037/0003-066X.33.5.517.a)
   Google Scholar
• Field , A. P. 2003 . The problems of using fixed- models of meta-analysis effects on real-world data . Understanding Statistics , 2 : 77 – 96 . doi: 10.1207/S15328031US0202_02
   Google Scholar
• Glass , G. V. 1976 . Primary, secondary, and meta-analysis of research . Educational Researcher , 5 : 3 – 8 . (doi:10.3102/0013189X005010003)
   Google Scholar
• Glass , G. V. 1978 . Integrating findings: The meta-analysis of research . Review of Research in Education , 5 : 351 – 379 .
   Google Scholar
• Glazerman , S. , Levy , D. and Myers , D. 2003 . Nonexperimental versus experimental estimates of earnings impacts . The Annals of the American Academy of Political and Social Sciences , 589 : 63 – 93 . (doi:10.1177/0002716203254879)
   Google Scholar
• Hedges , L. V. and Olkin , I. 1985 . Statistical methods for meta-analysis , New York , NY : Academic Press .
   Google Scholar
• Hedges , L. V. and Pigott , T. D. 2001 . The power of statistical tests in meta-analysis . Psychological Methods , 6 ( 3 ) : 203 – 217 . (doi:10.1037/1082-989X.6.3.203)
   Google Scholar
• Hedges , L. V. and Pigott , T. D. 2004 . The power of statistical tests for moderators in meta-analysis . Psychological Methods , 9 ( 4 ) : 424 – 445 . (doi:10.1037/1082-989X.9.4.426)
   Google Scholar
• Higgins , J. P.T. and Thompson , S. G. 2002 . Quantifying heterogeneity in a meta-analysis . Statistics in Medicine , 21 : 1539 – 1558 . (doi:10.1002/sim.1186)
   Google Scholar
• Hunter , J. E. and Schmidt , F. L. 2004 . Methods of meta-analysis: Correcting error and bias in research findings , 2 , Newbury Park , CA : Sage Publications .
   Google Scholar
• *Kieffer , M. J. , Lesaux , N. L. , Rivera , M. and Francis , D. J. 2009 . Accommodations for English language learners taking large-scale assessments: A meta-analysis on effectiveness and validity . Review of Educational Research , 79 ( 3 ) : 1168 – 1201 . doi: 10.3102/ 0034654309332490
   Google Scholar
• Lipsey , M. W. and Wilson , D. B. 1993 . The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis . American Psychologist , 48 ( 12 ) : 1181 – 1209 . doi:10.1037/0003-066X.48.12.1181
   Google Scholar
• MacLehose , R. R. , Reeves , B. C. , Harvey , I. M. , Sheldon , T. A. , Russell , I. T. and Black , A. M.S. 2000 . A systematic review of comparisons of effect sizes derived from randomized and non-randomised studies . Health Technology Assessment , 4 ( 34 ) : 1 – 154 .
   Google Scholar
• *Marsh , H. W. , Bornmann , L. , Mutz , R. , Daniel , H.-D. and O'Mara , A. 2009 . Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches . Review of Educational Research , 79 ( 3 ) : 1290 – 1326 . doi: 10.3102/0034654309334143
   Google Scholar
• *Marulis , L. M. and Neuman , S. B. 2010 . The effects of vocabulary intervention on young children's word learning: A meta-analysis . Review of Educational Research , 80 ( 3 ) : 300 – 335 . doi: 10.3102/0034654310377087
   Google Scholar
• Matt , G. E. and Cook , T. D. 2009 . “ Threats to the validity of generalized inferences ” . In The handbook of research synthesis and meta-analysis , Edited by: Cooper , H. , Hedges , L. V. and Valentine , J. C. 537 – 560 . New York , NY : Russell Sage .
   Google Scholar
• Moher , D. , Jadad , A. and Tugwell , P. 1996 . Assessing the quality of randomized controlled trials, Current issues and future directions . International Journal of Technology Assessment in Health Care , 12 : 195 – 208 . (doi:10.1017/S0266462300009570)
   Google Scholar
• *Möller , J. , Pohlmann , B. , Köller , O. and Marsh , H. W. 2009 . A meta-analytic path analysis of the internal/external frame of reference model of academic achievement and academic self-concept . Review of Educational Research , 79 ( 3 ) : 1129 – 1167 . doi: 10.3102/ 0034654309337522
   Google Scholar
• Mosteller , F. and Boruch , R. 2002 . Evidence matters: Randomized trials in education research , Washington , DC : Brookings Institute Press .
   Google Scholar
• Nave, B., Miech, E.J., & Mosteller, F. (1999). A rare design: The role of field trials in evaluating school practices. Paper presented at The American Academy of Arts and Sciences and Harvard University, Cambridge, MA.
   Google Scholar
• Pedhazur , E. J. 1997 . Multiple regression in behavioral research , 3 , Orlando , FL : Harcourt Brace .
   Google Scholar
• Pigott, T. (2006, April). Using retrospective power analysis in meta-analysis. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA.
   Google Scholar
• Pigott , T. 2012 . Advances in meta-analysis , New York , NY : Springer .
   Google Scholar
• *Rakes , C. R. , Valentine , J. C. , McGatha , M. B. and Ronau , R. N. 2010 . Methods of instructional improvement in algebra: A systematic review and meta-analysis . Review of Educational Research , 80 ( 3 ) : 372 – 400 . doi: 10.3102/0034654310374880
   Google Scholar
• Raudenbush , S. W. 2005 . Learning from attempts to improve schooling: The contribution of methodological diversity . Educational Researcher , 34 ( 5 ) : 25 – 31 . (doi:10.3102/0013189X034005025)
   Google Scholar
• *Ritter , G. W. , Barnett , J. H. , Denny , G. S. and Albin , G. R. 2009 . The effectiveness of volunteer tutoring programs for elementary and middle school students: A meta-analysis . Review of Educational Research , 79 ( 1 ) : 3 – 38 . doi: 10.3102/0034654308325690
   Google Scholar
• Robinson , D. 2004 . An interview with Gene Glass . Educational Researcher , 33 ( 3 ) : 26 – 30 . (doi:10.3102/0013189X033003026)
   Google Scholar
• Shadish , W. R. , Cook , T. D. and Campbell , D. T. 2002 . Experimental and quasi-experimental designs for generalized causal inference , Boston , MA : Houghton-Mifflin .
   Google Scholar
• Shadish, W.R., Luellen, J.K., & Clark, M.H. (2006). Propensity scores and quasi-experiments: A testimony to the practical side of Lee Sechrest. In R.R. Bootzin & P.E. McKnight (Eds.), Strengthening research methodology: Psychological measurement and evaluation (pp. 143–157). Washington, DC: American Psychological Association.
   Google Scholar
• Shadish, W., & Myers, D. (2004, November). Campbell collaboration research design policy brief. Retrieved April 27, 2009, from http://camp.ostfold.net/resources/methods_policy_briefs/
   Google Scholar
• Slavin , R. E. 1986 . Best-evidence synthesis: An alternative to meta-analytic and traditional reviews . Educational Researcher , 15 ( 9 ) : 5 – 11 . (doi:10.3102/0013189X015009005)
   Google Scholar
• Slavin , R. E. 2002 . Evidence-based education policies: Transforming educational practice and research . Educational Researcher , 31 ( 7 ) : 15 – 21 . (doi:10.3102/0013189X031007015)
   Google Scholar
• Slavin , R. E. 2004 . Educational research can and must address “what works” questions . Educational Researcher , 31 ( 1 ) : 24 – 26 .
   Google Scholar
• Slavin , R. E. 2008 . Perspectives on evidence-based research in education: What works? Issues in synthesizing educational program evaluations . Educational Researcher , 37 ( 1 ) : 5 – 14 . (doi:10.3102/0013189X08314117)
   Google Scholar
• *Slavin , R. E. , Lake , C. , Chambers , B. , Cheung , A. and Davis , S. 2009 . Effective reading programs for the elementary grades: A best-evidence synthesis . Review of Educational Research , 79 ( 4 ) : 1391 – 1466 . doi: 10.3102/0034654309341374
   Google Scholar
• *Slavin , R. E. , Lake , C. and Groff , C. 2009 . Effective programs in middle and high school mathematics: A best-evidence synthesis . Review of Educational Research , 79 ( 2 ) : 839 – 911 . doi: 10.3102/0034654308330968
   Google Scholar
• Smith , G. C.S. and Pell , J. P. 2003 . Parachute use to prevent death and major trauma related to gravitational challenge: Systematic review of ramdomised controlled trials . British Medical Journal , 327 : 20 – 27 .
   Google Scholar
• *Sosa , G. W. , Berger , D. E. , Saw , A. T. and Mary , J. C. 2011 . Effectiveness of computer-assisted instruction in statistics: A meta-analysis . Review of Educational Research , 81 ( 1 ) : 97 – 128 . doi: 10.3102/0034654310378174
   Google Scholar
• *van Steensel , R. , McElvany , N. , Kurvers , J. and Herppich , S. 2011 . How effective are family literacy programs? Results of a meta-analysis . Review of Educational Research , 81 ( 1 ) : 69 – 96 . doi: 10.3102/0034654310388819
   Google Scholar
• *Swanson , H. L. and Hsieh , C.-J. 2009 . Reading disabilities in adults: A selective meta-analysis of the literature . Review of Educational Research , 79 ( 4 ) : 1362 – 1390 . doi: 10.3102/ 0034654309350931
   Google Scholar
• *Tamim , R. M. , Bernard , R. M. , Borokhovski , E. , Abrami , P. C. and Schmid , R. S. 2011 . What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study . Review of Educational Research , 81 ( 1 ) : 4 – 28 . doi: 10.3102/0034654310393361
   Google Scholar
• Valentine , J. C. and Cooper , H. M. 2008 . A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The study design and implementation assessment device (Study DIAD) . Psychological Methods , 13 : 130 – 149 . (doi:10.1037/1082-989X.13.2.130)
   Google Scholar
• *Weiser , B. and Mathes , P. 2011 . Using encoding instruction to improve the reading and spelling performances of elementary students at risk for literacy difficulties: A best-evidence synthesis . Review of Educational Research , 81 ( 2 ) : 170 – 200 . doi: 10.3102/0034654310396719
   Google Scholar
• West, S., King, V., Caret, T., Lohr, K., McKoy, N., Sutton, S., & Lux, L. (2002). Systems to rate the strength of scientific evidence. Evidence report/technology assessment No. 47 AHRQ (Publication No. 02-E016). Rockville, MD: Agency of Healthcare Research and Quality.
   Google Scholar
• What Works Clearinghouse. (2011, September). WWC procedures and standards handbook (Version 2.1). Retrieved May 30, 2012, from http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf
   Google Scholar
• Wortman , P. M. 1994 . “ Judging research quality ” . In The handbook of research synthesis , Edited by: Cooper , H. and Hedges , L. V. 97 – 109 . New York , NY : Russell Sage Foundation .
   Google Scholar