Statistical control versus classification of study quality in meta-analysis

Abstract

A vexing problem for meta-analysts is how to account for the differential methodological quality of studies. We critically examine five approaches that deal with methodological quality. The first approach is to ignore the methodological quality of studies and aggregate all available research for which effect sizes (ES) can be calculated. The second approach is to use methodological quality as an inclusion criterion, thereby excluding all studies that fail to meet some predetermined level of quality. The third approach is to use methodological quality to classify included studies and report their findings separately. The fourth approach is to weight studies by methodological quality. And the fifth approach is to treat methodological quality as a predictor of ES by removing its influence from the collection of evidence. After discussing some of the limitations of the five approaches, we describe a method for treating methodological quality as a study feature, coding for it, and adjusting for its influence. A four-step procedure is illustrated using two examples. Step one: Are the ES homogeneous? Step two: does study quality explain the heterogeneity? Step three: which qualities of studies matter? Step four: how do we deal with the differences?

Keywords:

• meta-analysis
• systematic review
• quantitative synthesis
• quantitative review

Acknowledgements

The preparation of this paper was supported by grants awarded to the authors by the Social Sciences and Humanities Research Council of Canada and the Fonds québécois de la recherche sur la société et la culture, Province of Quebec. Address inquiries to Philip C. Abrami, Centre for the Study of Learning and Performance, Concordia University, 1455 de Maisonneuve Blvd. W., Montreal, Quebec Canada H3G 1M8, 514-848-2424 x2102, [email protected]. The authors express their appreciation to Hannah Rothstein, Herbert W. Turner III, Chad Nye and several anonymous reviewers for their useful and insightful comments on drafts of this paper.