Examining the consequential validity of standardized examinations via public perceptions: a review of mixed methods survey design considerations^†

^†This article is based in part on a previously presented conference paper by Hitchcock, Onwuegbuzie, & Koshaim (2012).

ABSTRACT

The use of standardized examinations to inform high-stakes decisions is a process that is followed in many countries, and there is merit to understanding the impact that such testing has on any given society. For this reason, some researchers have studied the societal impact of testing to understand the social validity and consequences of assessment. Information gleaned from such efforts can be used to inform testing practices as well as communication efforts with stakeholders. Assessing social validity, and aspects of the consequences of measurement, can be handled in part via survey studies. Because the use of testing is becoming more widespread, and across several nations, anyone interested in such inquiry will need to be able to develop culturally relevant surveys. For this reason, this methodological article describes an exploratory and sequential mixed methods research approach for survey development.

KEYWORDS:

• Standardized testing
• mixed methods
• measurement
• culturally relevant surveys
• social validity
• consequential validity

Notes

^† This article is based in part on a previously presented conference paper by Hitchcock, Onwuegbuzie, & Koshaim (2012).

1 Polls often entail short, simple items. In these cases some of the methods described here might be unnecessary. An operating assumption is that readers will be interested in surveying nuanced concerns among the citizenry in a given country.

2 – There is a philosophical orientation behind why the authors do not offer a simple set of polling questions that might be translated from prior surveys used in the USA and other countries. We assume cultural and contextual factors are so pervasive and complex that it is unrealistic to assume that key elements needed to write a survey are currently in place. Namely, it might be premature to know what questions to ask and how to go about asking them in a particular setting. After all, a simple problem with any survey is that failure to ask appropriate questions, in an appropriate manner, will yield data of limited utility. Students might, for example, have some strong opinions about being subjected to college entrance examinations, but we will not discover their views if we do not know what questions to pose.