Raising the Stakes on a Political Science Major Field Test: Evidence From a New Capstone Course

Abstract

Using a major field test (MFT) is a common method of program assessment. For accurate assessment to take place, however, students must be sufficiently motivated to perform. This paper studies a change in major requirements that associated a major field test to a grade in a political science capstone course. After controlling for possible confounders like Gender, Age, Ethnicity, and differences in GPA and ACT scores, a modest but statistically significant treatment effect was found. However, the fit between a student’s program GPA and MFT scores did not improve, leading to questions about how best to assess political science knowledge for program graduates.

Keywords:

• Program assessment
• major field test
• ETS
• capstone course
• curriculum
• high stakes testing

Notes

1 In our sample, the highest correlate with the ETS MFT overall score is a student’s score on the California Critical Thinking Skills Test (CCTST), a required university exit exam also taken in a student’s final semester (.71). This is followed by Best Composite ACT score (.58), our in-house measure of critical thinking based on the Washington State University Rubric typically measured in a student’s junior year (.52), and program GPA (.45). However, a student’s GPA outside the major is a weak correlate (.18).

2 These other measures could themselves be biased on account of gender, so it is difficult to draw any inferences about the ETS from this.

3 When we submit evidence of student learning as a part of our quadrennial program review process, the ETS informs 5 of 18 measures of learning outcomes reported. Other measures include research reports, critical thinking essays, literature reviews, an exit survey, and an alumni survey. Arguably, we are only able to track some of these other outcomes because the ETS substitutes for having to expend the time and effort on other forms on in-house assessment.

4 The remotely proctored 2-hour exam currently costs the university $53 per test (ETS 2023).

5 It is possible in this funding formula to receive approval for a locally developed MFT, but the process for developing the assessment takes three years and two external reviewers to approve.

6 The probability of scoring a perfect score while answering at random is.25¹³⁰

7 Initially, some class time was devoted to review quizzes, but these have been shifted online more recently, to allow more time in class for review and for questions.

8 We now know that the relationship between MFT score and the percentage of questions answered correctly is not quite linear, but the boundaries established are close enough, given the ranges in play.

9 This is where in-house major field tests are an improvement over standardized exams. The loss of comparability trades off with information about exactly where students are struggling. Our program’s practice quizzes make up for this loss to some extent.

10 It is certainly possible to get an in-house MFT peer-reviewed. There are significant advantages to doing so in terms of control and flexibility. Whether stakeholders who are not experts in the field (like state legislatures) or even parents of students trust this process is an open question.

Additional information

Notes on contributors

John LaForest Phillips

John LaForest Phillips is Department Chair for the Political Science and Public Management programs at Austin Peay State University. Prior to becoming chair, he was also the assessment coordinator and curriculum chair for the political science program for five years. His research of late has been on how to improve critical thinking and measures of critical thinking for political science programs. He has B.A.s in Political Economy and German Studies from Williams College and an M.A. and Ph.D. in Political Science from Brown University. Born in Paris, France, Phillips moonlights as a soccer referee for high school and travel soccer on the weekends. One of his proudest moments was refereeing a Special Olympics Unified Team game between Nashville S.C. and Chicago Fire at Geodis Park. He is currently working on learning a fifth foreign language (Swedish) and lives with his partner and two cats in Pleasant View, TN.