Abstract
This paper examines national gaps and trends in geography achievement in eighth grade from 1994 to 2018. Statistical models comprising student- and school-level variables were developed to predict achievement using data provided by the National Assessment of Educational Progress (NAEP). Although there were statistically significant relationships between achievement and school-level attributes such as geographic region and school sector, the magnitudes of the coefficients were relatively minor and inconsistent over time compared with student-level characteristics such as gender, race, ethnicity, and parental education. The results inform current policy directions and efforts to foster educational equity in K-12 geography.
Introduction
Policymakers commonly rely on data from assessment research to make informed decisions about the educational needs of young people in schools (Edelson, Shavelson, and Wertheim Citation2013). Assessments provide evidence of student achievement at multiple levels, from individual performance to data representing district, state, and national outcomes. The results of these assessments inevitably prompt discussions about why some students and jurisdictions perform well, and others less well.
Although the geography education research literature is rich with descriptive accounts of classroom practices and assessments aligned with individual lessons or curriculum units, studies of this nature have limited value for educational policy formulation aimed at achieving broader reforms in schooling, teacher education, and curriculum. For this reason, there have been calls for geographers to begin conducting research using data provided by large-scale school-based assessment studies (Downs Citation2012; Bednarz, Heffron, and Huynh Citation2013).
In the U.S., the National Assessment of Educational Progress (NAEP) collects achievement data from representative samples of students that, depending on the subject, permit generalization to state or national populations of learners. Since 1994, geography has been included in the NAEP program as a national-level assessment of what young people know and can do in the subject. The results of NAEP Geography assessments were cited in the U.S. Government Accountability Office’s 2015 study as evidence that “throughout the country, K-12 students may not be acquiring adequate skills in and exposure to geography, which are needed to meet workforce needs in geospatial and other geography-related industries” (GAO Citation2015). And yet, other than this GAO report, there has been minimal use of NAEP Geography data to address questions about educational policy and practice (Gribben, Schultz, and Woods Citation2019).
One of the key unanswered questions regarding NAEP’s geography reports concerns the nature of the achievement gaps that have appeared with every assessment. Simple disaggregation of achievement data, such as what we commonly see in NAEP reports, accounts neither for relevant covariates nor intercorrelation among student characteristics. In this article, we explore the extent to which national outcomes in geography are attributable to within school differences versus between school differences in student performance. Our statistical analysis is based on all available NAEP Geography datasets at the 8th-grade level to provide an empirical perspective of the significance of student- and school-level predictors of geography achievement in the United States from 1994 to 2018. We focus on the eighth grade as this was the student population available across all NAEP Geography assessments and reflects the level of geography achievement at the end of lower secondary education.
The present study is the first in a planned sequence of studies involving restricted-use NAEP Geography data. For this foundational study, we address three questions:
What is the variation in geography achievement within and between schools, and how does that change over time?
How are student characteristics associated with geography achievement, what are the gaps between student groups, and how has that changed over time?
How are school characteristics associated with geography achievement, what are the gaps between different school types, and how has that changed over time?
To provide the necessary context for understanding the methods we used to address these questions, we turn next to a brief summary of the underlying assessment theory and sampling methods utilized by NAEP to enable statistical inferences to a national population of geography learners. From there, we present the methodology and results of the statistical modeling of NAEP Geography data and discuss how this research advances the field’s knowledge and understanding of geography achievement beyond what we already know from NAEP reports. We conclude by discussing the results and what they suggest should be priorities for research in geography education moving forward, including investigations that NAEP is uniquely positioned to support.
Methodology
NAEP assessment design
NAEP assessments are administered to students who are selected via a multistage, stratified systematic random sampling design (US Department of Education Citation2018). Stage 1 of the sampling design is used to pick schools from the 50 states and other participating jurisdictions (e.g., Bureau of Indian Education schools, the District of Columbia, and Puerto Rico). Candidate schools are next selected in a stratified manner, where strata are created based on school characteristics such as location in rural, suburban, or urban areas, racial composition, and achievement level. Within these strata, schools are selected via systematic random sampling, but such that each school’s probability of selection is proportional to its size. In Stage 2, students are selected via systematic random sampling such that each student in a selected school has an equal probability of being selected to complete the assessment. Inverse probability sampling weights are computed to dampen the impact of oversampled students and heighten the impact of under-sampled students. This is done to align the sample selected with national demographics, thereby creating a representative sample of the target student population (e.g., all 8th graders in the U.S.).
NAEP assessments are administered using a balanced incomplete block design, meaning that students do not respond to every item prepared for the assessment. Rather, students respond to a subset of items, and item-response theory (IRT) procedures are used to generate multiple plausible values (PVs) for each student’s performance on both the overall geography assessment as well as on each specific content area represented in the assessment. These PVs are provided on a scale of 0–500 as well as on the IRT theta scale (approximate mean and standard deviation of 0 and 1, respectively). Each of these PVs represents an independent estimate of a student’s achievement. A given student’s mean PV for the overall assessment, or for any particular content domain, may serve as a point estimate of the student’s corresponding level of achievement overall or for that specific domain. The variance of the student’s PVs is related to the reliability/precision of the point estimate, with greater variance in PVs representing greater uncertainty about the student’s true achievement. These IRT-generated PVs thus account for heterogeneity of measurement precision/differential reliability of scores within the student sample.
Students who participate in NAEP assessments also respond to background questionnaires that are designed to capture information about their demographic attributes (e.g., race and ethnicity), socioeconomic status (e.g., eligibility for free or reduced-price lunch under the National School Lunch Program (NSLP), parental education), informational resources available at home (e.g., books, magazines, encyclopedias), and interests in and perceptions of the assessment subjects (e.g., the degree to which they like studying geography, use maps and globes, and expend effort on geography tests, etc.). These additional data permit researchers to explore relationships between achievement and the characteristics of the students participating in the assessment.
Multilevel statistical modeling
Recognizing that NAEP data are hierarchically structured with students nested in schools, we approached the quantitative analysis of NAEP data in phases using hierarchical linear modeling (HLM; Raudenbush et al. Citation2019) with PVs of achievement on the IRT theta scale (i.e., PVs of theta) as dependent variables. We used the automated features of the HLM 8 software package to run models for each of the available plausible values, average the coefficients, and calculate the correct standard errors according to the formulas of Little and Schenker (Citation1995). The HLM approach builds upon traditional regression by allowing regression parameters to vary across schools, as opposed to assuming a single regression line should explain all schools in the same manner. Conceptually, this approach estimates separate level 1 regression equations within each school. Here, student characteristics function as predictors of achievement. At level 2, the schools effectively become the observations, and level 1 coefficients become outcome variables. At level 2, school-level predictors can then function as predictors of these level 1 coefficients. The HLM software simultaneously estimates all level 1 and level 2 equations to arrive at parameter estimates that maximize the likelihood of the observed data. Through this approach, HLM accounts for school-based clustering effects. All analyses were conducted using NAEP sampling weights that account for the sampling design.
To address the first research question, we calculated the intraclass correlation coefficient (ICC) for the unconditional (null) model in each assessment year as a ratio of the amount of variance due to schools relative to the total variance. ICC values range between 0 and 1 and are generally interpreted as representing the proportion of total variance in the outcome that is accounted for by clustering. Higher values reflect greater between-group variability and lower values represent greater within-group variability (i.e., geography achievement among students within a school).
To address the second and third research questions, a consistent algorithm was followed in each assessment year to identify potential shifts in the strengths, directions, and statistical significance of relationships between geography achievement and student- and school-level factors over time. We modeled PVs of theta for the NAEP composite geography score as the outcome variable. Level-1 student predictors included gender, race/ethnicity, free/reduced-price lunch eligibility, English language learner (ELL) status, individualized education plan (IEP) status, age, number of books in home, and parental education. Level 2 school predictors included geographic region, urbanicity, percent free/reduced-price school lunch, percent racial and ethnic composition, percent ELL composition, and public/private school.
Reported regression coefficients are unstandardized effects of each predictor on the theta-scale achievement outcome. These coefficients represent estimated net effects conditional on the other variables included in the models. Because achievement is on the theta scale, each coefficient is interpreted as the predicted change in standard deviations of student achievement per unit change of the predictor while holding all other predictors constant. The regression coefficients for dichotomous predictor variables therefore represent estimated mean differences between the reference group and the group that is the focus of the predictor, holding other predictors constant. For this reason, Cohen’s (Citation1988) very general guidelines for the interpretation of effect size d are potentially useful; values of .10, .25, and .40 represent small, medium, and large effects, respectively. All effects, including intercepts, were treated as random in these HLM models. The model specifications for each yearFootnote1 are shown in .
Table 1. Variable specifications for NAEP Geography, 1994–2018.
Results
Within vs. between school variation in geography achievement
The unconditional ICC results for each study year ranged from 0.29 to 0.34 (). Thus, between-school differences in geography achievement consistently explain approximately 1/3 of the variance in geography achievement outcome over the study period. This means most of the variance in achievement is attributable to the differential performance of students within schools.
Table 2. Unconditional intraclass correlation coefficient results for NAEP Geography, 1994–2018.
Next, we present the achievement estimates associated with the student- and school-level predictor variables. For each assessment year, two results are reported: the regression results () and a set of descriptive statistics () for the full model.
Table 3. Regression results for variables predicting 8th grade student achievement in geography, 1994.
Table 4. Regression results for variables predicting 8th grade student achievement in geography, 2001.
Table 5. Regression results for variables predicting 8th grade student achievement in geography, 2010.
Table 6. Regression results for variables predicting 8th grade student achievement in geography, 2014.
Table 7. Regression results for variables predicting 8th grade student achievement in geography, 2018.
Table 8. 1994 NAEP Geography Level 1 and Level 2 descriptive statistics.
Table 9. 2001 NAEP Geography Level 1 and Level 2 descriptive statistics.
Table 10. 2010 NAEP Geography Level 1 and Level 2 descriptive statistics.
Table 11. 2014 NAEP Geography Level 1 and Level 2 descriptive statistics.
Table 12. 2018 NAEP Geography Level 1 and Level 2 descriptive statistics.
Relationships between geography achievement and student-level predictors
Level one predictor variables had the most consistent results across the study period.
Gender
There was a statistically significant relationship between gender and geography achievement in each assessment year. Compared with males, female students scored about .06 standard deviations lower on average on NAEP Geography in 1994; .14 standard deviations lower in 2001; .21 standard deviations lower in 2010; and .19 standard deviations lower in 2014 and 2018. Hence, the effect sizes for gender trended from relatively small to medium-sized over the study period.
Race and ethnicity
Since 1994, there has been a statistically significant relationship between geography achievement and race and ethnicity. First, compared with White students, Black students scored about .62 standard deviations lower on average on NAEP Geography in 1994; .60 standard deviations lower in 2001; .45 standard deviations lower than 2010; .54 standard deviations lower in 2014; and .60 standard deviations lower in 2018. In contrast, the conditional differences between White and Hispanic students trended from relatively large to medium-sized over time. Hispanic students scored about .35 standard deviations lower on average on NAEP Geography in 1994, .17 standard deviations lower in 2001; .14 standard deviations lower in 2010; .16 standard deviations lower in 2014; and .21 standard deviations lower in 2018.
Significant conditional differences in geography achievement also existed in some years for other racial and ethnic groups compared with White students. Asian students scored about 0.17 standard deviations higher on average on NAEP Geography in 2014 and about 0.15 standard deviations higher in 2018. Native American students scored about .29 standard deviations lower in 1994 and about .31 standard deviations lower in 2010. Pacific Islander students scored about .75 standard deviations lower in 1994. Students categorized as other race/ethnicity scored about .43 standard deviations lower in 2001. This category changed to “two or more races” in 2010, but a significant difference was no longer observed.
Eligibility for free or reduced-price lunchFootnote2
Relationships between geography achievement and eligibility for free/reduced-price school lunch were statistically significant in all years. The coefficients consistently represent medium-sized differences. Conditional on all other covariates, eligible students scored about .22 standard deviations lower on average than ineligible students on NAEP Geography in 1994; .15 standard deviations lower in 2001Footnote3; .14 standard deviations lower in 2010; .19 standard deviations lower in 2014; and .24 standard deviations lower in 2018.
English language learners (ELL)
Conditional differences between ELL and non-ELL students were consistently large over the study period. English language learners scored about .67 standard deviations lower on average on NAEP Geography in 1994; .47 standard deviations lower in 2001; .64 standard deviations lower in 2010; .74 standard deviations lower in 2014; and .67 standard deviations lower in 2018.
Students on an individualized education plan (IEP)
Similarly, conditional differences between IEP and non-IEP students were consistently large over the study period. IEP students scored about .73 standard deviations lower on average than students without an IEP on NAEP Geography in 1994; .6 standard deviations lower in 2001; .64 standard deviations lower in 2010; .78 standard deviations lower in 2014; and .72 standard deviations lower in 2018.
Modal age
The smallest conditional differences at the 8th grade were for modal age. Students who were above modal age scored about .06 standard deviations lower on average on NAEP Geography in 1994; .09 standard deviations lower in 2001; .07 standard deviations lower in 2010; and .06 standard deviations lower in 2014 and 2018.
Parental education and books in home
There were statistically significant relationships between geography achievement and two variables that commonly serve as proxy measures of socioeconomic background. We consistently estimated an increase in geography achievement with each unit increase in parental education and books in home. Effect sizes for books in the home were consistently larger than the effect sizes for parental education over the study period.
To summarize, the following variables were statistically significant predictors of geography achievement over the study period: gender, race/ethnicity (particularly Black and Hispanic relative to White students), free/reduced-price lunch eligible (FRLE), ELL, IEP, above modal age, parental education, and books in the home. depicts these gaps and trends.
Figure 1. Gaps and trends in geography achievement, in theta units by various student groups, 1994–2018.
Relationships between geography achievement and school-level predictors
Compared with student-level predictors, there was little consistency in the results of school-level predictors across the study period.
School composition
There was negligible evidence of school composition (i.e., racial/ethnic composition, percent ELL, and percent free/reduced-price lunch eligible) having a significant relationship with geography achievement over the study period, conditional on all other covariates.
Urbanicity
The only assessment year that had a significant conditional difference based on urbanicity was in 2018. Compared with schools in large cities, schools located in suburbs scored about .09 standard deviations higher on average on NAEP Geography while rural schools scored about .13 standard deviations higher.
Region
There was a statistically significant relationship between geography achievement and region in 1994 and 2001, conditional on all other covariates. Compared with schools in the northeast, schools in the central region scored about .26 standard deviations higher on NAEP Geography in 1994, on average. Schools in the western region scored about .16 standard deviations lower than schools in the northeast in 2001, yet in 2018 schools in the west scored .09 standard deviations higher.
School sector
There was a statistically significant relationship between geography achievement and school sector in 1994 and 2001. Compared with public schools, private schools scored about .22 standard deviations higher, on average, on NAEP Geography in 1994, conditional on all other covariates. However, in 2001, private schools scored about .15 standard deviations lower.
Missing data procedures
Variables with >5 percent missing data were parental education (in all years) and school composition (percent free/reduced-priced lunch and percent ELL in all years, and percent Black and percent Hispanic in 1994 and 2001). We manually calculated these missing school composition values in SPSS by aggregating student-level variables, which yielded complete data for all variables except percent free/reduced lunch in 2001, 2010, and 2014.
When models were run using only predictors with ≤ 5 percent missing data, there was a change in statistical significance for the following variables: above modal age in 1994; mid-size city, free/reduced-price lunch, free/reduced lunch NA, and above modal age in 2001; suburb and percent free/reduced-price lunch in 2010; and suburb, west, and Pacific Islander in 2018. While statistical significance changed for these predictors, the actual change in effect size (i.e., the size of the coefficient) was minor between the models using only complete variables and these models with missing data for parental education and percent free/reduced lunch. For predictors where statistical significance was unchanged, there were only minor differences in the magnitudes of the estimated effects between the two types of models.
Discussion
The purpose of the present study was to initiate large-scale quantitative research into gaps and trends in geography achievement using restricted NAEP datasets. Specifically, we predicted geography achievement using student- and school-level characteristics. The study does not address causality, only conditional differences in achievement. In this section, we first discuss our findings in relation to what other education studies have documented with regard to inequality in student outcomes. We continue with a brief discussion of research into inequality in educational opportunity to set the stage for further research.
Inequality in student outcomes
Our models of geography achievement provide empirical estimates that represent statistically significant, independent (net) effects of student- and school-level predictor variables. When comparing the level one and level two estimates, the results for the student-level predictors were the most consistent and often much larger. This is an important discovery that will be critical to future attempts to improve geography achievement by giving special attention to the characteristics of the learners, as opposed to focusing on differences in school types.
The results of our study are largely consistent with the results from other disciplines that have developed models of achievement using NAEP data. Students of color, females, and students who are eligible for free or reduced-price lunch often attain significantly lower scores than their counterparts on NAEP assessments for U.S. history (Heafner and Fitchett Citation2015; Fitchett, Heafner, and Lambert Citation2017) and economics (Heafner, VanFossen, and Fitchett Citation2019). The same is true for students who receive language and learning accommodations and whose parents did not attend college. These studies also documented achievement gaps by race and ethnicity after controlling for learning accommodation and socioeconomic status.
The National Academies study Monitoring Educational Equity points to the especially serious challenges in education experienced by students with learning disabilities and those who are less fluent in English (National Academies of Sciences, Engineering, and Medicine Citation2019). Indeed, these student groups had some of the largest gaps in geography achievement over our study period. Students with learning disabilities are much less likely to participate in advanced or honors coursework and far more likely to experience less time learning standards-based content, less instructional time, and less content coverage than students without learning disabilities (Kurz et al. Citation2014). Similarly, ELL students may experience linguistic barriers that substantially reduce the time they spend learning academically rigorous content (Abedi and Herman Citation2010; Callahan and Shifrer Citation2016; Umansky Citation2016).
In contrast to the student-level predictors, there were few consistently significant relationships between geography achievement and school-level predictors, conditional on the student-level variables included in our study. School composition variables with regard to student demographics did not consistently predict achievement in geography. This is a notable finding because other NAEP social studies research has reported incrementally negative associations between social studies achievement and percentages of students who are Black and Hispanic at the school level (Fitchett, Heafner, and Lambert Citation2017; Heafner and Fitchett Citation2015; Heafner, VanFossen, and Fitchett Citation2019). Here again, further research is needed to explore the possible reasons behind these subject area differences.
Inequality in educational opportunity
A major unknown in need of further research concerns the extent to which the observed inequalities in student geography outcomes can be explained by inequalities in educational opportunity. We feel the most productive path forward is to frame research into this issue using the concept of opportunity to learn (OTL), which emerged several decades ago from large-scale assessment studies administered by the International Association for the Evaluation of Educational Achievement (IEA) (Schmidt, Zoido, and Cogan Citation2013). OTL has been defined by some researchers as the notion that “the time a student spends learning something is related to what the student learns” (Schmidt, Zoido, and Cogan Citation2013, 10). Beyond time, conceptions of OTL considers relationships between student achievement and curriculum coverage and exposure (e.g., Schmidt and Maier Citation2009), teacher expertise and beliefs (e.g., Blömeke et al. Citation2014), and instructional delivery (e.g., Stevens Citation1996; Tarr et al. Citation2006).
Addressing disparities in OTL has become a driver of educational reform and policy developments in recent decades (Schmidt, Burroughs, Zoido and Houang Citation2015; McDonnell, Citation1995). The next phase of our NAEP Geography research will incorporate contextual OTL variables into the current two-level predictive model to investigate the extent that student achievement varies systematically with OTL-related factors. We anticipate this future research will advance the field’s understanding of the extent that opportunity to learn variables account for national gaps and trends in geography achievement since 1994. As a byproduct, this future work will also reveal the extent to which achievement gaps and contextual effects remain after controlling for OTL variables.
Conclusion
Schooling is commonly viewed as a great equalizer – a means available to anyone to “acquire the requisite knowledge to escape and achieve a better position in society economically, through better employment, as well as politically or socially as a well-equipped and informed citizen” (Schmidt, Zoido, and Cogan Citation2013). Our initial statistical analysis of NAEP Geography makes it clear that all is not equal in U.S. geography education. The coefficients estimated in our study are indicators of inequality in student outcomes, with weak geography knowledge acquisition persisting over a quarter century to the primary detriment of young people who are female, Black, Hispanic, or living in less privileged social and economic circumstances.
Our research findings come at a time when teachers are facing new professional demands associated with contemporary education policy and an increasingly diverse K-12 student body. According to a recent National Academies report (National Academies of Sciences, Engineering, and Medicine Citation2020), teachers are increasingly expected to have deep subject matter expertise; prepare students for the workforce by engaging them in authentic applications of content to solve problems; practice culturally responsive pedagogy by accounting for student diversity and the experiences students bring to school; and enact curriculum goals set in state and district-level learning and content standards. Geographers are accordingly taking steps toward supporting geography teachers in these ways, such as through research-practice partnerships in culturally diverse school and university environments (Dony et al. 2019; Solem et al. Citation2021) and curriculum approaches tailored to student geographies and aspirations (Larsen et al. Citation2021; Schlemper et al. Citation2018).
As geography education researchers begin the quest to identify the root causes of inequality in student outcomes, they will do so at a time of massive global changes to Earth’s climate, ecosystems, and land-use. These changes are so widespread that many scientists argue we are now living in a new ‘human epoch’ – the Anthropocene (Lewis and Maslin Citation2015). Beyond the climate crisis are myriad other indicators of a rapidly changing planet: the contemporary and connected challenges implied by the ‘age of acceleration’, including the ‘new space race’ generating enormous volumes of spatial data and greater varieties of digital mapping and geospatial technologies (Baiocchi and Welser Citation2015; Downs Citation2016); globalization and the information society in the digital age; and the rise of an ‘impulse society’ associated with fast capitalism (Walkington et al. Citation2018).
The NAEP program is the nation’s only large-scale assessment study that measures the geography that geographers believe students should know and be able to do. NAEP Geography data are and will likely remain the best available source to determine the extent that young people are acquiring knowledge to become citizens capable of applying geography to address complex social and environmental problems. The global changes associated with the Anthropocene are fundamentally geographical in nature and have become part of young people’s lives everywhere, underscoring the imperative of providing equitable access to a geography education of the highest epistemic quality.
Acknowledgments
Dr. Russell Weaver provided helpful assistance with the description of the NAEP sampling design and assessment procedures.
Additional information
Funding
Notes on contributors
Michael Solem
Michael Solem is a Professor of Geography at Texas State University. He co-directs the National Center for Research in Geography Education and serves the American Association of Geographers (AAG) as Senior Advisor for Geography Education. In 2015, he was awarded AAG Gilbert Grosvenor Honors in Geographic Education.
Phillip Vaughan
Phillip Vaughan is a research scientist with the Methodology, Measurement, and Statistical Analysis (MMSA) division at Texas State University. He specializes in structural equation modeling (SEM), hierarchical linear modeling (HLM), and other advanced statistical techniques.
Corey Savage
Corey Savage is a researcher at the American Institutes for Research. He studies civic education, teacher policy, and curricular reform.
Alessandro S. De Nadai
Alessandro S. De Nadai is an Assistant Professor of Psychology at Texas State University. His work seeks to translate new developments in data science into improved behavioral outcomes.
Notes
1 In 1994 the books in home variable was binary >25 books (comparison category <25 books). Beginning in 2001 the books in home variable became ordinal. 1994 and 2001 the urbanicity variables were mid-sized city, large town, small town, and rural (comparison category = large city). Beginning in 2010 the urbanicity variables were suburban, town, and rural (comparison category = city).
2 NAEP classifies students as “free or reduced-price lunch N/A” in two cases: (1) school records were not available, or (2) the school did not participate in the National School Lunch program. Increased accuracy in school reporting has reduced the percentage of students classified in this group since 1994.
3 Statistical significance was observed in the negligible missing data model.
References
- Abedi, J., and J. Herman. 2010. Assessing English language learners’ opportunity to learn mathematics: Issues and limitations. Teachers College Record 112 (3):723–46.
- Baiocchi, D., and W. Welser, IV. 2015. The democratization of space: New actors need new rules. Foreign Affairs 94 (3):98–104.
- Bednarz, S., S. Heffron, and N.T. Huynh (eds.). 2013. A road map for 21st century geography education: Geography education research (a report from the Geography Education Research Committee of the road map for 21st century geography education project). Washington, DC: Association of American Geographers.
- Blömeke, S., F. Hsieh, G. Kaiser, and W. Schmidt. 2014. International perspectives on teacher knowledge, beliefs and Opportunities to Learn. Dordrecht: Springer.
- Callahan, R. M., and D. Shifrer. 2016. Equitable access for secondary English learner students: Course taking as evidence of EL program effectiveness. Educational Administration Quarterly: EAQ 52 (3):463–96.
- Cohen, J. 1988. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers.
- Dony, C. C., A. Nara, S. Rey, M. Solem, and T. Herman. 2019. Encoding geography: Building capacity for inclusive geocomputational thinking with geospatial technologies. California State University Geospatial Review 16:2–3.
- Downs, R. 2016. Bringing geography back to life: the role of the geospatial revolution in schools. Geography 101 (2):77–84. doi: https://doi.org/10.1080/00167487.2016.12093988.
- Downs, R. M. 2012. The NAEP Geography Report 2010: What will we do next? Journal of Geography 111 (1):39–40. doi: https://doi.org/10.1080/00221341.2011.615412.
- Edelson, D. C., R. J. Shavelson, and J. A. Wertheim (eds.). 2013. A road map for 21st century geography education: Assessment (a report from the Geography Education Research Committee of the road map for 21st century geography education project). Washington, DC: National Geographic Society.
- Fitchett, P. G., T. L. Heafner, and R. G. Lambert. 2017. An analysis of predictors of history content knowledge: Implications for policy and practice. Education Policy Analysis Archives 25:65. doi: https://doi.org/10.14507/epaa.25.2761.
- Government Accountability Office. 2015. K-12 education: Most eighth grade students are not proficient in geography (GAO-16-7). Washington, DC: United States Government Accountability Office.
- Gribben, M. A., S. R. Schultz, and A. P. Woods. 2019. National-only NAEP results (arts, civics, economics, geography, technology and engineering literacy, and U.S. history): Who uses them, why, and how. Human Resources Research Organization Report 2019 No. 007, Alexandria, VA: Human Resources Research Organization.
- Heafner, T. L., and P. G. Fitchett. 2015. An opportunity to learn US history: What NAEP data suggest regarding the opportunity gap. The High School Journal 98 (3):226–49. doi: https://doi.org/10.1353/hsj.2015.0006.
- Heafner, T. L., P. J. VanFossen, and P. G. Fitchett. 2019. Predictors of students’ achievement on NAEP-economics: A multilevel model. The Journal of Social Studies Research 43 (4):327–41. doi: https://doi.org/10.1016/j.jssr.2019.01.003.
- Kurz, A., S. N. Elliott, C. J. Lemons, N. Zigmond, A. Kloo, and R. J. Kettler. 2014. Assessing opportunity to learn for students with disabilities in general and special education classes. Assessment for Effective Intervention, 40(1): 24–39.
- Larsen, T., M. Solem, J. Zadrozny, and R. G. Boehm. 2021. Contextualizing powerful geographic knowledge in higher education: Data-driven curriculum design to interweave student aspirations with workforce applications. International Research in Geographical and Environmental Education, 516: 1–14. doi: https://doi.org/10.1080/10382046.2021.1902622.
- Lewis, S., and M. Maslin. 2015. Defining the anthropocene. Nature 519 (7542):171–80. doi: https://doi.org/10.1038/nature14258.
- Little, R. and N. Schenker. 1995. Missing data. In G. Arminger, C. C. Clogg & M. E. Sobel (Eds.), Handbook of Statistical Modeling for the Social and Behavioral Sciences (pp. 39–76). New York: Plenum Press.
- McDonnell, L. M. 1995. Opportunity to learn as a research concept and a policy instrument. Educational Evaluation and Policy Analysis 17 (3):305–22. doi: https://doi.org/10.3102/01623737017003305.
- National Academies of Sciences, Engineering, and Medicine. 2019. Monitoring educational equity. Washington, DC: The National Academies Press. doi: https://doi.org/10.17226/25389.
- National Academies of Sciences, Engineering, and Medicine. 2020. Changing expectations for the K-12 Teacher Workforce: Policies, preservice education, professional development, and the workplace. Washington, DC: The National Academies Press. doi: https://doi.org/10.17226/25603.
- Raudenbush, S. W., A. S. Bryk, Y. F. Cheong, R. Congdon, and M. Du Toit. 2019. HLM 8: Hierarchical linear and nonlinear modeling. Chicago, IL: Scientific Software International.
- Schlemper, M. B., V. C. Stewart, S. Shetty, and K. Czajkowski. 2018. Including students’ geographies in geography education: Spatial narratives, citizen mapping, and social justice. Theory & Research in Social Education 46 (4):603–41. doi: https://doi.org/10.1080/00933104.2018.1427164.
- Schmidt, W. H., N. A. Burroughs, P. Zoido, and R. T. Houang. 2015. The role of schooling in perpetuating educational inequality: An international perspective. Educational Researcher 44 (7):371–86. doi: https://doi.org/10.3102/0013189X15603982.
- Schmidt, W. H., and A. Maier. 2009. Opportunity to learn. In Handbook of education policy research, ed. G. Sykes, B. Schneider, and D. N. Plank, 541–59. New York, NY: Routledge.
- Schmidt, W. H., P. Zoido, and L. S. Cogan. 2013. Schooling matters: Opportunity to learn in PISA 2012. OECD Education Working Papers, No. 95, Paris, France: OECD Publishing.
- Solem, M., C. Dony, T. Herman, K. León, A. Magdy, A. Nara, W. Ray, S. Rey, and R. Russell. 2021. Building Educational Capacity for Inclusive Geocomputation: A Research-Practice Partnership in Southern California. Journal of Geography 120 (4):152–9. doi: https://doi.org/10.1080/00221341.2021.1933140.
- Stevens, F. I. 1996. The need to expand the opportunity to learn conceptual framework: Should students, parents, and school resources be included? Paper presented at the annual meeting of the American Educational Research Association, New York, NY, April.
- Tarr, J. E., Ó. Chávez, R. E. Reys, and B. J. Reys. 2006. From the written to the enacted curricula: The intermediary role of middle school mathematics teachers in shaping students’ opportunity to learn. School Science and Mathematics 106 (4):191–201. doi: https://doi.org/10.1111/j.1949-8594.2006.tb18075.x.
- Umansky, I. M. 2016. Leveled and exclusionary tracking: English learners’ access to academic content in middle school. American Educational Research Journal 53 (6):1792–833. doi: https://doi.org/10.3102/0002831216675404.
- US Department of Education. 2018. NAEP technical documentation on the Web. US Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP). https://nces.ed.gov/nationsreportcard/tdw/sitemap.aspx (Accessed 31 October, 2021).
- Walkington, H., S. Dyer, M. Solem, M. Haigh, and S. Waddington. 2018. A capabilities approach to higher education: Geocapabilities and implications for geography curricula. Journal of Geography in Higher Education 42 (1):7–24. doi: https://doi.org/10.1080/03098265.2017.1379060.