Improving introductory economics course content and delivery improves outcomes for women

Abstract

The presentation of economics in introductory courses has been highlighted as potentially exacerbating the underrepresentation of women in economics. The authors study the impact of a gender-neutral change in content and instruction in introductory economics courses intended to increase student engagement. By implementing meaningful applied problems and structured group work, their intervention focuses on the students’ perceptions of “what” economics is and “how” economics is used. Using institutional data from 8,727 students over nine semesters, they find that the intervention improved women’s grades relative to men’s in both Introductory Microeconomics and Macroeconomics and eliminated underperformance by women in Introductory Macroeconomics relative to men at baseline. These effects are evidence that course content and delivery impact the experiences and outcomes of female students in economics education.

Keywords:

• Economics instruction
• gender
• inclusion
• undergraduate education

JEL CODES:

• A22; I21; I29; J16

Economics education has a problem—students, instructors, and society are all pointing out how inaccessible and irrelevant introductory economics education has become relative to what is interesting to students and needed in the real world (Inman 2014; Pühringer and Bäuerle 2018; Reisz 2016; Wolfers 2019; Thornton 2020; Oudija 2022). This problem is not new; for decades, people have been pointing out the limitations of how introductory economics is taught (e.g., Heyne 1995), and while some gains have been made, even in 2020, the main methods of teaching introductory economics are “chalk-and-talk” lectures that do not address topics of interest to students (Asarta, Chambers, and Harter 2021). There have been widespread calls for moving away from pure lecture approaches in university instruction, moving instead to problem-based learning (Hmelo-Silver 2004) or collaborative learning (Chi and Wylie 2014). While many individual instructors and departments have made efforts to increase the interactivity and relevance of introductory economics education to their students, large time and cost barriers, as well as uncertainty about the benefit of such changes, have prevented broader adoption of these improved teaching techniques (Henderson and Dancy 2007).

However, there may be an additional benefit to adopting these more interactive and relevant teaching methods—improving gender diversity throughout the economics pipeline.¹ Economics suffers from a gender representation problem—the further one goes along the academic pathways from introductory courses to full professorships, the proportion of women at each step decreases (Avilova and Goldin 2018; Buckles 2019).² The traditional way in which economics is taught likely has both causal and exacerbating effects on this underrepresentation (Bayer et al. 2020). For example, biased and narrow content matter focusing on theory, ignoring topics of greater interest to women, and diminishing the contributions of women to economics theory and practice negatively influences perceptions of the relevance of economics to female students (Ferber 1995; Jensen and Owen 2001). Non-inclusive teaching techniques, i.e., the heavy use of lecture, individual learning through reading and homework, and few high-stakes exams, that differentially diminish a sense of self-efficacy in female students (Jensen and Owen 2001) are also common in current introductory economics pedagogy (Asarta, Chambers, and Harter 2021; Gartner and Schneebaum 2023).³ Furthermore, while women typically receive higher grades than men in high school and university as an overall GPA (Conger 2015; O’Dea et al. 2018) and as grades within many STEM courses (Matz et al. 2017; O’Dea et al. 2018), they have tended to receive lower grades than men in introductory economics courses (Anderson, Benjamin, and Fuss 1994; Borg and Stranahan 2002). Given that performance in introductory economics courses is a strong determinant of course persistence and degree selection (Ahlstrom and Asarta 2019; Arnold 2020), the combination of having higher grades in other courses and lower grades in economics courses could be a particularly strong signal to major in other areas, particularly in combination with other signals of not belonging that women already face (Wang, Eccles, and Kenny 2013; Kugler, Tinsley, and Ukhaneva 2021).

In this article, we provide evidence that the combination of (1) changing course content to be more focused on real-world applications and (2) changing pedagogical methods to be more interactive learning together improve the performance of women students in introductory economics courses to better match their overall stronger performance in most university courses, which could then increase rates of their further study of economics at the undergraduate level. The University of Pittsburgh’s economics department implemented an intervention in the recitations of multi-sectioned large Introductory Microeconomics and Macroeconomics courses in which students engaged in more group learning, with a greater focus on contributing ideas rather than just correct answers and engaging with real-world data and examples. While these changes to recitations were primarily designed to generally increase all students’ engagement in recitations and to help all students understand the broader implications of the material learned in class (Josephson et al. 2019), we find that women’s overall course grades in these economics courses increased, relative to men’s, after the intervention was put into place. Women’s grades increase by about 0.2 (on a 4.0 scale) after the intervention, with no change in men’s grades. This is nearly the equivalent of moving from a minus to a whole grade or from a whole grade to a plus.⁴

This article adds to the nascent literature understanding of how course content and delivery impact diversity in economics. There are very few papers addressing this specific topic. Owen and Hagstrom (2021) evaluate the impact of changing the undergraduate curriculum in economics to include classes focused on empirical economics and economic inequality and find that women receive higher grades under the revised curriculum. Espey (2018) finds that men and women perform differently in Introductory Microeconomics courses, including team-based learning, a specific form of interactive learning, with women benefiting from teams that are more cooperative. While both of these interventions are productive, they both require substantial changes, i.e., adding courses to the curriculum and changing the way lectures are taught and graded, which may be prohibitively difficult for many departments. We provide evidence on a less-invasive intervention that could be conducted within the existing curricula structure and only changed how recitations were conducted and graded. We show that this relatively less intense change has positive benefits for women’s outcomes in these introductory economics courses.

Our article also contributes to the extensive literature studying other classroom interventions geared at improving diversity in economics. In 2015, the Undergraduate Women in Economics Challenge (UWE) provided funds for a randomized control trial of interventions to improve the gender gap in economics in undergraduate institutions in the United States, including disseminating information about the major and related career fields, providing female role models and mentoring, and explicitly encouraging women to major in economics (Avilova and Goldin 2018).⁵ However, the results of such interventions are mixed: Li (2018) found that information about academic and professional career paths, along with information about relative academic standing and a “nudge” from a professor or advisor to encourage women to continue in economics increased the number of women who chose to take economics classes beyond the introductory courses, while Pugatch and Schroeder (2021) found that such nudges had no effect on women while increasing the likelihood that men would continue in economics. Porter and Serra (2020) found that short exposure to female alumnae role models in introductory classes increased the likelihood that women would major in economics, while men’s likelihood was not affected.⁶ Most of these interventions might best be considered small non-pedagogical changes focused on encouraging women specifically to continue study in economics and choose economics-related careers. We provide evidence on an alternative tack for improving women’s representation in economics that acts not on “who” economists are but “what” economics is.

We build from the broader literature in research into pedagogical advancements that increase diversity in non-diverse fields. In STEM disciplines with similar or worse gender gaps, such as physics, computer science, and many areas of engineering, a variety of interventions have been tried to improve gender gaps by implementing changes in undergraduate courses. Many of these interventions have focused on showing women that they belong in that discipline, such as mentoring and same-gender instructors (Bettinger and Long 2005; Carrell, Page, and West 2010; Griffith 2010), while others have focused on changing the pedagogical methods to emphasize interactive learning and collaboration in introductory physics courses (Lorenzo, Crouch, and Mazur 2006). While there is evidence that these techniques have been effective in reducing gender gaps in STEM disciplines, there is limited evidence as to their efficacy in economics.

Intervention

As part of a university-wide effort to improve large lecture classes, in the fall of 2018, we implemented an intervention that changed the pedagogical methods and content in our Introductory Microeconomics and Macroeconomics classes with the goal of increasing student engagement and having students utilize more higher-level thinking skills. Studies have shown that active learning, a learning process in which students are participating in ways other than simply listening or taking notes, increases examination scores and reduces failure rates in STEM disciplines (Freeman et al. 2014). In particular, collaborative learning strategies are especially effective (Springer, Stanne, and Donovan 1999; Chi and Wylie 2014). In addition, there is ample evidence in the STEM pedagogy literature that learning is improved by the use of real-world problems (for a review, see Hmelo-Silver 2004). Research also shows that these practices increase the retention of women in these disciplines (Lewis 2011; Rodriguez, Potvin, and Kramer 2016; Di Tommaso et al. 2021). We wanted to exploit both of these potential pedagogical improvements (i.e., collaborative active learning and real-world problems).

We decided to focus on the recitation sections as the locus of these changes because research from the STEM disciplines shows that interactive learning is logistically challenging in large classes and tends to be more effective in smaller class settings (Walker and Warfa 2017). Our transformation of the recitations had two main components: (1) new lesson plans and materials for all of the weekly recitations in both Microeconomics and Macroeconomics that incorporated interactive and collaborative learning and utilized real-world examples and data, and (2) enhanced teacher training for the graduate student teaching assistants who lead the recitations, to train them in teaching using techniques that support interactive and collaborative learning.

Recitation plans

Before the intervention, the content and format of the recitations had varied somewhat by faculty instructor and graduate teaching assistant. However, most commonly, the teaching assistant answered questions and demonstrated solutions for the assigned homework problems or additional practice problems that were similar to the homework problems. Sometimes they gave a review lecture. Students received 5–10 percent of their course grade for attending recitation. The learning was generally not active or interactive. Students listened, took notes, or worked individually on assigned problems.

In order to maximize the benefit of the new recitation plans, we collaborated with instructional design experts at the University’s Center for Teaching and Learning to design plans that included active learning, applying economics to rich real-world problems, and analysis that required higher-level thinking. We specifically planned activities that included using mathematical economic models for problem-solving, applying these models to analyze economic situations in the real world, analyzing economic data, and practicing the application of economic concepts and skills in novel situations. Instructors across the department each worked on creating one or more specific recitations using the design plan. The design plan forced each instructor to think carefully about how each recitation activity fits into the learning objective and how to effectively scaffold each step of these activities. Completed recitation plans served as a detailed guide for the recitation teaching assistants: 1) clear and specific learning objectives for the class that tie specifically to each included activity; and 2) a clear guide for leading the class, including preparation before the recitation, an outline with guidelines for time to be spent on each task, and notes about possible pitfalls. In most cases, group work paused intermittently for larger group discussion, and specific notes were included about how to lead this larger group discussion. Each lesson included some work to be handed in at the end of the session. This helped to guarantee participation from all students.

Activities included analyzing current events found in newspaper articles and videos, analyzing real economic data, participating in economic simulations and experiments, discussing readings, and analyzing case studies (see corresponding author’s website for an example recitation plan: https://www.malloryavery.com/research). These included classroom simulations and experiments in which everyone could participate and problems about news stories or issues that would be familiar and relevant to most students, such as the price of concessions in sports stadiums and the impact of ride-sharing services. The goal was to increase engagement and develop higher-order thinking skills while changing the focus of course material away from the usual hypothetical economics examples and toward more real-world topics with a broader interest range and greater applicability to current social issues, business, and other topics of students’ current interests.

In each recitation session, we gave students the opportunity to work with their peers in pairs or small groups. Groupwork pragmatically reduces the number of independent learners the TA needs to support, and a group will be less likely to get stuck than individuals working alone. Further, group work has generally been found to allow students to learn more and higher-level content through interactive discussion (Chi and Wylie 2014) and master more complex skills (Kirschner, Paas, and Kirschner 2009). The group work also allowed students to build connections with their peers and with the teaching assistant, which could potentially improve their sense of belonging and present economics as a discipline that involves working with others—two factors that have been identified as relevant to attracting women to economics (Werner et al. 2005).

Each recitation had a learning product that could be assessed at the end of the session. Students received 5–10 percent of their course grade for this product rather than just for attendance. This was particularly important for assessing the higher-level skills in the course as the large lecture size made it difficult to include such non-standardized material on the exam assessments.

Teaching assistant (TA) training

Because the intervention was delivered in the recitation sections by TAs and it involved a kind of pedagogical approach that was unfamiliar to many of the TAs, the training of the TAs was a very important component in the success of the intervention. The economics department had an existing course for graduate students on teaching economics, which was required for all graduate students. The course used an apprenticeship model focused on discussion, observation of practice lessons, and feedback but did not include instruction on pedagogical theory and techniques. We modified this course with the help of the Center for Teaching and Learning to focus on active learning teaching techniques with sessions on presentation skills, facilitating group work, and teaching case studies.

Faculty instructors then worked closely with the TAs on preparing to teach each lesson. They met weekly with their TAs to preview that week’s recitation plan—explaining the teaching methods that were to be used and answering questions about the learning objectives, methods, and materials for that recitation. In these meetings, faculty also collected TA feedback on the previous week’s recitations, encouraging the TAs to reflect on the extent to which the recitation had met the learning objectives of the plan and soliciting their suggestions on how to improve the lesson. The primary goal of these meetings was to facilitate high-quality implementation of the new lesson plans and ensure that student engagement and learning goals were being met.

Intervention summary

See table 1 for a summary of how the intervention did and did not change the students’ experience in the course. The result for students was a course with greater expectations for student engagement during recitations, with more active and collaborative learning. For recitation TA instructors, the recitation followed a carefully crafted plan that they reviewed with the instructor prior to the recitation.

Table 1. Intervention summary.

Class Element	Before Intervention	After Intervention
Lecture pedagogy and student behavior	Large lecture (200–260 students) with limited student interaction (i.e., predominantly listening and note-taking)
Lecture content	Standard introductory economic theory
Homework	Multiple choice and problem solving (numerical answer)
Exams	Multiple choice and problem solving (numerical answer)
Recitation % of final grade	5–10 percent of course grade (set by instructor)
Recitation grade basis	Attendance	Completion of group work
Recitation instructor (TA) behaviors	Homework review, lecture review, and demonstration of practice problems	Posing questions, facilitation of group work, coaching problem solving
Recitation student behavior	Listening and note taking, asking clarifying questions, or individual problem solving	Active learning and group work
Recitation content	Theory and abstract hypothetical examples	Theory and detailed real-world cases (e.g., news articles or real economic data)
Recitation instructor (TA) training	Focused on organizing a coherent lecture and designing a syllabus	Focused on supporting active and collaborative learning

Notes: This table describes how students’ experiences in the introductory economics courses changed and did not change with the intervention.

Data summary

This intervention was implemented in the economics department on the main campus of the University of Pittsburgh, a large, public, urban, residential R1 university in the northeastern United States with over 19,000 undergraduates enrolled each semester. Students at this university are representative of the typical residential, public university student: 95 percent are full-time, 66 percent are in-state students, and 83 percent are aged 18–21. As of fall 2019, the undergraduate student body was 55 percent female, 68 percent white, 5 percent African American, 11 percent Asian American, 4 percent Hispanic or Latino/a, and 5 percent international students of various origins.

The economics department is housed in the School of Arts and Sciences. While the School of Arts and Sciences is the largest school (58 percent by undergraduate enrollment in fall 2019), the university hosts various other schools, the largest of which are the School of Engineering (15 percent) and the College of Business Administration (10 percent). Students are admitted into a school and must choose majors from within that school unless they switch schools. However, students can take classes across schools, and students from all schools can take Introductory Microeconomics and Macroeconomics.

Depending on major and school, Introductory Microeconomics and Macroeconomics may be required or may count toward general graduation requirements. Both introductory economics courses are required for economics majors and minors and may be taken in either order. Undergraduates in the College of Business Administration are required to take both introductory courses and must start with Microeconomics. In addition, either introductory course can be counted toward the general education requirement for students in the School of Arts and Sciences or the School of Engineering. Our study population consists of all full-time students at the main campus who were registered in any class in a fall or spring semester from fall 2015 through fall 2019.⁷ These nine semesters comprise six pre-intervention semesters (fall 2015–spring 2018) and three post-intervention semesters (fall 2018–fall 2019).

Demographics of the university vs. the introductory economics class population

In this section, we compare the demographics of students who take either introductory economics class to those who never take such an economics class. In order to analyze the impact of our intervention on students’ decisions to take later economics courses, we include in our sample only full-time students at the main campus who have taken any class from fall 2015 to fall 2019.

Panel A of table 2 compares the demographics of students in our sample who have ever taken an introductory economics class to those who have not. Students who take introductory economics classes are less likely to be female, more likely to be white, less likely to belong to an underrepresented minority (URM) group, more likely to be Asian American, and more likely to be international students compared to the general student body.⁸ They are also less likely to be first-generation students. These differences raise the issue of recruiting both female students and underrepresented minority students into the introductory economics classes, which will be necessary to close the gender gap at this step in the pipeline. Aside from possible enhanced reputational effects, which we expect will be minimal given the proportion of first-time freshmen in the introductory economics courses, addressing this issue is beyond the scope of our intervention.

Table 2. Summary statistics.

	Panel A: School Sample			Panel B: Economics Sample
Variables	No Intro Econ	Intro Econ	Difference	Pre-Intervention	Post-Intervention	Difference
Female	0.48 (0.00)	0.38 (0.01)	0.09 (0.01)***	0.38 (0.01)	0.39 (0.01)	0.01 (0.01)
White	0.66 (0.00)	0.71 (0.00)	−0.06 (0.01)***	0.72 (0.01)	0.70 (0.01)	−0.02 (0.01)**
URM	0.11 (0.00)	0.10 (0.00)	0.01 (0.00)***	0.10 (0.00)	0.11 (0.01)	0.01 (0.01)
Asian American	0.08 (0.00)	0.10 (0.00)	−0.02 (0.00)***	0.10 (0.00)	0.11 (0.01)	0.01 (0.01)
International	0.03 (0.00)	0.06 (0.00)	−0.03 (0.00)***	0.06 (0.00)	0.07 (0.00)	0.01 (0.01)
First Generation	0.46 (0.00)	0.37 (0.01)	0.08 (0.01)***	0.38 (0.01)	0.36 (0.01)	−0.02 (0.01)*
Low Income	0.50 (0.00)	0.40 (0.01)	−0.10 (0.01)***	0.41 (0.01)	0.38 (0.01)	−0.03 (0.01)***
3+ in Family	0.94 (0.00)	0.95 (0.00)	0.02 (0.00)***	0.95 (0.00)	0.95 (0.00)	0.00 (0.00)
HS GPA	3.87 (0.01)	3.89 (0.01)	−0.02 (0.01)**	3.87 (0.01)	3.94 (0.01)	0.07 (0.01)***
SAT Verbal/100	6.42 (0.00)	6.53 (0.01)	−0.11 (0.01)***	6.52 (0.01)	6.55 (0.01)	0.03 (0.01)**
SAT Math/100	6.47 (0.01)	6.68 (0.01)	−0.21 (0.01)***	6.66 (0.01)	6.73 (0.01)	0.07 (0.02)***
Took AP Micro	0.06 (0.00)	0.06 (0.00)	0.00 (0.00)	0.06 (0.00)	0.06 (0.00)	0.00 (0.01)
AP Micro Score	3.85 (0.03)	3.30 (0.05)	0.55 (0.05)***	3.33 (0.06)	3.23 (0.08)	−0.10 (0.10)
Took AP Macro	0.06 (0.00)	0.07 (0.00)	−0.01 (0.00)***	0.07 (0.00)	0.08 (0.01)	0.01 (0.01)
AP Macro Score	3.81 (0.03)	3.26 (0.05)	0.55 (0.05)***	3.25 (0.06)	3.28 (0.08)	0.03 (0.01)
Arts & Sciences	0.54 (0.00)	0.69 (0.00)	−0.14 (0.01)***	0.69 (0.01)	0.67 (0.01)	−0.03 (0.01)**
Engineering	0.13 (0.00)	0.13 (0.00)	0.00 (0.00)	0.14 (0.00)	0.11 (0.01)	−0.02 (0.01)***
Business	0.06 (0.00)	0.17 (0.00)	−0.11 (0.00)***	0.15 (0.00)	0.20 (0.01)	0.05 (0.01)***
First Time Freshman				0.58 (0.01)	0.66 (0.01)	0.08 (0.01)***
First Micro Grade				2.79 (0.02)	2.94 (0.02)	0.15 (0.03)***
First Micro DFW				0.17 (0.00)	0.13 (0.01)	−0.05 (0.01)***
First Macro Grade				2.88 (0.02)	3.07 (0.03)	0.19 (0.03)***
First Macro DFW				0.15 (0.00)	0.08 (0.01)	−0.07 (0.01)***
N	29052	8727	37779	5860	2867	8727

Notes: Means and standard errors in parentheses for different variables. The school sample includes all students who were registered in any classes between fall 2015 and fall 2019. The no intro econ sample includes those students who never took an introductory economics course within that timeframe. The intro econ sample includes all students who took an introductory economics course within that timeframe. Difference is the no intro econ group mean minus the intro econ group mean, as well as the significance of the t-test comparing those two means. The economics sample includes all students who registered for their first introductory economics course between fall 2015 and fall 2019. Pre-intervention is all students who took their first introductory economics course before the intervention started in fall 2018. Post-intervention is all students who took their first introductory economics course after the intervention started in fall 2018. Difference is the post-intervention mean minus the pre-intervention mean, as well as the significance of the t-test comparing those two means.

***p < 0.01; **p < 0.05; *p < 0.1

Students who take introductory economics classes seem to be marginally better prepared academically than their peers. While they do not have substantially different high school GPAs, students who take introductory economics courses have higher SAT verbal and math scores.⁹ On the other hand, when looking at Advanced Placement tests, students who take either introductory economics course tend to have lower scores conditional on taking the exam. This is because students who do well in these exams place out of the introductory classes. However, it is relevant to note that taking the AP Micro and Macro tests is relatively uncommon in both our school sample and our economics subsample, with only about 7 percent of students having taken these exams.

Students from the School of Arts and Sciences and the College of Business Administration are well-­represented in the introductory economics courses. The representation of engineering students does not vary across the two groups.

Demographics of introductory economics students before and after the intervention

The intervention was implemented in the fall of 2018. The pre-intervention data include students who took their first introductory economics class from fall 2015 through spring 2018, and the post-intervention data include students who took their first introductory economics class from fall 2018 through fall 2019. Panel B of table 2 presents the demographics of the pre-intervention and post-intervention samples. The primary demographic of concern in this article, gender, does not change across this sample in this time frame. This allays any concerns about concurrent efforts to recruit more women into the economics major or substantial changes in recruitment by gender at the university. However, other demographics do vary across this time period; specifically, the population taking introductory economics classes becomes less white. Given that almost three-quarters of students in our economics sample are white, we will restrict our final sample to white students. We make this restriction because we anticipate that intersectionality may be important in understanding the outcomes for women due to this intervention, but given the small sample of nonwhite students, we do not have the power to investigate this intersectionality properly. Instead of assuming that women of all races have the same reaction to this intervention, we instead make the decision to restrict our sample to white students, while acknowledging that this limits the scope of our results and recognizing the importance of studying the outcomes for nonwhite students, both overall and interacted with gender. We do include a basic analysis in figure A1, but this analysis is underpowered and should be interpreted with caution.

Figure 1. Pre- to post-intervention differences by student gender.

Academic preparedness seems to improve in this time period as well. Students who take introductory economics courses have higher high school GPAs and math SAT scores in the post-intervention period compared to the pre-intervention period. However, the likelihood of having taken an Advanced Placement economics exam or the average score conditional on having taken the exam did not change in this period. These are factors that we will control for in our main analysis.

Students who take these courses are more likely to be from the College of Business Administration and less likely to be from the School of Arts and Sciences or the School of Engineering in the post-intervention period compared to the pre-intervention period. This may represent a greater shift within the school toward being in the College of Business Administration. We choose to restrict our sample to students in the School of Arts and Sciences for multiple reasons. First, we believe students in the School of Arts and Sciences are more likely to be voluntarily taking these economics courses out of interest or to consider economics as a major, whereas students from the College of Business Administration and the School of Engineering take these courses either as required classes or to satisfy general education requirements. Second, we believe that the students who are in the School of Arts and Sciences are the most relevant population to discuss in terms of the representation problem in economics, as they are the ones who would potentially progress in economics, conditional on their experiences in their introductory courses. Additionally, students who take introductory economics after the intervention are more likely to be first-time freshmen. As such, we will control for this in our main analyses.

Given the substantial changes in demographics and preparedness that occur across these two samples, we will include these controls in our main analyses. However, we are primarily concerned with whether these controls change with the intervention differentially by gender. Figure 1 shows the pre- and post-­intervention means for the academic preparation variables separately by gender.¹⁰ Due to some small differences found in some of the coefficients, primarily in having taken the AP Macro exam, we believe controlling for these incoming variables will be most important. We add those controls separately to evaluate the relative importance of adding those controls on our regression results.

Results

Outcomes of introductory economics students before and after the intervention

Before discussing our main results, we will discuss how the outcomes of students who take introductory economics courses before and after the intervention vary. This analysis is cursory means-level comparisons and is only used to get a sense of the general population effects of the intervention.

Panel B of table 2 presents the means of various outcomes in the pre- and post-intervention groups. In general, it seems like outcomes in introductory courses are better after the intervention compared to before the intervention. Post-intervention students get 0.15 better grades in Introductory Microeconomics and 0.19 better grades in Introductory Macroeconomics on their first try and are 5 (7) percentage points less likely to get a D or F or to withdraw Introductory Microeconomics (Macroeconomics).¹¹ These results indicate that the intervention seems to have positively benefited the population of introductory economics students both in terms of getting better grades and in being less likely to have bad grade outcomes.

The intervention effects on course grades

We now turn our attention to the primary concern of our analysis: the impact of the intervention by gender, with an eye toward understanding how outcomes of women differentially respond to the intervention. We begin by analyzing the mean grade changes from pre- to post-intervention by gender and race in the full sample of students who took an introductory economics course between fall 2015 and fall 2019. As we can see in figure 2, there were minimal changes in grades among white male students, while all other demographic groups, including white women, showed improvements in mean grade, particularly for Introductory Macroeconomics.¹² This is encouraging, as it indicates that while white men did not necessarily benefit, they were not harmed, nor was any other group that is not considered in the remainder of the analyses. The lower panel of figure 1 also reflects these outcomes in the final sample focusing on white students in the School of Arts and Sciences, showing that women’s grades in both courses differentially improve after the intervention. Additionally, figure 3 presents these results for white students in the School of Arts and Sciences by year and shows that for both introductory courses, the structural break is after the intervention, providing evidence that the intervention is likely the cause of this change. However, due to the differential change in incoming characteristics by gender that we found in figure 1, we rely on regression results to evaluate this outcome fully.

Figure 2. Grades by gender and ethnicity, pre- and post-intervention.

Figure 3. Grades over time by gender.

We run regressions in which we can control for potentially varying student characteristics in order to evaluate the impact of the intervention on student grades in line with the following equation: (1) $$y_i={\beta }_0+{\beta }_1\text{*}\mathit{\text{Femal}}{e}_i+{\beta }_2\text{*}\mathit{\text{In}}\text{t}\mathit{\text{erventio}}{n}_i+{\beta }_3\text{*}\mathit{\text{Femal}}{e}_i\text{*}\mathit{\text{Interventio}}{n}_i+\gamma X_i+{ϵ}_i,$$(1) in which $y_i$ presents individual students’ outcome variables of either their course grade or getting a D, F, or withdraw in their first attempt at either an Introductory Microeconomics or Macroeconomics course. The “First” in these outcome variables indicates that it is the first attempt at these courses and does not include retakes. ${\mathit{\text{Female}}}_i$ is an indicator variable that is equal to 1 when the student is female and 0 when the student is male.¹³ ${\mathit{\text{Intervention}}}_i$ is an indicator variable that is equal to 1 when the student made their first attempt at that particular course, either Introductory Microeconomics or Macroeconomics, in line with the course referenced in $y_i$, during the intervention period, i.e., fall 2018 or later, and 0 otherwise. $X_i$ is a vector of controls. In the first model, we consider only structural controls for the student’s academic year when they first took that economics course, whether freshman, sophomore, junior, or senior, or beyond. The second model adds prior academic experience controls such as their high school GPA, their math and verbal SAT scores, and whether they took AP Micro and AP Macro. This model also includes controls for whether they took either introductory course at a satellite campus prior to taking the course at the main campus. The third model adds prior economics coursework controls for whether they took Introductory Microeconomics or Macroeconomics first. The final model adds family characteristics controls such as whether their family reports having a low income, whether they are a first-generation college student, and whether there are three or more individuals in the student’s family. ${\beta }_1,{\beta }_2,$ and ${\beta }_3$ present our coefficients of interest, while ${\beta }_0$ indicates the constant, $\gamma$ represents the vector of coefficients for the vector of controls $X_i$, and ${ϵ}_i$ indicates the error term. Our primary analysis is OLS regressions for both grades and the outcome of getting a D, F, or W, but we include Logit regressions for the binary outcomes of getting D, F, or W in either course in appendix A.¹⁴ We find substantially similar results using this alternative framework.

In table 3, Models 1a–4a present the primary coefficients of interest for regressions evaluating the impact of the intervention on Introductory Microeconomics grades, and Models 1b–4b present the same for Introductory Macroeconomics grades. The different columns indicate different sets of controls, starting with a model with only structural controls to the last column that includes all of the controls. The controls of concern, those of high school GPA and Math SAT scores, are incorporated in Models 1b and 2b for Introductory Microeconomics grade and Introductory Macroeconomics grade regressions, respectively.¹⁵ Using these regressions, we first find evidence of gender disparities in grades in Introductory Macroeconomics, but not Introductory Microeconomics.¹⁶ We also find no evidence of an impact of the intervention on the grades of male students. However, we find that the intervention raises grades for women by 0.15 to 0.22 points (out of a 4.0 scale), although this is marginally insignificant in some of the Macroeconomics regressions. To put this change in context, a 0.2 point change could bring someone from a minus grade to a full grade, or from a full grade to a plus grade. Additionally, it more than closes any preexisting gender gaps in grades and results in a reversal of such gaps, with women performing better than men in the classes.

Table 3. Grades and DFW rates.

	(1)		(2)		(3)		(4)
Overall Grades
Micro	Model 1a		Model 2a		Model 3a		Model 4a
Female	−0.02	(0.04)	−0.03	(0.05)	−0.03	(0.05)	−0.03	(0.05)
Intervention	0.04	(0.05)	0.01	(0.05)	0.02	(0.05)	0.02	(0.05)
Female X Intervention	0.22	(0.09)**	0.15	(0.08)*	0.15	(0.08)*	0.15	(0.08)*
Macro	Model 1b		Model 2b		Model 3b		Model 4b
Female	−0.08	(0.05)	−0.11	(0.05)**	−0.10	(0.05)**	−0.10	(0.05)*
Intervention	0.03	(0.06)	−0.01	(0.06)	0.01	(0.06)	0.01	(0.06)
Female X Intervention	0.22	(0.10)**	0.18	(0.10)*	0.15	(0.10)*	0.15	(0.09)
DFW
Micro	Model 1c		Model 2c		Model 3c		Model 4c
Female	0.01	(0.02)	0.02	(0.02)	0.02	(0.02)	0.02	(0.02)
Intervention	−0.03	(0.02)	−0.02	(0.02)	−0.02	(0.02)	−0.02	(0.02)
Female X Intervention	−0.10	(0.04)***	−0.08	(0.03)**	−0.07	(0.03)**	−0.07	(0.03)**
Macro	Model 1d		Model 2d		Model 3d		Model 4d
Female	0.02	(0.02)	0.02	(0.02)	0.02	(0.02)	0.02	(0.02)
Intervention	−0.01	(0.03)	0.00	(0.02)	−0.01	(0.02)	−0.01	(0.02)
Female X Intervention	−0.05	(0.04)	−0.04	(0.04)	−0.03	(0.04)	−0.03	(0.04)

Notes: Separately for Micro and Macro as well as separately for overall grades and DFW rates, estimated regression beta coefficients (and standard errors) for the main effect of gender, main effect of intervention pre vs. post, and interaction of intervention by gender across models that included (1) only structural controls, (2) the addition of prior academic experience controls, (3) the addition of prior economics coursework controls, and (4) the addition of family characteristics controls. Final sample of white students from the School of Arts and Sciences who took an introductory economics course between fall 2015 and fall 2019.

***p < 0.01; **p < 0.05; *p < 0.1

We can also evaluate the impact of the intervention on the distribution of course grades. The top panel of figure 4 presents the changes in the percentage of students in each grade category that occurred with the intervention, split by course and the gender of the student. Both courses saw an increase at the top of the grade distribution and a decrease at the grade distribution bottom for both genders. For Microeconomics, the growth was in As and Bs and is larger for female students. For Macroeconomics, the growth was specifically in As, and it is especially large for female students.

Figure 4. Grade distributions and DFW evolution by gender.

We also may be concerned where these students are moving from. In particular, we would want these increases at the top end of the distribution to come from decreases in the bottom end of the distribution, in particular, the Ds, Fs, and Ws (sometimes collected as DFW). We see that there is some decrease in the DFW rate, particularly for women, but much of the decrease is happening in the C grade range. This suggests that while some of the worst outcomes are being averted, particularly for women, most of the change in grades is coming from a hollowing out of the middle outcomes toward the better outcomes. There does appear to be a small increase in the failure rate in Macroeconomics for both male and female students; this may involve a redistribution of Ws to Fs or small differences in student characteristics over time.

Turning specifically to these most negative outcomes, the DFW rate, we can evaluate if the intervention had any specific impact on these outcomes, and if so, if it differentially impacted men and women. Considering the temporal trends, the lower panel of figure 4 suggests that the intervention may differentially inhibit the overall decline in DFW rates for men but not for women in Introductory Microeconomics, while there were no differences for Introductory Macroeconomics. Columns 1c–4c and 1d–4d of table 3 present regressions of the interaction between the intervention and gender for the DFW rate in Introductory Microeconomics (1c–4c) and Macroeconomics (1d–4d) with the associated controls.¹⁷ We can see here that there does not seem to be a differential DFW rate by gender prior to the intervention. In Microeconomics, there seems to be no change in the DFW rate for men and a 7 to 10 percentage point decrease for women. However, this does indicate the generation of a gender gap in the DFW rate for Introductory Microeconomics, with women having lower DFW rates than men. In Introductory Macroeconomics, there is no statistically detectable change in the DFW rate for either men or women with the intervention.

Discussion and conclusion

Gender gaps in economics are pervasive and persistent, and they start as early as the first economics course a student takes in their undergraduate studies. While there have been many interventions attempted at this level to change in students’ minds the image of who an economist is (i.e., that economists are women) (Carrell, Page, and West 2010; Griffith 2010; Li 2018; Porter and Serra 2020), there have not been interventions that manipulate the presentation of what economics is (i.e., something that is interesting to women), through either course content or content delivery.

In this article, we evaluate an intervention that changed course content and how that content was presented and evaluated in the recitations of Introductory Microeconomics and Macroeconomics courses. This intervention made recitations more focused on group work that involved solving real-world problems using the economic models learned in class. Not only did this better integrate marginalized students, such as women, into the class discussion, but it also provided a broader range of examples of how economics can be used. While the intervention was not designed to close gender or any other demographic gaps, we thought such changes may be possible, given how the content of the course was broadened by the intervention.

We find that, prior to the intervention, women get similar grades to men in their introductory economics courses. When the intervention was implemented, we found women’s grades in their introductory economics courses exceeded those of men with similar characteristics and educational backgrounds. This is an important step in ensuring that women proceed along the economics pipeline, as prior literature has shown that women are particularly sensitive to grades when deciding whether or not to remain in economics. Furthermore, because women typically get better grades in non-economics courses, the increase in grades women receive here will potentially put their economics grades more in line with the rest of the grades they are receiving, giving them less of an indication that they should specialize outside of economics.

This intervention highlights the importance of considering course content and content delivery when trying to understand why gender gaps emerge and persist in economics. The traditional framing in economics courses of superficially focusing on business may be less interesting to women than other framings, such as those aimed at policy and welfare. However, this intervention also attends to pedagogy as a component of gender gaps.

Another area of future research would be to evaluate how this intervention impacted other demographic gaps, such as racial gaps, in economics course outcomes and progression. While we are able to show evidence that suggests that racial gaps in course performance in Introductory Microeconomics also improved, given the intervention (Miller-Cotto and Schunn 2020), we do not have the power to evaluate whether this was consistently the case across both courses or how the intersectionality between race and gender affect outcomes.¹⁸ Further, it is unlikely that we will be able to evaluate this program further in the near run due to the COVID-19 pandemic.

In evaluating interventions, dimensions other than effect sizes, such as cost and sustainability, should be considered. From a cost perspective, this intervention was expensive to develop (e.g., summer support for many faculty) but inexpensive to sustain. The high development cost could have been distributed over more time by having a smaller team gradually change the contents of recitations. However, this might have produced an incoherent experience for students across the transition years. Further, training TAs on a new approach is important and hard to justify along the way when only a small proportion of recitation contents are changed. From a sustainability perspective, there were benefits of the adapted implementation strategy of involving all relevant faculty in the development of the revision materials and then in the teaching using the revised materials: it created an institutional commitment to sustain the revisions across the gradual replacement of faculty and introduction of new policy changes. The research literature on the process of changing teaching across a department suggests that curriculum/pedagogy-focused efforts can be more successful than other kinds of changes because faculty see the value of working collectively, and sufficient data can be collected to robustly test the effects (Henderson, Finkelstein, and Beach 2010). It is an open question, however, whether the faculty involved then also changed other aspects of their teaching (e.g., how they teach upper-level undergraduate courses).

Several strategic elements were vital to the successful implementation of this large project. With the support of a grant from our provost, we were able to get active cooperation and collaborative design during a summer term from all the faculty who teach Introductory Microeconomics or Macroeconomics. This collective approach, in addition to enabling robust quantitative modeling of the effects of the intervention on student outcomes, allowed us to implement the new training for all of the teaching assistants, which simplified scheduling and also allowed us to take advantage of economies of scale. Faculty shared the workload of developing the new recitation plans and specialized in particular topics. It also allowed faculty to design plans collaboratively, which fostered innovation and an iterative revision process that we believe served to improve our results. The exchange of ideas through regular meetings and hallway conversations throughout the summer term was invaluable to faculty. Similarly, teaching assistants benefited from being in a teaching class with their peers and using the same lesson plans as their peers. They were able to assist each other because they were working on the same lessons at the same time. Finally, the collective approach will likely support sustainability of the revision since so many faculty are invested and personally connected to the revision.

Acknowledgments

The authors thank the extraordinary instructional team, David Hewitt, Svitlana Maksymenko, James Maloy, Kevin Shaver, Carey Treado, and Rita Zabelina, for their development of the intervention materials. They gratefully acknowledge the help of these experts at the Learning Research and Development Center and the University Center for Teaching and Learning at University of Pittsburgh: Joel Brady, Dana Miller-Cotto, Joe Horne, Lorna Kearns, and Lindsay Onufer. They also thank the graduate teaching assistants for their efforts to implement the new recitation curriculum and their helpful feedback, and they thank Randy Walsh and the participants of the dB-SERC lunch talk series for their useful comments.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

Funding for this project was provided by the Provost’s Office at University of Pittsburgh through a Course Incubator Grant.

Notes

1 The economics profession also has large racial gaps in terms of outcomes and progression. While we can identify those gaps in our pre-intervention sample and we find evidence to support the closing of some of those gaps due to our intervention, the population of the University of Pittsburgh is such that we do not have a large enough sample of underrepresented minority students to have the power to detect such changes. While it was a goal of this project to also consider racial gaps in economics education outcomes, the structural pedagogical changes due to the COVID-19 pandemic and its associated shift to remote instruction have effectively cut off the clean post-intervention sample and have made it impossible to do that additional analysis in the future.

2 For example, in high school, 55 percent of all Advanced Placement (AP) tests in the United States are taken by women, but only 45 percent of all AP Micro test takers and 42 percent of all AP Macro test takers are women (College Board 2020). Similarly, although women have consistently made up about 56 percent of U.S. undergraduates from the 1990s to 2019 (U.S. Department of Education 2020), only about 42 percent of U.S. students in introductory economics courses were women (Avilova and Goldin 2018). Subsequently, only 31 percent of U.S. undergraduate economics degrees were granted to women, and this proportion has declined from the 1990s through more recent years (Avilova and Goldin 2018; Bayer and Wilcox 2019). This pattern is replicated in other countries (Arnold 2020). Furthermore, 44 percent of incoming master’s students and 34 percent of incoming PhD students in economics in the United States are women, and only 35 percent of economics PhDs are granted to women (Siegfried 2010, 2020). In faculty careers, representation issues are even worse: 30 percent of assistant professors in economics are women, 26 percent of tenured faculty are women, and only 15 percent of full professors are women (Lundberg 2017). At our own institution, while 55 percent of undergraduate students are women, just 41 percent of students in the introductory economics courses are women, and only 28 percent of undergraduate economics degrees are granted to women.

3 These non-inclusive teaching techniques generally diminish students’ self-efficacy because students do not have opportunities to see whether their struggles with content are shared broadly by other students and can falsely attribute their own struggles as due to their own inadequacies rather than challenging concepts and/or poor pedagogy. Female students may be differentially impacted by this issue due to negative experiences in high school, the result being that women are more likely to begin the undergraduate introductory courses with lower self-efficacy in economics, which then changes engagement, response to challenging coursework, and grade outcomes (Ballard and Johnson 2005; Jakobsson 2012).

4 At the University of Pittsburgh, plus and minus grades are increments of 0.25. So, while a B grade is 3.0 on a 4-point scale, a B + is 3.25, and a B − is 2.75.

5 The UWE Challenge suggested that interventions include changing instructional content and presentation style, but so far none of the participating schools have reported on such an intervention.

6 We do consider the impact of instructor gender and TA gender on gaps in male and female outcomes but do not find any compelling evidence that instructor or TA gender matters. However, this may be due to the demographics of the instructors for the Introductory Microeconomics and Macroeconomics courses, with most of the Introductory ­Microeconomics instructors being female and almost all of the Introductory Macroeconomics instructors being male. Results are available upon request.

7 We exclude summer classes as they are substantially different from main-term classes. Specifically, they tend to be smaller and taught by graduate student instructors rather than full-time instructors. None of the summer classes in the post-intervention period were subject to the intervention.

8 Underrepresented minorities include those that identify as African American, Hispanic, American Indian, and ­Pacific Islander. Asian American includes both those who primarily identify as Asian and those who identify as both Asian and white. First-generation students are defined as having no parents who attended college.

9 For students who took the ACT rather than SAT, scores were converted to their SAT equivalent using the 2018 ­College Board SAT/ACT concordance tables found here: https://collegereadiness.collegeboard.org/pdf/guide-2018-act-sat-concordance.pdf.

10 This is done only for our final sample, which is white students in the school of Arts and Sciences.

11 Grades are based on a 4-point scale, where grades are given the following letters: A or A+ = 4.0, A− = 3.75, B+ = 3.25, B = 3.0, B− = 2.75, C+ = 2.25, C =2.0, C− = 1.75, D+ = 1.25, D = 1.0, D− = 0.75, F = 0. The measure of DFW, or getting a D or F or having to withdraw, is an important indicator for very bad course outcomes and generally means the student will not get credit for the course, will have to retake the course, or will have substantial penalties to their GPA.

12 Figure A1 shows the complete intersectionality of male/female within all four racial groups. The number of students in each category becomes very small and thus the means are poorly estimated, but the trends are suggestive of all groups except white males, showing benefits of the intervention in at least one of the two courses. The remaining analyses focus on the gender effect within white students because there are clear signs of gender x race/ethnicity intersectionality in the effects of the intervention but there is only enough power to estimate the gender effect among white students.

13 Given the reporting in our data, very few students report any gender identity other than male or female, and so we restrict our sample to these two groups.

14 Appendix A can be found on the corresponding author’s website (https://www.malloryavery.com/research).

15 See tables A1 and A2 for the full set of estimated coefficients from each model.

16 While not our primary model of interest, we also evaluate the gender gaps in grade anomalies for these courses, defined as the gender gaps in grades conditional on the student’s semester GPA excluding these classes. We find evidence of a gender gap in grade anomalies, with women performing significantly worse in both courses when controlling for their same-semester GPA from their other classes. This is true if we control for the entire same-semester GPA or only their Arts and Sciences GPA, their GPA in other social sciences classes, or, for Macroeconomics grade, their GPA in their sciences classes. These gaps are substantial, ranging from about a 0.2 gap in the Microeconomics course grades to a 0.3 gap in the Macroeconomics course grades. Results are available upon request.

17 Full regression outputs are presented in tables A3 and A4. Logit regressions are presented in tables A5 and A6. They present similar results: the Logit results for Microeconomics indicate the intervention decreased women’s DFW rate by 43–47 percent, or by 7 to 8 percentage points relative to the mean DFW rate of 17 percent; women do not experience a statistically significant change in their DFW rate in Macroeconomics.

18 Figure A1 indicates that while women seem to benefit across all racial groups from the intervention, it is not clear that non-white women benefit differentially compared to non-white men. It is not clear, ex ante, what we should expect in this environment: benefits from this intervention may be additive across intersectional minority identities, resulting in women having increases in performance relative to men for URM, Asian Americans, and international students; alternatively, the intervention may benefit all marginalized individuals the same, whether they have intersectional identities or not. While we do not have the power to truly study this topic, we do run regressions on grade, interacting race and gender identities in tables A7 and A8. We find mixed results—in some cases, it seems indicative that women may differentially benefit regardless of race, whereas in other cases, it seems that benefits to the entire racial group crowd out benefits to women from that racial group.