Impact of Lesson Study and Fractions Resources on Instruction and Student Learning

Abstract

Lesson study, especially when conducted with support from targeted resources, is often identified as a potentially effective means to support teacher learning. In this study, 80 school-based teams of educators, representing 80 classrooms of third- or fourth-grade students in the US, were randomly assigned to one of four conditions: lesson study supported by fractions resources, lesson study only, fractions resources only, or a comparison group. Fractions instruction was improved in the two conditions that received the fractions resources. Average student performance on a fractions test did not increase relative to the comparison group in any of the three intervention conditions. However, moderation analyses suggest that lesson study, both with and without fractions resources, may have improved student performance in one state, which represented half of the sample.

Keywords:

• Elementary school
• mathematics
• experimental design
• hierarchical linear models
• fractions

Fractions are a challenging topic for students (National Mathematics Advisory Panel, 2008; OECD, 2014). Fractions knowledge predicts proficiency in algebra, even when analyses adjust for other facets of mathematical knowledge, and fractions competence is often identified as a “gatekeeper” topic for secondary-level mathematics (Booth & Newton, 2012; Siegler et al., 2013). Only 20% of U.S. elementary teachers rate their own fractions knowledge as strong or very strong (Ward & Thomas, 2006), a finding that may explain why simply providing collegial work sessions for teachers to collaborate on improving fractions instruction is not sufficient to improve teachers’ mathematical knowledge of fractions (Gearhart et al., 1999; Saxe et al., 2001).

Despite billions of dollars spent every year on teacher professional development in the US and frequent claims that PD is an important lever by which to improve instruction and student learning, very few PD programs are rigorously evaluated for impact on classroom teaching and student learning (Fermanich, 2002; Garet et al., 2016; Hill et al., 2013; Odden et al., 2002; Schoen et al., 2018a; TNTP, 2015; U.S. Department of Education, 2014; Wei et al., 2010; Yoon et al., 2007). A 2014 report found that only two of 643 studies of mathematics PD found positive effects on student mathematics proficiency when the study design met rigorous scientific criteria (Gersten et al., 2014). Lesson study with support from a fractions resource kit (Lewis & Perry, 2017) was the subject of one of those two studies.

The study described here is a conceptual replication (Schmidt, 2009) of the Lewis and Perry (2017) randomized trial. The intervention in the Lewis and Perry study involved mailing fractions-lesson study binders to teams of teachers who used them to conduct lesson study. The binders were designed to support teacher-led lesson study with the support of a set of resources to support teaching of fractions with an emphasis on linear representations of fractions concepts. Teacher and student-level results in the intervention condition were contrasted with those where teams of teachers conducted lesson study on a topic of their choosing or chose both the topic and a professional learning method other than lesson study. Lewis and Perry reported a positive impact of lesson study on both teachers’ and students’ fractions knowledge, but the study was not designed to separate the effects of the two main intervention components: the lesson study process and the fractions resources.

This conceptual replication study uses four treatment conditions—all of which were provided with some opportunity for teachers to learn about teaching fractions. This study was designed to enable evaluation of the independent and combined impacts of lesson study (a process for teacher learning) and content resources (fractions resources for teachers to study) on classroom instruction and student learning in fractions. This study also involved a larger sample, a slightly modified version of the measure of student outcomes, and a lead author and methodologist who represent a third-party (i.e., not the program developers or implementers) evaluation team. Table 1 is provided to help orient the reader to some of the key similarities and differences between the current study and the Lewis and Perry study.

Table 1. Comparison of present study with Lewis and Perry (2017) study.

	Lewis and Perry (2017)	Present study
Conditions	C1: LS + FR	C1: LS + FR
	C2: C(-F)	C2: LS (+F)
	C3: LS(-F)	C3: FR C4: C (CCSSM)
Randomization	Volunteer teams of teachers with 4–9 members	Volunteer teams of teachers with 3–9 members
Sample	39 Teams 213 Teachers 1,162 students* with pretest 1,059 students* with pre and posttest 37% in Grades 2 or 3 (389) 19% in Grade 4 (203) 44% in Grade 5 (467) C1: 57% Grade 5 C2 and C3: 37% Grade 5 11 states + DC *Students in classes of all teachers who taught a research lesson (66 teachers total)	80 Teams 322 Teachers 1,407 students with pretest* 1,119 students with pre and posttest* 45% in Grade 3 (629) 55% in Grade 4 (778) 10 states (56% in FL) *Only one RLT per team, measures obtained only for students in RLT classes
Main outcome measures	Teachers’ fractions knowledge^a Students’ fraction knowledge^b	Mathematics instruction (IQA)^c Students’ fraction knowledge (EFTv2.2)^d
Effect size estimates	Teachers (LS + FR vs. C and LS): 0.19 Students (LS + FR vs. C and LS): 0.49		LS + FR v C	LS v C	FR v C
		IQA	0.82	0.21	0.73
		EFT	−0.08	−0.12	−0.19

Note. LS + FR = fractions resources and support for conducting lesson study. C(-F) = Business as usual with request not to study fractions. LS(-F) = Lesson Study with request not to study fractions. LS(+F) = Lesson Study with encouragement to study fractions. FR = Fractions Resources. C (CCSSM) = Business as usual with encouragement to study the way fractions are taught per the Common Core standards.

^a Researcher created measure using items from various published assessments and research articles.

^b Researcher created measure using items from published research, NAEP, Japanese textbooks and California Standards Test.

^c IQA (Boston, 2012).

^d EFT v2.2 (Schoen et. al., 2017b). The post-test used by Lewis and Perry (2017) for grade 3 had 14 items in common with EFT v2.2. The grade 4 test had 16 items in common.

Linear representations in fractions instruction

Linear representations of fractions, such as number line and distance measurement, may help students see fractions as numbers with magnitude, not just as objects in a set or pieces of a whole, allowing students to integrate knowledge of fractions and whole numbers into a unified number system (Lamon, 2005; Morris, 2000; Saxe et al., 2013; Siegler et al., 2010). Linear representations of fractions are emphasized in textbooks of some high-achieving Asian countries (Grow-Maienza & Beal, 2005; Ma, 1999; Tokyo Shoseki Co., 2015). Historically, U.S. textbooks have not emphasized linear representations (Lewis et al., 2011; Watanabe, 2007).

The Common Core State Standards for Mathematics (CCSSM) and related standards adopted by most U.S. states within the last decade emphasize the number line as a means to integrate fractions (as rational numbers) and whole numbers in elementary education. These standards also deemphasize use of discrete wholes (e.g., pizzas) and set models (e.g., 2 of 6 buttons) to represent fractions (NGACPB & CCSSO, 2010). Further, CCSSM emphasizes students’ foundational understanding of unit fractions (fractions with a numerator of one), the relationship of the unit fraction to non-unit fractions and to the referent unit (e.g., two-thirds is made up of two one-thirds; three one-thirds make one) and equipartitioning (dividing the referent unit into same-size unit fractions). In summary, the elementary-level CCSSM emphasize linear representations of fractions and the use of unit fractions as basic units to compose/decompose other numbers—changes that represent a departure from prior U.S. practice and a potential challenge for teachers to implement as intended (Common Core Standards Writing Team, 2013; Sidney et al., 2019). Several studies have evaluated the effects of interventions designed to incorporate linear representations of fractions into instruction using experimental designs and reported positive impact on student achievement on tests of fractions knowledge (Fuchs et al., 2017; Jayanthi et al., 2021; Lewis & Perry, 2017).

Lesson study

In lesson study—a teacher-driven approach to professional learning that is used widely in Japan—teachers conduct collaborative “Study-Plan-Teach-Reflect” cycles designed to improve classroom instruction (Lewis & Hurd, 2011; Takahashi & McDougal, 2014). The team of participating teachers may include all teachers at a given grade level, or, as in the current study, any team of interested volunteers. Teachers typically begin the cycle by studying the relevant content and teaching materials, laying out their immediate and long-term goals for students, and choosing one lesson within the planned unit to serve as a “research lesson” that will be collaboratively planned and then taught by one team member to that member’s students. Research lessons are so named, because teams enact their hypotheses about good instruction by designing a lesson, conducting the lesson, gathering data during implementation of the lesson, and discussing student responses to the lesson (Lewis et al., 2019). Revising and reteaching the lesson is an optional feature of lesson study, common in the U.S. but less so in Japan (Fujii, 2014).

Study of the academic content and teaching materials—kyouzai kenkyuu—is integral to lesson study in Japan (Takahashi et al, 2005; Yoshida & Jackson, 2011). During kyouzai kenkyuu, teachers study documents such as teacher’s manuals, content frameworks, and research reports to learn about both the subject matter and its teaching and learning (Doig et al., 2011; Takahashi et al., 2005). Japanese teacher’s manuals provide support for kyouzai kenkyuu by identifying and discussing the key mathematical ideas in each instructional unit, anticipating likely student solution strategies and linking them to the key mathematical ideas, and situating the unit within a multi-year trajectory of mathematical learning (Lee & Zusho, 2002; Lewis et al., 2011; Miyakawa, 2011; Takahashi, 2021). Lesson study engages teams of teachers in studying research-based instructional resources with a particular emphasis on planning their use and examining their impact on practice. Lewis and Perry (2017) found that lesson study can increase U.S. teachers’ content knowledge for instruction and improve student learning.

Teaching materials found outside Japan do not always provide good support for kyouzai kenkyuu. For example, comparison of the treatment of quadrilateral area in the teacher’s manual for widely used Japanese and U.S. textbooks revealed that 28% of the statements in the Japanese teacher’s manual focused on student thinking, but only 1% of the statements in the U.S. manual focused on student thinking (Lewis et al., 2011). Similarly, 10% of the statements in the Japanese teacher’s manual focused on the rationale for instructional decisions (e.g., the choice not to show gridlines when asking students to consider how to find the area of a rectangle), whereas none of the statements in the U.S. teacher’s manual provided such rationale. This stark difference in curriculum-embedded professional-learning resources can limit the quality and coherence of various phases in the lesson-study cycle when U.S. teachers conduct lesson study.

Lesson study with fractions resources: Program theory of change

Enactment of new instructional approaches is difficult (Cohen, 2011), and fractions are a mathematically challenging topic for teachers (Ward & Thomas, 2006). In the intervention reported here and in the previous randomized trial (Lewis & Perry, 2017), the program theory of change focuses on two major elements hypothesized to improve fractions instruction and therefore student learning: (1) research-based resources on fractions that highlight the role of linear representations and (2) a lesson-study process for collaborating with colleagues to study the resources and improve fractions instruction. The research-based fractions resources were intended to expand the fractions knowledge available to teachers (especially related to linear representations and student thinking), and the lesson-study process was intended to support the challenging work of translating these resources into high-quality instruction. The theory of change suggests that the two elements together—resources and a collaborative process for studying them and using them to plan instruction—should be more effective at improving fractions teaching and student learning than either element in isolation.

Research questions

The study was designed to investigate the following research questions (RQs). We consider RQ1 and RQ2 to be confirmatory and RQ3 to be exploratory.

RQ1. What are the independent and combined impacts of lesson study and fractions resources on fractions instruction, as measured by the Instructional Quality Assessment (IQA)?

RQ2. What are the independent and combined impacts of lesson study and fractions resources on student learning in fractions, as measured by the Early Fractions Test (EFT)?

RQ3. To what extent are effects in fractions instruction and student achievement moderated by the following factors: (i) student (average) pretest score, (ii) grade level, (iii) geographic location, and (iv) average baseline fractions knowledge of the participating teacher team?

Method

The study reported here used a randomized controlled trial research design to investigate the impact of the lesson-study process, conducted with and without fractions mathematical resources. Eighty school-based teams of educators agreed to participate in the present study—about twice the number included in the study by Lewis & Perry (2017)—but the teams were distributed across four experimental conditions (rather than three as in the prior trial). As described in detail below, at least one educator on each team was a third- or fourth-grade teacher responsible for fractions instruction during the study period. The teams were assigned to one of four conditions: lesson study supported by fractions resources (LS + FR); lesson study only (LS); fractions resources only (FR); and a comparison group (C).

Description of the intervention and counterfactual conditions

As in Lewis and Perry (2017) study, the intervention reported here consisted of condition-specific binders mailed to participating sites in three intervention conditions and one control condition. The binder contents differed by experimental condition and were expected to catalyze different activities. The binders for the four conditions represent the four possible combinations of providing or not providing support in the form of (1) a professional learning process (lesson study) and (2) fractions content resources. Every participating teacher in all four conditions received their own binder. In all four conditions, the binders encouraged teachers to improve their fractions instruction and to log the time, materials, and processes they used to do so.

Table 2 shows the binder contents by condition. The binder for the control condition provided the CCSSM fractions standards, suggested that teachers examine curriculum materials and research to improve their fractions instruction, and recommended some general resources (e.g., consulting a district math specialist, visiting the website of the National Council of Teachers of Mathematics). For the sites in the two conditions assigned to lesson study (i.e., LS + FR, LS), the binders also included step-by-step guidance on conducting a lesson study cycle, starting with developing norms and processes to work as a team, and then going through the four lesson-study phases: study the curriculum and content; plan a series of lessons; teach the lessons, where at least one lesson represents a research lesson that is observed by the whole team, and reflect on data from the lesson and learning during the whole cycle. For educators in the two conditions that received fractions resources (i.e., LS + FR, FR), the binder also included specific fractions resources (curriculum examples, video, and research summaries; see Table 2) accompanied by discussion prompts (relabeled as “reflection” prompts in the FR condition, where participants were expected to study these materials individually). The selected fractions resources emphasized linear representation of fractions as a support for students in understanding fractions as numbers that can be placed on a number line (Saxe, 2007; Saxe et al., 2013; Watanabe, 2002, 2007; Wu, 2011). Each condition also received condition-specific restrictions, reproduced verbatim in Table 2.

Table 2. Binder contents by condition and condition-specific prompts and restrictions.

Binder contents	Condition
	LS + FR	LS	FR	C
Copies of fractions standards in the CCSSM	X	X	X	X
Lesson study resources Overview of lesson study; Protocols to support norm-setting, note-taking, observation, discussion; Lesson planning template	X	X
Fractions resources, including mathematical tasks, research references, and curriculum materials 3 fractions tasks to solve, consider student responses (from NAEP, research) Table summarizing research on six challenges in understanding fractions, with student thinking examples for each 8 visual representations of the fraction ¾ and prompts to compare them Hands-on task for teachers: Express “mystery length” as a fraction of a meter Grade 3 and 4 fractions units from a Japanese textbook Video of series of fractions lessons by Japanese teacher (to U.S. students) at grades 3 & 4 Outline of Teaching Through Problem-solving lesson flow (also modeled on video)	X		X
Condition	Prompt and restrictions
LS	This guide does not provide mathematical resources, so we encourage you to seek out mathematical resources to support your lesson study on fractions, with the following exception: Please refrain from using Japanese curriculum materials on fractions during the study period.
FR	Please use whatever method you wish to learn from these resources, with the following exception: Please refrain from using these resources for lesson study (i.e., for planning, observing, and discussing a lesson with colleagues) during the study period. (Conducting lesson study on a topic other than fractions is fine.) Please feel free to do individual self-study or work as a team, as you prefer, using whatever other methods you think will be effective.
C	We encourage you to seek out mathematical resources to support your work on fractions, and to use whatever learning methods you find effective, with the following exceptions: Please refrain from using Japanese curriculum materials on fractions during the study period. Please also refrain from conducting lesson study on fractions (i.e., refrain from planning, observing, and discussing a lesson on fractions with colleagues) during the study period. (Conducting lesson study on a topic other than fractions is fine.) Please feel free to do individual self-study or work as a team, as you prefer, using whatever other methods you think will be effective.

Note. Prompts and restrictions are reproduced verbatim from binder materials. LS + FR = Lesson Study with Fractions Resources, LS = Lesson Study Only, FR = Fractions Resources only, C = Practice-as-usual comparison.

The binder sent to teachers in the LS + FR condition was similar to that used previously by Lewis & Perry (2017). Two major additions were the inclusion of CCSSM excerpts on fractions and video of an additional series of lessons (at fourth grade). To accommodate these new materials while keeping binder length reasonable, several binder sections that were provided to sites in the previous study were eliminated, including five short investigations that teachers could conduct with their students.

Procedures

The target population for the study included teams of educators in U.S. public schools—regular or charter—with at least one educator responsible for teaching mathematics to third- or fourth-grade students.

Recruitment

Information about the opportunity to participate in the research study was advertised through two main channels. Every known school-district mathematics specialist and elementary-school principal in Florida was notified of the opportunity through an e-mail from the first author. The notice was also shared by the second author through several listservs focused on lesson study and through informal conversations at professional conferences. All the sites located outside Florida were recruited through the listservs and professional conferences. Potential participants were directed to an online survey, where they could access the consent form approved by the Institutional Review Board at Florida State University (FSU). When at least three educators (including at least one third- or fourth-grade teacher of mathematics) for a given school site had applied to participate and provided informed consent, research permission was requested from the relevant school district. Only one team per school site was allowed to participate.

Recruitment occurred in two cohorts. The initial cohort (Cohort 1) consisted of 68 sites that met the eligibility criteria for the 2016-2017 school year. Before randomization, we obtained positive parental consent and student assent for 1,211 students from these sites. The sites in this cohort were randomized in Fall 2016, after the student pretests were administered and the consent forms returned, and their associated data were collected during the 2016-2017 school year. To increase the number of sites in the study, a second cohort of 12 sites (with a total of 196 consenting students) that met the eligibility criteria (Cohort 2) was recruited to participate in the 2017-2018 school year. Apart from the year delay, the procedures in both cohorts were similar.

Incentives

Each team was promised a $4000 stipend for study completion (to be used for substitutes, team member stipends, or other professional learning expenses designated by the team). Each team was also promised a video camera to be used in data collection; the site could keep the camera with the stipulation that it be available for follow-up video-recording.

Eligibility criteria

To be eligible, at least three and at most eight teachers at a school site had to provide informed consent to participate, including at least one third- or fourth-grade teacher responsible for teaching mathematics. Only one team per school site was eligible. The district and school principal also had to approve the terms of the study. One of the consenting third- or fourth-grade mathematics teachers agreed to serve as the research-lesson teacher (RLT).¹ RLTs also agreed to obtain parental consent for students in their own classrooms, teach a series of three fractions lessons to be video-recorded, and administer a pre- and postintervention fractions assessment to their students. The students of the other participating teachers were not eligible for the study. For sites assigned to the LS or LS + FR conditions, the RLT was also expected to teach the research lesson, which is a crucial part of the lesson study process, as one of the series of three videotaped lessons.

Consent process

The RLTs distributed parental consent and student assent forms to the students in their mathematics classes and mailed the returned forms to the research team with the student pretests and class rosters. The study included only those students who had both parental consent and student assent before the sites were notified of their assigned treatment condition.

Randomization

The study employed a randomized block design. The unit of randomization was the team of teachers who qualified for enrollment in the study based on the criteria listed above. Teams in eligible schools identified the RLT and corresponding class of students at the beginning of the school year (before randomization). Schools were grouped into blocks of four with the goal of achieving within-block homogeneity with respect to the following three criteria in descending order of importance: state, grade level of the designated RLT, and proportion of students eligible for free or reduced-price lunch. Every block could not be exactly homogenous with respect to those three variables. Each site in a given block was assigned at random to one of the four assignment conditions and notified of its treatment condition after consent for participation and baseline data were obtained for both teachers and students at that site.

Participants

Table 3 presents school characteristics for the full sample, split by condition. Ten states and two grade levels of students are represented in the sample; 56% of the sites were in Florida. State, grade level, and the percentage of students in the school who were eligible for free or reduced-price lunch were relatively balanced across the four conditions.

Table 3. School characteristics.

	Treatment condition
	LS + FR (n = 20)	LS (n = 20)	FR (n = 20)	C (n = 20)	Total (N = 80)
School proportion FRL
Mean	.64	.57	.69	.64	.64
SD	.24	.26	.26	.23	.25
Min	.06	.15	.20	.23	.06
Max	1.00	.96	1.00	.97	1.00
State
Arizona	1	0	0	0	1
California	2	4	1	4	11
Colorado	1	0	0	0	1
Florida	11	10	12	12	45
Illinois	3	4	2	2	11
Indiana	0	0	1	0	1
New Jersey	1	1	1	1	4
New York	1	1	1	1	4
Oklahoma	0	0	1	0	1
Texas	0	0	1	0	1
RLT grade level
Grade 3	12	8	9	9	38
Grade 4	8	12	11	11	42

Note. FRL = Free or reduced-price lunch. RLT = Research Lesson Teacher. LS + FR = Lesson Study with Fractions Resources, LS = Lesson Study Only, FR = Fractions Resources only, C = Practice-as-usual comparison.

Table 4 presents information about all participating teachers. Only teachers who had consented to participate in the study at the time that their schools were randomized were included. The mean number of years of teaching experience was 13.5 (SD = 8.0). Sample characteristics were relatively balanced across the four conditions. The RLT sample was similar to the total sample of teachers in its composition, and the RLT sample was relatively balanced in characteristics across conditions. Table 5 presents information about participating RLTs. Table 6 presents information pertaining to the sample of participating students.

Table 4. Participating teacher characteristics.

	Sample, split by condition
	LS + FR (n = 74)	LS (n = 86)	FR (n = 80)	C (n = 82)	Total (N = 322)
Primary teaching role
Regular classroom^a	69 (.93)	66 (.77)	62 (.77)	65 (.79)	262 (.81)
Varying exceptionalities^b	3 (.04)	5 (.06)	5 (.06)	5 (.06)	18 (.06)
Math coach^c	2 (.03)	6 (.07)	6 (.08)	7 (.09)	21 (.07)
Other^d	0 (.00)	6 (.07)	2 (.03)	4 (.05)	12 (.04)
Departmentalization
Teaches all subjects	48 (.65)	51 (.59)	51 (.64)	54 (.66)	204 (.63)
Teaches only math	25 (.34)	26 (.30)	23 (.29)	25 (.31)	99 (.31)
Does not teach math	0 (.00)	5 (.06)	2 (.03)	3 (.04)	10 (.03)
Grade level primarily taught
Kindergarten	0 (.00)	1 (.01)	0 (.00)	2 (.02)	3 (.01)
Grade 1	0 (.00)	0 (.00)	3 (.04)	0 (.00)	3 (.01)
Grade 2	3 (.04)	6 (.07)	1 (.01)	3 (.04)	13 (.04)
Grade 3	32 (.43)	30 (.35)	33 (.41)	34 (.42)	129 (.40)
Grade 4	22 (.30)	32 (.37)	24 (.30)	25 (.31)	103 (.32)
Grade 5	13 (.18)	11 (.13)	14 (.18)	11 (.13)	49 (.15)
Grade 6	2 (.03)	2 (.02)	1 (.01)	6 (.07)	11 (.03)
Grade 7	1 (.01)	0 (.00)	0 (.00)	1 (.01)	2 (.01)
Highest degree earned
Associate’s degree	1 (.01)	0 (.00)	2 (.01)	0 (.00)	2 (.01)
Bachelor’s degree	39 (.53)	35 (.41)	36 (.45)	43 (.52)	153 (.48)
Master’s degree	26 (.35)	36 (.42)	34 (.43)	33 (.40)	129 (.40)
Specialist degree	7 (.10)	11 (.13)	5 (.06)	6 (.07)	29 (.09)
Areas of certification
Elementary Education	67 (.91)	74 (.86)	68 (.85)	71 (.87)	280 (.87)
PreK/Primary Education	7 (.10)	14 (.16)	9 (.11)	12 (.15)	42 (.13)
Middle Grades Mathematics	6 (.08)	8 (.09)	6 (.08)	5 (.06)	25 (.08)
Secondary Mathematics	2 (.03)	3 (.04)	1 (.01)	2 (.02)	8 (.03)
ESOL/Bilingual/Dual-language	20 (.27)	36 (.42)	28 (.35)	37 (.45)	121 (.38)
Varying Exceptionalities^b	13 (.18)	21 (.24)	20 (.25)	21 (.26)	75 (.23)
Previous experience with lesson study	12 (.16)	14 (.17)	15 (.19)	22 (.27)	63 (.20)
Years of teaching experience	13.8 ± 8.3	15.2 ± 8.1	11.1 ± 7.0	13.5 ± 8.0	13.4 ± 8.0
Baseline questionnaire completion
Completed	73 (.99)	82 (.95)	76 (.95)	82 (1.00)	313 (.97)
Did not complete	1 (.01)	4 (.05)	4 (.05)	0 (.00)	9 (.03)

Note. Statistics are presented as frequency (percentage) for categorical variables and mean ± standard deviation for numerical variables. LS + FR = Lesson Study with Fractions Resources, LS = Lesson Study Only, FR = Fractions Resources only, C = Practice-as-usual comparison.

^a Regular classroom teachers teach core content but may have classrooms where gifted and talented students, students with disabilities, and/or English language learners are enrolled.

^b Varying exceptionalities indicates specialized instruction for gifted and talented and students with disabilities.

^c Includes math coaches and curriculum specialists.

^d Other includes teachers of noncore subject areas, administrators, and ESOL specialists.

Table 5. Research lesson teacher characteristics.

	Treatment condition
	LS + FR (n = 20)	LS (n = 19^c)	FR (n = 20)	C (n = 20)	Total (N = 79)
Primary teaching role
Regular classroom^a	19 (.95)	15 (.79)	16 (.80)	19 (.95)	69 (.87)
Varying exceptionalities^b	1 (.05)	2 (.11)	1 (.05)	0 (.00)	4 (.05)
English language learners	0 (.00)	2 (.11)	0 (.00)	1 (.05)	3 (.04)
Math coach	0 (.00)	0 (.00)	3 (.15)	0 (.00)	3 (.04)
Departmentalization
Teaches all subjects	15 (.75)	13 (.68)	15 (.75)	14 (.70)	57 (.72)
Teaches only math	5 (.25)	6 (.32)	5 (.25)	6 (.30)	22 (.28)
Grade level primarily taught^c
Grade 3	12 (.60)	8 (.42)	9 (.45)	9 (.45)	38 (.48)
Grade 4	8 (.40)	11 (.58)	11 (.55)	11 (.55)	41 (.52)
Highest degree earned
Bachelor’s degree	9 (.45)	5 (.26)	7 (.35)	12 (.60)	33 (.42)
Master’s degree	9 (.45)	11 (.58)	12 (.60)	6 (.30)	38 (.48)
Specialist degree	2 (.10)	3 (.16)	1 (.05)	2 (.10)	8 (.10)
Areas of certification
Elementary Education	19 (.95)	18 (.95)	18 (.90)	17 (.85)	72 (.91)
PreK/Primary Education	1 (.05)	2 (.11)	3 (.15)	3 (.15)	9 (.11)
Middle Grades Mathematics	0 (.00)	2 (.11)	1 (.05)	1 (.05)	4 (.05)
Secondary Mathematics	0 (.00)	1 (.05)	0 (.00)	0 (.00)	1 (.01)
ESOL/Bilingual/Dual-language	5 (.25)	10 (.53)	6 (.30)	11 (.55)	32 (.41)
Varying Exceptionalities^b	5 (.25)	7 (.37)	4 (.20)	4 (.20)	20 (.25)
Previous experience with lesson study	3 (.20)	2 (.11)	4 (.20)	2 (.10)	11 (.14)
Years of teaching experience	17.0 ± 10.6	13.5 ± 6.6	11.1 ± 7.7	15.5 ± 8.8	14.3 ± 8.7

Note. Statistics are presented as frequency (percentage) for categorical variables and mean ± standard deviation for numerical variables. LS + FR = Lesson Study with Fractions Resources, LS = Lesson Study Only, FR = Fractions Resources only, C = Practice-as-usual comparison.

^a Regular classroom teachers teach core content but may have classrooms where gifted and talented students, students with disabilities, and/or English language learners are enrolled.

^b Varying exceptionalities indicates specialized instruction for gifted and talented and students with disabilities.

^c One RLT did not complete the pretest demographic survey. Percentages in this column are presented out of the 19 respondents.

Table 6. Consenting student characteristics for sample.

	Sample split by condition
	LS + FR (n = 378)	LS (n = 362)	FR (n = 324)	C (n = 343)	Total (N = 1,407)
Gender
Male	170 (.45)	178 (.49)	130 (.40)	141 (.41)	619 (.44)
Female	176 (.47)	181 (.50)	163 (.50)	161 (.47)	681 (.48)
Missing	32 (.09)	3 (.08)	31 (.10)	41 (.12)	107 (.07)
English language learner
ELL	34 (.09)	87 (.24)	43 (.13)	78 (.23)	242 (.17)
Non-ELL	313 (.83)	273 (.75)	214 (.66)	244 (.65)	1,024 (.73)
Missing	31 (.08)	2 (.06)	67 (.21)	41 (.12)	141 (.10)
Grade level
Grade 3	210 (.56)	129 (.36)	145 (.45)	145 (.42)	629 (.45)
Grade 4	168 (.44)	233 (.64)	179 (.55)	198 (.58)	778 (.55)
Preintervention test completed
Completed	367 (.97)	356 (.98)	309 (.95)	332 (.97)	1,364 (.97)
Did not complete	11 (.03)	6 (.02)	15 (.05)	11 (.03)	43 (.03)
Postintervention test completed
Completed	331 (.88)	314 (.87)	273 (.84)	311 (.91)	1,229 (.87)
Did not complete	47 (.12)	26 (.07)	51 (.16)	32 (.09)	178 (.13)

Note. Statistics are presented as frequency (proportion) for categorical variables. LS + FR = Lesson Study with Fractions Resources, LS = Lesson Study Only, FR = Fractions Resources only, C = Practice-as-usual comparison.

Data collection

Teacher surveys and assessments

After at least three educators at a school had signed up and designated a third- or fourth-grade RLT, they were invited to complete the Web-based baseline assessments, which included knowledge assessments and demographic information. At the end of the year, the teachers in all four conditions completed a postintervention assessment, which again included the knowledge assessment and asked teachers to report on their experiences with professional learning over the past year.

School and student demographics

Demographic information for individual students with parental permission was provided by teachers on a class-roster form that was mailed to researchers along with student pretests, parental-consent, and student-assent forms.

Videos of classroom instruction

Each site was asked to videorecord the first three lessons of the fractions unit taught by the designated RLT, one of which was expected to be the designated research lesson in the LS + FR and LS conditions. Students without consent for videorecording were seated outside the camera view.

Student assessments

Student fractions assessments were administered by teachers in the fall and spring of their participating school year to students taught by the RLT. The same test forms were administered to both third- and fourth-grade students. Administered by teachers after the fractions unit was completed, the postintervention test form was different and more difficult than the pre-intervention form, but a set of anchor items were consistent on both test forms.

Measures

Teacher knowledge of subject matter

The computer-based Knowledge for Teaching Elementary Fractions (K-TEF) tests measure teacher knowledge of fractions at the elementary level. The test contains 19 items: 12 drawn from the Learning Mathematics for Teaching (2004) project and 7 adapted from other sources (Beckmann, 2005; Newton, 2008; Saderholm et al., 2010; Schifter, 1998; Ward & Thomas, 2015; Zhou et al. 2006). Using response data for the K-TEF tests provided by 277 teachers in fall 2016 (i.e., Cohort 1 in this study), Schoen et al. (2018b) found that the K-TEF measures a single, dominant factor. Coefficient $\alpha$ for the K-TEF pretest scores with this sample was .76.

Implementation

A full investigation of implementation is beyond the scope of the current study, but we provide some highlights of the implementation study in the main body of this article and more information in the Supplementary Online Materials. We focus on three facets of implementation, all three of which are central to the theory of change for the intervention: evidence of implementation of a research lesson, evidence of the use of the materials in the fractions resource kit, and evidence of the use of linear representations of fractions during fractions instruction.

Linear representations of fractions

A total of 181 fractions-lesson videos representing 68 sites were received (3 videos per site were requested), and all were coded to reveal whether lessons used linear representations of fractions. If any lesson in any video from the site used a linear fraction representation for at least one-half of the lesson, the site received a 1 for use of linear representations of fractions. Otherwise, the site was given a score of 0 on use of linear representations of fractions.

Evidence of a research lesson

We created a dichotomous variable for each site to indicate whether we observed either of the following conditions in the submitted videos of classroom instruction for each site: (a) two or more adult observers were present for the lesson or (b) at least one adult observer was focused on students and taking notes.

Mystery strip activity

The Mystery Strip activity comes from a Japanese textbook (Hironaka & Sugiyama, 2006) and was included in the fractions resources (both as a problem for teachers to try to solve and as a task featured in a set of lesson videos provided in the resources). The Mystery Strip activity is not widely known or used in the U.S., so its use in a submitted lesson video provides a strong indicator that the RLT looked at the provided fractions resources. Use of the Mystery Strip activity was coded from classroom videos by means of a dichotomous variable that recorded its use in any submitted lesson video (1 for presence, 0 for absence). Sites were not specifically asked to use the Mystery Strip activity in lessons. Therefore, we did not expect that all sites in the LS + FR or FR conditions would use the Mystery Strip activity in their videos, but teachers in the other two conditions (i.e., LS, C) should show no evidence of awareness of the Mystery Strip. Although presence of the Mystery Strip activity suggests the participant did use the fractions resources, absence of the Mystery Strip does not necessarily imply failure to use them.

Classroom instruction in fractions

Videos of fractions instruction were rated by means of the Instructional Quality Assessment (IQA; Boston, 2012). The IQA provided a treatment independent (Slavin & Madden, 2011) measure of classroom instruction, and it served as the primary outcome of interest for the confirmatory RQ1. Seven of the nine established IQA scales (i.e., Potential of the Task, Implementation of the Task, Student Discussion, Teacher Linking, Student Linking, Teacher Press, Student Providing) were used in scoring the videos. Each rubric was scored from zero to four. The first lesson video submitted by each of the 68 sites that returned videos was selected to be coded. The primary observer coded all 68 videos, and a second observer coded 15 of these. Discrepant scores between the two observers’ ratings were reconciled by the two observers and input as the final scores for those 15 videos. For all other videos, the primary observer’s ratings were used. We computed linear-weighted kappa (Cohen, 1968) using SPSS (version 27) for the two raters and found it to be 0.686 (p = .000). The arithmetic mean of the ratings on the seven rubrics was used as the overall IQA score.

Student Performance on fractions tests

Two versions of the Early Fractions Test (EFT; Schoen et al., 2017a, 2017b) were used—one version administered before the intervention, the other version administered after the intervention. Lewis and Perry (2017) used an earlier version of the EFT in the initial efficacy study. Items on the EFT tests correspond to the grade-level expectations in the CCSSM (NGACBP & CCSSO, 2010) for third- and fourth-grade students, including linear representations of fractions (including the number line), referent unit, magnitude comparison, partitioning and iterating, and performing addition on fractions presented in symbolic notation.

Using response data for the EFT tests provided by 68 schools in Cohort 1, Schoen et al. (2017a, 2017b) found that the EFT tests measure a single dominant factor. Coefficient $\alpha$ for the EFT pretest scores with the Cohort 1 sample was .85; the posttest coefficient $\alpha$ was found to be .84 with the Cohort 1 sample. Scoring procedures used a two-parameter logistic model based on item-response theory, scores were not vertically scaled across pre- and posttest waves of data collection, and scores were generated by the expected a priori estimator (Schoen et al., 2017a, 2017b).

Data analysis

Attrition analysis

The Institute of Education Science’s What Works Clearinghouse (WWC) publishes standards for determining whether the combination of overall and differential attrition rates is likely to bias estimates of average causal effects from randomized experiments (U.S. Department of Education, 2017). We provide an attrition analysis in the Supplementary Online Materials.

RQ1: Effects on classroom instruction

These analyses use data from the 68 RLT teachers who submitted at least one video of fractions instruction in their classrooms. The model for the IQA outcome was an ordinary least squares regression model with FLORIDA, PCTFRL, the school average student pretest value, a dummy variable indicating that most of the students in the RLT teacher’s classroom were in fourth grade, treatment dummy variables, and block dummy variables. The effect size for IQA was computed based on the residual standard error from an ordinary least squares model where the only independent variables are dummies for treatment status and cohort.

RQ2: Effects on Student achievement in fractions

This analysis used data from the 1,199 students who provided complete pre- and postintervention test data for the (EFT). A two-level hierarchical linear model with students nested within schools was estimated. Specifically, we estimated models with the following specifications:

Level 1: Y_jk = τ_0k + τ_1k (PREC) _jk +u_0jk.

Level 2: τ_0k = γ₀₀ + γ₀₁ (PREM)_k + γ₀₂ (FLORIDA)_k + γ₀₃ (PCTFRL)_k + γ₀₂ (GRADE4)_k + Σγ_0T (TREAT_T)_k + Σγ_0M(BLOCK_M)_k + r_00k

τ_1k = γ₁₀

Y_jk represents the EFT student postintervention test score, PREC is the school-mean-centered EFT student preintervention test score, PREM is the school mean of the EFT preintervention test score, FLORIDA is an indicator variable equal to 1 if the school is located in Florida and 0 otherwise, PCTFRL is a continuous variable equal to the percentage of students eligible for free or reduced-price lunch at the school, GRADE4 is an indicator variable equal to 1 if the grade level of the RLT classroom was composed of fourth-grade students, TREAT_T (T = 1, …, 3) are dummy variables for the three active treatment conditions (LS + FR, LS, and FR), and BLOCK_M (M = 1,…, 19) are dummy variables for the randomization blocks.

Effect sizes associated with each active treatment condition are reported as the coefficients for the treatment dummies divided by the unconditional standard deviation of the outcome variable of interest. The unconditional standard deviation is computed as the sum of the square root of the two estimated variance components from a two-level model where the only independent variables are dummies for treatment status and cohort.

RQ3: Moderation by student pretest, teacher pretest, grade level, and school location

This research question addresses moderating effects of the four variables listed relative to each of the outcome variables considered in RQ1 (classroom instruction) and RQ2 (student achievement). The continuous moderators (student pretest and site mean for K-TEF pretest) were z-scored before entering statistical models. The student grade moderator was coded 1 if most of the students in the RLT’s classroom were in fourth grade and 0 otherwise. The school-location moderator was coded 1 if the school was in the state of Florida and 0 otherwise. (Approximately half of the sites were in Florida.) The binary moderators were not z-scored.

For the Florida variable, site-mean K-TEF score, and RLT grade-level moderators, the moderating variable was defined in the same way regardless of whether IQA or EFT measured the outcome. For the student preintervention test EFT moderator, the individual student test was used as the moderator when EFT was the outcome, and the site-mean student test was used when IQA was the outcome.

Moderator models were specified by elaboration of the models used to answer research questions 1 and 2. Specifically, interactions between the moderator in question and each of the three dummy treatment variables were included in the models. Otherwise, the statistical models used were unchanged.

Results

Implementation of the intervention components

While not a research question, implementation of an intervention, including the implementation and experiences of participants in the counterfactual condition during the intervention period, is an important consideration for interpreting results of a randomized controlled trial. Table 7 presents results of our three implementation indicators for each treatment condition overall and disaggregated by whether the site was in Florida. A more complete description of our implementation analyses can be found in the Supplemental Online Materials.

Table 7. Implementation activities observed in videos (by location and randomization status).

			Proportion (and number) of sites exhibiting evidence of:
Condition	Location	Submitted video	Linear representations	Mystery strip activity	Research lesson
LS + FR	Florida	9	0.67 (6)	0.33 (3)	0.56 (5)
	Non-Florida	7	0.71 (5)	0.43 (3)	0.43 (3)
	Overall	16	0.69 (11)	0.38 (6)	0.50 (8)
LS	Florida	10	0.70 (7)	0.00 (0)	0.60 (6)
	Non-Florida	7	0.43 (3)	0.00 (0)	0.71 (5)
	Overall	17	0.59 (10)	0.00 (0)	0.65 (11)
FR	Florida	10	0.80 (8)	0.30 (3)	0.00 (0)
	Non-Florida	7	0.86 (6)	0.29 (2)	0.14 (1)
	Overall	17	0.82 (14)	0.29 (5)	0.06 (1)
C	Florida	11	0.55 (6)	0.00 (0)	0.00 (0)
	Non-Florida	7	0.71 (5)	0.00 (0)	0.57 (4)
	Overall	18	0.61 (11)	0.00 (0)	0.22 (4)

Note. LS + FR = Lesson Study with Fractions Resources, LS = Lesson Study Only, FR = Fractions Resources only, C = Practice-as-usual comparison. Mystery strip activity was provided in the intervention materials to conditions LS + FR and FR and was not expected to be seen in LS or C conditions. Conditions LS and LS + FR were expected to implement a research lesson; conditions FR and C were asked to refrain from lesson study on fractions during the study period. Linear representations of fractions during fractions instruction are central to the theory of instruction in the LS + FR intervention, but they could also be present in practice-as-usual instruction.

Linear representations of fractions were observed in videos of fractions instruction in all four conditions. It was observed in a higher proportion of videos from the FR and LS + FR conditions than in the LS or C conditions. We did not perform tests to examine whether these differences are statistically significant, because we did not design the study of implementation for inferential statistics. We note that the point estimate for implementation rates of linear representations of fractions instruction is lowest in the condition that conducted lesson study without the infusion of fractions resources and highest in the condition that was provided fractions resources and asked teachers to study them on their own (i.e., not in collaboration with the others at their site).

We found a lower-than-expected rate of implementation of research lessons in the LS and LS + FR conditions and a higher-than expected rate in the other two conditions, especially in the C condition. This suggests low implementation rate in the two conditions involving lesson study and a potential contamination of the C condition, especially in sites located outside of Florida.

The Mystery Strip activity was not observed in the two conditions that did not receive the activity as part of the intervention. We did observe the activity being used in videos of fractions instruction in the two conditions that received that as part of the fractions resources. We remind the reader that—unlike the research lesson—those groups were not specifically asked to submit videos of the Mystery Strip lesson in use. The presence of the activity in some of the videos does suggest that the sites used the fractions resources. The absence of the activity in the LS and C conditions suggests minimal diffusion of the fractions resources into the LS or C conditions.

RQ1: Effects on fractions instruction

The first row of Table 8 shows estimates of the effects of each of the three treatment conditions on fractions instruction as compared with the comparison condition (C). Fractions instruction at sites assigned to both the conditions that received fraction resources scored significantly higher on the IQA than that of sites in the comparison condition. The effect-size estimate was also positive for the LS condition, but the magnitude was much smaller, and the p-value was much greater than .05 for that comparison. The first row of Table 9 shows the raw means and standard deviations for the IQA measure, separately for each treatment group.

Table 8. Treatment contrasts for fractions instruction and student learning.

	LS + FR vs. C			LS vs. C			FR vs. C
	β	SE	ES	β	SE	ES	B	SE	ES
Instruction
IQA	0.633*	0.271	0.822	0.156	0.262	0.203	0.559*	0.271	0.726
Student learning
EFT	–0.067	0.132	–0.075	–0.107	0.135	–0.120	–0.171	0.138	–0.192

Note. β for IQA is from OLS regression. β for EFT is from 2-level HLM regression. SE = Standard Error, ES is Cohen’s d. LS + FR = Lesson Study with Fractions Resources, LS = Lesson Study Only, FR = Fractions Resources only, C = Practice-as-usual comparison.

*p < .05

Table 9. Mean, standard deviation, and sample size by treatment condition for the analytic samples used to answer research questions 2 and 3.

	Condition
	LS + FR		LS		FR		C
Variable	Mean	SD	Mean	SD	Mean	SD	Mean	SD
Classroom instruction
Instructional Quality (IQA) Assessment postintervention test (n = 68)	2.91	0.78	2.58	0.77	2.86	0.70	2.33	0.80
Student achievement
Early Fractions Test (EFT) Pre-intervention test	–0.11	0.88	0.20	0.87	0.15	0.99	0.05	0.93
Early Fractions Test (EFT) Postintervention test (n = 1,199)	–0.10	0.94	0.14	0.88	0.02	0.99	0.13	0.88

Note. LS + FR = Lesson Study with Fractions Resources, LS = Lesson Study Only, FR = Fractions Resources only, C = Practice-as-usual comparison.

^a The EFT pre- and postintervention test forms were different, the form used at after intervention was more difficult than the form used before, and the theta estimates for the two tests were not calibrated to the same scale (Schoen et al., 2017a; 2017b).

RQ2: Effects on student performance on the fractions test

The last row of Table 8 shows that students taught by comparison-condition teachers performed best on the EFT, but none of the pairwise differences were statistically significant. The effect size where the LS + FR group is compared to control was estimated to be −0.075, that where the LS group is compared to control was −0.12, and that where the FR group is compared to control was −0.19. The last two rows of Table 9 show the raw means and standard deviations for the EFT measure at pre- and post-intervention test, separately by treatment group. Overall, there is little evidence of stable differences among the experimental conditions with respect to student performance on the EFT.

RQ3: Moderator effects

We present the results of moderator analyses in Table 10, which shows uncorrected p-values and four moderators for three pairwise comparisons of interest. For each outcome variable, therefore, 12 moderation effects are estimated. This multiplicity problem makes it difficult to distinguish signal from noise in our moderator analyses, because the magnitudes of certain effects will appear different from those of others just by chance, due to the number of effects estimated. If a Bonferroni correction was applied to account for the 12 comparisons, none of the results would be statistically significant at the conventional 0.05 level. Even if the more liberal Benjamini-Hochberg (1995) procedure was used, one would not reject any of the null hypotheses unless the false discovery rate was set to a high value of 0.27 (that is to say, the lowest q value is 0.27). Nonetheless, the moderation analyses do present some suggestive results.

Table 10. Moderator analyses with instructional quality assessment and early fractions test as outcomes.

	LS + FR		LS		FR		LS + FR × Mod		LS × Mod		FR × Mod
Outcome	β	SE	β	SE	β	SE	β	SE	β	SE	β	SE
Individual student Early Fractions Test preintervention test
Instructional Quality Assessment	0.643*	0.295	0.133	0.280	0.556+	0.281	0.063	0.347	0.150	0.355	0.028	0.262
Early Fractions Test	−0.052	0.133	−0.099	0.135	−0.168	0.135	0.076	0.063	−0.043	0.064	0.044	0.069
Fourth grade
Instructional Quality Assessment	0.550	0.388	−0.342	0.435	0.331	0.396	0.059	0.656	0.918	0.667	0.315	0.560
Early Fractions Test	0.026	0.195	−0.071	0.224	−0.336	0.209	−0.241	0.309	−0.067	0.341	0.361	0.283
Site mean mathematical knowledge for teaching
Instructional Quality Assessment	0.462	0.292	0.113	0.264	0.564+	0.288	−0.333	0.378	−0.530+	0.288	−0.066	0.297
Early Fractions Test	−0.13	0.13	−0.115	0.132	−0.179	0.140	−0.052	0.15	−0.293+	0.15	0.012	0.145
Florida
Instructional Quality Assessment	−0.020	0.404	−0.576	0.444	0.273	0.437	1.21*	0.546	1.193*	0.586	0.537	0.571
Early Fractions Test	−0.429*	0.195	−0.524*	0.2	−0.316	0.203	0.662*	0.267	0.74**	0.277	0.293	0.278

Note. LS + FR = Lesson Study and Resource Kit, LS = Lesson Study Only, FR = Fractions Resources only, C = Practice-as-usual comparison. SE = Standard Error. When Early Fractions Test is outcome, β and SE estimates are from two level hierarchical linear model. When Instructional Quality Assessment A is outcome, β and SE estimates are from OLS regression model. Models condition on additional covariates as described above in data analysis section. Continuous moderators (student preintervention test score and site mean mathematical knowledge for teaching) are z-scored. The Fourth-grade moderator is coded 1= fourth grade, 0 = third grade. The Florida moderator is coded 1 = Florida, 0 = all other states. The unconditional standard deviation of the Instructional Quality Assessment variable was 0.77 and that of the Early Fractions Test variable was 0.90.

⁺p <.10

*p < .05**p <.01

We first discuss moderator analyses where student performance on the EFT was the outcome. There is no evidence to suggest that treatment effects vary systematically with respect to student pretest scores. Although not statistically significant, some evidence suggests that the negative contrast between FR and C students was mainly due to negative effects in third grade. The fitted means for fourth-grade students in the FR and C groups are, in fact, almost identical. Some evidence also suggests that, among the sites in the LS condition, the average teacher knowledge of fractions of the lesson study team is negatively related to treatment effects, suggesting that, relative to similarly situated control group students, students performed comparatively worse on the test if they had high-knowledge teachers who were asked to engage in lesson study without the support of the fractions resources.

The most striking results for the EFT outcome involved the moderating effect of state. Evidence suggests substantial differences in the impact of lesson study between Florida and non-Florida sites. For schools not in Florida, the estimate of the impact of both LS and LS + FR relative to control was negative and statistically significant (without multiplicity corrections). The estimated effect sizes were −0.58 for LS and −0.48 for LS + FR for non-Florida schools. The effect estimated for the FR condition relative to control was also negative, but it was smaller and not statistically significant. On the other hand, a large positive interaction was estimated for the moderating effect of a location in Florida. For both LS and LS + FR, the moderating effect of Florida is positive, statistically significant, and greater in absolute value than the estimated negative impact in states other than Florida. In other words, evidence suggests that impacts of lesson study were, on average, negative outside of Florida but positive within Florida (although the positive-within-Florida estimates are not statistically significant). The same pattern of positive moderation was observed for the FR condition, but it was a much smaller effect. Even after this positive moderating effect was accounted for, the average treatment effect estimate for FR vs. C within Florida was negative.

Some evidence also indicates heterogeneity in effects across certain moderators for the classroom-instruction outcome. Like the student outcome, it is not moderated by the (average) student pretest. Some evidence indicates a positive treatment effect on IQA in the LS group for fourth-grade teachers and a negative effect for third-grade teachers, but the result is not statistically significant, even without multiplicity corrections. Slightly more evidence indicates moderation with respect to the average level of fractions knowledge in the participating teacher group. Moderation again seems to occur only for the LS group; outcomes were worse (relative to the comparison condition) for groups with a higher average level of fractions knowledge.

Again, the most striking results came from examination of state as a moderator. For sites in states other than Florida, a small, not statistically significant, negative impact of treatment on IQA was evident for the LS and LS + FR groups, but state had a strong moderating effect. Within Florida, effect size estimates were positive, statistically significant, and greater than 0.5 for the LS, LS + FR and the FR conditions.

Discussion

Impact of fractions resources on instruction

A statistically significant (p < .05) effect on classroom instruction as measured by the IQA was found in the two conditions that received fractions resources, but a similar effect was not found in the condition assigned to lesson study alone. The resources provided to the sites in the two fractions resources conditions are more extensive than those U.S. teachers typically receive as part of curriculum adoption (Remillard, 2000); they included research-based summaries of students’ fractions challenges, fractions tasks, curriculum materials highlighting linear representations of fractions, video of fractions instruction, and prompts to discuss (or reflect on) all of these. The positive impact on instruction in the two conditions that received the fractions resources provides evidence that mathematical resources and the opportunity to study them—whether individually or collaboratively—can increase quality of instruction.

Effects on student learning

The average effects of each of the three intervention conditions on student performance on fractions tests were negative and close to zero in all three cases. Additional explorations are needed to reveal the reason for these overall null effects, which contrast with findings of a prior study (Lewis & Perry, 2017) of a similar intervention with a similar outcome measure.

The positive effects on IQA (for the two conditions that received fractions resources) and negative effects on student achievement are perplexing and raise questions about the association between IQA scores and student learning. The analyses are not presented here, but we sought to determine whether IQA scores predict student learning, and we found a small, positive, statistically significant (α < .05) correlation between IQA scores and student performance on the postintervention EFT when adjusted for the preintervention performance (Authors, manuscript under review). Many researchers in mathematics education are using IQA in their studies. Only a small subset of them have reported on empirical associations between IQA scores and student achievement in mathematics, but those that have reported on the association report a positive association (Lewis et al., manuscript under review; Matsumura et al., 2008; Quint et al., 2007). However, the association between IQA and student performance found in the current study may be driven by a confounding factor that we have not yet identified. Our results underscore the need for further empirical study to identify the measurable components that lead to both improved instruction and increased student learning in fractions and the causal links between the two.

Moderators of intervention impact

None of the four moderators—student fractions pretest, grade level, site mean of teachers’ mathematical knowledge, and state (i.e., Florida, non-Florida)—showed significant impact after correction for multiplicity. The moderator analyses do, however, suggest positive impact of the two lesson-study conditions on instructional and student outcomes in Florida and negative average impacts in non-Florida sites.

To further explore explanations for the treatment-by-Florida interaction, we split the implementation data into Florida and non-Florida sites (Table 7). We found that all four of the C condition sites that presented evidence of a research lesson (i.e., that failed to abide by the control-condition restrictions) were outside Florida. A lack of adherence to the assigned condition could explain both the Florida moderator and the apparent discrepancy between the findings of the present study and those of the previous randomized controlled trial of lesson study with fractions resources. We note that the Florida sites were recruited through a general call for research-study participants that was sent by the first author to district mathematics curriculum supervisors and principals across the state. The sites outside Florida were recruited by the second author through lesson study networks and professional learning networks. This difference in recruiting method may explain why sites outside of Florida more frequently engaged in lesson-study-type activities, even when asked to refrain from doing so. In this study, the location of sites (within or outside Florida) and the method of recruiting sites is confounded. So we cannot be sure whether the state context, the recruitment method, or some combination of the two produced the differences between sites within and outside Florida.

Limitations

The present study included teams of educators who volunteered, limiting generalizability to non-volunteers. The marginal levels of fidelity to the intervention, especially in the LS + FR condition, and lack of a sharp contrast across conditions make inferences about the underlying program theory of change difficult. The three videorecorded lessons were not selected at random from the full unit on fractions and may not represent the full experience of students in fractions, possibly explaining the positive effects on classroom instruction and null effects on student learning. The improvement in instructional quality in the LS + FR and FR conditions might not extend to other topics in mathematics or to other grade levels, because the materials were specific to early fractions.

Conclusions

Although assignment to lesson study alone did not improve instruction, assignment to the two conditions that received mathematical resources (with and without lesson study) did produce significantly higher IQA scores. This result contributes to a growing collection of findings that professional-development opportunities that are, by design, combined with curriculum resources or other actionable materials (e.g., lesson plans, tasks to be used in formative assessment) may be more likely to have positive effects on teaching and learning (Garet et al., 2016; Harris et al., 2015; Jacob et al., 2017; Lang et al., 2014; Lonigan et al., 2011; Penuel et al., 2011; Rosenfeld et al., 2019; Schoen & Koon, 2021). These apparent improvements in instruction, however, did not lead to improvements in student performance on fractions tests in this study.

We observed lower-than-expected implementation of lesson study in the two lesson-study conditions, substantial use of linear fractions representations in all conditions, and substantial collaboration in all conditions, including some lesson-study-like activities in the control condition. Binders and guidance about expected and impermissible activities failed to produce the expected contrast across conditions. We imagined that providing the fractions standards in the control condition binder would have little impact, since these standards are readily available. Clearer guidance and establishment of mutual expectations for the activities in the C condition might have yielded a different outcome. Implementation data suggest that some sites/teachers may need more, or different, support for implementation of the materials, especially in the LS + FR condition.

Evidence of a treatment-by-state interaction in the present study provides a reminder that context may matter. This program was designed to support local adaptation and agency, and it appears to have had varied results in different settings or with participants recruited in different ways. Another follow-up study may be warranted to increase fidelity of implementation of the designed programs and better reveal the conditions under which the interventions can be expected to have the intended effects.

Disclosure statement

The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

Data availability statement

The data that support the findings of this study are openly available via ICPSR at https://doi.org/10.3886/ICPSR119304.v1 [ICPSR 37080].

Additional information

Funding

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant # R305A150043 to Mills College.

Notes

1 While no research lessons were taught in sites assigned to the FR or C conditions we still refer to teachers performing the tasks described in the next sentences as “RLTs.”