Oral History in Evaluation: A New Partnership to Expand and Enhance Both Fields

ABSTRACT

Evaluators utilize a variety of qualitative and quantitative methods to assess outcomes and make recommendations; however, we were unable to find any reference in the literature regarding the incorporation of oral histories for evaluation purposes. Based on our use of oral histories to evaluate a Critical Race Theory (CRT)-based undergraduate research and mentoring program, we learned how oral histories can be used effectively—and meaningfully—for evaluations seeking a deeper understanding of the program participants’ lived experiences. By sharing our (and our narrators’) experiences as we delved into this new methodology, as well as lessons learned about the benefits, limitations, and considerations when incorporating oral histories into evaluations, we show how both can be adapted to honor oral history while also meeting our evaluation needs.

KEYWORDS:

• cultural research
• evaluation
• evaluator
• methodology

Though storytelling has sustained historical memory for tens of thousands of years, and individuals from a variety of fields, including journalism, psychology, education, history, and anthropology, among others, have used oral history to understand and document personal experiences covering topics ranging from significant historical events, such as wars and slavery, to more personal focuses, such as the history and experiences of a family or organization,¹ oral history is a new methodology for the field of evaluation. In fact, despite the varied data-collection methods evaluators use, we were unable to find reference to oral history’s methodological use—with its foundational goal of gaining a deep understanding of individuals’ personal experiences through open-ended exploration—in the evaluation literature at all.² The purpose of this paper is to share our experience as we learned about, conducted, and analyzed oral history interviews as part of our evaluation of a Critical Race Theory (CRT)-based undergraduate research and mentoring program. Due to the nature of the program we evaluated, we chose oral history methods most closely associated with narrative inquiry because of their ability to raise the voices and honor the experiences of traditionally marginalized and minority groups. We hope that seeing the rich information our narrators shared with us will further expand the application of oral history by encouraging other evaluators to incorporate this method, when appropriate, into their evaluation plans. Further, we propose that evaluation’s use of its methodology could benefit the field of oral history by increasing its recognition and employment opportunities, broadening its scope, and improving its funding.

So what is evaluation? The American Evaluation Association broadly defines evaluation as “a systematic process to determine merit, worth, value, or significance.”³ More specifically, evaluators collect data using quantitative (e.g., surveys and test scores) and/or qualitative (e.g., interviews and focus groups) methods to answer questions such as: Did a program meet its goals? Is a policy effective? What improvements are recommended? For example, we may use attendance reports and testing data to determine whether an after-school program for academically delayed youth increased participants’ school attendance and test scores, then interview the staff to gauge if the program is run according to best practices. Or we may survey employees to measure their level of satisfaction with a professional development activity and conduct focus groups to determine if and how they are implementing what they learned. Evaluators can come from a variety of settings; some are internal to a program or business that they evaluate exclusively, while others are independent contractors that can choose to evaluate one program or several. As the evaluators for BUILD PODER (Building Infrastructure Leading to Diversity Promoting Opportunities for Diversity in Education and Research), we are part of an independent evaluation group at the University of Mississippi called the Center for Research Evaluation. We work with universities, private foundations, and state and federal government agencies to provide actionable answers to important questions about the impact of their work.

Though this is a very basic way of describing a multifaceted field, it begins to illuminate how oral history could benefit it. Still, as stated earlier, we were surprised that we could not find the use of oral history in any published evaluation manuscript. A comparative historiography, based on the works of oral historian Alessandro Portelli and evaluator Susan Giancola sheds some light on the likely reason for this. In short, it is a matter of objectivity versus subjectivity. Evaluation, well before becoming a formal field, was based on objective quantitative inquiry. The earliest evidence of systematic evaluation, dating back to 2200 BCE, included employee proficiency testing. This practice continued in various forms for roughly four thousand years, around which time the first student grading system was developed in the 1700s. After this, when reaching beyond evaluating people to also evaluating programs, methods still tended to be quantitative in nature.⁴ In fact, Giancola explains, “Due to the quantitative nature of evaluative systems through the mid-1800s, many educators and lawmakers equated assessment and measurement to evaluation. That is, evaluation was narrowly seen as the quantitative assessment of outcomes.”⁵ This was the case until about fifty years ago when program evaluation became a defined field and sociologists introduced the use of qualitative methods. Still, even the qualitative methods evaluators use today (most broadly, interviews) strive to produce objective, quantifiable (via coding) data. Oral history, on the other hand, tends to be highly subjective. Portelli notes that oral history is about the meaning behind experiences rather than the mere facts about events they portray. This means that the “facts” of the stories are true in the sense that they are the individual’s recollection and interpretation of their own experience but cannot always be objectively validated.⁶ This does not mean they are useless to an evaluator, but rather that they must be put into context and used alongside other, more objective, measures to paint a richer, fuller picture of stakeholders’ experiences.

As the field of evaluation continues to develop and respond to contextual needs, so too do its methods. As King and Alkin point out, “The field has acknowledged the critical importance of diversity and inclusion and of evaluators’ need to focus on issues of power and equity. This is directly related to the broader acceptance of multiple approaches to knowledge creation.”⁷ David Fetterman, for example, uses ethnography “to explore rich, untapped sources of data not mapped out in the research design.”⁸ His research involves field work that immerses him into the lives and cultures of the people and programs he evaluates, and like oral history, seeks and embraces insider perspective. As we will show using a program evaluation we conducted as an example, oral history can similarly be a useful knowledge-creation tool for programs and evaluators seeking deeper participant involvement and more meaningful experiential insight than traditional evaluative methods, such as surveys and interviews, typically provide. While we recognize the differences between an evaluation and oral history, we will share how we adapted each to honor oral history while also meeting our evaluation needs.

Program Background

The purpose of our particular evaluation is ultimately to determine the extent to which an undergraduate research training program known as BUILD PODER, based in CRT, leads to student success in pursuing graduate studies and a research-related career, particularly in the area of health disparities.

Housed at California State University Northridge, BUILD PODER is one of ten institutions funded by the National Institutes of Health to engage, support, and retain undergraduate students from diverse backgrounds in biomedical research and other science, technology, engineering, and mathematics (STEM)-related fields.⁹ BUILD PODER provides various on- and off-campus activities, CRT-based research mentoring, tuition assistance, monthly stipends, and degree-progress advising to meet its site-specific goals of increasing students’ research self-efficacy, confidence, science identity, and interest in pursuing biomedical-related graduate programs. Through the lens of CRT, BUILD PODER develops initiatives on and off campus to support students in succeeding in their undergraduate education and identifying appropriate research mentors and projects, navigating graduate school application processes, and, later, transitioning to their graduate programs. Beyond that, BUILD PODER introduces and reframes traditional research approaches through workshops and webinars based on health and social equity. That is, researchers of various disciplines are encouraged to develop research questions that focus on health equity, while at the same time considering lived experiences of diverse students as an asset to their research projects.

Method Choice

We sought a well-grounded, yet meaningful, qualitative data-collection method to complement and enhance quantitative data—from annual surveys and academic tracking—in a mixed-methods evaluation approach, which not only incorporated the strengths of both quantitative and qualitative methods, but also supported the use of methods that are best suited to answer the evaluation questions.¹⁰ With this in mind, we considered details about the program we were evaluating to find the most appropriate qualitative data-collection method.

As stated above, the training program we were evaluating is grounded in CRT, an intellectual movement that emerged out of law in the 1970s and began to gain currency in the late 1980s and the 1990s.¹¹ Of all disciplines, education was one of the first to draw upon CRT with the publication of pedagogical theorist Gloria Ladson-Billings and social scientist William Tate’s seminal piece, “Toward a Critical Race Theory of Education,” in 1995.¹² Drawing on insights from legal studies,¹³ interdisciplinary scholar Daniel Solórzano articulated the five main themes that form the basic perspectives, research methods, and pedagogy that would eventually be popularized and become what are known to many today as the five tenets of CRT in education:¹⁴ the centrality and intersectionality of race and racism; the challenge to dominant ideology; the commitment to social justice; the centrality of experiential knowledge; and the interdisciplinary perspective.¹⁵ Other core CRT tenets include interest convergence, counter-storytelling, and whiteness as property.¹⁶

CRT scholars in education have produced an abundance of research, including how race or racism have influenced a student’s science identity, the experiences of Latino male engineering undergraduates within an engineering retention program, the integration of faculty of color in academia, college admissions in relation to affirmative action, and how the legacy of institutional racism impacts students of color negatively.¹⁷ By centering the perspectives of marginalized groups in discussions, these research initiatives enrich mainstream understandings of problems and offer new insights that have been wanting heretofore.

In response to the traditionally poor STEM outcomes for marginalized and minority students, BUILD PODER’s approaches of matching students with trained faculty mentors, placing them in research labs on and off campus, exposing them to careers in the STEM fields, developing a sense of community among student cohorts, and providing financial support align with more recent initiatives at other US academic institutions, from high school to colleges and universities, aimed to encourage STEM education among students from diverse backgrounds. For example, work conducted among African American students attending a STEAM (science, technology, engineering, arts, and mathematics) high school found that the lack of minority students in STEM fields was due to an overall lack of information and exposure to the fields, relatable role models, and support from families who may not understand the value of college.¹⁸ More specifically, because working has allowed their family to survive—and perhaps even thrive—throughout history, they may not understand the worth of going to college or graduate school, and particularly spending money to do so. A longitudinal study of 1,420 undergraduate and graduate minority students from fifty different US universities found that good mentorship and research experiences were positively related to students’ science identity, self-efficacy, and values.¹⁹

Because BUILD PODER student participants are primarily female (71%) and the majority are Latinx (55%) and/or first-generation college students (58%),²⁰ program leaders use a CRT-based approach to help them overcome inherent barriers to college and career success often faced by individuals with similar backgrounds. As such, to evaluate this program we needed to find an appropriate methodological approach that challenged dominant perspectives, highlighting alternate views and experiences. Oral history seemed to be the perfect methodological choice because, “While in-depth interviews tend to be topic- or issue-focused, oral history interviews often cover an extensive part of a participant’s life, seeking to uncover processes and link individual experiences with the larger context in which those experiences occur.”²¹ For our purposes, rather than conducting traditional interviews that would utilize a narrow set of restrictive questions that provide information only about experiences within the training program, our more-fluid and more-personal oral history interviews encouraged students to share stories about their upbringing and cultural background, as well as how those experiences and personal identities framed their academic, research, and mentoring experiences. As stated earlier, our narrative inquiry approach to oral history also allowed us to help raise the students’ voices, so often unheard due to their marginalized and minority status (see Table 1 for demographic breakdown).

Table 1. Participant Demographic Statistics

Gender	Percentage, n
Female	76%, n=28
Male	24%, n=9
Race/Ethnicity	Percentage, n
Asian/Asian American	19%, n=7
Black/African American	5%, n=2
Hispanic/Latinx	54%, n=20
Pacific Islander	3%, n=1
White/Caucasian	11%, n=4
Other (specified as “Filipinx American”, “Egyptian”, or “Latino/Middle Eastern”	8%, n=3

As previously noted, oral history is not a common evaluation method. We believed, however, that oral history’s application in gaining a deeper understanding of topics of interest and community experiences²² would be a useful method as we attempted to learn about our participants’ life experiences and identities, and how these might impact their college and career success. We were not disappointed. As shared later in the Preliminary Findings section, information students shared during our first two years of oral history interviews allowed us to draw important conclusions about their college and career experiences, as well as program-related outcomes. We plan to conduct follow-up interviews over the next three years to track these students, and others as they join the training program, to continue connecting how their identities and experiences—both in the past and in the training program—shape their career decisions. While not the sole component of our evaluation (we also incorporate surveys and secondary data as complementary quantitative methods for more objectively measurable outcomes), oral histories provided us rich first-hand data, as well as positive experiences for both the interviewers and narrators.

Our Oral History Experience

Here we describe our two-year-long journey (so far) learning about and using oral history. We hope the reader will understand that, while some of our terminology and processes may not fully reflect those of veteran oral historians, we have put considerable effort into acquiring training and preparation from an oral historian so we could replicate the essence of this method in which we see great value.

Interviewer Experience: Planning

The interview team consisted of five individuals in year 1 and seven in year 2, representing a variety of gender, national, racial/ethnic, sexual, and academic identities. Though each interviewer had prior experience in evaluation and/or interviewing techniques, none of us had specific experience in oral history, so we sought advice from Jessica Wilkerson, at the time an oral historian at the University of Mississippi, who helped us determine that using oral history as our method of gaining insight into student background and experiences was fitting. Having never used this method before, our team then acquired Jessica’s assistance in teaching us what we needed to know to conduct oral histories effectively. Our training in year 1 consisted of a three-hour, in-person workshop that included such topics as background and ethical considerations, recording equipment, question development, and interview technique (including ethical interviewing). Jessica further explained the importance of understanding interviewer-narrator dynamics.²³ Year 1 interviewers and new ones that joined the team in year 2 reviewed Jessica’s training resources in a self-study format to prepare for the year 2 interviews. New interviewers also practiced interviewing techniques with those that had experience interviewing in year 1 to ensure their capability.

We then developed the questions (see protocols in appendix 1) with Jessica’s input and the following goal in mind: giving students a safe space to talk about their past and current experiences in an effort to evaluate the training program that aims to impact their future. The training program itself, guided by CRT, prioritizes embracing the participants’ personal histories, encouraging them to use these experiences and cultural assets to change the scientific landscape. This link between our methodological goal and BUILD PODER’s priorities were key to using oral history as a component of our evaluation successfully.

We elected not to request that narrators bring related artifacts or documents to simplify the process as we got our feet wet with this new method, but we hope to include them as an option in future years to add another layer of meaning-making. We did, however, utilize information about the training program’s history and setting, provided by the program team. We gained further knowledge about these and related cultures (of the university, the training program, and narrators) through a site visit that included exploratory conversations with the program team and student participants.

With ethical and privacy considerations in mind, the University of Mississippi’s Institutional Review Board reviewed our protocols, which opened with detailed information, including the interview’s purpose, as well as confidentiality and anonymity assurances, such as being assigned a pseudonym or code so no names would be associated with the stories we shared. Though recordings were transcribed, we replaced names with codes in the transcript. Before proceeding with the interview, we obtained consent to participate from each narrator and explained that they could stop the interview or withdraw consent at any time.

Interviewer Experience: Interviews

We intended to conduct the interviews face-to-face; however, due to COVID-19 restrictions we conducted them via Zoom with our cameras turned on during the interview. We encouraged (though did not require) that narrators do the same to simulate a face-to-face experience and allow us to respond to facial and behavioral cues throughout the interviews. We also recorded the interviews, with narrators’ consent, using Zoom’s audio/video-recording capabilities and cellular phone recordings for backup.

By the end of our evaluation’s second year, our original twenty-eight narrators (with the exception of one who did not participate in year 2) had participated in two one- to two-hour interviews. Since the training program added new students each year, we too added narrators, with an additional nine participating in their first interview during year 2. We changed most interview questions for narrators’ second interview to get a greater breadth of experiences and updates but were consistent in terms of ensuring questions were open-ended, which allowed us to capture rich stories, even for those who seemed hesitant to share initially. They eloquently recounted stories of resilience, often sharing freely about their successes, but also their struggles and flaws (see Preliminary Findings section for more details). Had we used a structured or semistructured interview, as is most typical in qualitative evaluation, our questions, and thus participant responses, would have been much more limited; we likely would have gotten only surface-level information.

Our interview questions focused on the students’ experiences prior to, during, and—once they graduated—after college, without focusing much on the training program itself. This allowed us to focus on the students’ stories without making them feel like we were only interested in their BUILD PODER experiences. We were able to do this and still draw comparative conclusions about key program outcomes (e.g., experiences with mentors, feelings of science identity, sense of belonging, etc.) by using a control group of demographically matched non-program students. While a control group is not typically part of the oral history process, it helped us not only tell the stories about the program we were evaluating, but also allowed us to learn about how these students’ college experiences differed from those of similar non-program students and understand the university experience more broadly.

Interviewers completed a field note after each interview to capture immediate thoughts on the interview process (e.g., location, perceived comfort level, intersubjectivity) and content (e.g., recurring themes, absent constructs). As described in the following section, these field notes helped inform later analysis.

Interviewer Experience: Analysis

We found analyzing the oral history interview transcripts to be surprisingly challenging. We did not realize, until we began the analysis process in year 1, that we would have to adapt our typical interview-analysis process significantly. When analyzing traditional structured or semistructured interviews, we sorted the data into codes, often based on interview questions and common respondent themes. Because oral history interviews are much less structured, often including personalized questions and responses, we had to alter how we conceptualized and defined themes. Instead of finding common themes within the transcripts, we developed a priori (i.e., theoretical) themes based on information contained in the interview field notes, the literature on gaining knowledge from narratives,²⁴ and a sample analysis of four interview transcripts. Using these sources, we identified four primary themes—settings, characters, self, and stories—and then added additional subthemes during the coding process to add depth and richness to our interpretations. While this provided a workable coding structure for our first attempt, we recognized a need to adapt it for future analyses.

Thus, in year 2 we moved to an inductive constant comparison process,²⁵ whereby we noted patterns across interview transcripts to develop and refine our coding structure continually. We first developed a coding structure based on the interview protocol items and common themes found in the field notes and interviewer comments. Each member of the analysis team then coded four interviews while completing analytic memos (i.e., reflective notes about themes, ideas, and patterns that came up during the analysis process)²⁶ to identify necessary revisions to the preliminary coding structure. We then coded all interviews using the final coding structure and summarized the findings. While this analysis process was an improvement compared to year 1, we realized that the individualized nature of oral history led to a coding structure that consisted of a large number of themes with a small number of narrators endorsing each one. This made summarizing very time consuming, which could be a barrier for some evaluations. We recognize that it will be important for us and other evaluators to refine and streamline the analysis process as we continue to use oral history in the future.

As we prepared to share our analysis findings with the project team, we also recognized a tension between honoring individual stories—as oral history strives for and our narrators deserved—and respecting their anonymity—to ensure no harm would result from sharing their true thoughts and feelings about a program they were either still participating in or still required support from (e.g., letters of recommendation, graduate school application support). We found that by analyzing interviews individually, but with a comprehensive coding structure, and reporting not only aggregate findings but also individual (de-identified) quotes, we were able to ensure anonymity while also honoring individual stories. While recordings are currently saved in a secured Box folder to ensure privacy, we have informed the narrators that we may submit them to a repository in the future, with their permission, to share their stories with a wider audience.

Interviewer Experience: Preliminary Findings

Though we will follow our narrators’ stories with annual interviews throughout the remainder of our five-year project, we have learned a great deal from them over the past two years—far more than we could have learned from surveys or structured/semistructured interviews.

For example, roughly two-thirds of the narrators (program participants and non-members) revealed stories of struggling with mental health and well-being. In the following exchange, the narrator shared some deep feelings she had been having, which led to a related health problem.

Interviewer:

I know you mentioned, too, about having these, I forget what you called it, but “sad girl” moments? What did you mean by that?

Narrator:

Is it not normal to have three consecutive, just give up days on the floor? Is that normal? …

Interviewer:

So, these are days where you really can’t do anything. Just need to cry it out?

Narrator

Burnt out. Done. That’s all I can give. I’m going to sit here and do something mindless, like play solitaire or scroll on Instagram or read some book that I read when I was 10.

Interviewer

Mm-hmm (affirmative).

Narrator:

Because that’s the amount of “mmmm yeah” I have. Anything. That’s it. That’s all we got. I’ve been better. I used to care a lot more. A thing I’m supposed to say senior year of college. I used to care a lot more. I don’t know. I went to the doctor so many times because every time it was finals week or midterm week or something. Oh, I faint. All the time … Not so much this past year, because I’m always near furniture. I’m always sitting down. I don’t have to go anywhere, so it’s fine. But I used to barely sleep, just eat whatever junk was near me. And it was … I’d faint. And I don’t know if it’s the not caring or the always sitting down, but that hasn’t been happening as bad this year. So …

Interviewer

That’s good. I definitely want you to be safe and take care of yourself.

Narrator

Yeah. I don’t think I was doing the whole work-life balance thing very well.

Narrators also shared a great deal about the difficulty of choosing a career pathway, often explaining the ups, downs, twists, and turns as they describe the rollercoaster of making this important decision. Many, however, agreed that finding meaning in a career would allow them to feel of value to others.

Despite these struggles, many expressed a sense of belonging at the university, describing ten key factors that helped them feel that someone cared about them and that they were not alone: (1) relatedness, (2) diversity, (3) interest groups, (4) friendly and helpful people, (5) personal connections, (6) welcoming environment, (7) faculty interest in them, (8) similarity to others, (9) people who care about them, and (10) tools facilitating communication and connections. Conversely, the following ten factors hindered the sense of belonging for some: (1) unapproachable faculty, (2) feeling like they are wasting faculty members’ time, (3) being put down by faculty members, (4) feelings of self-worth, (5) lack of interpersonal connections, (6) large groups, (7) feeling like they do not fit in, (8) being unable to join for personal reasons, (9) others not understanding their background, and (10) explicit racist or sexist behavior toward them.

For BUILD PODER participants specifically, we learned about the various types of support they got from the program (e.g., aspirational, emotional, functional/financial, opportunities/connections) that then contributed to career-preparation, shared experiences, and race-conscious mentorship. Program participants, more than non-members, engaged in self-reflection that involved sophisticated deliberations about which graduate school options/opportunities to pursue and described stronger levels of persistence toward biomedical research careers.

Interviewer Experience: Lessons Learned

As this first experience with oral history was intended to help us improve our interview and analysis technique for additional interviews during the training program’s remaining three years, we reflected on the process to learn from it. For example, we learned that it is important for us to remember that the stories shared during oral history interviews are the narrators’ perceptions of their experiences as much as what actually happened. As Alessandro Portelli explains, “Oral sources tell us not just what people did, but what they wanted to do, what they believed they were doing, and what they now think they did.”²⁷ We are gaining insight from their perspective and their way of recalling the events they experienced. While evaluators strive to use tools with known validity, we consider the stories we hear through oral histories to be completely valid to the extent that they are how the narrator interprets their own experiences. This position is congruent with the CRT tenet that experiential knowledge is an asset that informs research. It is important to recognize, however, that taking narrators’ interpretations at face value also limits how certain we can be of the conclusions we draw from them. Therefore, a multimethod approach is optimal when including oral history in an evaluation.

We further learned that while we strive to pose general questions to elicit deeper responses, for an evaluator with specific outcomes in mind it is important to gear some questions toward those outcomes if the stories do not adequately cover those areas. For example, in year 1 we did not ask specifically about if/how mentors incorporate CRT into their mentoring practices. We hoped that this would come up organically as narrators talked about their mentors, but in many cases it did not. As we wanted to delve deeper into this area, we added more prompts about mentor behaviors related to culture/race in the year 2 interview. Similarly, in an effort to acquire more knowledge about how our narrators’ identities impacted their lives and choices, we added specific questions about this for year 2. We are cognizant of the line we need to tow between allowing narrators the freedom to talk about what they feel is most important while also gleaning information that is important to answering our evaluation questions.

Narrator Experience: Interviews

Throughout the oral history process, from recruitment to interview, we strived to express to participants that their stories held meaning and we were eager to hear them. We explained that understanding their background and other experiences helped us make connections between these experiences and their college and career decisions. On the whole, our narrators seemed to enjoy the opportunity to share their stories, perhaps even finding the experience cathartic. We drew this conclusion because most narrators provided an open and eloquent retelling of their past and current experiences, and even those that were somewhat hesitant in the beginning warmed up to the idea as the interview progressed. Leavy speaks of this potential effect oral history interviews could have on the narrators, ²⁸ noting that the opportunity to share so deeply about themselves is often affirming, rewarding, and even empowering. She further hypothesizes that narrators could gain understanding and self-awareness as they reflect on their life. We hope as well that narrators will be more engaged in all aspects of our evaluation because they see that we consider their perspective to be very important to the evaluative process.

Narrator Experience: Planning for the Future

As we have throughout the process thus far, we are prioritizing narrator experience as we make our plans for future interviews. One way we plan to do this is by eliciting new stories that allow us to make connections between those shared in the initial interviews, mainly focusing on background and early college experiences, with stories of more current experiences and plans for the future. We also plan to delve deeper and more specifically into the influence of race and culture, including barriers, strategies, and the mentoring relationship.

Given our population and the training program’s focus, race is an important theme in our evaluation. Thus, it is very important that we recognize and consider that our team of interviewers observed that narrators opened up more readily, especially regarding race, with our interviewers of color. Though it is impossible to know for sure, we believe that this is because narrators of color felt a connection with them.²⁹ We intentionally enlisted interviewers of color with that possibility in mind and were not surprised by this observation. In an attempt to grow our interviewer-narrator relationships for future interviews, we hope to keep (as much as possible) the same interviewer-narrator pairs each year. We made careful notes about important topics each narrator shared so that we can follow up on them to build further the relationships developed thus far. Our white interviewers, especially, will also continue to strive to create a safe space for narrators to talk about racial issues more comfortably.

Oral History in Evaluation: Benefits

Putting our particular project aside for a moment, we began to consider how oral history could benefit the field of evaluation. Evaluations vary in terms of several components, such as client and participant demographics, program goals, monetary resources, and timelines. Additionally, evaluation is a broad field with a wide range of applications, and each application requires careful consideration and planning to address the primary goals and evaluation questions. Evaluation questions and purpose drive the evaluator to choose what they believe to be the ideal type of evaluation, sometimes mixing several approaches, to meet their clients’ needs. There are a wealth of options from which the evaluator can choose (Patton lists 79), and the type of evaluation they choose influences which methods are most appropriate.³⁰ With all of this in mind, we considered which types of evaluations and evaluation components oral history would benefit most.

Program Goals

Program goals and theoretical foundation were influential in leading us to consider using oral histories in our evaluation. Our goals contain rich racial and cultural themes, such as how the program thrives to help students learn to overcome racial and cultural barriers along their career path and teach them how they can use their cultural assets to succeed. This focus pushed us to seek a method that would allow us to explore how racial and cultural themes influenced our participants. Oral history’s focus on storytelling and uncovering deeper meaning about individuals’ lived experiences provided the level of understanding that our evaluation required. It aligns with CRT’s counter-storytelling tenet that is used to elevate minority voices, perspectives, and experiences while allowing for the expression of nondominant narratives. In addition to its usefulness for evaluations involving themes of race and culture, oral history could provide similar insight for other themes such as gender, sexual orientation, or religion to capture experiences at the intersections of multiple oppressions that are often erased, as CRT scholars note.³¹

Participant Demographics

The themes above have a common thread of focusing on minority or marginalized groups. In this way, theme is often related to participant demographics and is why oral history was particularly useful for our evaluation. Not only were our goals tied to race and culture, but also the vast majority of our participants (95%)—like the broader BUILD PODER population—were from at least one marginalized group that has traditionally experienced a high level of social injustice (refer back to Table 1). Additionally, for groups that are often not listened to when they speak up about issues they have faced, oral history could provide them a welcome outlet in which to share their stories and ultimately be heard by those with the power to make institutional change. These benefits are consistent with CRT tenets that affirm one’s experiential knowledge and challenge dominant ideology. This is not to say that oral history is only appropriate when participants are from minority or marginalized groups. The key here is their common life-altering experience. Individuals from majority populations can also experience life-altering experiences of interest to evaluators, which can be acute (e.g., 9/11 or a devastating hurricane), or chronic (e.g., drug use/recovery, body image issues). Again, when considering if oral history is appropriate for a particular evaluation population, the key is whether participants have experienced a common life-altering experience, either chronic or acute, that relates to the goals of the evaluation.

With demographics in mind, we propose that oral history is optimal specifically for a number of participant groups, including those that are historically storytellers (native and aboriginal groups), students (they may speak more freely because it has less of an “exam” feel than a survey or more-structured interview, and it also provides them with a feeling of autonomy that they have craved much of their lives), and budding researchers (because the narrator often feels like their stories are an important part of the evaluation process).

Evaluation Types

Oral history, because of its usefulness in gaining meaning and perspective from those whose stories may not otherwise be heard or documented historically,³² would be particularly beneficial within specific types of evaluations, especially those that emphasize a cross-cultural or person-focused approach (see Table 2), which seek a deep level of understanding of their participants. Compared to a survey, oral history interviews tend to encourage the narrator to provide much more detail because the interviewer listens attentively and can ask follow-up questions for more clarity. Likewise, an oral history narrator typically discloses more personal revelations than a focus group participant because the one-on-one setting is more private and specifically aims to be a safe space for sharing. Differentiating oral histories from other types of interviews, Leavy places oral history within a continuum,³³ falling among the open-ended interview types, but less so than the minimalist biography interview. She describes oral history as “a collaborative exchange, in which the researcher and research participant are cocreators in the knowledge-building process,”³⁴ and delineates five instances in which oral history data would be most useful, two of which are particularly relevant to research and evaluation: contributing to understanding of topical areas and gaining “community” experiential knowledge. As qualitative methods go, oral history is optimal for participatory evaluations that aim to foster participant engagement and ownership. It is also particularly well-suited for the evaluation types in Table 2 because oral histories are intentionally open-ended to encourage the narrator to share their stories freely, while also being just structured enough for the interviewer to help guide the conversation toward greater understanding and meaning. However, in the next section we will explain why using oral history alone is not recommended for evaluation purposes.

Table 2. Appropriate Evaluation Types and Focuses for Oral History Use

Evaluation type	Focus^1
Culturally-responsive	“ … focusing on the influences of cultural context and factors on program processes and outcomes”
Diversity focus	“The evaluation gives voice to different perspectives on and illuminates various experiences with the program.”
Inclusive	“ … deliberate emphasis on including groups that have historically experienced oppression and discrimination on the basis of gender, culture, economic level, ethnicity or race, disability, sexual orientation, language, or religious preference.”
Personalizing	“Portrayal of people’s lives and work as contexts within which to understand a program.”
Social justice focus	“How effectively does the program address social justice concerns?”

¹Excerpts from more-comprehensive table in Patton, Utilization-Focused Evaluation, 300-305.

Limitations of Use within Evaluation

While oral history can be a good methodological fit for certain types of evaluations, as described above, it is important to consider the following limitations before incorporating oral history into an evaluation plan.

Not Stand-Alone

Using a mixed-methods approach with both qualitative and quantitative methods increases research rigor and validity;³⁵ thus, oral history should not be used as a stand-alone evaluation method. Rather, an evaluator can reinforce the meaningful information gained in the more-subjective oral history interview by supporting it with more-objective qualitative data. In our evaluation, we developed a survey to reach a larger number of participants (2020 survey n=271, 2021 survey n=281; 2020 oral history n=28, 2021 oral history n=37) and ask more (and more explicit) questions. These surveys provided breadth, while the oral history interviews provided depth. While the insight we learned from our oral history interviews has been (and no doubt will continue to be) invaluable to our evaluation, we also recognize that such data is strengthened by adding a complementary quantitative approach.

Time

Another important consideration for evaluators seeking to incorporate oral history into their evaluation plan is time. Oral history interviews typically require more time than other types of interviews—one to two hours each—and are most useful when each narrator participates in multiple interviews.

Another time-related issue is training. We encourage first-time oral history interviewers to seek out training from a seasoned oral historian, as we did, to build an appreciation for the method, and most importantly, learn related ethics and best practices. Similarly, we recommend that those new to this method set time aside to practice the art of conducting an oral history interview, as well as using your particular questions, in an effort to not miss out on important revelations in your first interviews by learning as you go.

Analyzing oral histories can also be a very time-consuming process, more so than traditional interview or focus-group analysis. Though oral histories are centered on a common event or experience, they focus on individual stories; thus, common themes do not always emerge clearly at first. This can lengthen the time needed to develop a coding system from which to analyze these stories.

Cost

Oral historians can be ideal consultants when embarking on an evaluation using oral history. As stated previously, their expertise is important throughout the training process, but oral historians could also be brought into an evaluation team to conduct the oral history interviews themselves. They are already experts in the field and would possess important skills that beginners may take a great deal of time to refine. That said, bringing in an oral historian includes its own limitation—cost. Whether the oral historian is brought in to train the interviewers or to conduct the interviews, the budget must also include these consulting services.

If the evaluator/team plans to conduct the oral history interviews themselves, the evaluation budget should also allow for the additional cost required to train for, conduct, transcribe, and analyze this time-consuming method (as detailed above).

Evaluation Types

While oral history is an appropriate addition to certain types of evaluations (noted above), it is not recommended for all evaluations. For example, oral history interviews are a highly inappropriate method for cost-benefit analyses that only compare costs with outcomes (benefits) quantitatively. Likewise, a personnel evaluation, which aims to determine staff members’ effectiveness in carrying out their responsibilities, would not benefit from incorporating oral history interviews. Even in cases where some qualitative data is useful, other types of interviews or focus groups would be preferable over oral histories if stories are irrelevant. In general, if life-altering experiences are not a key focus of the evaluation, oral history interviews would provide little to no value at great time and cost.

Conclusions

Oral history is “both the oldest type of historical inquiry, predating the written word, and the most modern.”³⁶ Despite its lack of use as an evaluation method in the past, we found that the stories we obtained through oral history added meaning and depth to our BUILD PODER program evaluation, particularly because of the culturally diverse participants and the program’s CRT focus, which seeks to give voice to those whose voice is often stifled. Though more subjective than the more frequently used evaluation methods, we believe oral history could similarly benefit a variety of evaluations, especially those that are more participatory and experience-based. In fact, in addition to enhancing our evaluation, we found that oral history directly supported what the program itself was trying to accomplish (e.g., raise voices, build empowerment, etc.), as during the oral history interview, the narrator is transformed in the moment. Many narrators shared stories with us that they previously had shared with few (if any) others, but through the interview process they became more empowered because of it.

As with any evaluation method, before including oral histories in an evaluation plan, evaluators should consider the evaluation’s purpose, goals, budget, timeline, and personnel. Oral history is most suited for evaluations whose purpose and goals could be supported by participants’ personal stories and experiences, but it requires a great deal of time and personnel (and thus, financial resources), not only for conducting the interviews themselves, but also for training, transcribing, and completing the analysis. It is also important to remember that, like most evaluation methods, oral history should not be used alone, but rather in tandem with other complementary data sources—preferably including objective quantitative ones.

We also highly recommend evaluators seek knowledge and consultation from a practicing oral historian, as we did, to gain a better understanding of how oral histories differ from more traditional interviews, specifically in terms of process and interviewer-narrator dynamics (e.g., intersubjective, dialogic). In addition to the knowledge and experience they possess from conducting oral histories themselves, oral historians also bring “skills in understanding and incorporating different perspectives,” which is important for all evaluative work, but is particularly important when conducting cross-cultural evaluations.³⁷

Implications for the Field of Oral History

While we have made the case for how and when oral histories could benefit the field of evaluation, we now suggest that their use in evaluation could benefit the field of oral history in several ways as well, including growing its recognition and employment opportunities, broadening its scope, and improving its funding.

Increased use by a variety of people for a variety of purposes will continue oral history’s recognition and evolution as a field. For example, as oral history becomes useful in new ways, more people will learn about and embrace it, making the field more acknowledged and “marketable.” In fact, a new line of employment could open to oral historians if evaluators begin including oral history in their evaluation plans but prefer to have an oral historian conduct them, rather than investing time and money in acquiring their own training. Additionally, while oral history has a rich tradition of qualitative documentation and data collection, its use in evaluation—alongside quantitative methods—could help oral history evolve by broadening the scope of the technique as part of a mixed-method approach. This approach, often preferred by funders, may help oral historians secure more funding by showing that what they learn from oral histories can be backed up by quantitative data, while also demonstrating that oral histories add much richer information than quantitative data alone.

Along these lines, the amount of funding available to support evaluation provides an advantage over most oral history projects. Over fifty years ago, Donald Swain cautioned would-be oral historians about the time and cost associated with oral history,³⁸ and this remains an issue for oral historians today. Oftentimes, even given the best of intentions, oral histories go un-transcribed and unshared for long periods of time—if not indefinitely—due to financial and time-related limitations.³⁹ Because evaluation is often better funded, and reporting on findings is expected, evaluators—and oral historians who complete oral histories for evaluation—are less likely to encounter these common challenges, ensuring that others will hear what narrators have to say. With that being oral history’s ultimate goal, we propose oral history’s use in evaluation to be a mutually beneficial endeavor.

Acknowledgments

This research was supported by grants funded by the National Institute of General Medical Sciences (NIGMS) Building Infrastructure Leading to Diversity (BUILD) Initiative, grant numbers TL4GM118977, UL1GM118976, and RL5GM118975. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIGMS.

Supplementary Material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/00940798.2023.2284391

Disclosure statement

No potential conflict of interest was reported by the author.

Additional information

Funding

This work was supported by the National Institute of General Medical Sciences (NIGMS) Building Infrastructure Leading to Diversity (BUILD) Initiative under Grant numbers UL1GM118976 and TL4GM118977. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIGMS.

Notes on contributors

Shannon R. Sharp

Shannon Sharp is an Evaluation Associate at the Center for Research Evaluation at the University of Mississippi. Email: [email protected].

Judith C. P. Lin

April Grayson is Director of Community and Capacity Building at Mississippi-based nonprofit The Alluvial Collective, where she facilitates dialogue and education programs, leads equity workshops, and oversees oral history, public history, and documentary projects. Over the past two decades, she has worked in film and video, oral history practice, and arts education, screening films in festivals, galleries, and museums. Email: [email protected].

Patchareeya P. Kwan

Sarah Mason is Director at the Center for Research Evaluation at the University of Mississippi. Email: [email protected].

Sarah Mason

Judith C. P. Lin is Research Associate at the Health Equity Research and Education (HERE) Center at California State University, Northridge. Email: [email protected].

Jessica Wilkerson

Jessica Wilkerson is the Stuart and Joyce Robbins Associate Professor of History at West Virginia University. She is the author of To Live Here, You Have to Fight: How Women Led Appalachian Movements for Social Justice (University of Illinois Press, 2019).

April Grayson

Patchareeya Kwan is an Associate Professor in the Department of Health Sciences at California State University, Northridge and one of the multiple Principal Investigators of the BUILD PODER Center funded by the National Institutes of Health.

Notes

^1. Erica Fugger, “The Practice and Application of Oral History: A Field Overview,” US History Scene, accessed August 24, 2021, https://ushistoryscene.com/article/oral-history-1/.

^2. The American Journal of Evaluation, the flagship journal of the American Evaluation Association, published a series of oral histories (The Oral History of Evaluation) from 2004 to 2017, focusing on the professional development of notable evaluators; however, they were not used as an evaluation method. For the most recent of these oral histories, see Melvin M. Mark et al., “The Oral History of Evaluation: The Professional Development of Thomas D. Cook,” American Journal of Evaluation 39, no. 2 (June 2018): 290–304.

^3. “What Is Evaluation?,” AEA Task Force, American Evaluation Association, accessed August 24, 2021, https://www.eval.org/Portals/0/What%20is%20evaluation%20Document.pdf.

^4. Susan P. Giancola, Program Evaluation: Embedding Evaluation into Program Design and Development (New York: SAGE Publications, 2021).

^5. Giancola, Program Evaluation, 33.

^6. Alessandro Portelli, “What Makes Oral History Different?,” in The Oral History Reader, 3rd ed., eds. Robert Perks and Alistair Thomson (New York: Routledge/Taylor & Francis, 2016), 48–58.

^7. Jean A. King and Marvin C. Alkin, “The Centrality of Use: Theories of Evaluation Use and Influence and Thoughts on the First 50 Years of Use Research,” American Journal of Evaluation 40, no. 3 (2018): 450

^8. David M. Fetterman, Ethnography: A Step-by-Step Guide, 4th ed. (Los Angeles, CA: SAGE Publications, 2020), 2.

^9. “Building Infrastructure Leading to Diversity (BUILD) Initiative,” National Institutes of Health: National Institute of General Medical Sciences, accessed August 25, 2021, https://www.nigms.nih.gov/training/dpc/pages/build.aspx.

^10. R. Burke Johnson and Anthony J. Onwuegbuzie, “Mixed Methods Research: A Research Paradigm Whose Time Has Come,” Educational Researcher 33 (2004): 13–36.

^11. Kimberle Crenshaw et al., eds., Critical Race Theory: The Key Writings That Formed the Movement (New York: The New Press, 1995); Kimberle Williams Crenshaw, “Twenty Years of Critical Race Theory: Looking Back to Move Forward,” Connecticut Law Review 43, no. 5 (July 2011): 1253–1354; Richard Delgado and Jean Stefancic, Critical Race Theory: An Introduction (New York: New York University Press, 2001).

^12. Gloria Ladson-Billings and William F. Tate, “Toward a Critical Race Theory of Education,” Teachers College Record 97, no. 1 (1995): 47–68. See also Gloria Ladson-Billings, “Just What Is Critical Race Theory and What’s It Doing in a Nice Field Like Education?” International Journal of Qualitative Studies in Education 11, no. 1 (2010): 7–24; William F. Tate, IV, “Critical Race Theory and Education: History, Theory, and Implications,” Review of Research in Education 22 (1997): 195–247.

^13. Mari J. Matsuda et al., Words That Wound: Critical Race Theory, Assaultive Speech, and the First Amendment (Abingdon, UK: Routledge, 1993), 3.

^14. Daniel G. Solórzano, “Images and Words That Wound: Critical Race Theory, Racial Stereotyping, and Teacher Education,” Teacher Education Quarterly 24, no. 3 (Summer 1997): 5–19; Daniel G. Solórzano, “Critical Race Theory, Race and Gender Microaggressions, and the Experience of Chicana and Chicano Scholars,” International Journal of Qualitative Studies in Education 11, no. 1 (1998): 121–36.

^15. Maria C. Ledesma and Dolores Calderón, “Critical Race Theory in Education,” Qualitative Inquiry 21, no. 3 (2015): 206–222; Carrie L. Saetermoe et al., “Critical Race Theory as a Bridge in Science Training: The California State University, Northridge BUILD PODER Program,” BMC Proceedings 11, Suppl. 12 (2017): 21; Tara J. Yosso, “Toward a Critical Race Curriculum,” Equity and Excellence in Education 35, no. 2 (2002): 93–107.

^16. Derrick A. Bell, “Brown v. Board of Education and the Interest-Convergence Dilemma,” Harvard Law Review 93, no. 3 (1980): 518–33; Daniel G. Solórzano and Tara J. Yosso, “Critical Race Methodology: Counter-Storytelling as an Analytical Framework for Education Research,” Qualitative Inquiry 8, no. 1 (2002): 23–44; Cheryl I. Harris, “Whiteness as Property,” Harvard Law Review 106, no. 8 (1993): 1707–1791.

^17. Mitchell J. Chang et al., “Considering the Impact of Racial Stigmas and Science Identity: Persistence among Biomedical and Behavioral Science Aspirants,” Journal of Higher Education 82, no. 5 (September-October 2011): 564–96; Katherine Rainey et al., “Race and Gender Differences in How Sense of Belonging Influences Decisions to Major in STEM,” International Journal of STEM Education 5, Article 10 (2018); Elvira J. Abrica et al., “Sense of Belonging and Community Building within a STEM Intervention Program: A Focus on Latino Male Undergraduates’ Experiences,” Journal of Hispanic Higher Education (November 2020); Kenya V. Beard and Wrenetha A. Julion, “Does Race Still Matter in Nursing? The Narratives of African-American Nursing Faculty Members,” Nursing Outlook 64, no. 6 (June 2016): 583–96; Octavio Villalpando and Dolores Delgado Bernal, “A Critical Race Theory Analysis of Barriers That Impede the Success of Faculty of Color,” in The Racial Crisis in American Higher Education: Continuing Challenges for the Twenty-First Century, eds. William A. Smith, Philip G. Altbach, and Kofi Lomotey (New York: SUNY Press, 2002), 243–69; Scott Finnie, “Affirmative Action in College Admissions: A Compelling Need and a Compelling Warning,” Ethnic Studies Review 30, no. 1 and 2 (2007): 93–102; William A. Smith, Tara J. Yosso, and Daniel G. Solórzano, “Racial Primes and Black Misandry on Historically White Campuses: Toward Critical Race Accountability in Educational Administration,” Educational Administration Quarterly 43, no. 5 (December 2007): 559–85.

^18. Edward C. Fletcher, Jr. and James L. Moore, III, “Lived Experiences of Low-Income, African American Males in a High School STEAM Academy: Implications for School Counselors,” Professional School Counseling 25, 1 part 4 (2021). https://doi.org/10.1177/2156759X211040030.

^19. Mica Estrada, Paul R. Hernandez, and P. Wesley Schultz, “A Longitudinal Study of How Quality Mentorship and Research Experience Integrate Underrepresented Minorities into STEM Careers,” CBE—Life Sciences Education 17, no. 1 (2018), 1.

^20. These data were provided by BUILD PODER leaders and reflect the 2019–2022 cohorts.

^21. Patricia Leavy, Oral History: Understanding Qualitative Research (Oxford, UK: Oxford University Press, 2011), 10.

^22. Leavy, Oral History, 24-25.

^23. Nora Waters, “An Invitation to the Intersubjective Exchange,” Oral History Master of Arts (OHMA), Columbia University, accessed July 22, 2022, http://oralhistory.columbia.edu/blog-posts/An%20Invitation%20to%20the%20Intersubjective%20Exchange.

^24. Steve R. Yussen and N. M. Ozcan, “The Development of Knowledge about Narratives,” Issues in Educational Psychology: Contributions from Educational Psychology 2 (1997): 1–68.

^25. “Constant Comparative Analysis,” APA Dictionary of Psychology, American Psychological Association, accessed December 20, 2022, https://dictionary.apa.org/constant-comparative-analysis; Brittany L. Peterson, “Thematic Analysis/Interpretive Thematic Analysis,” in The International Encyclopedia of Communication Research Methods, eds. Jörg Matthes, Christine S. Davis, and Robert F. Potter (Hoboken, NJ: John Wiley & Sons, 2017).

^26. Thomas Groenewald, “Memos and Memoing,” in The SAGE Encyclopedia of Qualitative Research, vol. 2, ed. Lisa M. Given (Thousand Oaks, CA: SAGE Publications, 2008), 505–6.

^27. Alessandro Portelli, “What Makes Oral History Different,” in The Oral History Reader, 3rd ed., eds. Robert Perks and Alistair Thompson (London, UK: Routledge, 2015), 52.

^28. Leavy, Oral History, 20.

^29. Belinda Bozzoli, “Interviewing the Women of Phokeng,” in The Oral History Reader, 3rd ed., eds. Robert Perks and Alistair Thompson (London, UK: Routledge, 2015), 212–22.

^30. Michael Quinn Patton, Utilization-Focused Evaluation, 4th ed. (Oxford, UK: Oxford University Press, 2008), 300–305.

^31. Kimberle Crenshaw, “Mapping the Margins: Intersectionality, Identity Politics, and Violence against Women of Color,” Stanford Law Review 43, no. 6 (1991): 1241–99; Vonzell Agosto and Ericka Roland, “Intersectionality and Educational Leadership: A Critical Review,” Review of Research in Education 42, no. 1 (2018): 255–85.

^32. Jessica Wilkerson, “Oral History Workshop: An Introduction to Best Practices” (workshop, University of Mississippi, Oxford, MS, February 4, 2020).

^33. Leavy, Oral History, 12.

^34. Leavy, Oral History, 9.

^35. Judith Schoonenboom and R. Burke Johnson, “How to Construct a Mixed Methods Research Design,” Kolner Z Soz Sozpsychol. 62, no. 2 (2017): 107–31.

^36. “Oral History: Defined,” Oral History Association, accessed August 24, 2021, https://www.oralhistory.org/about/do-oral-history/.

^37. Patton, Utilization-Focused Evaluation, 214.

^38. Donald Swain, “Problems for Practitioners of Oral History,” American Archivist 68, no. 1 (1965): 63–69.

^39. April Grayson (oral historian) in discussion with the lead author, June 2022.