Development and assessment of the STEM storytellers program

ABSTRACT

We developed and tested an NSF IGE-funded year-long program to improve the oral communication skills of STEM graduate students. By the end of the program, we organized our curricular materials into a 3C framework: connect, convey, and captivate. Within this framework, the Fellows learned rhetorical strategies, created podcasts, regularly practiced improvisation, and provided each other with extensive feedback about their work. The Fellows in cohorts one and three gave public presentations followed by a question-and-answer period. Formative and summative assessments were built into the curriculum, including the use of the Public Speaking Competency Rubric (PSCR), and a calculation of jargon. Fellows also longitudinally self-reported their anxiety levels and apprehension of public speaking. The Fellows performed well according to their PSCR scores, but they also reported a slight increase in anxiety levels; the project, as well as the study, were confounded by the COVID-19 pandemic. Public audiences rated cohort three highly on all 3C items. Graduate students in STEM perform research that is of interest and significance to society, and the STEM Storytellers program helped them better translate their research to engage these broader audiences in meaningful conversation.

KEYWORDS:

• Graduate education
• oral communication

Background and motivation

Throughout their graduate education, doctoral students are trained in a highly specific discipline to communicate their research using technically precise terminology, often with a narrow research focus. This scope of study narrows students’ perspectives so that they may become experts in their chosen field. This language-specific, or jargon-based, structure can create communication barriers that inhibit interdisciplinarity or career prospects, especially in careers outside academia (Bullock et al., 2019). While graduate school trains students to communicate with their disciplinary peers, the lack of broad science communication training may hinder students’ confidence and success in communication with the public. This may lead to an increase in anxiety, missed career opportunities, or poor public communication. As we have seen through the duration of the COVID pandemic, the need for effective two-way communication between scientists and the public is incredibly important. Effective dialogue about scientific uncertainty, how the nature of our understanding changes as new evidence is discovered, and even the breakneck pace of vaccine development were all areas that highlighted the importance of effective dialogue between scientists and the public at large.

Scientists report that when communicating with the public (Besley & Nisbet, 2013; Davies, 2008; Mooney, 2009), they consider this type of communication to be one-way – explicitly thinking of it in terms of conveying information, and not as a conversation or exchange. According to scientists in one UK-based study, this type of science communication is frequently viewed as a ‘dangerous’ balance between interest and truth: communication to the public needs to be ‘careful’ so that the public does not misunderstand or misuse the scientific information being dispensed by the speaker (Davies, 2008).

This approach is typically referred to as a ‘deficit model’ of communication, suggesting that the public simply needs adequate education on scientific topics fed to them by scientists with the relevant information. In a survey review by Besley and Nisbet, only about one-third of scientists surveyed in the UK think that the main barrier to greater science understanding is scientists themselves, due to both lack of communication skills and interest. Respondents to this survey were also nearly equally divided on whether or not their science is too difficult for the public to understand (Besley & Nisbet, 2013). This frame for understanding communication with non-experts is reinforced through insular communication during a student’s graduate education (Calarco, 2020). There have been efforts to improve these interactions, primarily by working with practicing scientists to reframe their role as partners in dialogue with the public.

Stakeholders have developed different avenues to facilitate these reframed interactions. Nerd Nite (2022) brings informal presentations to familiar spaces that engender community, fostering relationships between listeners and presenters. Café Scientific (2022) encourages experts to give short introductions and then engage in conversation through a lengthy question and answer period in an informal setting. Other initiatives like the Alan Alda Center for Communicating Science (2022), which was developed by Alda in partnership with the Stony Brook School of Journalism, Brookhaven National Lab, and Cold Spring Harbor Laboratory, focus on targeted training to develop skills for effective science communication.

To help scientists navigate the surprising or unexpected aspects of interacting with the public, there have also been related efforts to include improvisation training for scientists, including improvisation.science, with partnerships in the US and the EU, which focuses on teaching experts to ‘think on their feet’ through improvisation exercises. Improv-PHYS-ation (Watt, 2021) encourages learning structured around possibility, creativity, and acceptance of multiple ideas. Recognizing that there are other models beyond the deficit model for science communication opens other avenues for communicating science that is inclusive, builds community, creative, and impactful (Alterio & McDrury, 2003).

Many interventions focus on training scientists to communicate their research in the form of a story. This act of translation gives context to new information and presents scientific facts in a format that allows for an engaged connection from the audience. Storytelling has a long history of not merely entertaining an audience, but conveying some moral, history, or worldview: ‘Man is essentially a storytelling animal’ (MacIntyre, 1984). Telling a story engages the listener in a way that is more compelling than simply citing a long list of facts. And rhetorical structures like story arc, figures of speech, comparisons, and parallel structure can encourage active listening and information retention (Hadzigeorgiou, 2016). Stories structure intangible or novel ideas in ways that increase understanding and memory: they help listeners remember details and causality, organize their thoughts, and encourage them to invest in the outcome. However, the use of a story arc, that includes a clear beginning, middle, and end is quite different than the typical scientific presentation, which generally includes hypotheses, methods, findings, and a discussion. Shanahan et al. (2019) found that constructing a story arc around a scientific concept can change the way the information is perceived and alter the way people choose to act based on new scientific information. Shanahan found that listeners had a stronger affective response to information about flood risk that included a hero arc, or a victim to hero arc compared to other formats. The authors point out that the stronger affect is directly related to a greater likelihood to act. This type of communication can have deep real-world impacts. The use of a narrative arc to communicate science also invites scientists to ‘translate’ complex concepts into a structure that is familiar to their audience.

Another type of translation that allows complex concepts to fit a structure that is familiar to the audience is the use of rhetorical devices such as similes, metaphors, and analogies. Analogies have long played an important role in scientific reasoning and can be a powerful method for effectively conveying new information. When well-constructed with clear symbolism, rhetorical devices can add clarity without overly simplifying concepts (Pedro, 2021; Skopeliti et al., 2009). Based on a study done with children learning about the phases of the moon and the cause of seasons, Vosniadou and Skopeliti state, ‘ … analogies can be used without reservation to facilitate the learning of science from text and more generally in instruction’ (2019). Other researchers have also found that analogies can be useful for teaching abstract subjects such as electromagnetic waves (Podolefsky & Finkelstein, 2007). The use of familiar concepts to explain relationships, functions, importance, or barriers may be used to help lay audiences understand highly technical or abstract information. With the aid of rhetorical devices to decrease reliance on technical vocabulary or understanding of complex topics, scientists may decrease reliance on jargon to effectively convey scientific information.

The reduction of jargon in science communication is essential: one of the primary barriers to public engagement with science is the overuse of jargon in scientific presentations and writing (Alda, 2017; Leshner, 2018; Mooney, 2009). Based on a study of reading comprehension, Nation found that to gain adequate comprehension, readers need to understand 98% of the words they encounter (Nation, 2006). Bullock et al. tested this point by creating messages about the same three concepts (robot surgeons, self-driving cars, and 3-d bio-printing), using jargon-laden text with one group and jargon-free text with another group (Bullock et al., 2019). Defining jargon as, ‘specialized, technical, vocabulary terms that are associated with a situational context or purpose’, they found that, consistent with Nation, the use of jargon results in readers suffering from reduced processing fluency – lower ease with processing complex information. As a result, readers were more likely to find these new types of technology to be threatening. The perceived level of threat was measurably lower for the group of readers who did not have to process excessive jargon to understand the three new types of technology.

The development and use of strong analogies have been shown to improve understanding of new concepts in a range of age groups. Reducing the amount of jargon increases the public’s cognitive abilities to think through the consequences of technology, and it helps them to understand what the actual scientific message is. The use of a story situates the listener as an active agent and can align the science with the broader societal backdrop (Collins, 2021). Scientists will find audiences have a greater interest in and retention of the information they present if they translate the structure and vocabulary of the information presented and forego the deficit model of communication to embrace a more interactive approach such as the dialogue or ambassador models of science communication (Nadkarni et al., 2019; Reincke, 2020).

These skills are not part of the traditional training model for STEM graduate students. Recognizing this need, five instructors, a professional actor, and a scholarly communication librarian developed and implemented an NSF-funded, year-long course for STEM graduate students to improve their oral communication skills. We taught students skills and techniques to improve their ability to engage listeners when discussing scientific concepts. The training focused on the science these students were most familiar with their graduate research projects. Course work included lectures and readings about various types of rhetorical devices and improvisation. As homework, the students created audio recordings based on structured assignments each week. During the weekly meetings, students extensively discussed their research, trying out new metaphors and story arcs, while taking questions and receiving regular feedback from the entire group. As a culminating experience, they presented a short informal talk and Q and A on an aspect of their thesis research. After three cohorts of students completed the program, we have a fully developed curriculum, as well as formative and summative assessment tools for effectively teaching a range of communication skills to STEM graduate students.

STEM storytellers Fellowship

In 2017, we were awarded a National Science Foundation grant to establish a Fellowship program for STEM graduate students to develop science communication skills (NSF IGE Grant 1735124). The purpose of this program was to teach oral communication to STEM graduate students by facilitating their learning of public speaking, improvisational acting, reading body language, and extensive discussion and feedback. Participants in the program refined their communication skills through creating podcasts and speaking at public events.

The application process was competitive; students applied to the program by providing a brief, recorded audio summary of their thesis work. Transcripts of the audio recordings were limited to 500 words, encouraging students to consider how they would present their research in a concise manner. We modified the application process from year to year as we learned how best to evaluate applicants, and specifically, how to use application materials as a pre-test of the program’s impact on their skills. As an example, applicants for cohorts two and three filled out the Personal Report of Communication Apprehension (McCroskey, 1978, 1982), a measure of their comfort level with different kinds of public speaking before the intervention. Applicants also filled out a form gauging their level of interest in learning to talk with various target audiences. Finally, applicants were required to upload a form indicating that their main thesis advisor was aware that they were applying for this program. Eight applicants were selected as Fellows each year, committing to participate in the program for the academic year. Of the 24 students who initially started the program, 22 completed the full year. Most of the Fellows identified as female (16), and four were non-native English speakers, with overlap between these groups.

The instructors developed curricular materials that include units on storytelling techniques, use of analogies, improvisation, podcast creation and editing, and peer critique of the Fellows’ work (details within syllabus). This framework was solidified for cohort two based on feedback from our first cohort. Initially, we did not clearly communicate the learning objectives, and we did not explicitly demonstrate how each activity supported these objectives. To help clarify our objectives and ground students in our desired outcomes we added the following learning objectives:

• Applied exploration of Learning Goals through multiple recorded and rehearsed presentations

• Demonstrated skill with improvisational and storytelling concepts

• Increased understanding/facility with narrative methodologies, analogy

• Increased clarity of communication, minimization of the use of jargon

• Increased skill at understanding the priorities of target audiences

• Increased facility with communicating effectively to those priorities

The full course was two semesters, with weekly classes that lasted 90 min. Over the year, students typically met 28 times in total, according to the university’s semester structure. As a capstone to the course, Fellows created and delivered an informal 10-minute talk to the general public at a local venue, followed by a 10-minute question and answer period.

In the first semester of the training, we situated the Fellows as experts in their field and novices in many other areas to ground them in their knowledge and humility. We introduced the concepts of kindness, experimentation, creativity, and community before framing the course around the 3 Cs, improvisation, and iterative audio recording as a means to develop oral communication skills. Fellows spent time in the course learning to listen to their voices and gaining comfort with their individual styles of communication. Fellows were asked to record specific pieces of communication each week to practice composition, adhering to time limits, adapting for a specific audience, and giving and receiving feedback.

Figure 1. A brief outline of four narrative structures.

Each week the instructors guided discussion on specific factors of communication, such as excitement, humour, interest, what makes presentations boring, keystone facts, conceptual unity, or concept mapping. These discussions were informal and allowed Fellows, instructors, and investigators from the NSF grant to engage and learn together. Additionally, the Fellows participated in improvisation games that reinforced the communication skills of the week. Five weeks into the program, Fellows began to engage in structured peer feedback on audio recordings. The instructors set broad guidelines, including: ‘be open’, ‘ask questions’, ‘be kind’, ‘talk about the work rather than the person’, and ‘assume good intentions’.

In week seven, we introduced the Fellows to various types of storytelling techniques, with the goal of teaching students how to effectively connect with their audience. We discussed the importance of structure to provide suspense, interest, and cultural grounding to new information. Some structures include: the classical narrative structure (Oxford Reference, 2022), the hero’s journey (Campbell, 2003), the ‘and, but, therefore’ storytelling approach (Olson, 2013), and the Pixar storytelling model (Karen McKee, 2019). These structures each offer a framework that students may use to tell more compelling stories. See Figure 1 for a brief outline of each. We considered these examples of narrative structure, and, using fairy tales and children’s stories as specific artifacts, discussed the cultural differences in structure that feel familiar based on a listener’s background.

Next, the Fellows were encouraged to start thinking about how they might apply one or more of these techniques to discussing their own thesis work. We had them start small: they produced a two-to-three-minute recording explaining a commonly misunderstood aspect of their research. One second-year Fellow explained the misunderstanding that bees are all dying and used this fact as a starting point to discuss the diversity of types of bees and how commercial bee management can lead to massive die-offs. Her research focuses on how climate change affects pollinators, and her short recording used the hero’s journey story structure model to further understand the possibilities of bee diversification.

One student in Cohort three successfully employed the Pixar model in his public talk, but he continued to use a large amount of jargon. Despite the team’s efforts to encourage him to use less jargon, he did not substantively change his vocabulary in class, nor in any of his public talks. This student scored the lowest on both the PSCR and the public surveys.

Another aspect of the training is the development of analogies to aid understanding of new concepts. As an initial intervention to allow the Fellows to convey the concepts of their work without relying on jargon, they iterated and provided feedback on the utility of analogies that can be used to describe aspects of their research. Fellows used analogy to compare avalanche propagation to the thrill of romance, biofilm to skyscrapers, pest management to video games, and molecular structures to LEGO bricks. STEM graduate students often work on a small aspect of a larger problem, and the way they discuss their work and how it fits into the larger goals of their advisors’ labs tends to be esoteric and laden with jargon. Using analogies, the Fellows were often able to more clearly map how their project fits into the larger goals of the lab.

By developing analogies, the Fellows learned to help the listener understand a new concept by mapping it to a concept a listener already understands. These analogies were developed over several months and included conversations with the whole team about how well the analogies work, their limitations, and how relatable they are to the non-experts in the room. Through an iterative process of critique, the Fellows gained trust with each other and learned how to be accurate but not technical when discussing their work.

One very effective analogy was developed by a student who studies how bone density varies as a function of age: specifically, how three different types of bone cells create, maintain, and dismantle bones. To discuss osteocytes and osteoblasts more clearly, this Fellow drew an analogy to a construction site: some people on the job are builders, some work on dismantling, and others are bosses who oversee both builders and dismantlers. A construction site is more likely to fit into a familiar framework of understanding for an average listener. They can map the functions of a bone cell onto more familiar tasks and obtain a broad understanding of the functions of each cell type. At the end of the year, this student was rated very highly by the public for their understanding of her work. A student from Cohort one used an analogy to compare the interdependence of microbes to busy and diverse neighbourhoods. They used this concept of interdependence and microbial variety to explain the importance of understanding complex polymicrobial systems when treating antibiotic-resistant diseases.

We introduced improvisation exercises early in the curriculum to encourage students to develop active listening skills, become attentive to reactions from classmates, learn to follow a narrative and become comfortable with thinking quickly in front of an audience. These skills helped Fellows develop confidence and presence to captivate their audience. The improvisation games require paying attention to what is happening, listening carefully, and working toward making positive contributions to the game being played. For example, one improv game in the curriculum is called, ‘yes, and … ’ (Robbins Dudeck, 2018). One person begins a scene, and their partner builds on the idea using the phrase ‘yes, and … ’. The objective behind this exercise is to learn how to ‘accept’ what your partner ‘offers’ in the service of a successful scene, regardless of the content. Fellows have reported that they are better able to answer questions when giving professional talks as a direct result of the improvisation games we played in class. During one of the in-person talks, one Fellow very adeptly handled questions that were quite unrelated to her research, and afterward, she attributed this skill to the regular improv practice in class. One of our STEM faculty members found this exercise useful in the classroom immediately after learning it!

Another example of an improvisation game is called the three-headed expert. In this game, three people play the role of a single expert being interviewed by a fourth person. The ‘interviewer’ asks an open-ended question, and the ‘three headed expert’ answers by having each participant say one word at a time, acting as the expert’s ‘three heads’. This game requires careful listening by each ‘head’ of the expert, so the answer makes sense, forms complete sentences, and actually answers the question, even if in an unusual way. It also requires careful listening by the interviewer, who must ask follow-up questions that fit with the answers the expert has already provided. Additionally, the Fellows played simple, interactive or call-and-response type games to develop confidence in their own voice, comfort with low-level stress, attentive listening, and the importance of continuity. These in-class exercises were coupled with out-of-class homework, which consisted of recording and critiquing podcasts and writing guided journal entries. Examples of the journal prompts can be found in the syllabus weekly schedule.

Over the first semester, Fellows recorded successively more complex podcasts, receiving feedback on each iteration. Fellows were assigned two peers to provide feedback for each week. They practiced empathy, humility, and respect when critiquing each other’s work. The instructors modelled this practice and moderated discussions as needed to ensure a positive, constructive tone. In one podcast exercise, Fellows summarized a paper in their field for a variety of audiences: disciplinary peers, college graduates with some science training, and a general audience. The 3Cs are useful for categorizing successful communication in a structured format that Fellows can easily map to and use to structure discussions. About one month into the intervention, one Fellow received feedback on one of her recordings. She made the following journal note regarding the critique: ‘If you ask a question you must answer by the end of the talk || connection and captivation’. And, ‘Picture someone that you are talking to when recording audio so it comes of [sic] conversational.’ Hence the Fellows are recognizing that the critiques can improve the quality of their communication, and they could map this feedback directly to the 3C framework. Through this iterative process, the Fellows hone their messages, learn to embody the listener and develop a sense of how critiques can improve one’s work. At the end of the first semester, one Fellow wrote in their journal, ‘I believe I still need to improve on the structures of my stories so they are more captivating, but I have improved on this tremendously!’

Finally, the Fellows develop a ten-minute talk to be delivered to a public audience, along with a question-and-answer period. The second semester of the academic year is spent developing this conversation through writing, rehearsing, recording, iterating, and critiquing each other’s work (details in the syllabus). To scaffold this process, Fellows start with a one-minute elevator pitch, followed by an outline of the talk, then a captivating introduction, etc., until they develop the full talk to deliver. Continuing to work within the framework of ‘connect, convey, and captivate’, each iteration of their talk was analysed and workshopped to meet the 3C goals. Fellows delivered their talks several times in class and were critiqued by the group as a whole each time. This allowed the Fellows to see their talks as part of a conversation or a dialogue and not strictly as a conveyance of information from them to the audience. They also experienced the benefits of planning, rehearsal, and feedback from trusted colleagues for effective communication.

One exercise that allowed the Fellows to find perspective on a topic they know deeply, is to outline the talk based on content, and then chart it on a white board based on some other parameter. Given the wide latitude on the parameters of their charts, one Fellow chose to graph their talk in terms of excitement and information delivered on the y-axis versus time on the x-axis. She noticed that part of her talk was very low on excitement and very high on information delivered, and as a result, she reworked her talk to be more even on both as a function of time. This perspective improved each talk and the exercise modelled a way to consider an audiences’ perspective.

Fellows received instruction on how to create PowerPoint slides that were not too wordy, minimize the use of graphs, and avoid the use of animations (Corum, 2022). When they began putting together their slide shows in the spring, the slides were workshopped multiple times by the entire team.

The Fellows gave two public talks: the first was to undergraduate students at the university, and the second was for a public audience at an off-campus location. The first public talks were a preliminary practice in front of a live audience consisting mostly of college freshman. The audience consisted of several engineering students, which led to questions that were enthusiastic and detailed. Students expected answers and were eager to engage on a deep level, as were the Fellows.

The second talk was advertised to a broad general audience and held off campus. In both settings, Fellows employed skills developed over the year: they made eye contact with the audience, listened to and answered questions, and considered the extent to which they were connecting with, conveying to, and captivating their audiences.

The COVID-19 pandemic posed a significant challenge to the Storytellers program as the pandemic set in during the spring semester of our second cohort of Fellows. Before the pandemic started, cohort one gave public talks at a local brewery. Cohort two also had the opportunity to give their talks to a live audience as they had the opportunity to speak to an audience of college freshman in early March 2020 before the university switched to emergency remote learning. However, instead of a similar experience as Cohort one had at the local brewery, in April 2020 members of Cohort two made YouTube videos and posted them for public viewing. Cohort three gave talks live over the internet using the Webex video conferencing platform. Although this gave our Fellows the opportunity to deliver public talks while navigating the COVID-19 pandemic, as discussed below, this challenge confounded our evaluation of the impact of the program on Fellows’ communication skills.

Assessment

The program included both formative and summative assessments. We first discuss the assessments that we used for each cohort, shown in Table 1, and then present the results from these assessments demonstrating the outcomes of the STEM Storytellers program. Several statistics were calculated on the data obtained through our assessment; these statistics were used to test contrasts between time points as evidence of the impact of the program on our participants and for additional evidence of agreement among reviewers. Given the size and specific context of the program, we do not intend these statistics to be used for generalization to a broader population of STEM graduate students. Five assessments were used in this research, as shown in Table 2:

1. The Public Speaking Competency Rubric (PSCR) (Schreiber et al., 2012) was used to assess the final talks.

2. An R script was used to quantify the jargon used before and after the intervention with all three cohorts.

3. A short 6-item anxiety measure was used longitudinally to capture ‘snapshots’ over time of how Fellows’ levels of anxiety changed.

4. The Personal Report of Communication Apprehension was employed after being recommended to us partway through the program.

5. Public surveys were given to audience members, with the 3C framework reflected only in survey questions for cohort three.

Table 1. Assessments used for each cohort of STEM Storytellers.

Assessment	Cohort 1	Cohort 2	Cohort 3
PSCR (by PIs)	Yes	–	Yes
Jargon	Yes	Yes	Yes
Six item anxiety	10/23/18	10/4/19; 12/6/19; 3/3/20	10/27/20; 3/29/21
PRCA	–	Pre/Post	Pre/Post
Public surveys	Yes	–	Yes
Final talks	In person	YouTube	WebEx

Table 2. STEM Storytellers curricular materials and assessments within the 3C framework.

Connect	Convey	Captivate
	PSCR
	Public surveys
	Reflective Journal entries
Anxiety measures	Jargon calculations	storyboarding

Public speaking competence

We used the PSCR with Cohorts one and three to assess their public speaking and presentation skills in delivering their final talks to the public. An expanded rubric was provided to us by one of the authors of the PSCR. This rubric included seven items not included in the published rubric: tempo, volume, enunciation, facial expressions, eye contact, posture, and use of space.

Two rubric items that were pertinent only to in-person delivery were not assessed for Cohort three, because their talks were delivered using a virtual format (Webex).

Four STEM faculty members were trained and calibrated to use the PSCR by watching Three Minute Thesis competition entries on YouTube (Three Minute Thesis Queensland, 2022). They watched and rated a series of these short speeches, discussed differences in scores, and attempted to reach consensus on each aspect of the rubric. Written scores were not changed, but the discussions were considered for subsequent ratings.

Agreement among the scorers was very high each time the PSCR was used, reflective of the training provided to raters prior to using the rubric to score Fellows. We used three indices to estimate interrater agreement on the recommendation of LeBreton and Senter, as each measure on its own is subject to idiosyncratic limitations based on the calculation method (LeBreton & Senter, 2008). Scores indicated high agreement among raters across all three indices; the interested reader can see the Appendix for details on these indices.

On average, Cohort one scored just below and Cohort three scored just above proficient (level three), suggesting both Cohorts reached our goal for the Fellowship in terms of their public speaking skills. Overall, Cohort three appears to have scored higher than Cohort one, though the difference is within half a rubric point, reflecting that both cohorts showed solid public speaking skills by the end of the year. Cohort three scored higher on two items that pertain to ‘connect’ in our 3C framework (selects an appropriate topic, formulates an effective introduction) and three items that pertain to ‘convey’ (effective organizational pattern, compelling supporting materials, persuasive message). Our PSCR data show its utility among different raters as well as the level of proficiency of our Fellows as a result of completing the program. A full table showing the descriptive statistics for cohorts one and three on this rubric is provided in the Appendix.

Jargon

Using the open-source statistical analysis software R, we developed a script to quantify the amount of jargon in talks given by the Fellows both before and after the intervention (Willoughby et al., 2020). Rather than using Nation’s generic benchmark of two percent jargon or less in a full transcript, we opted to develop a more targeted approach to determining how much jargon each student used before and after taking the course. Using corpora, or collections of words from both the scientific literature and from everyday sources, the R script compares how many words in the input file (Fellow’s transcript) are in both everyday and scientific corpora versus just in the scientific corpus. Using a logarithmic approach, the script assumes that a word present in the scientific corpus and not in the English corpus is 1000 times more likely than not to be jargon. If the word is present in both corpora, it logarithmically compares how often is it present in each corpus. The script calculates a jargon score for each word in a given transcript, and then the jargon score for each word in a transcript is summed and averaged, so the total score can be compared to benchmarked values from other sources including TED talks, Supreme Court Decisions, ArXiv papers, and more. This script also creates a list of words that are likely jargon because they are present in the scientific corpus only and not the English corpus. Word clouds can be created as well, showing graphically how many times a certain word or word family has been used in an input file so jargon laden language can be reworded if necessary. An example of this from Cohort two is shown in Figure 2.

Figure 2. Word clouds for cohort two transcripts before intervention (left panel) and after intervention (right panel).

We assessed how Fellows’ use of jargon changed over their year in the program using this process. We compared a transcript of their recording for the initial application with the recordings of their public talks at the end of the year. We used the R script to compute the number of words in each transcript that was jargon (and the number that was not jargon) and computed a chi-square test of independence for each cohort to determine if their use of jargon changed significantly over the course of the year. In Figure 3, we present the proportion of jargon at each time point.

Figure 3. The chi-square test of independence showed, for all three cohorts, a statistically significant change in use of jargon; cohort 1: chisq(1) = 20.732, p < 0.001; cohort 2: chisq(1) = 20.381, p < 0.001; cohort 3: chisq(1) = 21.952, p < 0.001. In all three cohorts, the amount of jargon in Fellows’ prepared research talks decreased significantly; cohort one dropped from 2.8% jargon to .93%, cohort two dropped from 2.74% to .93%, and cohort three dropped from 2.03% to .65%. Further, Fellows’ public talks dropped below the minimum 2% jargon recommended by Nation (2006).

Anxiety

In addition to public speaking competence and reduced reliance on jargon, we were interested to know if Fellows were more confident in their public speaking abilities after the intervention. We provided two instruments to our Fellows to determine how their levels of anxiety fluctuated during their time in the Fellowship. We used the six-item version of the Speilberger State-Trait Anxiety Inventory (STAI) (Marteau & Bekker, 1992) to capture ‘snapshot’ measures of anxiety at specific time points during the Fellowship, and we used the PRCA with cohorts two and three as pre- and post-test of communication apprehension. We did not use the STAI consistently across cohorts because of the pandemic; we reported what we learned about each cohort’s anxiety levels based on how we used the instrument with each.

With cohort one, we provided the six-item STAI before and after the whole class listened to and critiqued the recordings Fellows made. Cohort one scores decreased from the pretest STAI (M = 45.833, SD = 16.499) to the posttest (M = 43.750, SD = 12.902). For reference, scores of 20–39 are considered low anxiety, 40–59 moderate anxiety, and 60–80 high anxiety (Marteau & Bekker, 1992). The cohort scored within the moderate anxiety range just before giving talks in front of the class. Their anxiety slightly decreased after the talks, reflecting that a portion of their overall anxiety was due to the anticipation of public speaking, t(8) = 0.4589, p = 0.6602.

Cohort two was provided the STAI at several time points over the academic year. The following Figure 4 shows the average score for the cohort, measured over time. A repeated-measures ANOVA analysis of these scores did not show a significant relationship between score and time, F(11) = 0.8, p = 0.636, but Fellows’ reported anxiety showed some variability over time. The lowest score was obtained in early October and the highest score was recorded the following week directly before giving speeches to their peers. The averages on the STAI then levelled out starting at the conclusion of the fall semester with a slight increase just before the talks to college freshmen.

Figure 4. STAI anxiety scores for cohort two (2019–2020).

We opted not to continue collecting data on anxiety as we expected the COVID-19 pandemic to confound any measure of public speaking anxiety with cohort two.

Cohort three took the STAI twice, once during the fall semester and once during the spring semester, just before their scheduled talks. Scores for cohort three increased from Fall to Spring. Although the average of their anxiety scores increased by ten points between semesters, this difference was not statistically significant p = 0.1949. Higher anxiety before public speaking is common, even with practice. The Fellows report that they increased awareness of their performance and that both increased their anxiety and the quality of their talks, as one Fellow shared with us: ‘I didn’t even know what I didn’t know to be nervous about (false sense of confidence?)’. Another student reported that they developed, ‘ … an increased awareness of my own flaws and more anxiety regarding all of my own communication weaknesses which were brought up and worked on during the fellowship.’

Responses to the statement ‘I am calm’ changed the most: the Fellows felt less calm in the spring semester than in the fall semester, p = 0.046. The other five scores did not change enough to be statistically significant.

Apprehension about communication

The PRCA was employed to assess apprehension about communication more specifically. We provided the PRCA to both cohorts two and three. The summary statistics and indicators of significant differences are shown in Table 3. The PRCA was provided with their applications to the program and then again in the spring semester just after completing their public talk videos. The Fellows reported a pre-intervention mean of 67.8 and a post-intervention score of 77. Both fall within the moderate range of apprehension, which is between 55 and 83.

Table 3. Summary statistics of the PRCA for cohorts 2 and 3 pre and post intervention.

	Pre	Post	P
Cohort two
Group discussion	18.4	20.2	0.223
Meetings	17.0	18.6	0.088
Interpersonal communication	17.4	20.0	0.042
Public speaking	15.0	18.2	0.225
Cohort three
Group discussion	17.4	21.9	0.054
Meetings	16.6	19.9	0.157
Interpersonal conversations	18.4	21.3	0.165
Public speaking	16.6	17.3	0.753

For cohort two, apprehension about communication increased across all four domains measured on the instrument, including public speaking, but only the increase for interpersonal communication was significant. Even though the instrument shows that apprehension about public speaking increased over the course of the Fellowship, overall apprehension about communication increased, and the increase for public speaking was not significant. We again take this to mean that Fellows became more conscientious about how they communicate and not that the program increased their apprehension. Early in the program, for example, one Fellow mentioned that their thesis advisor consistently refers to insects by their Latin names. As a result, they began using common terms when discussing research within the Fellowship, but they were apprehensive about slipping up in front of their advisor and using the common names in that situation. This, the Fellow feared, would cause their advisor to think less of them.

A similar pattern was observed for cohort three, though none of the changes in communication domain scores were significant.

Public surveys

Our primary goal in this Fellowship was to improve the abilities of STEM graduate students to discuss their research with broad audiences. To measure this, we provided evaluative surveys to audiences watching our Fellows’ public talks (cohorts one and three), developing simple surveys to gauge audience impression. For cohort one, the audience was asked to rate how well the speaker conveyed the importance of their research, how well they answered questions, and if their talk increased the audience member’s excitement for science. The audience, on average, rated speakers on each item between ‘agree’ (four) and ‘strongly agree’ (five). How Fellows handled audience questions was rated the highest (M = 4.641, SD = .588), followed by conveying the importance of their research (M = 4.432, SD = .686), and then increasing audience excitement for science (M = 4.418, SD = .803). These questions did not specifically address the 3C framework, thus they were revised for the public talks given by cohort three.

With cohort three, audience members were given the revised survey that prompted them to rate speakers on how well they understood the project that was presented (convey), the extent to which the talk sparked their curiosity about the topic (captivate), and the extent to which the audience member felt the speaker was speaking directly to them (connect). Again, speakers were rated highly, between four and five on average.

In this final cohort, Fellows were rated highest on how they connected with the audience (M = 4.602, SD = .643). This was followed by how well Fellows piqued audience members’ curiosity (M = 4.506, SD = .739), and then the extent to which the audience members felt they understood the topic (M = 4.438, SD = .719). We acknowledge that the audiences included many ‘friendly faces’ who rated these talks highly. However, we still interpret these scores to show that our Fellows engaged well with broad audiences, as more than half of the audience for each cohort rated each item at the highest level. We also interpret these scores to mean that the Fellows possessed skills that enabled them to engage with non-experts about their research.

Limitations

The largest limitation of this study was the worldwide COVID-19 pandemic, which caused specific aspects of the course and of the assessments to be altered. The pandemic altered US higher education modality weeks before the end of the second year, so cohort two gave only the campus talks before the University pivoted to emergency remote learning. Cohort three completed the entire course online via WebEx, except for a single class in person during the fall 2020 semester. Although communicating science virtually is important, we had not designed the program for that format.

Additionally, the pandemic disrupted our assessment. The 6-item anxiety measure was used sparingly after March 2020, because everyone was almost certainly feeling more anxiety than usual. Also, neither cohorts two nor three gave in-person talks to general audiences, making that measure of success difficult to compare directly to cohort one. We were able to apply the PSCR to cohorts one and three, both of whom gave talks in real time. Ultimately, we were unable to make as many direct comparisons between cohorts as we had initially planned.

Another notable limitation of this study is the small sample size. The program was designed to work with 24 students, and even though 22 students completed the program, the kinds of quantitative analyses we could perform with the data were limited. The qualitative feedback proved very useful. Since students applied for this program as additional coursework in their graduate education, we did not have a random sample to test whether the program caused a change in effective communication.

Finally, surveys given to the public in years one and three were developed for ease of use by a general audience and not tested for reliability. We did adopt several tested measures from the literature to assess students’ public speaking competence and anxiety, but we kept the audience assessments as simple as possible for quick administration following each public talk. We raise this limitation as a caution regarding the conclusions that can be drawn solely from the audience surveys.

Discussion and conclusions

Overall, our five assessments suggest that the STEM Storytellers curriculum is an effective intervention to improve the oral communication skills of STEM graduate students. Twenty-two students completed a year-long intervention focusing on ‘connect, convey, and captivate’. The public feedback in particular showed Fellows interacted effectively with their audience, with items pertaining to ‘connect’ scoring the highest with both cohorts one and three within a dialogue model of science communication. Based on the initial varied abilities of the Fellows, we can see that the overall skill level increased for both students who already excelled at science communication and those who struggled pre-intervention. We do note that one student in particular continued to use jargon pretty heavily even in their final talk, making their work difficult for the lay audience to understand and appreciate.

On average, the Fellows were able to reduce the amount of jargon in their talks from more than two percent on their application recordings to less than 1% in their final public talks, a reduction of over 50%. This change also aligns with Nation’s recommendation that jargon compose less than two percent of content in a public scientific presentation so as not to overwhelm a general audience with technical details. Beyond the metric of a reduction in jargon, Fellows became more comfortable sharing the story of their work in accurate but non-technical terms. The Fellows recognized the benefits of storytelling for conveying the importance of their research and that this practice captivates the listener and allows them to connect with both the speaker and the information presented. Fellows from every cohort mentioned their plans for utilizing story arcs in other parts of their professional lives, consistent with recommendations by Shanahan et al. (2019).

Many Fellows initially expressed trepidation with the activities due to their understanding of science in highly precise technical terms, a concern of oversimplifying their research, or misrepresenting their advisors’ research in public settings. Through practice Fellows built confidence in their ability to accurately convey information without relying on jargon or complex methodology. They also developed increased confidence and comfort in handling unexpected questions or comments through practicing improvisational acting (Robbins Dudeck, 2018). These improvements were best observed in their public talks as they shared their science with a broad audience.

Finally, every cohort increased their use of rhetorical devices when giving public talks, consistent with recommendations regarding effective communication (Vosniadou & Skopeliti, 2019). Through an iterative process, the Fellows refined their analogies, balancing accessibility with scientific accuracy.

Even though the COVID-19 pandemic disrupted the flow of the study, we found that graduate students enrolled in the STEM Storytellers program were more able to connect with and captivate their audience, and to better convey information regarding their research. The curriculum and assessments we have developed are licensed for reuse, [added as an additional document] and we encourage graduate educators to consider adopting these materials.

Recommendations for future use

The year-long shared experience helped build a sense of community among Fellows that resulted in them supporting each other’s development through constructive, critical feedback. The iterative conversation of editing and revising could be modelled in a stand-alone workshop, but to develop the foundation of confidence and trusted feedback that we believe most benefited our Fellows, the length of the program is essential. We anticipate the Fellows will continue to implement many of these lessons into future scientific talks, even in technical presentations.

It is essential that Fellows have the support of their advisors. Program instructors should build in timely communication to inform academic advisors of the effort and advancements of their students over the year, although we have heard anecdotally that the students’ improvements in oral communication were obvious to whole research groups.

Students who are trained in STEM disciplines generally enjoy the structure and regularity of clear learning outcomes. We learned from cohort one that the purpose of activities was often not apparent to them as they progressed through the year. In future cohorts, instructors more clearly shared the purpose and learning outcomes from each activity and the program as a whole. While we initially sought to shelter them from the technical ways these activities fit together to support communication development, we found that they enjoyed the structure and framework of connect, convey, captivate as touchstones for skill building.

It is our hope that other instructors and institutions choose to embrace and use the dialogue-based curriculum, along with the suite of assessment tools presented here.

Acknowledgements

The authors acknowledge the support of the National Science Foundation. The professional acto, Kent Davis, provided immense amounts of support and feedback for the Fellows.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by National Science Foundation of the US. Innovations in Graduate Education: [Grant Number 1735124].