Humanoid robot as an educational assistant – insights of speech recognition for online and offline mode of teaching

ABSTRACT

Technology has the potential to enhance the effectiveness of the teaching and learning process. With the integration of technological resources, educators can create dynamic and interactive learning environments that offer diverse learning methods. With the help of these resources, students may be able to understand any topic deeply. Incorporating humanoid robots provides a valuable approach that combines the benefits of technology with the personal touch of human interaction. The role of speech is important in education; students might face challenges due to accent and auditory problems. The display of the text on the robot's screen can be beneficial for students to understand the speech better. In the present study, our objective is to integrate speech transcription with the humanoid robot Pepper and to explore its performance as an educational assistant in online and offline modes of teaching. The findings of this study suggest that Pepper's speech recognition system is a suitable candidate for both modes of teaching, regardless of the participant's gender. We expect that integrating humanoid robots into education may lead to more adaptive and efficient teaching and learning, resulting in improved learning outcomes and a richer educational experience.

KEYWORDS:

• Pepper
• humanoid robot
• speech-to-text
• education
• online
• offline

1. Introduction

In past years, society has faced many challenges in education because of the widespread impact of the Covid-19 pandemic. The implementation of lockdown compelled us to shift from traditional offline to online teaching mode. With the help of technological advancements, the transition to online mode was feasible and acceptable (Rajeb et al. 2023; Trelease 2016). However, the online medium has issues such as limited personal interactions and the need for technology access (Joshi et al. 2022). Physical classrooms facilitate social interaction and real-time doubt clarification, whereas geographical distance might challenge some students. Thus, online and offline teaching have their advantages and disadvantages. Determining the superiority of any mode is difficult depending on many factors, including constraints and context.

Integrating advanced technologies such as social robots in physical and online teaching environments can enhance teaching methods and learning experiences. Past studies indicated the positive impact of including social robots in physical classrooms. The robots can engage the students and increase their participation during lectures. Few studies demonstrated the advantage of social robot inclusion for language learning (Alemi, Meghdari, and Ghazisaedy 2014; Kennedy et al. 2016; Randall 2019). Pandey, Subedi, and Mishra (2022) showed that incorporating the humanoid robot NAO was helpful for the advancement of language and reading skills of primary students in physical classrooms. In the other study, NAO robot inclusion in higher education was demonstrated to teach numerical methods (Cruz-Ramírez, García-Martínez, and Olais-Govea 2022).

In the present study, social robot Pepper was employed to evaluate its usefulness in online and offline mode of teaching. SoftBank Robotics developed a Pepper robot which can interpret and interact with its surroundings using cameras, microphones and sensors. Pepper is equipped with four microphones on its head, and it can catch sounds from multiple directions. Pepper can establish communication with humans using its speech recognition system. However, the facility of speech-to-text conversion is not integrated into Pepper's functionality, which means it is unable to transcribe the audio into text even though it has a display screen to show the text.

Transcription of speech into text can validate the accuracy of the speech recognition system. Speech-to-text conversion is being utilised in various domains, including the education sector. Rustam et al. (2014) comprehensive review shed light on the role and capabilities of speech-to-text conversion in education. Debnath et al. (2023) demonstrated the integration of speech-to-text transformation with audio-visual methods to support teaching to individuals with disabilities. There are many speech-to-text conversion tools available. Our previous research suggested Whisper as a dependable and effective open-source tool (Pande, Rani A, and Mishra 2023) for speech-to-text conversion.

The understandability of a delivered lecture depends on the clarity of speech and, hence, the quality of the audio received. The accuracy of audio might vary depending on the selected mode – online or offline and therefore, it is important to evaluate the accuracy of captured audio. To our belief, there is no such study wherein the quality of the delivered lecture by the same teacher for online and offline mode has been assessed. The involvement of robots during the lecture could enhance the learning experience. The use of robots in education has made substantial progress, but to the best of our knowledge, the utilisation of speech recognition coupled with transcription display on the robot's screen is still unexplored. Most of the existing work (Belpaeme et al. 2018; Li 2015) using humanoid robots such as Nao and Pepper involves the physical presence of participants (teacher and student) in the educational setting. This implies that social robot’s speech recognition capabilities have not been investigated for online teaching mode. These challenges motivate us to frame the following research questions.

RQ1: Will the integration of Pepper with Whisper be an effective solution for online and offline modes of teaching?

RQ2: Is there any difference between male and female voices on the developed system for online and offline mode?

To address the above research questions the objectives of this study is two fold. Firstly, this paper integrates Whisper with Pepper and evaluates the efficacy of Pepper's speech recognition system using various evaluation measures. Further, this study aims to provide insights into the performance of proposed system in online and offline teaching modes. To accomplish this goal, an experiment was undertaken in laboratory settings wherein a participant, acting as an educator, delivered lecture in both online and offline modes. In the offline mode, a person and a robot were positioned adjacent to each other, while in the online mode, the person was located remotely. In both scenarios, we assumed the presence of the robot in a classroom-like setting.

This paper is structured as follows. The next section describes the related works in this area. Methodology section includes data collection and the techniques employed. Further, data analysis and result section show the analysis of the result followed by discussions. Finally, the paper concludes with directions for future research.

2. Related works

To better understand the insights of speech recognition systems with the integration of social robots, it is important to explore the relevant studies in the field of education, including the role of social robots in various domains including education, mode of education – online and offline, impact of gender voice and importance of speech recognition.

2.1. The role of social robots in various domains

Social robots are popularly known for establishing communications with individuals. There are different domains where social robots are employed for communication purposes. One such field is healthcare, where social robots as a companions are utilised to provide emotional support. Abdollahi et al. (2022) utilised the social robot Ryan to engage and provide companionship to older people with depression and dementia by establishing communication through dialogues with them. Furthermore, they integrated artificial emotional intelligence with the robot to make it empathic and establish more effective dialogue communication. The social robot can be employed to establish interactions for the therapy of autistic children (Cabibihan et al. 2013).

The service industry, such as hotel and travel, are utilising social robots (Vishwakarma et al. 2024). The social robot can be used in the hotel industry for interacting with customers (Nakanishi et al. 2020). Pan et al. (2015) employed the Nao robot in the public space of a hotel for greeting purposes and to provide information about the hotel. Further, they analyzed the impact of robot’s speech on guests. Youssef et al. (2021) suggested the application of a social robot as a receptionist with conversational capabilities. Niemelä et al. (2019) explored the possibility of acceptance of social robot in shopping mall. Further, they suggested that there is a need for improved dialog capabilities in robot for better interaction with customers.

Social robots can be employed for entertainment purposes. Kory-Westlund and Breazeal (2019) findings suggest that robot’s speech impacted children’s engagement and entertainment. Vogt et al. (2019) utilised a social robot’s tablet to play educational games with children where the robot provided instructions and feedback.

2.2. Educational activities facilitated by social robots

Social robots are intelligently designed machines with the ability to interact with humans in a natural and convenient manner in different settings (Breazeal, Dautenhahn, and Kanda 2016). Belpaeme et al. (2018) conducted a systematic review exploring the roles of social robots in education by addressing their influence on learning outcomes, the impact of their physical appearance and their roles within educational settings. Li (2015) discussed the importance of physical presence of robot over telepresent robot and virtual agents. Furthermore, studies suggested that the physical presence of robots in educational settings has impact on learning enhancement (Kennedy, Baxter, and Belpaeme 2015) and cognitive learning (Leyzberg et al. 2012).

Lanzilotti et al. (2021) utilised Pepper robot in primary school to evaluate its impact on students’ engagement and cognition through coding activity. Moreover, they found it enjoyable and engaging. Donnermann, Schaper, and Lugrin (2022) demonstrated the benefits of integrating robot-assisted tutoring in higher education. Furthermore, they observed improvements in knowledge, motivation, and exam performance among students who received tutoring from robots, in contrast to those who had no exposure to a tutoring robot.

The social robots can be helpful in improving teamwork and communication among students through Robot-Supported Collaborative Learning (RSCL) activity. Shimada, Kanda, and Koizumi (2012) utilised social robot in a group of grade 6th students for collaborative learning and found the enhancement in children’s motivation. Buchem (2023) assessed the social learning experience and interaction quality in higher education level with the NAO robot and observed that RSCL game-based activity facilitated by robot was accepted by participants, regardless of their familiarity with the robot. Rosenberg-Kima, Koren, and Gordon (2020) employed both instructor and robot facilitation, it was observed through qualitative measures that human facilitation was better in terms of various aspects, but robot facilitation helped improve time management and efficiency. However, quantitative data suggested no significant differences.

In recent years, social robots have emerged as a promising tool to assist in language learning. Lee and Lee (2022) reviewed the efficacy of Robot-Assisted Language Learning (RALL) and observed that RALL had a positive impact on language learning compared to non-RALL conditions. In a different study, Alemi et al. (2015a) performed an experiment with high school students and compared the RALL group with non-RALL group. Furthermore, they showed that RALL group displayed reduced anxiety and a more positive attitude towards learning English vocabulary, finding the process enjoyable and effective. In another study, Alemi et al. (2015b) demonstrated that social robot NAO can be helpful for autistic children in learning a foreign language English.

Past studies demonstrated the effectiveness of social robot integration in educational settings. Such integration helps increase students’ engagement, cognition, learning and performance. This motivates us to include social robot for present study to provide students a comfortable environment where they can understand the lecture in a better way without any hesitation in asking.

2.3. Online and offline teaching

Previous studies have shown the mixed results of online and offline modes of teaching. Wiyono et al. (2021) analysed the data collected from teachers of elementary school and their finding suggested that offline communication is more efficient than online communication with students. Similarly, Daulatabad et al. (2022) showed that traditional offline classes were more effective than online classes. Zhu, Liu, and Hong (2022) examined the perception of online education among young children and their parents, and their findings illustrated the negative impact of online teaching. Furthermore, parents preferred offline education, but this perceivance varied by location and school type. On the other hand, Yen et al. (2018) findings indicated that undergraduate students performed equally well in three of teaching modalities that are face-to-face, online and blended. Veeraiyan et al. (2022) compared online and offline teaching for undergraduate dental students and found equivalent performance in both the modes. In contrast, Pei and Wu (2019) conducted a systematic review and suggested that online learning is more effective than offline learning in enhancing the knowledge and skills of undergraduate students. Rodríguez-Hidalgo and Sibrian (2022) conducted a qualitative study using interviews to evaluate if social robot Sima can be used as a pedagogical tool in the online classrooms. In another study, Sinai and Rosenberg-Kima (2022) utilised perception ratings of participants about social robot Nao inclusion as a motivator during online learning.

Past research revealed that the selection between online and offline mode is difficult because the impact of any of them varies based on scenarios. However, none of the studies highlighted the shortcomings of delivered lecture audio quality using social robots. In the present study, two experiments were conducted with each participant to understand the suitability of the mode for better capturing the audio by speech recognition system of the Pepper robot.

2.4. Impact of male and female voices

The effectiveness of understanding a lecture can be increased if the lecture is delivered with a clear and expressive voice. The modulation in voice is able to capture the attention of target audience successfully. Guldner et al. (2024) examined how intentional voice alterations affect a listener's impression of a speaker. Koch (2017) investigated the correlation between different teaching roles and various acoustic elements in teachers’ voices, considering a child-sensitive perspective. Yeshoda and Rajasudhakar (2023) findings suggested that female teachers’ voice exhibited more variations than male teachers’ voice as measures related to fundamental frequency (F0) as well as short and long term frequency perturbations were higher for females.

There are reports which suggest that impact of male and female voices on Automatic Speech Recognition (ASR) systems. Adda-Decker and Lamel (2005) illustrated that in the analysis of broadcast news and conversational telephone speech in both English and French, females exhibited higher accuracy compared to males in their voices. Similarly, Sawalha and Abu Shariah (2013) showed that Arabic ASR system perceived the female utterances more precisely than male. In contrast, Tatman (2017) findings suggested lower accuracy in the recognition of female speech in context of English dialect.

The above studies indicate that the selection between male and female voices is difficult for any mode of teaching. There is no conclusion about which voice is best for which medium. Considering this, we selected an equal number of participants from both genders.

2.5. Challenges with speech recognition

Accurate speech recognition is challenging for machines including social robots due to various factors (Mubin, Henderson, and Bartneck 2014). Speech variability is caused by several factors, such as accents, dialects, and speech disorders, which can greatly impact speech patterns and introduce variations in pronunciation. Furthermore, age is another contributing factor that can lead to differences in speech production (Bóna 2014; Das et al. 2013). Research studies found that social robot Nao faced problems in comprehending the voice of children (Kennedy et al. 2017; So and Lee 2023) and older adults (Keizer et al. 2019). Similarly, there are reports that demonstrated the challenges with speech recognition system of Pepper robot in understanding the utterences of children (Ekström and Pareto 2022) and elders (Carros et al. 2020).

Distance from the speaker device also poses significant issues for perfectly recognising the utterances. Previous studies have shown that speech accuracy is negatively impacted when a speaker moves away from the microphone (Nematollahi and Al-Haddad 2016; Rodrigues et al. 2019). Zahorik and Kelly (2007) suggested the noticeable affect in vocal intensity with increasing distance. Noise present in surroundings also affects the problems for speech recognition systems (Attawibulkul, Kaewkamnerdpong, and Miyanaga 2017; Meyer, Dentel, and Meunier 2013). Furthermore, teacher's raised voice in response to noise can lead to voice problems (Nusseck et al. 2022). The speech recognition systems of social robots such as Nao and Pepper are also susceptible to distance and noise. Kennedy et al. (2017) illustrated how increased distance from Nao robot affects speech recognition accuracy and noise. Similarly, Pande and Mishra (2023) findings also suggested that perceivance of utterances by Pepper’s speech recognition system is optimal at maximum distance of 1-meter. Komatsubara et al. (2014) also mentioned the problem in speech recognition due to noisy school environment during interaction with social robot Robovie.

2.6. Improvement in speech recognition

The enhancement in efficacy of speech recognition systems is important for better interactions between human and machines. Improvement in speech recognition systems in classroom settings can be achieved using Artificial Intelligence (AI) techniques. Previous studies indicated that AI play an important role in increasing speech recognition accuracy. Research showed that deep learning methods such as CNN (Abdel-Hamid et al. 2014), RNN (Graves, Mohamed, and Hinton 2013), DNN (Hinton et al. 2012) and transfer learning (Lin et al. 2021) are helpful in improving speech recognition precision. Rebai et al. (2017) utilised Data Augmentation (DA) and Ensemble Method (EM) methods to improve speech recognition within a framework of DNN architecture. Mehra, Ranga, and Agarwal (2024) proposed a late fusion technique that combines audio and image modalities to enhance speech command recognition. Mubin et al. (2012) suggested that performance of speech recognition can be increased using Robot Interaction Language (ROILA) and their findings indicated that ROILA outperformed English. Pande and Mishra (2023) incorporated speech-to-text conversion tool Whisper with Pepper robot to evaluate the speech recognition system performance through evaluation measures.

In previous studies many methods were provided for noise reduction to enhance the speech recognition accuracy. One of the popular methods used in various studies (Ayat, Manzuri-Shalmani, and Dianat 2006; Fu et al. 2023) is wavelet threshold. Sun et al. (2022) utilised joint acoustic echo cancellation method along with adaptive filter and DNN model. Liao et al. (2023) presented a method, ASR post-processing for readability (APR), to convert noisy ASR output into human readable text. Healy et al. (2013) developed an algorithm to segregate noise from speech to enhance speech intelligibility for listeners with hearing impairments. Li et al. (2023) demonstrated that speech recognition performance in noisy environment can be improved by incorporating lip reading skills and using the cross-modal fusion effect.

Research indicates that there are shortcomings with the speech recognition system of social robots, however, there are also solutions to improve that. In the present study, both the social robot and speech-to-text convertor Whisper were used to understand the statements spoken by participants accurately.

3. Methodology

The experiments were conducted in the Educational Technology Laboratory of NTNU Gjøvik, Norway. The laptop utilised in this experiment had an 11th Gen Intel(R) Core (TM) i5-1145G7 @ 2.60 GHz 1.50 GHz processor, 16 GB RAM, and was running Windows 10 operating system. The overall pipeline is shown in Figure 1, which are in sync with the following subsections.

Figure 1. Overview of the working of the proposed system with humanoid robot in offline and online settings.

3.1. Step 1 – demographics and experimental procedures

The experiment was conducted in two modes: online and offline. A set of selected statements were provided to participants to speak. These statements were also stored in a local computer as reference text. Four adult volunteer participants were recruited for this experiment, including two males and two females. The participants belong to the age group between 30 and 40, and they all are of Asian descent. Furthermore, their native language is not English. All the participants have experience of teaching at some point in their professional career. As it is evident from past studies that robot has difficulty understanding children and older people’s voice, those age groups were not recruited for experiment. People in the 30–40 age group were recruited because, in our opinion, most of them have some experience in teaching in an academic setting. They spoke the statements, that were received by Pepper. In online mode, the participant was at a remote location and connected through the web. In offline mode, the participant and robot were placed next to each other.

The objective of the experiment is only to evaluate the Pepper’s speech recognition capability. This experiment will serve as a prototype before conducting the experiment in real educational setting.

3.2. Step 2 – recordings of audio-transferring of recordings and speech-to-text conversion

This step involved three operations – recording participants’ audio by the speech recognition system of Pepper, transferring recordings to the local system and converting recordings to text to evaluate the efficacy by comparing it with the reference text.

Pepper captured the participants’ voices in both modes using its service ALAudioRecorder (Naoqi API documentation -ALAudioRecorder, http://doc.aldebaran.com/2-5/naoqi/audio/alaudiorecorder.html, last accessed 2023/05/07). ALAudioRecorder service provides a facility to record audio with the help of a robot's microphone, and the recording is stored in its system.

Transferring these recordings to the local system for further analysis is essential. Paramiko (Paramiko documentation, https://www.paramiko.org/, last accessed 2023/05/07) is a Python library that provides a way to transfer the files between Pepper and the local computer. Paramiko allows a secure connection with remote servers through SSH protocols.

Whisper (OpenAI Whisper, https://openai.com/research/whisper, last accessed 2023/05/07) is a useful open-source speech recognition model and has applications in various domains. It is trained using extensive audio recordings. Whisper has many capabilities, such as multilingual speech recognition, translation, and language identification. After transferring audio recordings captured by Pepper to a computer system, these recordings can be employed as input for Whisper. This empowers the generation of corresponding text files from the recorded audio, facilitated by the utilisation of Whisper.

3.3. Step 3 – feature generation: by comparing whisper-produced text with original text

The extraction of meaningful insights from text data is vital to observe the efficiency of speech recognition system. Two text files were saved in the computer: Whisper-generated text file and reference text file. Original statements were kept in reference text. Different metrics were used to evaluate the accuracy of recorded speech. These metrics play an important role in measuring predicted text quality, similarity, and accuracy.

ROUGE Score: ROUGE (Recall-Oriented Understudy for Gisting Evaluation) score (Lin 2004) is a metric that is used to assess the text quality of machine-produced text when compared with original texts. The ROUGE score has many types, such as ROUGE-1 (unigram overlap), ROUGE-2 (bigram overlap), and ROUGE-L (longest common subsequence). A higher value of the ROUGE score signifies a higher similarity between produced text and reference text. The precision, recall and F-score can be calculated for each of the ROUGE scores.

BLEU Score: The BLEU (Bilingual Evaluation Understudy) score (Papineni et al. 2002) is used for evaluating the quality of machine-produced texts by comparing them with reference texts. The more excellent score indicates a high quality of translations.

TF-IDF Similarity Score: The TF-IDF (Term Frequency-Inverse Document Frequency) (Christian, Agus, and Suhartono 2016) score computes the similarity on the basis of frequency terms and their significance across a corpus. A more excellent TF-IDF similarity score implies a better similarity between two texts.

Jaccard Similarity Score: The Jaccard similarity score (Bag, Kumar, and Tiwari 2019) determines the similarity between two documents. It computes the number of words present in the intersection divided by the number of words present in the union of these sets. The Jaccard similarity score varies from 0 to 1, where 0 means no similarity and 1 means very similarity.

WER: The Word Error Rate (WER) (Klakow and Peters 2002) metric can be applied to estimate the precision of automatic speech recognition systems. The difference between the number of words in reference text and inaccurately identified words in the predicted text can be measured. A smaller WER value suggests good accuracy in transcribing from speech to text.

3.4. Step 4 – removal of the robot-produced unwanted utterances

Pepper has the capability to respond to humans through natural conversations. It could be possible that sometimes, a robot can get some cues from any statement and start its own comments, even if it is not explicitly programmed for that. In such situations, there might be more statements in Whisper-generated text than reference text. It is necessary to remove extra sentences from the text.

There were twenty-four sentences in reference text. Each sentence present in Whisper generated text was compared with each sentence of reference text by retaining maximum values of the scores except WER values. The highest value of scores indicates the much similarity between the texts, while the lowest value of WER values indicates higher accuracy. A threshold of 0.7 was put on the Rouge score so that best-matched sentences could be identified and robot-generated extra comments could be discarded.

4. Results

To answer the research questions, the experiment was conducted in two phases with each of the participants. First, the connection between Pepper and the computer system was established using the Paramiko library. The four audio files were transferred to the computer system for further analysis. These audio files were transcribed into text files using Whisper. Predicted texts were compared with reference texts to evaluate the speech recognition capability of Pepper in online and offline modes. The features were generated corresponding to both text files using different scores. Additional robot responses were cleaned out by applying thresholds to every statement. It was found that the TF-IDF score was relatively high in all the experiments in comparison to other scores, therefore it is not shown from Figures 2–5 for better visualisation.

Figure 2. Analysis of features in the online mode of teaching with male participant.

Figure 3. Analysis of features in offline mode of teaching with male participant.

Figure 4. Analysis of features in the online mode of teaching with a female participant.

Figure 5. Analysis of features in the offline mode of teaching with a female participant.

Figure 2 illustrates the visualisation of six features for spoken statements by male participants through online mode. The selected values of features are placed in Table 1. It can be observed from the figure that the F-score of ROUGE-1 score is usually high for all text records. This implies that Pepper's speech recognition system with Whisper integration is efficient in acquiring individual words and unigrams from the spoken content. However, the variation of the ROUGE-2 score shows that in some instances, Pepper's speech recognition system can capture bi-grams magnificently. At the same time, in other cases, it struggles to capture bi-grams accurately. From the range of the ROUGE-L score, it can be inferred that the performance is good in catching the longest common subsequence based on whisper-generated text and reference text. BLEU scores highlight different scenarios. At times, Pepper's recorded audio perfectly matches the original speech, while there is no overlap between the two texts in some instances. However, in most instances, predicted and reference texts are moderately similar. The variation of Jaccard similarity scores signifies a moderate to high similarity between the words in the predicted and reference text. Varying range of WER values indicate instances of both perfect match and discrepancies between predicted text and original text.

Table 1. Features selected for male participants in the online mode of teaching.

Rouge1_fmeasure	Rouge2_fmeasure	RougeL_fmeasure	bleu_score	Jaccard_Similarity_Score	WER
0.9412	0.9333	0.9412	0.6004	0.5455	0.3333
0.8929	0.8519	0.8929	0.8266	0.8077	0.1071
0.7429	0.6061	0.7429	0.4225	0.5	0.3889
0.8182	0.7	0.8182	0.6989	0.6923	0.1818
0.8276	0.6786	0.8276	0.5197	0.6452	0.3333
0.9091	0.9032	0.9091	0.8702	0.8125	0.125
0.9091	0.8387	0.9091	0.7625	0.8235	0.125
0.8571	0.7368	0.8571	0.7071	0.8182	0.1
0.8	0.6429	0.8	0.4593	0.6667	0.2857
1.0	1.0	1.0	1.0	1.0	0.0
1.0	1.0	1.0	1.0	1.0	0.0
1.0	1.0	1.0	1.0	1.0	0.0
1.0	1.0	1.0	1.0	1.0	0.0
0.8889	0.8	0.8889	0.7612	0.8462	0.0769
0.8889	0.75	0.8889	0.7506	0.8	0.1111
0.6857	0.303	0.5714	0.0	0.4783	0.5882
0.7273	0.7097	0.7273	0.3595	0.5	0.9167
0.8718	0.7568	0.8718	0.6859	0.7368	0.1667
1.0	1.0	1.0	0.0	0.4	0.4286
1.0	1.0	1.0	0.726	0.6364	0.2222
0.8214	0.7407	0.8214	0.6824	0.6207	0.25
0.7458	0.5965	0.7458	0.4779	0.6061	0.3
0.8125	0.6667	0.8125	0.5681	0.6471	0.1875
0.8125	0.7333	0.8125	0.7368	0.7222	0.1875
0.7619	0.7368	0.7619	0.7598	0.6667	0.2
0.9286	0.8462	0.9286	0.8555	0.9286	0.0714
0.3462	0.24	0.3077	0.141	0.1707	3.0
0.7692	0.6667	0.7692	0.5784	0.6	0.2308
0.4898	0.2979	0.449	0.1412	0.303	1.2941
0.75	0.6364	0.75	0.5311	0.6	0.25
0.7887	0.6377	0.7606	0.4947	0.6774	0.3548

Figure 3 demonstrates the analysis of features generated by male participant's speech in offline mode. The selected values of features are placed in Table 2. It can be observed that there are high values for most of the records for F-scores of ROUGE-1, ROUGE-2, and ROUGE-L. This suggests that Pepper's speech recognition system is capable of acquiring both separate words and longer sequences. Further obtained varying values of the BLEU score and Jaccard score confirm that for most of the instances there is a significant similarity between the Whisper-generated texts and reference texts. Moreover, the low WER values suggest an accurate prediction of speech. These values together indicate a strong and accurate speech recognition system that successfully captures speech in offline mode.

Table 2. Features selected for male participants in the offline mode of teaching.

Rouge1_fmeasure	Rouge2_fmeasure	RougeL_fmeasure	bleu_score	Jaccard_Similarity_Score	WER
1.0	1.0	1.0	0.0	0.4	0.4286
0.8889	0.75	0.8889	0.3226	0.4167	0.4444
0.9455	0.9434	0.9455	0.8879	0.8462	0.1071
0.9189	0.8	0.9189	0.629	0.7778	0.1667
0.9375	0.871	0.9375	0.7975	0.871	0.1667
1.0	1.0	1.0	1.0	1.0	0.0
1.0	1.0	1.0	1.0	1.0	0.0
1.0	1.0	1.0	1.0	1.0	0.0
0.88	0.8696	0.88	0.68	0.6667	0.2857
0.7826	0.7619	0.7826	0.7103	0.6429	0.2727
0.8571	0.8462	0.8571	0.7577	0.6471	0.2143
1.0	1.0	1.0	1.0	1.0	0.0
0.9091	0.9	0.9091	0.8932	0.8182	0.0909
0.9286	0.9231	0.9286	0.8507	0.8571	0.1538
1.0	1.0	1.0	1.0	1.0	0.0
0.9412	0.875	0.9412	0.8282	0.8889	0.0588
1.0	1.0	1.0	0.0	0.4	0.4286
0.9474	0.8235	0.9474	0.6389	0.5833	0.3333
0.9643	0.963	0.9643	0.9622	0.92	0.0357
1.0	1.0	1.0	0.6899	0.8235	0.1111
1.0	1.0	1.0	1.0	1.0	0.0
0.9655	0.8929	0.9655	0.7823	0.8966	0.1
0.9825	0.9455	0.9825	0.7728	0.9231	0.1111
0.9375	0.9333	0.9375	0.9306	0.8667	0.0625
1.0	1.0	1.0	1.0	1.0	0.0
1.0	1.0	1.0	1.0	1.0	0.0
0.9286	0.8462	0.9286	0.7825	0.8667	0.0714
0.9524	0.8421	0.9524	0.7109	0.9091	0.0909
0.7857	0.6923	0.7857	0.645	0.6471	0.2143
1.0	1.0	1.0	1.0	1.0	0.0
1.0	1.0	1.0	1.0	1.0	0.0
0.96	0.8696	0.96	0.7677	0.9167	0.0769
1.0	1.0	1.0	1.0	1.0	0.0
0.8824	0.75	0.8824	0.6423	0.7895	0.1176
1.0	1.0	1.0	0.7825	0.8182	0.1
1.0	1.0	1.0	1.0	1.0	0.0
1.0	1.0	1.0	1.0	1.0	0.0

Figure 4 emphasises the visualisation of six performance indicators generated by female participants’ voices in online mode. The selected values of features are placed in Table 3. F-scores of ROUGE-1 and ROUGE-L suggest a significant capability of speech recognition systems in accurately identifying words and longer sequences. The F-score of ROUGE-2 implies a moderate to convincing acquiring bi-grams within the generated text by Whisper. BLEU score reveals varying ranges of matching between the predicted and reference texts, with some cases a perfect match while in others, there are fewer similarities. Most of the time, Jaccard's score is high; it shows a significant similarity of words in predicted and reference texts. WER values are mostly low, indicating that the recorded speech by Pepper is strongly like the original.

Table 3. Features selected for female participants in the online mode of teaching.

Rouge1_fmeasure	Rouge2_fmeasure	RougeL_fmeasure	bleu_score	Jaccard_Similarity_Score	WER
0.6667	0.64	0.6667	0.2646	0.3	jan.33
0.8929	0.8148	0.8929	0.7372	0.7407	0.1429
0.9444	0.8824	0.9444	0.6269	0.7222	0.1667
0.9091	0.9	0.9091	0.8932	0.8333	0.0909
0.918	0.8475	0.918	0.7203	0.7273	0.1667
0.9286	0.8889	0.9286	0.7637	0.8214	0.1111
0.8387	0.6897	0.8387	0.6423	0.75	0.1875
0.9375	0.8667	0.9375	0.8762	0.9333	0.0625
0.8571	0.8421	0.8571	0.7861	0.75	0.2
0.8276	0.6667	0.7586	0.5703	0.6471	0.2857
0.9524	0.9474	0.9524	0.7964	0.75	0.1818
0.8571	0.6923	0.8571	0.6298	0.75	0.1429
1.0	1.0	1.0	1.0	1.0	0.0
0.8182	0.7	0.8182	0.6875	0.5833	0.2727
1.0	1.0	1.0	1.0	1.0	0.0
0.7778	0.5	0.7778	0.3689	0.6364	0.2222
0.8889	0.7059	0.8889	0.4531	0.6364	0.2941
0.7619	0.6316	0.7619	0.4692	0.6154	0.3
0.88	0.7826	0.88	0.6652	0.7143	0.25
0.9474	0.8889	0.9474	0.7966	0.7778	0.1667
0.9143	0.8824	0.9143	0.6126	0.7	0.2581
1.0	1.0	1.0	0.0	0.4	0.4286
0.8889	0.75	0.8889	0.3051	0.3077	0.5556
0.9286	0.8889	0.9286	0.8646	0.8462	0.0714
1.0	1.0	1.0	0.8394	0.9375	0.0556
1.0	1.0	1.0	1.0	1.0	0.0
0.8182	0.7813	0.8182	0.4759	0.5526	0.5556
0.8387	0.7586	0.8387	0.6296	0.6471	0.25
1.0	1.0	1.0	0.8154	0.875	0.0625
0.8421	0.7059	0.8421	0.6717	0.7273	0.2
0.963	0.88	0.963	0.5799	0.6875	0.2143
0.9091	0.8	0.9091	0.7017	0.8333	0.0909
0.8571	0.7692	0.8571	0.6998	0.8	0.1429
1.0	1.0	1.0	1.0	1.0	0.0
1.0	1.0	1.0	1.0	1.0	0.0
0.9231	0.8333	0.9231	0.7612	0.8462	0.0769
1.0	1.0	1.0	1.0	1.0	0.0
0.9412	0.875	0.9412	0.8844	0.8889	0.0588
0.75	0.6364	0.75	0.3556	0.5714	0.6
1.0	1.0	1.0	1.0	1.0	0.0
0.8718	0.8108	0.8718	0.7332	0.7778	0.1667
0.9091	0.8438	0.9091	0.5615	0.6667	0.1935

Figure 5 exhibits the analysis evaluation metrics suggesting the accuracy of the speech recognition system of Pepper. The selected values of features are placed in Table 4. The values of F-scores of ROUGE-1 and ROUGE-L indicate the ability to obtain words and long sequences accurately. The F-score of ROUGE-2, which varies from moderate to high, implies that the speech recognition system's ability to identify bi-grams within the text also varies. The varying range of BLEU scores suggests both substantial and less similarity between the Whisper generated and original texts. For most of the instances, the Jaccard score is high, indicating a significant match of words. The inconsistent WER values demonstrate the divergence from the original speech.

Table 4. Features selected for female participants in the offline mode of teaching.

Rouge1_fmeasure	Rouge2_fmeasure	RougeL_fmeasure	bleu_score	Jaccard_Similarity_Score	WER
1.0	1.0	1.0	0.0	0.4	0.4286
1.0	1.0	1.0	0.5874	0.5	0.3333
0.8364	0.7547	0.8364	0.5722	0.6	0.2857
0.7119	0.5614	0.7119	0.3081	0.5152	0.4828
0.8333	0.6471	0.8333	0.5731	0.7222	0.1667
0.8182	0.8	0.8182	0.7861	0.6923	0.1818
0.8852	0.7797	0.8852	0.7226	0.75	0.1667
1.0	1.0	1.0	1.0	1.0	0.0
0.9375	0.8667	0.9375	0.6865	0.8125	0.125
0.9524	0.8421	0.9524	0.6989	0.75	0.2
1.0	1.0	1.0	0.7825	0.8667	0.0714
0.9091	0.8	0.9091	0.7017	0.8333	0.0909
0.8571	0.6154	0.7857	0.5565	0.75	0.2143
1.0	1.0	1.0	1.0	1.0	0.0
0.7857	0.7692	0.7857	0.6102	0.6667	0.5455
0.6154	0.5	0.6154	0.2928	0.4375	1.1111
0.9714	0.9091	0.9714	0.716	0.75	0.1765
0.9	0.8889	0.9	0.5254	0.6667	0.2
0.9167	0.8182	0.9167	0.7349	0.9167	0.0833
0.7727	0.6667	0.7727	0.5331	0.6667	0.4444
0.9412	0.9333	0.9412	0.6004	0.5455	0.3333
0.9444	0.8824	0.9444	0.6269	0.7647	0.1667
1.0	1.0	1.0	1.0	1.0	0.0
1.0	1.0	1.0	1.0	1.0	0.0
0.9375	0.8667	0.9375	0.8762	0.9333	0.0625
0.8571	0.8421	0.8571	0.7861	0.75	0.2
0.8387	0.7586	0.8387	0.6739	0.8125	0.2143
0.6667	0.5714	0.6	0.4561	0.4286	0.9286
1.0	1.0	1.0	1.0	1.0	0.0
1.0	1.0	1.0	1.0	1.0	0.0
1.0	1.0	1.0	1.0	1.0	0.0
0.8571	0.7273	0.8571	0.6654	0.75	0.1765
0.9	0.8889	0.9	0.8801	0.8182	0.1
1.0	1.0	1.0	1.0	1.0	0.0
1.0	1.0	1.0	1.0	1.0	0.0
0.9394	0.875	0.9394	0.4682	0.6129	0.2903

Figure 6 illustrates the average feature values of seven indicators for both genders in online and offline modes. The full table of average feature values is provided as supplementary data due to its size. The average F-score of ROUGE-1 indicates that Whisper-generated text from male and female online speeches are reasonably similar to the reference text. The same is true for the offline speech of a female. However, the F-score of ROUGE-1 corresponding to text of offline male speech signifies a strong matching with reference text. A similar kind of pattern is repeated for average F-scores of ROUGE-2 and ROUGE-L. The values of average BLEU scores suggest that only whisper-generated text for male offline speech matches significantly with the original text. However, male speech in online mode is moderately similar to reference text. The average value of the Jaccard Similarity score indicates that male speech in offline mode has a relatively higher degree of match with an original speech compared to others. At the same time, the male speech in online mode shows slight match with reference speech. However, the score of the online speech of male is relatively worse than the speech of female. The average values of WER suggest that whisper generated text and reference text match to each other quite well for both genders. The average value of TF-IDF similarity suggests that the speech recognition system is efficient in capturing male speech in offline mode. However, the values are acceptable for the offline mode of a female and an online mode of a male as well as a female. Furthermore, overall findings suggest that male voice is more suitable for offline mode than online mode, however for female voice, it is quite similar in all the cases. These results suggest that although there were differences in scores based on modes and gender but these variations are comparatively small.

Figure 6. Visualization of average features in online and offline modes of teaching.

5. Discussion

Several studies were conducted in the past that focused on the incorporation of technologies in the education system. Mavrikis et al. (2014) demonstrated how speech recognition technology can aid problem-solving and reflective learning. Grassini (2023) explored the potential of artificial intelligence and ChatGPT in the context of education. Some studies highlighted that including robots can affect the learning process (Chen et al. 2022; Engwall and Lopes 2022; Randall 2019). The present study provides essential insights into the humanoid robot Pepper as an educational assistant by assessing the speech recognition system for both online and offline modes. Further, it offers the opportunity to look over the capabilities and limitations of Pepper by integrating it with the speech-to-text conversion tool Whisper through the experiment results.

In relation to RQ1, the result shows that integration of Pepper and Whisper mostly achieves good scores for performance indicators while keeping low values of WER. Research findings show variability in the F-score values of ROUGE across different text records. These F-score values, namely ROUGE-1, ROUGE-2, and ROUGE-L, indicate how accurately the Pepper speech recognition system can capture individual words, bi-grams, and longer subsequences. When the F-score values are high, the speech recognition system performs well and accurately matches predicted and reference text (Topaz et al. 2022). This supports the idea that in integration with Whisper, Pepper can efficiently record and convert speech to text on both online and offline mode of teaching. The BLEU score measures the similarity between the predicted and reference text by comparing the overlap of n-grams. A high BLEU (Liu et al. 2003) score indicates accurate transcriptions of Pepper's recorded speech. This suggests that Pepper's speech recognition system successfully captures separate words and long sentences. The Jaccard Similarity scores indicate how much the generated text overlaps with the reference text regarding unique words. The consistently high (Pinto et al. 2012) scores suggest that Pepper's speech recognition system does an excellent job of picking up essential vocabulary from the spoken content. Even though there may be differences in how things are said, the overall system can extract crucial information from the reference text. The continuous high values of TF-IDF similarity suggest that the speech recognition system is sufficient to capture the audio. The Word Error Rate (WER) values, which show how much the predicted speech differs from the original, are usually less. This indicates the overall system is reasonably accurate at capturing and converting spoken language to text, with only minor differences between predicted and actual speech. Based on the feature values obtained in this study during the evaluation of speech recognition system, make Pepper a suitable candidate for online and offline both modes of teaching in a classroom like settings.

Previous studies also explored the integration of speech recognition systems in classroom settings for better teaching and learning experiences. Matre (2022) conducted a qualitative study for inclusion of speech-to-text (STT) technology in classrooms for helping students to write with the help of their voice. Their findings encompassed teachers’ experiences about this inclusion, revealing that this technology helpful for longer assignments, however the study also addressed some challenges during its implementation such as issues of inaccuracy, inappropriate use, and instances of whispering among students. In another study Matre and Cameron (2024) highlighted that STT is a useful way in enhancing the performance of writing of students with learning difficulties. Sun (2023) demonstrated that the use of ASR enabled experimental group to perform better than control group in learning a second language.

Regarding RQ2, research findings suggest that male voices are better suited for offline modes than for online modes. However, for female voices, there is little difference between the two modes. It is worth noting that although there are differences in scores based on gender and mode, these variations are relatively small. Existing literature is inconclusive in this area. Some studies (Adda-Decker and Lamel 2005; Sawalha and Abu Shariah 2013) suggested that performance of speech recognition is better for female voices than male voices. However another study by Tatman (2017) reported a contradictory result by showing lower accuracy in the speech recognition of female.

6. Conclusion, limitations and future work

A thorough analysis of Pepper's speech recognition system reveals its impressive ability to capture speech in both online and offline teaching environments accurately. With reference to RQ1, it has been observed that the overall system, including Pepper and Whisper, consistently achieves good scores on various performance indicators such as ROUGE F-score, BLEU, and Jaccard Similarity metrics, and TF-IDF similarity score while maintaining low WER values. In consideration to RQ2, for male and female voice in online and offline mode of teaching, the results of this study demonstrate minor differences in calculated scores. These findings demonstrate the overall system's potential to facilitate educational activities by capturing audio and converting speech-to-text with precision. These findings can contribute to the existing research on humanoid robots in education, highlighting their potential to serve as valuable educational assistants in different educational settings.

The proposed system with Pepper supports in understanding varying accents by transcribing spoken words into text and finally, displaying on Pepper’s display. However, there are a few challenges that may arise while utilising this system. The most prominent issue is related to Pepper's limited vocabulary, which can make it challenging for the system to comprehend complex words. Moreover, background noise can pose an obstacle, as it may inhibit Pepper's ability to perceive spoken utterances accurately. However, there are possibilities to overcome these issues. Challenges regarding noise can be addressed using the approach as described in this study related with unwanted utterances of Pepper. Pepper’s vocabulary can be enhanced, and background noise can be reduced by incorporating artificial intelligence techniques. Furthermore, the experiment conducted during the study was organised in a laboratory environment rather than in a real-world scenario. Therefore, we aim to improve Pepper's voice recognition capability by testing it in different environments along with including participants from diverse age groups in the future work. The inclusion of social robots as educational assistants has the potential to supplement the teaching and learning process.

The developed system with social robot and whisper has several implications. The presence of robot can enhance the students’ engagement and learning. We found that there is no major difference between the audio quality captured by robot, while teaching in offline and online mode. This will provide the teacher a flexibility to deliver lectures remotely as long as a robot as an educational assistant is present. If the teacher prefers to be present in-person, this will open up opportunities for guest lecturers to participate remotely. Furthermore, for teachers there will be no need to repeat their statements, and this will facilitate them to focus on delivering quality lecture. The students can have robot’s support at the time of their choice as they can listen to lecture recordings along with speech transcription.

In the future, additional features, such as the robot identifying when students have questions during a lecture, can be incorporated into the developed system by employing AI techniques. One such feature could be the ability to recognise raised hands, which would help in addressing student concerns. Furthermore, this system could assist in detecting the emotional state of students through facial expression, gesture and speech analysis. In case of any stress, social robot can provide them with emotional support and suggestions. The system could have the potential to provide solutions in response to particular query or suggestions to reduce their cognitive burden, ultimately enhancing the learning experience of students. All these additional functionalities will have the potential to make the proposed system more usable in classroom settings. The effectiveness of such a system could be tested by analyzing the collecting qualitative data such as questionnaires, surveys and interviews collected from both teachers and students.

Disclosure statement

No potential conflict of interest was reported by the author(s).