Designing human-centered learning analytics and artificial intelligence in education solutions: a systematic literature review

ABSTRACT

The recent advances in educational technology enabled the development of solutions that collect and analyse data from learning scenarios to inform the decision-making processes. Research fields like Learning Analytics (LA) and Artificial Intelligence (AI) aim at supporting teaching and learning by using such solutions. However, their adoption in authentic settings is still limited, among other reasons, derived from ignoring the stakeholders' needs, a lack of pedagogical contextualisation, and a low trust in new technologies. Thus, the research fields of Human-Centered LA (HCLA) and Human-Centered AI (HCAI) recently emerged, aiming to understand the active involvement of stakeholders in the creation of such proposals. This paper presents a systematic literature review of 47 empirical research studies on the topic. The results show that more than two-thirds of the papers involve stakeholders in the design of the solutions, while fewer papers involved them during the ideation and prototyping, and the majority do not report any evaluation. Interestingly, while multiple techniques were used to collect data (mainly interviews, focus groups and workshops), few papers explicitly mentioned the adoption of existing HC design guidelines. Further evidence is needed to show the real impact of HCLA/HCAI approaches (e.g., in terms of user satisfaction and adoption).

KEYWORDS:

• Human-centered design
• learning analytics
• artificial intelligence
• systematic literature review

1. Introduction

Current digital technologies enable the collection of fine-grained data on teaching and learning, which can potentially inform and recommend actions to a variety of stakeholders, including students, teachers, curriculum designers, and managers. Given this context, the Learning Analytics (LA) field has contributed to understanding and improving learning and its context, while Artificial Intelligence in Education (AIED) has focussed especially on simulating and predicting learning processes and behaviours. Both fields use the data generated from these systems and often apply similar analysis techniques such as machine learning (Rienties, Køhler Simonsen, and Herodotou 2020). For example, multiple technologies developed within the LA and AIED context have been successfully applied in educational settings to support task automation (Tsai et al. 2021), personalise learning (Chakraborty et al. 2021), and improve teacher awareness (Matcha et al. 2020). Consequently, during the last few years, LA and AIED have gained much attention.

Despite the rising interest in LA and AIED solutions (Salas-Pilco, Xiao, and Hu 2022), the widespread adoption of these technologies among stakeholders still remains limited (Sadallah et al. 2022). Among the multiple reasons behind this lack of adoption (e.g., costs, technical requirements, or institutional policies), several authors have pointed out the lack of contextual grounding of these solutions as one of the causes. This lack of grounding overlooks the pedagogical background and the stakeholder's actual needs (Dimitriadis, Martínez-Maldonado, and Wiley 2021; Sarmiento and Wise 2022). To overcome this limitation, many researchers emphasise the importance of adopting human-centered approaches where stakeholders are actively involved in the design, development and evaluation of LA/AIED solutions (Buckingham Shum, Ferguson, and Martinez-Maldonado 2019; Rodríguez-Triana et al. 2018). Such stakeholder involvement leads to better tailoring the solutions to the contextual requirements and needs, as well as to raising the reliability and trustworthiness of the systems (Martinez-Maldonado 2023), thereby removing barriers to their adoption. To address these issues, the subfields of Human-Centered LA (HCLA) and Human-Centered AI in Education (HCAI for brevity) have emerged to understand, inform and promote the design and development of human-centered solutions within their corresponding communities.

Nowadays, these two subfields are reaching a certain maturity, triggering the appearance of literature reviews to explore different aspects of HCLA/HCAI. Li and Gu (2023) carried out a systematic literature review about ethical design approaches and risks of HCAI. The literature review together with a Delphi study led to the identification of 8 potential HCAI risk indicators. Additionally, Sarmiento and Wise (2022) explored systematically the stakeholders' involvement in the co-design and participatory design of LA proposals, two core approaches in human-centered design. More concretely, their literature review identifies the stakeholders involved in the different design phases, as well as the tools and techniques used for that purpose. Both systematic reviews shed light on participatory approaches for AI and LA. However, to the best of our knowledge, in Li and Gu (2023) the authors did not explore how stakeholders are involved in the design and development of HCAI solutions. In the case of Sarmiento and Wise (2022), the authors particularly focussed on participatory design and co-design, leaving aside other terms used in the community (e.g., user-centered or human-centered design). Additionally, these papers do not offer systematic analyses of the specific methods and tools used to involve and collect data from the stakeholders while implementing the HC approaches. Furthermore, in both systematic reviews, there is no consideration of the learning theories applied to inform HCLA/HCAI research proposals and of the evaluation approaches employed on both HCLA and HCAI contributions.

We deem that the increasing number of research studies on HCLA and HCAI published in the last years presents a timely opportunity to obtain a global understanding of the HCLA/HCAI approaches employed in the literature. Such analysis can help identify current research gaps and potential research lines within the field. Thus, this paper reports a systematic literature review on HCLA/HCAI guided by the overarching research question: What is the current landscape of HCLA/HCAI empirical evidence?, which has been further subdivided into the following ones:

(RQ1)	Which learning theories/pedagogical approaches are considered in the design of HCLA and HCAI solutions?
(RQ2)	What are the main aims of the proposed HCLA and HCAI solutions?
(RQ3)	Which and how stakeholders are involved in the design and development of HCLA and HCAI solutions?
(RQ4)	Which methods and tools are used to design and develop HCLA and HCAI solutions?
(RQ5)	How are the HCLA and HCAI solutions being evaluated?
(RQ6)	What are the pros and cons of using HC approaches?

The rest of the paper is structured as follows. Section 2 describes the theoretical background behind the use of human-centered approaches for LA and AI. Section 3 presents the methodology followed to extract the 47 reviewed papers as well as the analysis approach. Section 4 reports the results of the review. Section 5 discusses the results according to the RQs and reflects on potential theoretical and practical implications for researchers and practitioners. Finally, Section 6 outlines the conclusions and limitations of this work.

2. Theoretical background

Recently, there has been a growing interest in human-centered design (HCD) within the Technology-Enhanced Learning field. HCD describes approaches that position the stakeholders and designers as collaborators in the creation of contextualised products (Zachry and Spyridakis 2016). According to Rouse (Rouse 2007), HCD should increase the capacities of humans, overcome their restrictions and foster technological acceptance. Dimitriadis, Dimitriadis, Martínez-Maldonado, and Wiley (2021) highlighted that, when employing HCD processes, researchers should guarantee the a) “agentic positioning” of the stakeholders, b) the consideration of the learning design cycle, and c) the pedagogical grounding on educational theories to guide the design of the desired solutions.

For decades, different terminologies have been used in the literature to describe human-centered approaches. All these approaches agree on the stakeholders' involvement during the design process to understand their perspectives, needs and context. However, while these terms are sometimes used interchangeably, they have different implications in practice (Buckingham Shum, Ferguson, and Martinez-Maldonado 2019):

• User-centered design: This approach sets the focus on the role of the stakeholders as 'users', and it refers to a design approach that concentrates on the usability of the design given the needs and experiences of users (Abras, Maloney-Krichmar, and Preece 2004). In this approach, the roles of the researcher, the designer and user are distinct. The user is not really a part of the design team (Sanders and Stappers 2008).

• Participatory design: This approach examines the users' needs and requirements by empowering them to take an active role in shaping the products, services or systems (Bødker et al. 2022). Compared to user-centered design, in participatory design, the roles of the designer and the researcher are not independent and the user becomes a critical component of the design process, yet without participating in the decision-making (Könings, Seidel, and van Merriënboer 2014; Sanders and Stappers 2008).

• Co-design: In this approach, designers and stakeholders (without design experience) collaborate during the design and development of a product, emphasising their collaboration and shared decision-making (Sanders and Stappers 2008). In co-design, each participant is considered an expert when it comes to their own experiences, thus, they draw upon their practical, experiential, and conceptual knowledge in the design process (Cavignaux-Bros and Cristol 2020).

The fields of HCLA and HCAI emerged to create actionable LA/AI solutions attending to the stakeholders' needs to overcome existing barriers, e.g., providing interpretations of the visualised data which may be challenging depending on the stakeholders' data literacy (Dimitriadis, Martínez-Maldonado, and Wiley 2021). HCLA refers to the use of LA to support the needs and goals of learners, instructors, and other stakeholders in the education process, in a way that the LA solutions suit the users and not the other way around (Buckingham Shum, Ferguson, and Martinez-Maldonado 2019). HCLA examines data related to student learning, such as engagement with course materials, performance on assessments, and interaction with peers and instructors. The shift regards the commitment of the stakeholders as co-designers, who are expected to participate in the design of LA solutions, regardless of their level of technical expertise or prior experience with analytics tools. This involves designing dashboards and visualisations that are easy to understand and interpret, as well as providing support and training to learners and instructors to help them make sense of the data.

Together with HCLA, there is a growing body of AI works that advocate HCD. AI has been applied extensively across sectors like healthcare, medicine, finances, and security (Salas-Pilco, Xiao, and Hu 2022). As far as it concerns the field of education, AI is being used through intelligent tutoring systems and virtual assistants, automated scoring on assignments, adaptive learning systems and predictive analytics. Shifting to HCD, HCAI supports the design and development of AI proposals that prioritise the needs, values, and goals of humans (Shneiderman 2020).

Both LA and AIED researchers have proposed conceptual tools such as processes (Knight et al. 2015; Martinez-Maldonado et al. 2022), strategies (Dollinger and Lodge 2018), and frameworks (Chatti et al. 2020; Martinez-Maldonado et al. 2015) for HC design. However, it is not clear how HCLA and HCAI approaches are put into practice.

3. Methodology

The literature review followed the guidelines proposed by Kitchenham and Charters (2007) to answer the aforementioned research questions. Although these guidelines were initially conceived for the software engineering field, they are typically used in other research fields including LA (e.g., Matcha et al. 2020) and AIED (e.g., Hooshyar, Yousefi, and Lim 2019). These guidelines structure the review process in three phases: planning, conducting, and reporting. This section summarises the planning and conducting phase, and the following sections report and discuss the results.

The decisions taken during the planning phase include the selection of the databases, the search string, the search location, the time window, and the inclusion and exclusion criteria (see Table 1). The databases included (a) digital libraries that typically publish articles related to this area (i.e., Scopus, which also includes articles listed in IEEE Xplore; and Web of Science, which also indexes articles listed in ScienceDirect); (b) journals with specific focus on LA/AIED (i.e., the Journal of Learning Analytics and the International Journal of Artificial Intelligence in Education); and (c) conferences with specific focus on LA/AIED (namely, LAK, AIED, CHI and EDM).

Table 1. Decisions taken during the SLR planning phase.

Category	Decision	Reason
Databases	Scopus	Scopus indexes papers in IEEE Xplore and WoS papers in ScienceDirect, thus including the main databases of TEL journals. JLA, IJAIED, LAK, CHI, AIED and EDM are forums that publish LA/AI studies in the educational landscape.
	Web of Science
	J. of Learning Analytics
	Int J. of AIED
	LAK Conference Proc.
	CHI Conference Proc.
	EDM Conference Proc.
	AIED Conference Proc.
Search String	[‘human-centered design’ OR ‘centred design’ OR ‘participatory design’ OR ‘co-design’] AND [‘learning analytics’ OR (‘artificial intelligence’ AND ‘education’)]	The search string includes the term ‘human-centered’ and the synonyms related to human-actions (e.g., participatory design, co-design) reported in relevant studies. We used the terms ‘-centred’ to detect combinations such as ‘user-’, ‘teacher-’ or ‘student-’. Furthermore, we used the terms associated with the research fields under study (i.e., ‘learning analytics’ or ‘artificial intelligence’) and the term ‘education’ that regards the frame of the systematic literature review.
Search Location	Title, abstract and keywords (metadata or abstract if restriction)	We expect that publications describing human-centered LA or AI studies mention it in the title abstract and/or keywords. Since there were variations in the way each search engine applied the query, to guarantee the same filtering criteria, we cross-checked that the title, abstract and keywords of each paper satisfied the query.
Time Restrictions	No restrictions	We aim to find all possible publications without tying to a specific time frame.
Inclusion Criteria	Empirical studies where LA and AI solutions are designed using human-centered approaches	We focus our research questions on empirical studies of HCLA and HCAI. Theoretical or conceptual papers are not relevant to answer the posed research questions.
Exclusion Criteria	Secondary studies (reviews); Papers under 5 pages; Project summaries; Non-English papers; Duplicate papers	We aim to analyse primary studies describing complete empirical experiences in English.

The terms used in the query included the main HCD approaches introduced in the previous section plus ‘centred design’, to detect similar terms such as user, teacher or student-centred design. Moreover, we included the terms representing the domains (i.e., LA and AIED). As a result, we used the following query: [‘human-centered design’ OR ‘centred design’ OR ‘participatory design’ OR ‘co-design’] AND [‘learning analytics’ OR (‘artificial intelligence’ AND ‘education’)]. The search was performed in December 2022 without imposing any time constraints. Whenever possible, we narrowed down the query to the paper title, abstract and keywords, obtaining a total number of 294 publications (see Figure 1). Since there were variations in the way each search engine applied the query, to guarantee the same filtering criteria, we cross-checked that the title, abstract and keywords of each paper satisfied the query.

Figure 1. Overview of the systematic literature review process followed.

Publications were then screened according to the inclusion and exclusion criteria by manually reading the title, abstract and keywords. During this process, we selected primary studies describing complete empirical experiences in English where LA/AI solutions were designed using HC. Moreover, we excluded papers under 5 pages, project summaries, and duplicate papers. From the total number of 294 initial publications, 47 were included in this literature review (see Figure 1). The full contents of the selected publications were then read.

Content analysis has been conducted employing etic categories, i.e., predetermined categories established prior to data analysis (Given 2012). Then the coding scheme was developed based on the established RQs. To ensure the reliability of our findings, we implemented the following strategies (Guba 1981): (a) Peer debriefing by involving the research team to review the coding scheme together, (b) Triangulation among researchers respecting the interpretation of the data.

The literature review involved 6 researchers (see author list) who actively participated in the filtering and data analysis process of different primary studies. To assess the inter-researcher consistency of the coding scheme, a random paper was chosen to be coded by all reviewers prior to the data analysis phase (Kitchenham and Charters 2007). Once a common understanding of the identified categories was set, the remaining papers were distributed among the reviewers for individual coding. The decision on whether to include/exclude dubious cases was solved through a joint discussion and a second researcher reading the paper in depth. Finally, the different topics under review were allocated to 3 researchers who cross-checked the coding of each paper and discussed with the whole team the potential discrepancies found. An overview of the final codification is available in Appendix.

4. Results

This section provides an overview of the reviewed papers and summarises the results of the literature review in relation to the aforementioned research questions.

4.1. Sample overview

Looking at the publication date, while the first HCLA (Knight et al. 2015) and HCAI (Long, Aman, and Aleven 2015; Santos and Boticario 2015a, 2015b) empirical studies date back to 2015, most papers were published starting from 2019 (37, 78.72%). Figure 2 depicts the reviewed articles, including the publication type and year. In terms of the research domain, few are in the artificial intelligence or data mining domains (8, 17,02%), while most papers (43, 82.98%) pertain to the learning analytics field.

Figure 2. Number of studies per publication type and year.

Additionally, we further analysed the relationships among the authors of the papers. In Figure 3, each bubble represents a different author (168 in total), the size of the bubble represents the number of papers by each author (spanning between 1 and 5), and the clusters reflect a coauthoring relationship. Some collective efforts yielded multiple papers by the same group of authors (e.g., Aleven and Martínez-Maldonado co-authored 5 and 4 papers respectively), indicating their leadership in the field of HCLA/HCAI. Nevertheless, there are also many other different teams that contributed to the HCLA/HCAI literature, showing that a wider community is adopting human-centered approaches.

Figure 3. Social network analysis of the reviewed papers based on co-authorship.

According to the publication type, 31 (65.96%) were published in conference proceedings, 13 (27.66%) in scientific journals and 3 (6.38%) in workshop proceedings. Among the conference papers, 11 (35.48%) were published in GGS¹ and/or CORE² indexed conferences, being LAK and EC-TEL conferences the most frequent venues of publication (with 9 and 7 papers, respectively). Also, among the journal papers, 5 (38.46%) were published in impact factor journals according to the Web of Science. The Journal of Learning Analytics was the most popular venue.

Among the different types of HCD approaches described in Section 2, the most frequent terminology used was co-design (15, 31.91%), followed by user-centered design (11, 23.40%) and participatory design (8, 17.02%). Interestingly, many papers used several of them interchangeably within the same study (10, 26.74%), sometimes as synonyms (e.g. Sadallah et al. 2022).

4.2. Learning theories and design aspects

Out of the 47 papers retrieved, 27 (57.45%) did not mention which learning theories or pedagogical approaches characterised the learning context and/or informed their LA solutions. Figure 4 depicts the identified learning theories. Some papers included more than one learning theory, e.g., Kilińska, Kobbelgaard, and Ryberg 2019).

Figure 4. Learning theories identified in the review.

The majority of the papers with a reference to theories used Collaborative Learning and its instances (9, 45.00%) -which entail working in pairs or groups to engage in discussions about concepts or to seek solutions to problems- as a theoretical basis for the design of the LA/AI solutions. For instance, Hoffmann et al. (2022) discuss the use of Computer Supported Collaborative Learning, while Martinez-Maldonado et al. (2022) refer to Collocated Collaborative Learning to inform the contextual design of their LA proposal. 4 papers mention Self-Regulated Learning (4, 20.00%) -which fosters learners' self-reliance as they progress through their learning process (Zimmerman 2000)- to ground their work. Additionally, further papers draw upon Constructivism (2, 10.00%), as in Huh et al. (2022) who use Vygotsky's Sociocultural Theory of Cognitive Development, and Porject-Based Learning (2, 10.00%). Single mentions regard the Cognitive Theory of Multimedia Learning (i.e. Revano and Garcia 2021) or the Motivation Theory (i.e. Long, Aman, and Aleven 2015).

Also, in some cases (11, 55.00%), the authors took into consideration specific aspects of the learning design to contextualise further their solutions. Examples are the actors/participants, the resources/objects, the learning objectives/goals, the social level (individual, group, classroom or institutional), the learning tasks and their types, time-related aspects, and teacher expectations about the students work. Building on the particularities of the learning design, the researchers adopted various pedagogical approaches, such as Adaptive Learning (i.e., Holstein, McLaren, and Aleven 2019a), Open-Ended Learning (i.e., Beheshti et al. 2020), Blended (i.e., Aleven et al. 2016) or Active Learning (i.e., Alzoubi et al. 2021).

Figure 5 presents the distribution of theories employed in the included papers. As observed in the graph, from 2018 there are more mentions to the pedagogical underpinning among the encountered papers. Keeping in mind the small numbers depicted, this result is just indicative of the potential awareness of the importance of the learning theories and learning design in grounding the LA/AI proposals.

Figure 5. Evolution of the number of publications involving learning theories.

4.3. Purposes of the HCLA/HCAI solutions

Inspired by the reference model proposed by Chatti et al. (2013) to characterise LA solutions, we analysed the paper contributions according to their type (what?), target users (who?), and purpose (why?).

What? According to the results, most of the studies propose standalone or embedded LA Dashboards (21, 44.68%), being followed by the design of entire LA Tools (13, 27.66%) and the identification of relevant indicators for different purposes (6, 12.77%). Other purposes also include the design of AI agents/systems (Holstein, McLaren, and Aleven 2019a; Huh et al. 2022), recommender systems (Santos and Boticario 2015b), and virtual assistants (Lister et al. 2021). Almost half of the analysed papers (22, 46.81%) do not report the learning platform where the proposed HCLA/HCAI contribution will be implemented. From the ones that report it, we can highlight that 10 refers to web-based applications, 6 to Learning Management Systems, 3 to Intelligent Tutoring Systems and 3 to mobile applications.

Who? The final users of the targeted solutions are solely teachers (19, 40.43%), solely students (18, 38.30%), or both of them (4, 8.51%), at all different educational levels (i.e., primary, secondary and tertiary). The remaining papers target other stakeholders, or a combination of teachers with other stakeholders (5, 10.64%), including, for example, educational managers (Eradze, Rodriguez Triana, and Laanpere 2017), parents (Huh et al. 2022) or museum visitors (Beheshti et al. 2020). Further details about the target users are provided in the following subsection.

Why? To better understand the human-LA/AI tandems, we categorised the HCLA/HCAI contributions based on the framework proposed by Soller et al. (2005). This framework identifies three types of tools: mirroring tools, which only display indicators; metacognitive tools, which compare the desired and the current results of the selected indicators; and guiding tools, which offer advice based on an interpretation of the indicators. In addition, we have included the category of ”intelligent systems” to refer to those contributions where the decision-making is done automatically. The results revealed that although mirroring tools are the most frequent purposes of HCLA solutions (e.g., Ahn et al. 2019), there are also guiding tools (e.g., Ouatiq et al. 2022) and intelligent systems (e.g., Long, Aman, and Aleven 2015).

4.4. Stakeholders and involvement in HC processes

The active positioning of stakeholders is one of the key aspects of HCD processes (Dimitriadis, Martínez-Maldonado, and Wiley 2021). The analysed data showed that teachers are the main participants involved in the human-centered processes (14, 29.79%). The teachers' involvement was particularly strong in HCLA/HCAI proposals addressing Higher Education (12 out of the 14 proposals), with only 2 HCLA solutions targeting K12 educational settings. Students were the second key actors either alone (9, 19.14%) or together with the teachers (9, 19.14%). A considerable number of papers also involved other types of participants in the HCD process either alone (4, 8.51%), together with students (3, 6.38%), together with teachers (5, 10.64%) or together with both teachers and students (3, 6.38%). IT experts, teaching assistants, school managers, project partners, and developers were among these stakeholders. One study (Huh et al. 2022) involved parents as well.

Figure 6 visualises the connection between the stakeholders who participated in the process of co-designing the HCAI/HCLA proposals and the end users aiming to take advantage of the final proposal. Based on the evidence gathered, it appears that almost half of the examined papers invited the final users as co-designers of the HCLA/HCAI solutions (25, 53.20%). However, in other cases, either additional stakeholders participated in the design or development of the proposals (15, 31.91%) or the involved stakeholders were different from the targeted ones (7, 14.90%). For example, Zhou, Sheng, and Howley (2020) and Romero et al. (2021) created AI and LA solutions to support students' algorithm understanding and self-regulated learning, respectively. Yet, in the first case, the involved stakeholders in the co-designing process were teachers and in the second case students together with project managers and developers.

Figure 6. Sankey diagram displaying the involved stakeholders in the design and development of HCLA/HCAI solutions and their target users (N = 47).

Attending the techniques and methods applied to the HCD processes, interviews seem to be the most prominent one (28, 56.57%), followed by co-design sessions and workshops (24, 51.06%) and surveys and questionnaires (13, 27.66%). Other techniques regarded low-fidelity (3, 6.38%) and paper-based prototyping (2, 4.26%), observations (3, 6.38%), Wizard of Oz (2, 4.26%), etc. Additionally, different types of cards were recurrently used to help participants express their needs and opinions (9, 19,15%), being some card decks explicitly developed for co-designing LA solutions (e.g., Alvarez, Martinez-Maldonado, and Shum 2020).

Each method served different purposes during the applied HCD processes. In our analysis, we followed the Human-Centered Indicator Design (HCID) framework (Chatti et al. 2020) to better understand the role of each technique/method in the design and development of HCLA and HCAI solutions. The framework consists of four phases: (a) define a goal/question that aims to understand the users' needs, (b) ideate that aims to support the co-creation phase, (c) prototype and d) test by getting feedback from the users. Figure 7 depicts how many times the techniques and methods were encountered in the publications and the purpose they served in accordance with HCID framework categories (Chatti et al. 2020). In terms of stakeholders' involvement in each of the HCID phases, our results indicate that human positioning happened mainly during the ‘Define the goal’ phase (44, 93,61%), and less participation was planned during the following three phases, i.e., 26 papers involved stakeholders during the ‘Ideate’ phase (55,32%), 22 during prototyping (46,80%) and 20 during the ‘Test & Evaluation’ phase (42,55%).

Figure 7. Techniques and methods applied in HCD processes together with the purpose they served in accordance with the HCID framework.

From the total number of 47 papers explored, 14 (29.79%) of them used concrete conceptual frameworks or models which were followed while designing the HCLA/HCAI solutions. Among these proposals we encounter the LATUX framework (Martinez-Maldonado et al. 2015), employed for co-designing and co-developing LA visualisations in Conijn, Van Waes, and van Zaanen (2020) and Holstein, McLaren, and Aleven (2019b); the Human-Centered Indicator Design (HCID) classification (Chatti et al. 2020) for the co-creation of LA indicators; and the Learning Analytics Translucence Elicitation Process (LAT-EP) (Martinez-Maldonado et al. 2022) to co-develop HCLA systems.

4.5. HCLA/HCAI evaluation and usage in authentic settings

Attending the evaluation mentioned in the retrieved papers, almost half of them (20, 42.55%) reported the effects of the implementation of the HCLA/HCAI proposals. For instance, Pishtari, Rodríguez-Triana, and Väljataga (2021) conducted an evaluative study to gather insights from stakeholders' perceptions about the developed prototypes -on analytics for LD in location-based settings. 5 educators, 2 researchers and 2 managers served as study informants. The results shed light on the stakeholders' different needs, e.g., the educators desired a location-based tool to monitor and evaluate the learning design during their actual practice. Santos and Boticario (2015a) co-designed guidelines to support personalised recommendations for online courses. They later evaluated their proposal within two experiences of different online courses. Their findings showed that the guidelines proposed could guide the design of personalised recommendations. At the same time, we encounter two cases (4.26%), i.e., Martinez-Maldonado et al. (2022), and Buckingham Shum, Ferguson, and Martinez-Maldonado (2019), where the authors mentioned that the evaluation of the proposals had been reported somewhere else.

In most cases, the HCAI/HCLA solutions were not applied in authentic settings (31, 65.96%) or this aspect was not clear in the paper (2, 4.36%). The rest of the papers report empirical studies framed in real educational settings (13, 27.66%). For example, Romero et al. (2021) developed a feedback tool for medical students following a user-centered design approach. They later evaluated the tool within an undergraduate course. According to the results, the students perceived positively the provided feedback and considered the tool environment as easy to use. Again, one study, i.e., Martinez-Maldonado et al. (2022), cited a prior work which described in detail the evaluation at the authentic setting (2.13%).

Regarding the evaluation techniques and methods employed, the users adopted an active role in all of the proposals. The type of study varied from interviews, controlled/lab experiments, to user tests and focus groups. Figure 8 presents the applied evaluation approaches, as reported in the research papers.

Figure 8. Evaluation methods and approaches as identified from the final pool of the included papers.

Out of 47 papers under review, 20 reported evaluation and 2 of them mentioned that the evaluation is reported elsewhere (namely, Buckingham Shum, Echeverria, and Martinez-Maldonado 2019; Martinez-Maldonado et al. 2022). Grouping the evaluation goals per topic, we found that the most frequent topic was assessing whether the solutions satisfied the user needs, e.g., looking at the effectiveness, usefulness, utility and requirement satisfaction (10, 50.00%). Authors also used the evaluation to gather ideas for future improvement (e.g., extracting user concerns, identifying core and missing aspects, eliciting further needs, or evaluating preferences) (8, 40.00%), and looked at how stakeholders used the solutions, assessing aspects such as acceptance, usability and user experience (8, 40.00%). Measuring the impact on stakeholder behaviour/engagement and evaluating understanding/interpretability appeared both with the same frequency (6, 30.00%). Other evaluation goals that gathered less attention were: the applicability, feasibility or actionability of the solutions (3, 15.00%); the impact on the learning gain (3, 15.00%); and the accuracy of the solutions (2, 10.00%).

4.6. Pros and cons of using HC approaches

In order to get a critical perspective about the benefits and challenges of using HC approaches, we extracted the pros and cons elicited in the reviewed papers, mainly looking at the discussion and conclusion sections. In total, 31 papers included this kind of reflection.

Among the positive implications, most papers highlighted: (1) the collection of meaningful and relevant idea, and the participants' capability to envision solutions, contributing to decisions which might not have been adapted to the stakeholders if only relying on the development team (14, 45.16% out of 31 papers); and, (2) the understanding gained of the stakeholders' values, needs, challenges, opportunities, requirements, existing practices, desired functionalities and preferences (13, 41.94%). The authors also perceived that the HC approaches had an impact, leading to more useful, effective, responsive and inclusive solutions (6, 19.36%); on the stakeholders engagement and satisfaction (5, 16.13%); and on the student performance (1, 3.23%). The reviewed papers also report that the involvement of the participants: (*) helped them to better understand the affordances and usage of the solutions (6, 19.35%), (*) contributed to their literacy and practice (2, 6.45%), and (*) promoted communication among different groups of stakeholders (1, 3.23%). Moreover, some authors pointed out that giving stakeholders power on the design process, raised their perception of appropriation and ownership (2, 6.45%). Regarding the benefits for the design team, the reviewed papers highlight that the acquired knowledge not only informed and triggered design decisions but also raised awareness of the impact of the design decisions (3, 9.68%).

However, the aforementioned benefits also came with some obstacles. Converging and coping with the diversity of needs, ideas and expectations about the designed product seems to be the most common problem (8, 25.81%). Sometimes, addressing all requests and implementing all proposals was not feasible and required management of expectations as well as mediating efforts. Additionally, in some cases, leaving aside some participants' ideas decreased their willingness to get implicated in later stages of the design. Also, there were several problems related to the stakeholders' difficulties in the design process such as expressing their needs, imagining their actual experience, lack of critical perspective while assessing their own designs, and lack of stakeholders' interest or motivation in the design tasks (7, 22.58 %). Furthermore, several authors highlighted that focussing on addressing the specific needs of the participants does not guarantee covering the core needs of the target audience, and the proposed solutions may not be transferable to other contexts (5, 16.13%): The common tension between contextualising and generalising. Last but not least, some authors noted that the implementation of HC approaches was a tedious and time-consuming effort (4, 12.90%).

5. Discussion

This section summarises the main lessons learnt from each research question, establishes connections with the existing literature and elaborates on the implications for the research community.

5.1. Main findings

The importance of theoretical grounding for creating sound and impactful LA and AI solutions has been often echoed in the scientific literature (Dawson et al. 2019). According to the findings of this study, similar concerns persist in the design process of the reviewed HCLA/HCAI solutions. In our analysis, collaborative learning and SRL have been found among the dominant theories informing the proposals discussed in the final pool of papers. This finding could be elucidated given the rise of digital and network technologies that indicated the benefits of social learning and the importance of learners' self-regulation (Jones 2015). Nevertheless, in 70% of the reviewed studies, there was no reference to learning or pedagogical theories. Our findings align with those of Topali et al. (2023) and Khalil, Prinsloo, and Slade (2022) about the limited grounding of LA research on learning theories. Although HCD approaches may lead to solutions that better cater to the needs of the target users, these solutions without a solid theoretical foundation may not afford to create the expected impact. Therefore, it is essential to establish a balanced approach during the design process that considers both users' perspectives and the theoretical principles to ensure effective and impactful solutions. At the same time, we observed that after 2018, more studies of our final pool reported a theoretical grounding on learning theories. Considering the limited data, this outcome merely serves as a hint regarding the increasing recognition of the significance of learning theories and learning design in underpinning LA/AI proposals.

Our study reported that teachers were the most relevant stakeholders involved in the co-design of the reviewed solutions, closely followed by the students, and other types of stakeholders, which aligns with the findings of the systematic literature review conducted by Sarmiento and Wise (2022). These findings are expected since teachers and students were the target user groups for most design products. However, despite teachers being the most frequently involved stakeholders, students can also play a crucial role in the co-design process, as they are the end-users of many HCLA/HCAI tools, and their input might contribute to creating more effective proposals. Interestingly, teachers and other stakeholders (e.g., researchers) often contribute to the design of tools for students but not the other way around. Students are rarely involved in the design of tools devoted to other stakeholders even if those LA/AI tools are usually about their learning process. Thus, future work could consider engaging students, for instance, to verify that the LA/AIED tools reflect and contribute to improving their learning experience. Additionally, it is noteworthy that a diverse group of stakeholders were involved including IT experts, school managers, and developers, which suggests that HC approaches require efforts from individuals beyond the target users. Nevertheless, considering this distinct group of stakeholders identified, there is the inherent risk of accommodating an excessive number of diverse and potentially conflicting needs (Steen 2012). This risk arises from the complexity and diversity within educational settings, where students, educators, administrators, parents, and other stakeholders may have distinct and sometimes competing interests, preferences, and priorities. Accordingly, satisfying a multitude of diverse desires can pose several challenges and limitations within the context of educational design research, such as the design of broad and unfocused solutions or less feasible ideas than those generated by designers alone, as also reported by Potvin et al. (2017) or Lang and Davis (2023) among others. To mitigate this issue, it seems critical for researchers and designers to be coherent in the design process by defining clear design goals, considering the educational objectives and/or by conducting a needs assessment to identify and prioritise the most pressing requirements and challenges faced by the target user group.

Differently from the work of Sarmiento and Wise (2022), we found out that most papers focussed on the Higher Education context rather than K12 or preschool education. This result might be argued, due to the rather less focus on LA/AI in K12 education compared to Higher Education (Kovanovic, Mazziotti, and Lodge 2021) and the relatively recent attention to HCD in educational design. Therefore, including young learners in HCD stands as a promising opportunity for the community.

Interestingly, most of the HCD approaches concerned stakeholders' participation during the goal definition and the problem understanding (93,62%), while fewer papers involved actively the human agents during the ideation, prototyping or testing of the proposals. Prinsloo and Slade (2016) highlighted the potential risk to teachers' and students' agency when their needs and desires are excluded from the design and development of analytical solutions. To overcome such a challenge, Martinez-Maldonado (2023) stressed the importance of equally positioning the human agents not only as brainstorming informants but during the whole process of HCD as equal co-designers of the targeted solutions. Building on our results, while stakeholders can serve as informants for brainstorming their problems and needs before the tool design, their experience could also inform about what aspects should be monitored, which meaningful data should be collected, the critical course design aspects affecting the learning process, etc.

We also noted several examples of tools (e.g., Sadallah et al. 2022) and frameworks specifically developed to systematize HCD processes such as LATUX (Martinez-Maldonado et al. 2015), HCID (Chatti et al. 2020) and LAT-EP (Martinez-Maldonado et al. 2022). However, while these efforts are relevant and meaningful as they provide other researchers with valuable guidelines to follow, less than a third of the reviewed papers embraced an HCD framework in the design and development processes. That is, many researchers relied on the research goals to intuitively determine how to involve the stakeholders in the tasks, phases, and procedures. While these efforts are worthy, using established HCD frameworks can promote a more comprehensive and effective design process. Furthermore, a first glimpse of the papers suggests that the usage of the HCD terms included in the query (‘co-design’, ‘human-centered design’ and ‘participatory design’) may not have been consistent across authors. Thus, future work should explore the underlying meaning of these terms, looking at the definitions provided, as well as analysing the connections between the terms and the specific HC methods, extending the work by Lang and Davis on what LA authors mean when referring to human-centeredness (Lang and Davis 2023).

Despite the important role that evaluation plays in the acceptance for publication of research papers, the majority of the reviewed papers do not report any evaluation of their LA/AEID proposals nor their usage in authentic studies. This signs that the proposals (and the field) may be still in an early stage. These results are in line with other LA (Larrabee Sønderlund, Hughes, and Smith 2019; Schwendimann et al. 2017) and AI reviews (Ouyang, Zheng, and Jiao 2022). To support this process, initiatives such as the human-centered evaluation framework for explainable AI proposed by Donoso-Guzmán et al. (2023) could be of great support. Also, it is noteworthy that, despite the emphasis of HC approaches on creating solutions aligned with the user needs and context, only 10 papers evaluated whether these goals were satisfied. Furthermore, while researchers often claim using HC approaches to promote adoption, only 3 papers assessed the impact on the stakeholders' practice, and only 1 paper looked at the adoption in the longer term (Ahn et al. 2021). Thus, to better understand the added value of using HC approaches, it would be necessary to assess not only to what extent the proposed solutions have satisfied the user needs but also the impact of HCD in the adoption of LA/AEID solutions.

Also related to the maturity of the HCLA/HCAI solutions, in a preliminary analysis of the development stage of the solutions, we have observed that most of them reached the stage of mockups, followed by experimental prototypes and few were fully working tools when the papers were written. However, this is an aspect requiring further investigation since often this information is not clear in the publication.

As raised by Martinez-Maldonado (2023), the implementation of HCD methods in LA/AIED systems has its own challenges. Also, the empirical evidence from other fields has reported on the downsides of HC methods vs methods without stakeholder involvement (Lang and Davis 2023), e.g., ending up with more but less feasible ideas than those generated by designers alone (Potvin et al. 2017). The reviewed papers have elicited several pros and cons of using HC approaches which vary from study to study. The same can be observed in the existing literature: e.g., while some studies report cost reduction throughout the development life cycle and easier-to-use systems (Bevan, Bogomolni, and Ryan 2001; Karat 1997) other studies did not identify those improvements (Hirasawa et al. 2010). Those differences lead us to highlight that there is no universal HCD recipe; thus, how and when stakeholders should be involved should be carefully adapted to each study to get the best out of these methods.

5.2. Theoretical and practical implications

Building on the findings discussed above, it is worth mentioning that the practical and theoretical implications of this study can be beneficial primarily for LA/AIED researchers. However, these implications may also be useful for tool providers, tool designers and educational stakeholders (e.g., practitioners or decision makers). The emerged implications extend into the following three dimensions:

• The importance of pedagogically grounding the HCLA/HCAI proposals.

• The need to support the stakeholders' participation within the HCD processes.

• The necessity for a thorough empirical assessment of the effects of the HCLA/HCAI proposals in authentic settings.

First, the majority of the examined proposals do not explicitly take into account pedagogical aspects, and only a few studies considered the learning objectives, the task type, the learning subject matter, and the context when designing and developing the HCLA/HCAI proposals. Prior studies stressed the importance of the alignment of LA/AI solutions with theoretical foundations and the learning design to foster their effectiveness in educational practice (Gašević, Kovanović, and Joksimović 2017; Ouyang, Zheng, and Jiao 2022; Rodríguez-Triana et al. 2015). Omitting such theoretical aspects in LA/AI applications could potentially result in technological determinism. Learning theories elucidate the learning process and explain the mechanisms behind how and why individuals learn. Thus, the pedagogical grounding clarifies what should be measured, why and at what moment by defining e.g. where to apply LA or AI and which data should be collected within the learning and teaching process (Banihashem et al. 2019). Additionally, the particularities of the learning design (e.g., the difficulty of the activities or the course structure) affect the learning and teaching process. Thus, their consideration is deemed crucial when designing technological solutions for educational purposes. du Boulay (2000) highlighted the importance of the pedagogical underpinning when applying AI in education to address learners' specific needs to determine what to offer and when, and to support teachers' and students' own agency. At the same time, having a pedagogical input in LA and AIED fosters a better framing of the students' and teachers' captured behaviours (Rodríguez-Triana et al. 2015; Wong et al. 2019).

The second aspect regards the stakeholders' support when participating in HCD processes. The stakeholders' involvement in the design of analytics and AI tools has been considered sometimes unproductive, given participants' relatively limited data and LA/AI literacy (Martinez-Maldonado 2023). Existing literature has reported, for example, that teachers often require additional assistance in reflecting on learners' data, selecting and fine-tuning the thresholds (Fernandez Nieto et al. 2022; Rienties et al. 2018) and connecting such data to their course learning design (Mangaroska and Giannakos 2019). Our findings indicated the use of interviews for brainstorming as the most employed HCD technique, and the initial phase of the goal definition as the most common moment of the stakeholders' co-presence. However, as Martinez-Maldonado (2023) highlighted, stakeholders' lived experiences, their design visions and practical knowledge should be supported and become an equal part of the whole HCD process and of decision-making. To support that inclusion, further research is necessary to better understand how and when stakeholders should be involved and when not to, as raised by Lang and Davis (2023).

Third, the evolution of HCLA/ HCAI requires further empirical research to assess the real-world impact of proposed solutions (e.g., in terms of satisfaction of user needs and user adoption), ensure ethical considerations and trust, inform decision-making, and contribute to the ongoing innovation in the field of educational technology. Recent literature highlighted a gap between the potential of LA/AI and its actual implementation in real cases (Ouyang, Zheng, and Jiao 2022). Conducting empirical studies in authentic settings will allow for the assessment of the real-world impact of HCLA/HCAI interventions on student learning outcomes and teaching practices. Additionally, it will permit the reconsideration of human-centered approaches followed to satisfy the needs of the involved stakeholders. Additionally, empirical studies will foster further data-driven evidence, allowing researchers, practitioners, and even policymakers to make more informed decisions about the effectiveness and feasibility of research solutions. Indeed, many proposals fail to be applied in authentic settings since they do not respond to the complexities of the educational system.

5.3. Limitations

This systematic literature review does not come without limitations, and caution has to be taken when interpreting our findings. The query, the selection of databases and the inclusion/exclusion criteria may have left out relevant papers. Terminology-wise, apart from ‘co-design’, ‘participatory design’, and ‘human-centered design’, researchers may have used other terms such as ‘co-create’ or ‘co-creation’, while referring to the user involvement in their proposals. For instance, not including both UK and US variants of ‘human-centered’ and ‘human-centered’ may lead to overlooking papers. Regarding the domain, our query targeted papers explicitly referring to ‘learning analytics’ or ‘artificial intelligence’ in ‘education’. However, there may have been related works in the areas of educational data mining, intelligent tutoring systems, adaptive systems, smart learning environments, etc. Furthermore, the inclusion of the keywords variations (e.g., codesign and co-design, and artificial intelligence or AI) as well as not restricting the query to title, abstract and keywords could have potentially returned additional papers. Dealing with the venues, we may have overlooked certain conferences, journals and databases that are not directly related to LA and AI but which eventually publish studies on these topics.

Regarding the coding process, despite the involvement of several researchers, and the codification training with one of the articles, the remaining articles were coded by one single person. In order to minimise this limitation, we cross-checked dubious cases among the whole team.

Finally, we want to highlight the unexpectedly small number of HCAI papers under review in comparison with the HCLA ones. Interestingly, most of the detected ones are only conceptual or theoretical and do not satisfy the inclusion criteria of applying HC approaches in practice. In any case, we hypothesise that AIED authors may have applied HC approaches without mentioning it in the paper title, abstract or keywords. Also, we have noticed that in order to comply with the length limitations of the papers, LA and AIED authors may have opted for omitting the details of their HC approaches. Thus, the reviewed papers may just represent the top of the iceberg in terms of HCLA and HCAI research solutions, which may not reflect the reality of the LA/AIED industry.

6. Conclusions

This study has reviewed reported practices in HCLA and HCAI research, analysing their pedagogical contextualisation, the purpose of the contributions, the stakeholder involvement in the design and evaluation, as well as the methods and tools used to support HC design. The results confirmed that, beyond the existence of conceptual and theoretical papers promoting HC approaches in LA and AIED, the community is actively involving stakeholders in the design and, to some extent, in the evaluation of their contributions. Among the reviewed papers, HC approaches were usually intended for the identification of requirements/needs, especially for the design of LA dashboards that might be integrated within web-based tools and learning management systems. Teachers and students were the main target users, with the purpose of supporting their awareness and decision-making (e.g., Aleven et al. 2016; Verbert et al. 2020). Interestingly, the stakeholder involvement went beyond the final users and sometimes included project managers, developers, as well as teachers (e.g., when the tools were devoted to students).

While multiple stakeholder-involvement techniques were used in the reviewed papers (mainly interviews, focus groups and workshops), few of them adopted existing HC design guidelines (e.g., LATUX). Thus, the research community could also benefit from compiling conceptual HCLA/HCAI contributions, guidelines -such as the principles of human-centered design proposed by the Interaction Design Foundation-, and existing standards (for instance, the ISO 9241-210:2019 standard on human-centered design for interactive systems) which may inform future studies and potentially increase the effectiveness of the stakeholder involvement. Also, as pointed out in previous reviews, researchers should bear in mind the importance of the pedagogical contextualisation of their HCLA and HCAI proposals, as well as the need for evaluation and usage in authentic settings. After this first attempt to understand the current landscape of HCLA/HCAI research, future work should further analyse aspects such as the accuracy of the usage of the HC terms (e.g., extending Lang and Davis (2023) with other terms than ‘human-centered’); the stakeholders' involvement in the implementation of the solutions (Rodríguez-Triana et al. 2018); and the level of automation of the contributions (Molenaar 2022). Furthermore, to release the potential of human-centered approaches, it is paramount to take into account already identified challenges of human-centered design (e.g., ensuring representative participation, considering the stakeholders' expertise and lived experiences in LA/AIED design) while applying them in HCLA/HCAI solutions (Martinez-Maldonado 2023), as well as the particularities of each study context. Last but not least, further studies would be necessary to better understand the application of human-centered approaches by the ed-tech industry, similar to what has been done for the manufacturing industry (Brückner et al. 2023; Nguyen Ngoc, Lasa, and Iriarte 2022) or the e-health context (van Velsen, Ludden, and Grünloh 2022).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research is partially funded by MCIU/AEI/10.13039/501100011033 and FSE+ under grants PID2020-112584RB-C32 and RYC2022-037806-I, and by the BAGEP Award of the Science Academy (Türkiye).

Appendix. Paper codification

Table

Title	Contribution (type)	Contribution (place)	Contribution (rank)	Year	LA or AI	Purpose (what-verb)	Purpose (what-object)	Purpose (who)	Purpose (why)
Schmitz et al. (2018). ‘Make It Personal!’ -- Gathering Input from Stakeholders for a Learning Analytics-Supported Learning Design Tool	conference	EC-TEL	–	2018	LA	develop	LA tool	HE teachers & students	personalise feedback
Pishtari, Rodríguez-Triana, and Väljataga (2021). A Multi-Stakeholder Perspective of Analytics for Learning Design in Location-Based Learning	journal	IJMBL	–	2021	LA	develop	authoring tool	practitioners	support the design of activities
Knight and Anderson (2016). Action-oriented, accountable, and inter(active) learning analytics for learners	workshop	LAL	–	2016	LA	develop	learning platform	students	shape online community with professionals
Oliver-Quelennec et al. (2022). Adapting Learning Analytics Dashboards by and for University Students	conference	EC-TEL	GSS 3 / CORE B	2022	LA	identify & design	LA indicators & LA dashboard	students	cover student expectations
Echeverria et al. (2022). An Exploratory Evaluation of a Collaboration Feedback Report	conference	LAK	GGS A- / CORE A	2022	LA	identify	LA indicators	students	support collaboration
Zhou, Sheng, and Howley (2020). Assessing post-hoc explainability of the BKT algorithm	conference	AIES	–	2020	AI	design & implement	interactive explanation	students	increase learning
Person et al. (2021). Co-design of a Learning Analytics Tool by Computer Scientists and Teachers: The Difficult Emergence of a Common World	conference	IEA	–	2021	LA	design	LA dashboard	high school teachers	NA
Holstein, McLaren, and Aleven (2019b). Co-Designing a Real-Time Classroom Orchestration Tool to Support Teacher-AI Complementarity	journal	JLA	–	2019	AI	design	LA tool (wearable)	K-12 teachers	support teacher orchestration
Ahn et al. (2021). Co-Designing for Privacy, Transparency, and Trust in K-12 Learning Analytics	conference	LAK	GGS A- / CORE A	2021	LA	design	LA dashboard	teachers	support teacher facilitation
Chalvatza, Karkalas, and Mavrikis (2019). Communicating learning analytics: Stakeholder participation and early stage requirement analysis	conference	CSEDU	–	2019	LA	design	LA tool	teachers, parents & managers	facilitate actionable decision-making
Hou, Nagashima, and Aleven (2022). Design a Dashboard for Secondary School Learners to Support Mastery Learning in a Gamified Learning Environment	conference	EC-TEL	GSS 3 / CORE B	2022	LA	design & refine	LA dashboard	HE students	promote mastery learning
Romero et al. (2021). Design and usability testing of an in-house developed performance feedback tool for medical students	journal	BMC Med. Educ.	JCR SSCI-Q2 JCR SCIE-Q2	2021	LA	design & develop	interface	HE students	track students' progress and skills for SRL

Table

Title	Learning theories / Pedagogical approaches	Considered aspects of the LD	Which stakeholders are involved	How stakeholders are involved	Applied target platform	Co-design tools	Terminology used	Evaluation (yes/no)	Applied in real scenario (yes/no)
Schmitz et al. (2018). ‘Make It Personal!’ -- Gathering Input from Stakeholders for a Learning Analytics-Supported Learning Design Tool	self-regulated learning	NA	teachers, students, TEL experts	focus groups, surveys	LA4LD tool (web app)	surveys	UCD	no	no
Pishtari, Rodríguez-Triana, and Väljataga (2021). A Multi-Stakeholder Perspective of Analytics for Learning Design in Location-Based Learning	NA	learning task type	teachers, educational instructors	a) contextual inquiry; b) interviews; c) design workshop	web-based mobile applications	storyboards, interviews	CD	yes	no
Knight and Anderson (2016). Action-oriented, accountable, and inter(active) learning analytics for learners	NA	NA	students	surveys, interviews	online learning environment	surveys, interviews	CD, PD	no	no
Oliver-Quelennec et al. (2022). Adapting Learning Analytics Dashboards by and for University Students	NA	learning context (study duration level), moment in the semester	students	co-design sessions	NA	paper-based prototypes	CD	no	no
Echeverria et al. (2022). An Exploratory Evaluation of a Collaboration Feedback Report	NA	NA	students	semi-structured interviews	NA	interviews	HCD, UCD	yes	no
Zhou, Sheng, and Howley (2020). Assessing post-hoc explainability of the BKT algorithm	NA	unclear	teachers	co-design sessions (brainstorming, usability testing, semi-structured interviews)	NA	paper-based prototypes, affinity diagrams, interviews	UCD	yes	no
Person et al. (2021). Co-design of a Learning Analytics Tool by Computer Scientists and Teachers: The Difficult Emergence of a Common World	NA	NA	teachers	interviews, workshop (brainstorming, game-based design)	NA	game-based design tool, interviews, cards, paper-based prototypes	CD, PD	no	no
Holstein, McLaren, and Aleven (2019b). Co-Designing a Real-Time Classroom Orchestration Tool to Support Teacher-AI Complementarity	adaptive learning	NA	teachers	(a) card sorting, interviews, field observations; (b) speed dating sessions; (c) sketching; (d) field observations, behavioural mapping, interviews	Lumino (AR app)	LATUX, paper-based prototypes, digital prototypes, cards, storyboards, mockups, speed-dating field observations, interviews	CD	yes	yes
Ahn et al. (2021). Co-Designing for Privacy, Transparency, and Trust in K-12 Learning Analytics	NA	NA	teachers, instructional coaches, district leaders	co-design sessions, semi-structured interviews	NA	interviews	CD	no	no
Chalvatza, Karkalas, and Mavrikis (2019). Communicating learning analytics: Stakeholder participation and early stage requirement analysis	NA	NA	teachers, TA, school manager, school committee member	workshop (focus groups)	NA	scenarios, focus groups, workshops	UCD	no	no
Hou, Nagashima, and Aleven (2022). Design a Dashboard for Secondary School Learners to Support Mastery Learning in a Gamified Learning Environment	NA	NA	students	(a) co-design sessions; (b) user testing	Gwynette (an ITS for algebra learning)	paper-based prototypes, interviews	CD	yes	no
Romero et al. (2021). Design and usability testing of an in-house developed performance feedback tool for medical students	SRL	NA	technical developers, project managers (researchers), medical students	(a) questionnaires, paper prototypes; (b) usability testing, (c) SUS	LevelUp (web app)	surveys	UCD	yes	unclear

Table

Title	Contribution (type)	Contribution (place)	Contribution (rank)	Year	LA or AI	Purpose (what-verb)	Purpose (what-object)	Purpose (who)	Purpose (why)
Beheshti et al. (2020). Design Considerations for Data-Driven Dashboards: Supporting Facilitation Tasks for Open-Ended Learning	conference	CHI	GSS 1 / CORE A*	2020	LA	design	LA dashboard (mobile)	museum visitors	provide formative feedback
Holstein, McLaren, and Aleven (2019a). Designing for Complementarity: Teacher and Student Needs for Orchestration Support in AI-Enhanced Classrooms	conference	AIED	GSS 2 / CORE A	2019	AI	design and use	AI-based system	K-12 teachers	support orchestration
Revano and Garcia (2021). Designing Human-Centered Learning Analytics Dashboard for Higher Education Using a Participatory Design Approach	conference	HNICEM	–	2021	LA	create	LA dashboard	HE students	provide progress awareness
Ahn et al. (2019). Designing in context: Reaching beyond usability in learning analytics dashboard design	journal	JLA	–	2019	LA	design & develop	LA dashboard	middle school teachers	support teacher pedagogical practices
Sadallah et al. (2022). Designing LADs That Promote Sensemaking: A Participatory Tool	conference	EC-TEL	GSS 3 / CORE B	2022	LA	design	LA dashboard	secondary school teachers	NA
Martinez-Maldonado et al. (2022). Designing translucent learning analytics with teachers: an elicitation process	journal	Interact. Learn. Environ.	JCR SSCI-Q1	2022	LA	design	LA dashboard	students	provide automated feedback
Knight et al. (2015). Developing a learning analytics dashboard for undergraduate engineering using participatory design	conference	ASEE	–	2015	LA	design & develop	LA dashboard	high school students	inform students' use
Aleven et al. (2016). Developing a teacher dashboard for use with intelligent tutoring systems	workshop	IWTA	–	2016	LA	create	LA dashboard	teachers	inform teachers' decisions
Hoffmann et al. (2022). Development of Actionable Insights for Regulating Students' Collaborative Writing of Scientific Texts	conference	EC-TEL	GSS 3 / CORE B	2022	LA	define & design	LA indicators & LA dashboard	teachers	monitor collaborative learning
Nazaretsky et al. (2022). Empowering Teachers with AI: Co-Designing a Learning Analytics Tool for Personalized Instruction in the Science Classroom	conference	LAK	GGS A- / CORE A	2022	LA	design	LA tool	teachers	build pedagogically meaninful explanations
Treasure-Jones, Dent-Spargo, and Dharmaratne (2018). How do students want their workplace-based feedback visualised in order to support self-regulated learning? Initial results & reflections from a co-design study in medical education	conference	EC-TEL (practitioners' track)	–	2018	LA	refine	learning platform	students	support self-regulated learning
Chatti et al. (2020). How to design effective learning analytics indicators? a human-centered design approach	conference	EC-TEL	–	2020	LA	identify	LA indicators	HE students	support students in course selection

Table

Title	Learning theories / Pedagogical approaches	Considered aspects of the LD	Which stakeholders are involved	How stakeholders are involved	Applied target platform	Co-design tools	Terminology used	Evaluation (yes/no)	Applied in real scenario (yes/no)
Beheshti et al. (2020). Design Considerations for Data-Driven Dashboards: Supporting Facilitation Tasks for Open-Ended Learning	open-ended learning	task type	facilitators (final users)	(a) observations, focus group; (b) co-design sessions	mobile app	field notes	PD	no	no
Holstein, McLaren, and Aleven (2019a). Designing for Complementarity: Teacher and Student Needs for Orchestration Support in AI-Enhanced Classrooms	NA	NA	K-12 teachers & students	Participatory speed dating approach	NA	low-fidelity prototypes, storyboards	PD	no	no
Revano and Garcia (2021). Designing Human-Centered Learning Analytics Dashboard for Higher Education Using a Participatory Design Approach	multimedia learning theory	NA	teachers, students	focus group	learning management system	low-fidelity prototypes, field notes	HCD, PD	no	no
Ahn et al. (2019). Designing in context: Reaching beyond usability in learning analytics dashboard design	collaborative learning	NA	researchers, designers, developers	(a) interviews; (b) brainstorming sessions; (c) co-des- ign sessions; (d) interviews	Edsight (web app)	interviews, low-fidelity prototypes, sticky notes, affinity diagrams, journey maps	HCD, PD	no	no
Sadallah et al. (2022). Designing LADs That Promote Sensemaking: A Participatory Tool	NA	unclear	teachers, administrator, instructional designer, researchers	workshop	NA	cards, paper-based prototypes	CD, HCD, PD	no	no
Martinez-Maldonado et al. (2022). Designing translucent learning analytics with teachers: an elicitation process	collocated collaborative learning	classroom dynamics, learning constructs, teacher questions, expectations and hypotheses	teachers, subject coordinators	card-sorting, focus groups, low-fidelity prototyping, co-design sessions, semi-structured interviews	NA	focus groups, cards, low-fidelity prototypes, interviews	CD	no (reported elsewhere)	no (reported elsewhere)
Knight et al. (2015). Developing a learning analytics dashboard for undergraduate engineering using participatory design	NA	NA	students	co-design session	NA	low-fidelity prototypes, field notes, focus groups	HCD, PD	no	no
Aleven et al. (2016). Developing a teacher dashboard for use with intelligent tutoring systems	blended learning	NA	teachers	contextual inquiry, speed dating, usability testing	CTAT and Lynnette (ITS infrastructure)	storyboards, mockups	UCD	no	yes
Hoffmann et al. (2022). Development of Actionable Insights for Regulating Students' Collaborative Writing of Scientific Texts	CSCL	NA	teachers	focus groups, interviews	LabNbook (webapp)	focus groups, interviews, mockups	HCD, UCD	no	no
Nazaretsky et al. (2022). Empowering Teachers with AI: Co-Designing a Learning Analytics Tool for Personalized Instruction in the Science Classroom	NA	NA	K-12 teachers	interviews	PeTeL (LMS)	interviews	PD	yes	yes
Treasure-Jones, Dent-Spargo, and Dharmaratne (2018). How do students want their workplace-based feedback visualised in order to support self-regulated learning? Initial results & reflections from a co-design study in medical education	SRL, workplace-based learning	NA	teachers, students	co-design workshops	myPAL (LMS)	workshops, interviews, paper-based prototypes, digital prototypes	CD	no	yes
Chatti et al. (2020). How to design effective learning analytics indicators? a human-centered design approach	NA	course characteristics	teachers, students	co-design sessions	Course Insights (web app)	interviews, paper-based prototypes, surveys	CD	yes	unclear

Table

Title	Contribution (type)	Contribution (place)	Contribution (rank)	Year	LA or AI	Purpose (what-verb)	Purpose (what-object)	Purpose (who)	Purpose (why)
Conijn, Van Waes, and van Zaanen (2020). Human-centered design of a dashboard on students' revisions during writing	conference	EC-TEL	–	2020	LA	design	LA dashboard (actionable)	teachers	display students' information
Cerro Martinez, Guitert Catasus, and Romeu Fontanillas (2020). Impact of using learning analytics in asynchronous online discussions in higher education	journal	ETHE	JCR SSCI-Q1	2020	LA	define, design & implement	LA indicators & LA tool	teachers	support asynchronous online discussion
Vezzoli, Mavrikis, and Vasalou (2020). Inspiration Cards Workshops with Primary Teachers in the Early Co-Design Stages of Learning Analytics	conference	LAK	–	2020	LA	unclear	LA tool	primary school students	unclear
Michos et al. (2020). Involving teachers in learning analytics design: Lessons learned from two case studies	conference	LAK	–	2020	LA	create	LA tool	teachers	guide in the design and running of courses
Alvarez, Martinez-Maldonado, and Shum (2020). LA-DECK: A Card-Based Learning Analytics Co-Design Tool	conference	LAK	–	2020	LA	unclear	unclear	unclear	unclear
Verbert et al. (2020). Learning analytics dashboards: The past, the present and the future	conference	LAK	–	2022	LA	create	LA dashboard (actionable)	teachers	support decision-making
Kilińska, Kobbelgaard, and Ryberg (2019). Learning analytics features for improving collaborative writing practices: insights into the students' perspective	conference	ICEM	–	2019	LA	design	LA tool	high school students	gain awareness on collaborative writing
Brun et al. (2019). Learning analytics made in France: the METAL project	journal	IJILT	–	2019	LA	design	LA dashboard	secondary school teachers & students	have better awareness
Long, Aman, and Aleven (2015). Motivational design in an intelligent tutoring system that helps students make good task selection decisions	conference	AIED	CORE A	2015	AI	identify	LA indicators	ITS	improve problem selection
Valkanova et al. (2016). Opening the black box of practice-based learning: Human-centered design of learning analytics	workshop	crossLAK	–	2016	LA	design	LA tool	teachers & students	support awareness during practice-based learning activities
Coughlan, Lister, and Freear (2019). Our Journey: Designing and utilising a tool to support students to represent their study journeys	conference	INTED	–	2019	LA	design	LA tool	students	represent own study journey
Santos and Boticario (2015a). Practical guidelines for designing and evaluating educationally oriented recommendations	journal	C&E	JCR SSCI-Q1 / JCR SCIE-Q1	2015	AI	compile	guidelines	unclear	produce personalised recommendations

Table

Title	Learning theories / Pedagogical approaches	Considered aspects of the LD	Which stakeholders are involved	How stakeholders are involved	Applied target platform	Co-design tools	Terminology used	Evaluation (yes/no)	Applied in real scenario (yes/no)
Conijn, Van Waes, and van Zaanen (2020). Human-centered design of a dashboard on students' revisions during writing	NA	NA	teachers, students	round-table sessions	NA	LATUX, discussion groups, prototypes, sticky notes	HCD	yes	no
Cerro Martinez, Guitert Catasus, and Romeu Fontanillas (2020). Impact of using learning analytics in asynchronous online discussions in higher education	collaborative learning	teachers' perceptions	teachers	co-design sessions, surveys	LA tool for the UOC campus (LMS)	discussion groups, surveys, prototypes	PD	yes	yes
Vezzoli, Mavrikis, and Vasalou (2020). Inspiration Cards Workshops with Primary Teachers in the Early Co-Design Stages of Learning Analytics	NA	teacher pedagogical ideas, questions and expectations	teachers	workshops	NA	workshops, cards	CD	no	no
Michos et al. (2020). Involving teachers in learning analytics design: Lessons learned from two case studies	collaborative learning	LD decisions	teachers	interviews, focus groups, questionnaires	NA	interviews, focus groups, surveys	CD	no	no
Alvarez, Martinez-Maldonado, and Shum (2020). LA-DECK: A Card-Based Learning Analytics Co-Design Tool	NA	learning objective and context	students, educators, researchers, developers	semi-structured interviews	NA	interviews, cards	CD	no	no
Verbert et al. (2020). Learning analytics dashboards: The past, the present and the future	NA	NA	researchers	workshops	NA	workshops, cards, sticky notes	PD	no	no
Kilińska, Kobbelgaard, and Ryberg (2019). Learning analytics features for improving collaborative writing practices: insights into the students' perspective	problem-based learning, project-based learning, collaborative learning	NA	students	co-design sessions, workshops	Google Docs (web app)	workshops, paper-based prototypes, cards	CD, PD	yes	no
Brun et al. (2019). Learning analytics made in France: the METAL project	NA	NA	teachers, students	teacher LAD: co-design sessions, students LAD: co-design sessions, interviews	LMS and ‘behaviour sensors’	focus groups, paper-based prototypes	CD	no	no
Long, Aman, and Aleven (2015). Motivational design in an intelligent tutoring system that helps students make good task selection decisions	coself-regulated learning, motivational theories	NA	K-12 students	interviews	ITS	interviews	UCD	no	yes
Valkanova et al. (2016). Opening the black box of practice-based learning: Human-centred design of learning analytics	practice-based learning	NA	K-12 teachers	interviews	NA	interviews	HCD	no	no
Coughlan, Lister, and Freear (2019). Our Journey: Designing and utilising a tool to support students to represent their study journeys	NA	NA	students, student union representatives, external staff	workshops	Our Journey (web app)	surveys	PD	no	no
Santos and Boticario (2015a). Practical guidelines for designing and evaluating educationally oriented recommendations	NA	NA	ICT manager, teachers, students	interviews, focus groups (brainstorming sessions)	NA	interviews, focus groups	UCD	yes	yes

Table

Title	Contribution (type)	Contribution (place)	Contribution (rank)	Year	LA or AI	Purpose (what-verb)	Purpose (what-object)	Purpose (who)	Purpose (why)
Eradze, Rodriguez Triana, and Laanpere (2017). Semantically Annotated Lesson Observation Data in Learning Analytics Datasets: a Reference Model	journal	ID&A	–	2017	LA	design	LA model	teachers	support context-aware semantical annotations
Huh et al. (2022). Service Design of Artificial Intelligence Voice Agents as a Guideline for Assisting Independent Toilet Training of Preschool Children	journal	ADR	–	2022	AI	create	AI agent	pre-school students	support toilet training
de Quincey et al. (2019). Student Centred Design of a Learning Analytics System	conference	LAK	–	2019	LA	design & develop	LA dashboard	students	personalise feedback and real-time support
Lister et al. (2021). Taylor, the Disability Disclosure Virtual Assistant: A Case Study of Participatory Research with Disabled Students	journal	Educ. Sci.	–	2021	AI	design	virtual assistant	students (disabled)	solve administrative issues
Alzoubi et al. (2021). TeachActive Feedback Dashboard: Using Automated Classroom Analytics to Visualize Pedagogical Strategies at a Glance	conference	CHI	–	2021	LA	design	LA dashboard	high school teachers	improve pedagogical practices
Buckingham Shum, Ferguson, and Martinez-Maldonado (2019). The Multimodal Matrix as a Quantitative Ethnography Methodology	conference	ICQE	–	2019	LA	design	LA tool	teachers & students	support awareness on collaborative learning
Pozdniakov et al. (2022). The Question-driven Dashboard: How Can We Design Analytics Interfaces Aligned to Teachers' Inquiry?	conference	LAK	GGS A- / CORE A	2022	LA	design & develop	LA tool	teachers	support awareness of students' collaboration
Ouatiq et al. (2022). Towards the Co-Design of a Teachers' Dashboards in a Hybrid Learning Environment	conference	IRASET	–	2022	LA	design	LA dashboard	HE teachers	assist students' tracking & predict their results
Dabbebi, Gilliot, and Iksal (2019). User centred approach for learning analytics dashboard generation	conference	CSEDU	–	2019	LA	design	LA dashboard	teachers	monitor students' performance & support teachers' decision-making
Santos and Boticario (2015b). User-centred design and educational data mining support during the recommendations elicitation process in social online learning environments	journal	Expert Syst.	JCR SCIE-Q3	2015	DM	develop	recommender system	students	produce accurate social-oriented recommendations
Eradze et al. (2020). Contextualising Learning Analytics with Classroom Observations: a Case Study	journal	ID&A	–	2020	LA	design & develop	LA dashboard	managers	better understand the adoption of digital learning resources

Table

Title	Learning theories / Pedagogical approaches	Considered aspects of the LD	Which stakeholders are involved	How stakeholders are involved	Applied target platform	Co-design tools	Terminology used	Evaluation (yes/no)	Applied in real scenario (yes/no)
Eradze, Rodriguez Triana, and Laanpere (2017). Semantically Annotated Lesson Observation Data in Learning Analytics Datasets: a Reference Model	NA	resources, activities, possible actions	teachers	participatory design session (semi-structured interviews, guided focus groups)	NA	focus groups	PD	no	no
Huh et al. (2022). Service Design of Artificial Intelligence Voice Agents as a Guideline for Assisting Independent Toilet Training of Preschool Children	Vygotsky's sociocultural theory	NA	pre-school teachers, parents and children	(a) interviews; (b) prototyping; (c) experiment	AI agent for toilet training	interviews, prototypes	CD	yes	no
de Quincey et al. (2019). Student Centred Design of a Learning Analytics System	NA	NA	students	focus groups, interviews	dashboard for LMS	focus groups, interviews	UCD	yes	yes
Lister et al. (2021). Taylor, the Disability Disclosure Virtual Assistant: A Case Study of Participatory Research with Disabled Students	NA	NA	students	(a) co-design sessions; (b) survey; (c) co-design workshops	virtual assistant for university website	interviews, discussion forums, surveys, paper-based prototypes	PD	yes	no
Alzoubi et al. (2021). TeachActive Feedback Dashboard: Using Automated Classroom Analytics to Visualize Pedagogical Strategies at a Glance	active learning	NA	teachers	interview, co-design meetings	NA	interviews, surveys, prototypes	UCD	no	no
Buckingham Shum, Ferguson, and Martinez-Maldonado (2019). The Multimodal Matrix as a Quantitative Ethnography Methodology	collaborative learning	NA	teachers, students	co-design sessions	nursing team simulation	unclear	CD	no (reported elsewhere)	yes
Pozdniakov et al. (2022). The Question-driven Dashboard: How Can We Design Analytics Interfaces Aligned to Teachers' Inquiry?	collaborative learning	NA	teacher assistants, students	Interviews with teachers, inquiry, interface prototyping, interface validation	Zoom and Google Docs (web apps)	interviews, prototypes	HCD	yes	yes
Ouatiq et al. (2022). Towards the Co-Design of a Teachers' Dashboards in a Hybrid Learning Environment	NA	NA	teachers	surveys, participatory workshops, feedback interviews	hybrid learning environment	surveys, interviews	CD, PD	yes	no
Dabbebi, Gilliot, and Iksal (2019). User centred approach for learning analytics dashboard generation	NA	NA	teachers	survey, semi-structured interviews and co-design workshops	Tactileo Map (mobile app)	surveys, interviews, focus groups, mockups	UCD	yes	no
Santos and Boticario (2015b). User-centred design and educational data mining support during the recommendations elicitation process in social online learning environments	collaborative learning	NA	teachers, students	questionnaires, interviews, scenarios, focus groups, card sorting	dotLRN (LMS)	surveys, interviews, focus groups, cards	UCD	yes	yes
Eradze et al. (2020). Contextualising Learning Analytics with Classroom Observations: a Case Study	NA	participants, artifacts, activities social level	project managers	semi-structured questionnaires, interviews	standalone systems (web app)	surveys, interviews	CD	yes	yes