Designing human-centered learning analytics and artificial intelligence in education solutions: a systematic literature review

Figures & data

Table 1. Decisions taken during the SLR planning phase.

Category	Decision	Reason
Databases	Scopus	Scopus indexes papers in IEEE Xplore and WoS papers in ScienceDirect, thus including the main databases of TEL journals. JLA, IJAIED, LAK, CHI, AIED and EDM are forums that publish LA/AI studies in the educational landscape.
	Web of Science
	J. of Learning Analytics
	Int J. of AIED
	LAK Conference Proc.
	CHI Conference Proc.
	EDM Conference Proc.
	AIED Conference Proc.
Search String	[‘human-centered design’ OR ‘centred design’ OR ‘participatory design’ OR ‘co-design’] AND [‘learning analytics’ OR (‘artificial intelligence’ AND ‘education’)]	The search string includes the term ‘human-centered’ and the synonyms related to human-actions (e.g., participatory design, co-design) reported in relevant studies. We used the terms ‘-centred’ to detect combinations such as ‘user-’, ‘teacher-’ or ‘student-’. Furthermore, we used the terms associated with the research fields under study (i.e., ‘learning analytics’ or ‘artificial intelligence’) and the term ‘education’ that regards the frame of the systematic literature review.
Search Location	Title, abstract and keywords (metadata or abstract if restriction)	We expect that publications describing human-centered LA or AI studies mention it in the title abstract and/or keywords. Since there were variations in the way each search engine applied the query, to guarantee the same filtering criteria, we cross-checked that the title, abstract and keywords of each paper satisfied the query.
Time Restrictions	No restrictions	We aim to find all possible publications without tying to a specific time frame.
Inclusion Criteria	Empirical studies where LA and AI solutions are designed using human-centered approaches	We focus our research questions on empirical studies of HCLA and HCAI. Theoretical or conceptual papers are not relevant to answer the posed research questions.
Exclusion Criteria	Secondary studies (reviews); Papers under 5 pages; Project summaries; Non-English papers; Duplicate papers	We aim to analyse primary studies describing complete empirical experiences in English.

Figure 1. Overview of the systematic literature review process followed.

Figure 2. Number of studies per publication type and year.

Figure 3. Social network analysis of the reviewed papers based on co-authorship.

Figure 4. Learning theories identified in the review.

Figure 5. Evolution of the number of publications involving learning theories.

Figure 6. Sankey diagram displaying the involved stakeholders in the design and development of HCLA/HCAI solutions and their target users (N = 47).

Figure 7. Techniques and methods applied in HCD processes together with the purpose they served in accordance with the HCID framework.

Figure 8. Evaluation methods and approaches as identified from the final pool of the included papers.

Table

Title	Contribution (type)	Contribution (place)	Contribution (rank)	Year	LA or AI	Purpose (what-verb)	Purpose (what-object)	Purpose (who)	Purpose (why)
Schmitz et al. (2018). ‘Make It Personal!’ -- Gathering Input from Stakeholders for a Learning Analytics-Supported Learning Design Tool	conference	EC-TEL	–	2018	LA	develop	LA tool	HE teachers & students	personalise feedback
Pishtari, Rodríguez-Triana, and Väljataga (2021). A Multi-Stakeholder Perspective of Analytics for Learning Design in Location-Based Learning	journal	IJMBL	–	2021	LA	develop	authoring tool	practitioners	support the design of activities
Knight and Anderson (2016). Action-oriented, accountable, and inter(active) learning analytics for learners	workshop	LAL	–	2016	LA	develop	learning platform	students	shape online community with professionals
Oliver-Quelennec et al. (2022). Adapting Learning Analytics Dashboards by and for University Students	conference	EC-TEL	GSS 3 / CORE B	2022	LA	identify & design	LA indicators & LA dashboard	students	cover student expectations
Echeverria et al. (2022). An Exploratory Evaluation of a Collaboration Feedback Report	conference	LAK	GGS A- / CORE A	2022	LA	identify	LA indicators	students	support collaboration
Zhou, Sheng, and Howley (2020). Assessing post-hoc explainability of the BKT algorithm	conference	AIES	–	2020	AI	design & implement	interactive explanation	students	increase learning
Person et al. (2021). Co-design of a Learning Analytics Tool by Computer Scientists and Teachers: The Difficult Emergence of a Common World	conference	IEA	–	2021	LA	design	LA dashboard	high school teachers	NA
Holstein, McLaren, and Aleven (2019b). Co-Designing a Real-Time Classroom Orchestration Tool to Support Teacher-AI Complementarity	journal	JLA	–	2019	AI	design	LA tool (wearable)	K-12 teachers	support teacher orchestration
Ahn et al. (2021). Co-Designing for Privacy, Transparency, and Trust in K-12 Learning Analytics	conference	LAK	GGS A- / CORE A	2021	LA	design	LA dashboard	teachers	support teacher facilitation
Chalvatza, Karkalas, and Mavrikis (2019). Communicating learning analytics: Stakeholder participation and early stage requirement analysis	conference	CSEDU	–	2019	LA	design	LA tool	teachers, parents & managers	facilitate actionable decision-making
Hou, Nagashima, and Aleven (2022). Design a Dashboard for Secondary School Learners to Support Mastery Learning in a Gamified Learning Environment	conference	EC-TEL	GSS 3 / CORE B	2022	LA	design & refine	LA dashboard	HE students	promote mastery learning
Romero et al. (2021). Design and usability testing of an in-house developed performance feedback tool for medical students	journal	BMC Med. Educ.	JCR SSCI-Q2 JCR SCIE-Q2	2021	LA	design & develop	interface	HE students	track students' progress and skills for SRL

Table

Title	Learning theories / Pedagogical approaches	Considered aspects of the LD	Which stakeholders are involved	How stakeholders are involved	Applied target platform	Co-design tools	Terminology used	Evaluation (yes/no)	Applied in real scenario (yes/no)
Schmitz et al. (2018). ‘Make It Personal!’ -- Gathering Input from Stakeholders for a Learning Analytics-Supported Learning Design Tool	self-regulated learning	NA	teachers, students, TEL experts	focus groups, surveys	LA4LD tool (web app)	surveys	UCD	no	no
Pishtari, Rodríguez-Triana, and Väljataga (2021). A Multi-Stakeholder Perspective of Analytics for Learning Design in Location-Based Learning	NA	learning task type	teachers, educational instructors	a) contextual inquiry; b) interviews; c) design workshop	web-based mobile applications	storyboards, interviews	CD	yes	no
Knight and Anderson (2016). Action-oriented, accountable, and inter(active) learning analytics for learners	NA	NA	students	surveys, interviews	online learning environment	surveys, interviews	CD, PD	no	no
Oliver-Quelennec et al. (2022). Adapting Learning Analytics Dashboards by and for University Students	NA	learning context (study duration level), moment in the semester	students	co-design sessions	NA	paper-based prototypes	CD	no	no
Echeverria et al. (2022). An Exploratory Evaluation of a Collaboration Feedback Report	NA	NA	students	semi-structured interviews	NA	interviews	HCD, UCD	yes	no
Zhou, Sheng, and Howley (2020). Assessing post-hoc explainability of the BKT algorithm	NA	unclear	teachers	co-design sessions (brainstorming, usability testing, semi-structured interviews)	NA	paper-based prototypes, affinity diagrams, interviews	UCD	yes	no
Person et al. (2021). Co-design of a Learning Analytics Tool by Computer Scientists and Teachers: The Difficult Emergence of a Common World	NA	NA	teachers	interviews, workshop (brainstorming, game-based design)	NA	game-based design tool, interviews, cards, paper-based prototypes	CD, PD	no	no
Holstein, McLaren, and Aleven (2019b). Co-Designing a Real-Time Classroom Orchestration Tool to Support Teacher-AI Complementarity	adaptive learning	NA	teachers	(a) card sorting, interviews, field observations; (b) speed dating sessions; (c) sketching; (d) field observations, behavioural mapping, interviews	Lumino (AR app)	LATUX, paper-based prototypes, digital prototypes, cards, storyboards, mockups, speed-dating field observations, interviews	CD	yes	yes
Ahn et al. (2021). Co-Designing for Privacy, Transparency, and Trust in K-12 Learning Analytics	NA	NA	teachers, instructional coaches, district leaders	co-design sessions, semi-structured interviews	NA	interviews	CD	no	no
Chalvatza, Karkalas, and Mavrikis (2019). Communicating learning analytics: Stakeholder participation and early stage requirement analysis	NA	NA	teachers, TA, school manager, school committee member	workshop (focus groups)	NA	scenarios, focus groups, workshops	UCD	no	no
Hou, Nagashima, and Aleven (2022). Design a Dashboard for Secondary School Learners to Support Mastery Learning in a Gamified Learning Environment	NA	NA	students	(a) co-design sessions; (b) user testing	Gwynette (an ITS for algebra learning)	paper-based prototypes, interviews	CD	yes	no
Romero et al. (2021). Design and usability testing of an in-house developed performance feedback tool for medical students	SRL	NA	technical developers, project managers (researchers), medical students	(a) questionnaires, paper prototypes; (b) usability testing, (c) SUS	LevelUp (web app)	surveys	UCD	yes	unclear

Table

Title	Contribution (type)	Contribution (place)	Contribution (rank)	Year	LA or AI	Purpose (what-verb)	Purpose (what-object)	Purpose (who)	Purpose (why)
Beheshti et al. (2020). Design Considerations for Data-Driven Dashboards: Supporting Facilitation Tasks for Open-Ended Learning	conference	CHI	GSS 1 / CORE A*	2020	LA	design	LA dashboard (mobile)	museum visitors	provide formative feedback
Holstein, McLaren, and Aleven (2019a). Designing for Complementarity: Teacher and Student Needs for Orchestration Support in AI-Enhanced Classrooms	conference	AIED	GSS 2 / CORE A	2019	AI	design and use	AI-based system	K-12 teachers	support orchestration
Revano and Garcia (2021). Designing Human-Centered Learning Analytics Dashboard for Higher Education Using a Participatory Design Approach	conference	HNICEM	–	2021	LA	create	LA dashboard	HE students	provide progress awareness
Ahn et al. (2019). Designing in context: Reaching beyond usability in learning analytics dashboard design	journal	JLA	–	2019	LA	design & develop	LA dashboard	middle school teachers	support teacher pedagogical practices
Sadallah et al. (2022). Designing LADs That Promote Sensemaking: A Participatory Tool	conference	EC-TEL	GSS 3 / CORE B	2022	LA	design	LA dashboard	secondary school teachers	NA
Martinez-Maldonado et al. (2022). Designing translucent learning analytics with teachers: an elicitation process	journal	Interact. Learn. Environ.	JCR SSCI-Q1	2022	LA	design	LA dashboard	students	provide automated feedback
Knight et al. (2015). Developing a learning analytics dashboard for undergraduate engineering using participatory design	conference	ASEE	–	2015	LA	design & develop	LA dashboard	high school students	inform students' use
Aleven et al. (2016). Developing a teacher dashboard for use with intelligent tutoring systems	workshop	IWTA	–	2016	LA	create	LA dashboard	teachers	inform teachers' decisions
Hoffmann et al. (2022). Development of Actionable Insights for Regulating Students' Collaborative Writing of Scientific Texts	conference	EC-TEL	GSS 3 / CORE B	2022	LA	define & design	LA indicators & LA dashboard	teachers	monitor collaborative learning
Nazaretsky et al. (2022). Empowering Teachers with AI: Co-Designing a Learning Analytics Tool for Personalized Instruction in the Science Classroom	conference	LAK	GGS A- / CORE A	2022	LA	design	LA tool	teachers	build pedagogically meaninful explanations
Treasure-Jones, Dent-Spargo, and Dharmaratne (2018). How do students want their workplace-based feedback visualised in order to support self-regulated learning? Initial results & reflections from a co-design study in medical education	conference	EC-TEL (practitioners' track)	–	2018	LA	refine	learning platform	students	support self-regulated learning
Chatti et al. (2020). How to design effective learning analytics indicators? a human-centered design approach	conference	EC-TEL	–	2020	LA	identify	LA indicators	HE students	support students in course selection

Table

Title	Learning theories / Pedagogical approaches	Considered aspects of the LD	Which stakeholders are involved	How stakeholders are involved	Applied target platform	Co-design tools	Terminology used	Evaluation (yes/no)	Applied in real scenario (yes/no)
Beheshti et al. (2020). Design Considerations for Data-Driven Dashboards: Supporting Facilitation Tasks for Open-Ended Learning	open-ended learning	task type	facilitators (final users)	(a) observations, focus group; (b) co-design sessions	mobile app	field notes	PD	no	no
Holstein, McLaren, and Aleven (2019a). Designing for Complementarity: Teacher and Student Needs for Orchestration Support in AI-Enhanced Classrooms	NA	NA	K-12 teachers & students	Participatory speed dating approach	NA	low-fidelity prototypes, storyboards	PD	no	no
Revano and Garcia (2021). Designing Human-Centered Learning Analytics Dashboard for Higher Education Using a Participatory Design Approach	multimedia learning theory	NA	teachers, students	focus group	learning management system	low-fidelity prototypes, field notes	HCD, PD	no	no
Ahn et al. (2019). Designing in context: Reaching beyond usability in learning analytics dashboard design	collaborative learning	NA	researchers, designers, developers	(a) interviews; (b) brainstorming sessions; (c) co-des- ign sessions; (d) interviews	Edsight (web app)	interviews, low-fidelity prototypes, sticky notes, affinity diagrams, journey maps	HCD, PD	no	no
Sadallah et al. (2022). Designing LADs That Promote Sensemaking: A Participatory Tool	NA	unclear	teachers, administrator, instructional designer, researchers	workshop	NA	cards, paper-based prototypes	CD, HCD, PD	no	no
Martinez-Maldonado et al. (2022). Designing translucent learning analytics with teachers: an elicitation process	collocated collaborative learning	classroom dynamics, learning constructs, teacher questions, expectations and hypotheses	teachers, subject coordinators	card-sorting, focus groups, low-fidelity prototyping, co-design sessions, semi-structured interviews	NA	focus groups, cards, low-fidelity prototypes, interviews	CD	no (reported elsewhere)	no (reported elsewhere)
Knight et al. (2015). Developing a learning analytics dashboard for undergraduate engineering using participatory design	NA	NA	students	co-design session	NA	low-fidelity prototypes, field notes, focus groups	HCD, PD	no	no
Aleven et al. (2016). Developing a teacher dashboard for use with intelligent tutoring systems	blended learning	NA	teachers	contextual inquiry, speed dating, usability testing	CTAT and Lynnette (ITS infrastructure)	storyboards, mockups	UCD	no	yes
Hoffmann et al. (2022). Development of Actionable Insights for Regulating Students' Collaborative Writing of Scientific Texts	CSCL	NA	teachers	focus groups, interviews	LabNbook (webapp)	focus groups, interviews, mockups	HCD, UCD	no	no
Nazaretsky et al. (2022). Empowering Teachers with AI: Co-Designing a Learning Analytics Tool for Personalized Instruction in the Science Classroom	NA	NA	K-12 teachers	interviews	PeTeL (LMS)	interviews	PD	yes	yes
Treasure-Jones, Dent-Spargo, and Dharmaratne (2018). How do students want their workplace-based feedback visualised in order to support self-regulated learning? Initial results & reflections from a co-design study in medical education	SRL, workplace-based learning	NA	teachers, students	co-design workshops	myPAL (LMS)	workshops, interviews, paper-based prototypes, digital prototypes	CD	no	yes
Chatti et al. (2020). How to design effective learning analytics indicators? a human-centered design approach	NA	course characteristics	teachers, students	co-design sessions	Course Insights (web app)	interviews, paper-based prototypes, surveys	CD	yes	unclear

Table

Title	Contribution (type)	Contribution (place)	Contribution (rank)	Year	LA or AI	Purpose (what-verb)	Purpose (what-object)	Purpose (who)	Purpose (why)
Conijn, Van Waes, and van Zaanen (2020). Human-centered design of a dashboard on students' revisions during writing	conference	EC-TEL	–	2020	LA	design	LA dashboard (actionable)	teachers	display students' information
Cerro Martinez, Guitert Catasus, and Romeu Fontanillas (2020). Impact of using learning analytics in asynchronous online discussions in higher education	journal	ETHE	JCR SSCI-Q1	2020	LA	define, design & implement	LA indicators & LA tool	teachers	support asynchronous online discussion
Vezzoli, Mavrikis, and Vasalou (2020). Inspiration Cards Workshops with Primary Teachers in the Early Co-Design Stages of Learning Analytics	conference	LAK	–	2020	LA	unclear	LA tool	primary school students	unclear
Michos et al. (2020). Involving teachers in learning analytics design: Lessons learned from two case studies	conference	LAK	–	2020	LA	create	LA tool	teachers	guide in the design and running of courses
Alvarez, Martinez-Maldonado, and Shum (2020). LA-DECK: A Card-Based Learning Analytics Co-Design Tool	conference	LAK	–	2020	LA	unclear	unclear	unclear	unclear
Verbert et al. (2020). Learning analytics dashboards: The past, the present and the future	conference	LAK	–	2022	LA	create	LA dashboard (actionable)	teachers	support decision-making
Kilińska, Kobbelgaard, and Ryberg (2019). Learning analytics features for improving collaborative writing practices: insights into the students' perspective	conference	ICEM	–	2019	LA	design	LA tool	high school students	gain awareness on collaborative writing
Brun et al. (2019). Learning analytics made in France: the METAL project	journal	IJILT	–	2019	LA	design	LA dashboard	secondary school teachers & students	have better awareness
Long, Aman, and Aleven (2015). Motivational design in an intelligent tutoring system that helps students make good task selection decisions	conference	AIED	CORE A	2015	AI	identify	LA indicators	ITS	improve problem selection
Valkanova et al. (2016). Opening the black box of practice-based learning: Human-centered design of learning analytics	workshop	crossLAK	–	2016	LA	design	LA tool	teachers & students	support awareness during practice-based learning activities
Coughlan, Lister, and Freear (2019). Our Journey: Designing and utilising a tool to support students to represent their study journeys	conference	INTED	–	2019	LA	design	LA tool	students	represent own study journey
Santos and Boticario (2015a). Practical guidelines for designing and evaluating educationally oriented recommendations	journal	C&E	JCR SSCI-Q1 / JCR SCIE-Q1	2015	AI	compile	guidelines	unclear	produce personalised recommendations

Table

Title	Learning theories / Pedagogical approaches	Considered aspects of the LD	Which stakeholders are involved	How stakeholders are involved	Applied target platform	Co-design tools	Terminology used	Evaluation (yes/no)	Applied in real scenario (yes/no)
Conijn, Van Waes, and van Zaanen (2020). Human-centered design of a dashboard on students' revisions during writing	NA	NA	teachers, students	round-table sessions	NA	LATUX, discussion groups, prototypes, sticky notes	HCD	yes	no
Cerro Martinez, Guitert Catasus, and Romeu Fontanillas (2020). Impact of using learning analytics in asynchronous online discussions in higher education	collaborative learning	teachers' perceptions	teachers	co-design sessions, surveys	LA tool for the UOC campus (LMS)	discussion groups, surveys, prototypes	PD	yes	yes
Vezzoli, Mavrikis, and Vasalou (2020). Inspiration Cards Workshops with Primary Teachers in the Early Co-Design Stages of Learning Analytics	NA	teacher pedagogical ideas, questions and expectations	teachers	workshops	NA	workshops, cards	CD	no	no
Michos et al. (2020). Involving teachers in learning analytics design: Lessons learned from two case studies	collaborative learning	LD decisions	teachers	interviews, focus groups, questionnaires	NA	interviews, focus groups, surveys	CD	no	no
Alvarez, Martinez-Maldonado, and Shum (2020). LA-DECK: A Card-Based Learning Analytics Co-Design Tool	NA	learning objective and context	students, educators, researchers, developers	semi-structured interviews	NA	interviews, cards	CD	no	no
Verbert et al. (2020). Learning analytics dashboards: The past, the present and the future	NA	NA	researchers	workshops	NA	workshops, cards, sticky notes	PD	no	no
Kilińska, Kobbelgaard, and Ryberg (2019). Learning analytics features for improving collaborative writing practices: insights into the students' perspective	problem-based learning, project-based learning, collaborative learning	NA	students	co-design sessions, workshops	Google Docs (web app)	workshops, paper-based prototypes, cards	CD, PD	yes	no
Brun et al. (2019). Learning analytics made in France: the METAL project	NA	NA	teachers, students	teacher LAD: co-design sessions, students LAD: co-design sessions, interviews	LMS and ‘behaviour sensors’	focus groups, paper-based prototypes	CD	no	no
Long, Aman, and Aleven (2015). Motivational design in an intelligent tutoring system that helps students make good task selection decisions	coself-regulated learning, motivational theories	NA	K-12 students	interviews	ITS	interviews	UCD	no	yes
Valkanova et al. (2016). Opening the black box of practice-based learning: Human-centred design of learning analytics	practice-based learning	NA	K-12 teachers	interviews	NA	interviews	HCD	no	no
Coughlan, Lister, and Freear (2019). Our Journey: Designing and utilising a tool to support students to represent their study journeys	NA	NA	students, student union representatives, external staff	workshops	Our Journey (web app)	surveys	PD	no	no
Santos and Boticario (2015a). Practical guidelines for designing and evaluating educationally oriented recommendations	NA	NA	ICT manager, teachers, students	interviews, focus groups (brainstorming sessions)	NA	interviews, focus groups	UCD	yes	yes

Table

Title	Contribution (type)	Contribution (place)	Contribution (rank)	Year	LA or AI	Purpose (what-verb)	Purpose (what-object)	Purpose (who)	Purpose (why)
Eradze, Rodriguez Triana, and Laanpere (2017). Semantically Annotated Lesson Observation Data in Learning Analytics Datasets: a Reference Model	journal	ID&A	–	2017	LA	design	LA model	teachers	support context-aware semantical annotations
Huh et al. (2022). Service Design of Artificial Intelligence Voice Agents as a Guideline for Assisting Independent Toilet Training of Preschool Children	journal	ADR	–	2022	AI	create	AI agent	pre-school students	support toilet training
de Quincey et al. (2019). Student Centred Design of a Learning Analytics System	conference	LAK	–	2019	LA	design & develop	LA dashboard	students	personalise feedback and real-time support
Lister et al. (2021). Taylor, the Disability Disclosure Virtual Assistant: A Case Study of Participatory Research with Disabled Students	journal	Educ. Sci.	–	2021	AI	design	virtual assistant	students (disabled)	solve administrative issues
Alzoubi et al. (2021). TeachActive Feedback Dashboard: Using Automated Classroom Analytics to Visualize Pedagogical Strategies at a Glance	conference	CHI	–	2021	LA	design	LA dashboard	high school teachers	improve pedagogical practices
Buckingham Shum, Ferguson, and Martinez-Maldonado (2019). The Multimodal Matrix as a Quantitative Ethnography Methodology	conference	ICQE	–	2019	LA	design	LA tool	teachers & students	support awareness on collaborative learning
Pozdniakov et al. (2022). The Question-driven Dashboard: How Can We Design Analytics Interfaces Aligned to Teachers' Inquiry?	conference	LAK	GGS A- / CORE A	2022	LA	design & develop	LA tool	teachers	support awareness of students' collaboration
Ouatiq et al. (2022). Towards the Co-Design of a Teachers' Dashboards in a Hybrid Learning Environment	conference	IRASET	–	2022	LA	design	LA dashboard	HE teachers	assist students' tracking & predict their results
Dabbebi, Gilliot, and Iksal (2019). User centred approach for learning analytics dashboard generation	conference	CSEDU	–	2019	LA	design	LA dashboard	teachers	monitor students' performance & support teachers' decision-making
Santos and Boticario (2015b). User-centred design and educational data mining support during the recommendations elicitation process in social online learning environments	journal	Expert Syst.	JCR SCIE-Q3	2015	DM	develop	recommender system	students	produce accurate social-oriented recommendations
Eradze et al. (2020). Contextualising Learning Analytics with Classroom Observations: a Case Study	journal	ID&A	–	2020	LA	design & develop	LA dashboard	managers	better understand the adoption of digital learning resources

Table

Title	Learning theories / Pedagogical approaches	Considered aspects of the LD	Which stakeholders are involved	How stakeholders are involved	Applied target platform	Co-design tools	Terminology used	Evaluation (yes/no)	Applied in real scenario (yes/no)
Eradze, Rodriguez Triana, and Laanpere (2017). Semantically Annotated Lesson Observation Data in Learning Analytics Datasets: a Reference Model	NA	resources, activities, possible actions	teachers	participatory design session (semi-structured interviews, guided focus groups)	NA	focus groups	PD	no	no
Huh et al. (2022). Service Design of Artificial Intelligence Voice Agents as a Guideline for Assisting Independent Toilet Training of Preschool Children	Vygotsky's sociocultural theory	NA	pre-school teachers, parents and children	(a) interviews; (b) prototyping; (c) experiment	AI agent for toilet training	interviews, prototypes	CD	yes	no
de Quincey et al. (2019). Student Centred Design of a Learning Analytics System	NA	NA	students	focus groups, interviews	dashboard for LMS	focus groups, interviews	UCD	yes	yes
Lister et al. (2021). Taylor, the Disability Disclosure Virtual Assistant: A Case Study of Participatory Research with Disabled Students	NA	NA	students	(a) co-design sessions; (b) survey; (c) co-design workshops	virtual assistant for university website	interviews, discussion forums, surveys, paper-based prototypes	PD	yes	no
Alzoubi et al. (2021). TeachActive Feedback Dashboard: Using Automated Classroom Analytics to Visualize Pedagogical Strategies at a Glance	active learning	NA	teachers	interview, co-design meetings	NA	interviews, surveys, prototypes	UCD	no	no
Buckingham Shum, Ferguson, and Martinez-Maldonado (2019). The Multimodal Matrix as a Quantitative Ethnography Methodology	collaborative learning	NA	teachers, students	co-design sessions	nursing team simulation	unclear	CD	no (reported elsewhere)	yes
Pozdniakov et al. (2022). The Question-driven Dashboard: How Can We Design Analytics Interfaces Aligned to Teachers' Inquiry?	collaborative learning	NA	teacher assistants, students	Interviews with teachers, inquiry, interface prototyping, interface validation	Zoom and Google Docs (web apps)	interviews, prototypes	HCD	yes	yes
Ouatiq et al. (2022). Towards the Co-Design of a Teachers' Dashboards in a Hybrid Learning Environment	NA	NA	teachers	surveys, participatory workshops, feedback interviews	hybrid learning environment	surveys, interviews	CD, PD	yes	no
Dabbebi, Gilliot, and Iksal (2019). User centred approach for learning analytics dashboard generation	NA	NA	teachers	survey, semi-structured interviews and co-design workshops	Tactileo Map (mobile app)	surveys, interviews, focus groups, mockups	UCD	yes	no
Santos and Boticario (2015b). User-centred design and educational data mining support during the recommendations elicitation process in social online learning environments	collaborative learning	NA	teachers, students	questionnaires, interviews, scenarios, focus groups, card sorting	dotLRN (LMS)	surveys, interviews, focus groups, cards	UCD	yes	yes
Eradze et al. (2020). Contextualising Learning Analytics with Classroom Observations: a Case Study	NA	participants, artifacts, activities social level	project managers	semi-structured questionnaires, interviews	standalone systems (web app)	surveys, interviews	CD	yes	yes

Schmitz, M., M. Scheffel, E. van Limbeek, R. Bemelmans, and H. Drachsler. 2018. “‘Make It Personal!’ -- Gathering Input from Stakeholders for a Learning Analytics-Supported Learning Design Tool.” 13th European Conference on Technology Enhanced Learning, Lyon, France, 297–310.

 Google Scholar

Pishtari, G., M. J. Rodríguez-Triana, and T. Väljataga. 2021. “A Multi-Stakeholder Perspective of Analytics for Learning Design in Location-Based Learning.” International Journal of Mobile and Blended Learning 13:1–17. https://doi.org/10.4018/IJMBL.

 Web of Science ®Google Scholar

Knight, S., and T. D. Anderson. 2016. “Action-Oriented, Accountable, and Inter(Active) Learning Analytics for Learners.” CEUR Workshop Proceedings, Edinburgh, UK, Vol. 1596, 47–51.

 Google Scholar

Oliver-Quelennec, K., F. Bouchet, T. Carron, K. Fronton Casalino, and C. Pinçon. 2022. “Adapting Learning Analytics Dashboards by and for University Students.” In European Conference on Technology Enhanced Learning, 299–309. Springer.

 Google Scholar

Echeverria, V., M. Wong-Villacres, X. Ochoa, and K. Chiluiza. 2022. “An Exploratory Evaluation of a Collaboration Feedback Report.” 12th International Conference on Learning Analytics and Knowledge, 478–484. https://dl.acm.org/doi/proceedings/10 1145/3506860.

 Google Scholar

Zhou, T., H. Sheng, and I. Howley. 2020. “Assessing Post-Hoc Explainability of the BKT Algorithm.” AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA, 407–413.

 Google Scholar

Person, J., C. Vidal-Gomel, P. Cottier, and C. Lecomte. 2021. “Co-Design of a Learning Analytics Tool by Computer Scientists and Teachers: The Difficult Emergence of a Common World.” In Congress of the International Ergonomics Association, 524–533. Springer.

 Google Scholar

Holstein, K., B. M. McLaren, and V. Aleven. 2019b. “Co-Designing a Real-Time Classroom Orchestration Tool to Support Teacher–ai Complementarity.” Journal of Learning Analytics 6: 27–52. https://doi.org/10.18608/jla.2019.62.3.

 Google Scholar

Ahn, J., F. Campos, H. Nguyen, M. Hays, and J. Morrison. 2021. “Co-Designing for Privacy, Transparency, and Trust in K-12 Learning Analytics.” In 11th International Conference on Learning Analytics & Knowledge, 55–65. ACM.

 Google Scholar

Chalvatza, F., S. Karkalas, and M. Mavrikis. 2019. “Communicating Learning Analytics: Stakeholder Participation and Early Stage Requirement Analysis.” In 11th International Conference on Computer Supported Education, 339–346. SciTePress.

 Google Scholar

Hou, X., T. Nagashima, and V. Aleven. 2022. “Design a Dashboard for Secondary School Learners to Support Mastery Learning in a Gamified Learning Environment.” 17th European Conference on Technology Enhanced Learning, Toulouse, France, 542–549.

 Google Scholar

Romero, Y. R., H. Tame, Y. Holzhausen, M. Petzold, J. V. Wyszynski, H. Peters, M. Alhassan-Altoaama, M. Domanska, and M. Dittmar. 2021. “Design and Usability Testing of An in-House Developed Performance Feedback Tool for Medical Students.” BMC Medical Education 21 (1): 354. https://doi.org/10.1186/s12909-021-02788-4.

 PubMed Web of Science ®Google Scholar

Beheshti, E., L. Lyons, A. Mallavarapu, B. Wallingford, and S. Uzzo. 2020. “Design Considerations for Data-Driven Dashboards: Supporting Facilitation Tasks for Open-Ended Learning.” Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, Hawai, USA, 1–9.

 Google Scholar

Holstein, K., B. M. McLaren, and V. Aleven. 2019a. “Designing for Complementarity: Teacher and Student Needs for Orchestration Support in AI-Enhanced classrooms.” 20th International Conference on Artificial Intelligence in Education, Chicago, IL, USA, 157–171.

 Google Scholar

Revano, T. F., and M. B. Garcia. 2021. “Designing Human-Centered Learning Analytics Dashboard for Higher Education Using a Participatory Design Approach.” 2021 IEEE 13th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management, Manila, Philippines, 1–5.

 Google Scholar

Ahn, J., F. Campos, M. Hays, and D. DiGiacomo. 2019. “Designing in Context: Reaching Beyond Usability in Learning Analytics Dashboard Design.” Journal of Learning Analytics 6:70–85. https://doi.org/10.18608/jla.2019.62.5.

 Google Scholar

Sadallah, M., J. M. Gilliot, S. Iksal, K. Quelennec, M. Vermeulen, L. Neyssensas, O. Aubert, and R. Venant. 2022. “Designing LADs That Promote Sensemaking: A Participatory Tool.” 17th European Conference on Technology Enhanced Learning, Toulouse, France, 587–593.

 Google Scholar

Martinez-Maldonado, R., D. Elliott, C. Axisa, T. Power, V. Echeverria, and S. B. Shum. 2022. “Designing Translucent Learning Analytics with Teachers: An Elicitation Process.” Interactive Learning Environments 30 (6): 1077–1091. https://doi.org/10.1080/10494820.2019.1710541.

 Web of Science ®Google Scholar

Knight, D., C. Brozina, E. Stauffer, C. Frisina, and T. Abel. 2015. “Developing a Learning Analytics Dashboard for Undergraduate Engineering Using Participatory Design.” 2015 ASEE Annual Conference and Exposition Proceedings, Seattle, Washington, USA, 26.485.1–26.485.11.

 Google Scholar

Aleven, V., F. Xhakaj, K. Holstein, and B. M. McLaren. 2016. “Developing a Teacher Dashboard for use with Intelligent Tutoring Systems.” In 4th International Workshop on Teaching Analytics, 15–23. CEUR.

 Google Scholar

Hoffmann, C., N. Mandran, C. d'Ham, S. Rebaudo, and M. A. Haddouche. 2022. “Development of Actionable Insights for Regulating Students' Collaborative Writing of Scientific Texts.” In European Conference on Technology Enhanced Learning, 534–541. Springer.

 Google Scholar

Nazaretsky, T., C. Bar, M. Walter, and G. Alexandron. 2022. “Empowering Teachers with AI: Co-Designing a Learning Analytics Tool for Personalized Instruction in the Science Classroom.” In 12th International Conference on Learning Analytics & Knowledge, 1–12. ACM.

 Google Scholar

Treasure-Jones, T., R. Dent-Spargo, and S. Dharmaratne. 2018. “How do Students Want their Workplace-Based Feedback Visualized in Order to Support Self-Regulated Learning? Initial Results & Reflections From a Co-Design Study in Medical Education.” CEUR Workshop Proceedings. CEUR Workshop Proceedings, Vol. 2193.

 Google Scholar

Chatti, M. A., A. Muslim, M. Guesmi, F. Richtscheid, D. Nasimi, A. Shahin, and R. Damera. 2020. “How to Design Effective Learning Analytics Indicators? A Human-Centered Design Approach.” In 15th European Conference on Technology Enhanced Learning, 303–317. Springer.

 Google Scholar

Conijn, R., L. Van Waes, and M. van Zaanen. 2020. “Human-Centered Design of a Dashboard on Students' Revisions During Writing.” In 15th European Conference on Technology Enhanced Learning, 30–44. Springer.

 Google Scholar

Cerro Martinez, J. P., M. Guitert Catasus, and T. Romeu Fontanillas. 2020. “Impact of Using Learning Analytics in Asynchronous Online Discussions in Higher Education.” International Journal of Educational Technology in Higher Education 17 (1): 1–18. https://doi.org/10.1186/s41239-020-00217-y.

 Web of Science ®Google Scholar

Vezzoli, Y., M. Mavrikis, and A. Vasalou. 2020. “Inspiration Cards Workshops with Primary Teachers in the Early Co-Design Stages of Learning Analytics.” In 10th International Conference on Learning Analytics & Knowledge, 73–82. ACM.

 Google Scholar

Michos, K., C. Lang, D. Hernández-Leo, and D. Price-Dennis. 2020. “Involving Teachers in Learning Analytics Design.” In 10th International Conference on Learning Analytics & Knowledge, 94–99. ACM.

 Google Scholar

Alvarez, C. P., R. Martinez-Maldonado, and S. B. Shum. 2020. “LA-DECK: A Card-Based Learning Analytics Co-Design Tool.” In 10th International Conference on Learning Analytics & Knowledge, 63–72, ACM.

 Google Scholar

Verbert, K., X. Ochoa, R. D. Croon, R. A. Dourado, and T. D. Laet. 2020. “Learning Analytics Dashboards.” In 10th International Conference on Learning Analytics & Knowledge, 35–40. ACM.

 Google Scholar

Kilińska, D., F. V. Kobbelgaard, and T. Ryberg. 2019. “Learning Analytics Features for Improving Collaborative Writing Practices: Insights into the Students' Perspective.” In Digital Turn in Schools–Research, Policy, Practice: Proceedings of ICEM 2018 Conference, 69–81, Springer.

 Google Scholar

Brun, A., G. Bonnin, S. Castagnos, A. Roussanaly, and A. Boyer. 2019. “Learning Analytics Made in France: The Metal Project.” The International Journal of Information and Learning Technology 36 (4): 299–313. https://doi.org/10.1108/IJILT-02-2019-0022.

 Web of Science ®Google Scholar

Long, Y., Z. Aman, and V. Aleven. 2015. “Motivational Design in an Intelligent Tutoring System that Helps Students Make Good Task Selection Decisions.” In 17th International Conference on Artificial Intelligence in Education, 226–236. Springer.

 Google Scholar

Valkanova, N., M. Cukurova, A. Berner, K. Avramides, and M. Mavrikis. 2016. “Opening the Black Box of Practice-Based Learning: Human-centered Design of Learning Analytics.” In CrossLAK, 40–46. Citeseer.

 Google Scholar

Coughlan, T., K. Lister, and N. Freear. 2019. “Our Journey: Designing and Utilising a Tool to Support Students to Represent Their Study Journeys.” 13th Annual International Technology, Education and Development Conference, Valencia, Spain, 3140–3147.

 Google Scholar

Santos, O. C., and J. G. Boticario. 2015a. “Practical Guidelines for Designing and Evaluating Educationally Oriented Recommendations.” Computers and Education 81:354–374. https://doi.org/10.1016/j.compedu.2014.10.008.

 Web of Science ®Google Scholar

Eradze, M., M. J. Rodriguez Triana, and M. Laanpere. 2017. “Semantically Annotated Lesson Observation Data in Learning Analytics Datasets: A Reference Model.” Interaction Design and Architecture (s) Journal-IxD&A 33:75–91. https://doi.org/10.55612/s-5002-000.

 Google Scholar

Huh, J., S. Ann, J. Hong, M. Cui, J. Y. Park, Y. Kim, B. Sim, and H. K. Lee. 2022. “Service Design of Artificial Intelligence Voice Agents as a Guideline for Assisting Independent Toilet Training of Preschool Children.” Archives of Design Research 35 (3): 81–93. https://doi.org/10.15187/adr.2022.08.35.3.81.

 Google Scholar

de Quincey, E., C. Briggs, T. Kyriacou, and R. Waller. 2019. “Student Centred Design of a Learning Analytics System.” In 9th International Conference on Learning Analytics & Knowledge, 353–362. ACM.

 Google Scholar

Lister, K., T. Coughlan, I. Kenny, R. Tudor, and F. Iniesto. 2021. “Taylor, the Disability Disclosure Virtual Assistant: A Case Study of Participatory Research with Disabled Students.” Education Sciences 11: 584. https://doi.org/10.3390/educsci11100587.

 Web of Science ®Google Scholar

Alzoubi, D., J. Kelley, E. Baran, S. B. Gilbert, A. K. Ilgu, and S. Jiang. 2021. “TeachActive Feedback Dashboard: Using Automated Classroom Analytics to Visualize Pedagogical Strategies at a Glance.” In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery.

 Google Scholar

Buckingham Shum, S., R. Ferguson, and R. Martinez-Maldonado. 2019. “Human-Centred Learning Analytics.” Journal of Learning Analytics 6:1–9.

 Google Scholar

Pozdniakov, S., R. Martinez-Maldonado, Y. S. Tsai, M. Cukurova, T. Bartindale, P. Chen, H. Marshall, D. Richardson, and D. Gasevic. 2022. “The Question-Driven Dashboard: How Can We Design Analytics Interfaces Aligned to Teachers' Inquiry?” In 12th International Conference on Learning Analytics & Knowledge, 175–185. ACM.

 Google Scholar

Ouatiq, A., B. Riyami, K. Mansouri, M. Qbadou, and E. S. Aoula. 2022. “Towards the Co-Design of a Teachers' Dashboards in a Hybrid Learning Environment.” 2nd International Conference on Innovative Research in Applied Science, Engineering and Technology, Meknes, Morocco, 1–6.

 Google Scholar

Dabbebi, I., J. M. Gilliot, and S. Iksal. 2019. “User Centered Approach for Learning Analytics Dashboard Generation.” International Conference on Computer Supported Education, Heraklion, Greece.

 Google Scholar

Santos, O. C., and J. G. Boticario. 2015b. “User-Centred Design and Educational Data Mining Support During the Recommendations Elicitation Process in Social Online Learning Environments.” Expert Systems 32 (2): 293–311. https://doi.org/10.1111/exsy.v32.2.

 Web of Science ®Google Scholar

Eradze, M., M. J. Rodríguez-Triana, N. Milikic, M. Laanpere, and K. Tammets. 2020. “Contextualising Learning Analytics with Classroom Observations: A Case Study.” Interaction Design & Architecture, 71–95. https://doi.org/10.55612/s-5002-000.

 Web of Science ®Google Scholar