Relationships between engagement, achievement and well-being: validation of the engagement in higher education scale

ABSTRACT

Addressing the factors associated with students’ underperformance, dropout rates, and mental health challenges is a pressing concern for university institutions. Favorable conditions for student engagement emerges as a potential solution to mitigate these issues. Therefore, there is a need for instruments that assess the multiple dimensions of the construct and relate them to significant variables in the lives of students and institutions. This study presents the initial validation of the Higher Education Engagement Scale (EiHES) in a sample of 760 students who responded to an online survey. Results of both exploratory and confirmatory factor analyses using two random subsamples revealed that the scale comprises six dimensions of engagement: academic learning, online, cognitive, social with teachers, social with peers, and affective. All subscales showed adequate reliability indicators. Measurement invariance across gender was established up to the metric models. Evidence of validity was supported by associations between engagement with academic achievement and subjective well-being. The EiHES provides a comprehensive perspective of student engagement and appears to be a suitable instrument for assessing the construct in Portuguese university students. The discussion included limitations and suggestions for future research.

KEYWORDS:

• Student engagement
• higher education
• scale validation
• academic achievement
• subjective well-being

Introduction

In higher education, underachievement, dropout rates, and mental health are major concerns, prompting institutions, researchers, and decision-makers to address these challenges (Gago et al. 2023). In Portugal, the dropout rate exceeds 20%, and only half of students complete a three-year degree in four years (OECD 2022). Also, there are gender disparities in academic experiences and educational attainment, with women outnumbering men at bachelor’s and master’s levels (OECD 2023). Furthermore, the prevalence of stress, anxiety, and depression among university students is substantial, with rates increasing further following the COVID-19 pandemic (Laranjeira et al. 2022).

Engagement has been identified as a powerful antidote to academic failure, school dropout, and disaffection (Appleton, Christenson, and Furlong 2008). The literature suggests associations between university students’ engagement with a range of desired academic, social, behavioral, and emotional learning outcomes, such as better grades (Ketonen et al. 2019), critical thinking and cognitive development (Kilgo, Ezell Sheets, and Pascarella 2015), persistence in courses (Kuh et al. 2008), feelings of happiness (Boulton et al. 2019), and enhanced mental health (Kotera and Ting 2021). According to Kuh and colleagues (2007), engagement is a factor that impacts students’ success at university and constitutes a fundamental dimension of success itself. This perspective underscores the interconnectedness of student engagement with other critical factors, such as the acquisition of knowledge, development of skills and competencies, and the attainment of educational goals and academic accomplishments.

Recognizing these valuable insights from the research literature, higher education institutions have prioritized efforts to enhance student engagement through the implementation of institutional policies and practices (Trolian 2024). Creating conditions conducive to student engagement is a way for universities to promote academic performance, general well-being and high standards of qualifications and skills.

Due to its relevance in the educational field, student engagement has been a topic of considerable and extensive research through schooling and across diverse educational contexts (Christenson, Reschly, and Wylie 2012; Trolian 2024). The present study seeks to provide an empirical contribution to this field within the context of higher education The introduction will be structured into several sections Firstly, we will delve into the concept and dimensionality of student engagement. Next, we will explore evidence linking engagement to two key learning and affective outcomes, which were selected based on the consistent body of empirical evidence showing the contribution of different dimensions of engagement to students’ academic achievement and subjective well-being (Trolian 2024). Then, considerations on measuring of engagement will be presented. Lastly, we will outline the purpose of the study and present the hypotheses.

Student engagement: A multidimensional construct

Student engagement is a meta-construct that has attracted the attention of researchers, practitioners, and policymakers over the world (e.g. Lam et al. 2012). It can be broadly defined as ‘the student's psychological investment in and effort directed toward learning, understanding, or mastering the knowledge, skills, or crafts that academic work is intended to promote’ (Newmann, Wehlage, and Lamborn 1992, 12). One of the key aspects of engagement lies in its malleability, as it arises from the dynamic interaction between individuals and their environment. Hence, academic and social conditions, such as school practices, have the potential to either enhance or hinder it (Fredricks, Blumenfeld, and Paris 2004).

Most researchers agree that engagement is multifaceted, yet the number and nature of its dimensions lacks consensus. One of the major challenges in studying student engagement is what Reschly and Christenson (2012) refer to as the ‘jangle fallacy’. Various taxonomies have proposed different labels for engagement dimensions, yet at times, these labels are used interchangeably or are encompassed within one another, such as in the case of academic and behavioral dimension. Additionally, there are instances where terminology remains consistent, but the operationalization of dimensions varies, as seen with cognitive or affective engagement (Wong and Liem 2022).

Engagement models typically range from two (Skinner and Belmont 1993) to four or more components (e.g. Reeve and Tseng 2011). One of the most widespread frameworks in the literature, proposed by Fredricks, Blumenfeld, and Paris (2004), suggests that engagement is a fusion of behavior, cognition, and emotion, reflecting students’ actions, thoughts, and feelings. Later, Fredricks and colleagues (2016) also introduced a social dimension, which involves collaborative work with peers in the classroom. Although engagement components are conceptually distinct from each other, they tend to be associated to a certain extent (Fredricks, Blumenfeld, and Paris 2004).

Another widely recognized model of engagement is proposed by Finn and Zimmer (2012), who identified four major dimensions: academic, cognitive, social, and affective. Despite ongoing debate over which taxonomy more accurately captures engagement features, this article adopts Finn and Zimmer’s four-dimensional model, which was further extended by Zhoc et al. (2019) to accommodate innovations associated with technology use in higher education. In the following sections, we will provide a detailed description of each dimension and subdimension.

Academic engagement

Academic engagement brings together observable behaviors that directly influence learning, such as attending classes, attentiveness, completing homework, and participating in class activities (Finn and Zimmer 2012). Zhoc et al. (2019) proposed that academic engagement could be divided into two subdimensions: academic learning and online engagement. The latter encompasses learning-directed behaviors using the Internet and other digital platforms. The incorporation of technology into the learning process represents one of the latest shifts in higher education, whether through the delivery of online courses or the integration of digital technologies into traditional face-to-face settings (Dumford and Miller 2018). Research suggests that the use of information technologies is strongly associated with other forms of engagement and is predictive of improved learning outcomes (Yu et al. 2022).

Cognitive engagement

Cognitive engagement definitions draw from two distinct bodies of literature. The first focuses on the investment in learning, such as asking questions, clarifying concepts, persisting with difficult tasks, and going beyond assigned requirements (Finn and Zimmer 2012). However, this perspective overlaps with some indicators of behavioral or academic engagement. For instance, persistence could be considered an indicator of both cognitive and behavioral dimensions (Fredricks and McColskey 2012).

The second perspective conceptualizes cognitive engagement as a self-regulated learning process. This includes the use of cognitive (e.g. connecting new ideas to existing knowledge) and metacognitive strategies (e.g. setting goals, planning, monitoring progress) (Blumenfeld, Kempler, and Krajcik 2006). The application of these strategies facilitates understanding and retention of learning materials, thereby leading to higher levels of performance (Greene 2015). In this paper, we operationalize this dimension according to the self-regulatory perspective, due to the crucial role of self-regulated learning in higher education settings (Dresel et al. 2015).

Social engagement

Social engagement is defined by Finn and Zimmer (2012) as the extent to which a student adheres to classroom rules, encompassing behaviors such as punctuality and respectful interactions with teachers and peers. Building on this definition, Zhoc et al. (2019) proposed a subdivision into social engagement with teachers and with peers.

Social engagement with teachers. This dimension refers to the interactions with teachers in the learning context, particularly in the classroom. Students who feel supported by their relationships with teachers, demonstrate greater self-confidence and higher performance in their courses (Micari and Pazos 2012). Positive student–faculty relationships are connected with college persistence and completion (Hoffman 2014).

Social engagement with peers. At university, social engagement with peers represents a vital area of academic, social, and emotional life. This dimension covers interactions with friends and peers, which can occur in the classroom or in social spaces such as associations, clubs or accommodations. Recognizing the diversity of peer interactions, Zhoc et al. (2019) proposed two other subdimensions: peer engagement, which pertains to collaboration among peers for learning and knowledge purposes (e.g. group work), and beyond-class engagement, which encompasses students’ interactions outside the classroom (e.g. extracurricular activities at the university). Research shows that the quality of relationships with peers is associated with students’ GPA, persistence in their studies (Goguen, Hiester, and Nordstrom 2010), and adjustment to university life (Maunder 2018).

Affective engagement

Some authors refer to this dimension as emotional engagement, defining it as feelings of enthusiasm and interest versus boredom and anxiety experienced in the classroom (Skinner and Belmont 1993). Others, such as Finn and Zimmer (2012), use the term affective engagement, characterizing it as feelings of belonging and valuing the school. In this paper, we align with the latter definition of affective engagement as the sense of belonging, relatedness, and identification with the university. Previous studies (e.g. Pittman and Richmond 2007) have shown that students more emotionally connected to the university exhibit better psychological and academic adjustment.

Student engagement and academic achievement

Academic achievement is a multidimensional construct that indicates the extent to which a student has successfully performed or attained specific educational goals across various subject areas and learning domains. In school settings, grades are the most typical measure of achievement (Steinmayr et al. 2014). Academic achievement is a widely researched topic given its ability to predict future outcomes, including employability (Byrne 2022).

In most theoretical frameworks (e.g. Fredricks, Blumenfeld, and Paris 2004), engagement emerges as one of the most immediate predictors of achievement. In a sample of pre-university students, levels of attendance, effort, and the quality of cognitive, metacognitive, and self-regulatory learning strategies predicted their university performance (van Rooij, Jansen, and van de Grift 2017). A recent meta-analysis (Wong et al. 2024) revealed a robust correlation between student engagement and academic achievement (r = .33). In their findings, behavioral engagement emerged as the strongest predictor of achievement, followed by cognitive and affective engagement, respectively.

Student engagement and subjective well-being

Most studies exploring the link between student engagement and well-being tend to adopt a hedonic approach, focusing on subjective well-being (SWB) (Wong et al. 2024). SWB is defined as the pursuit of happiness, encompassing both cognitive and affective assessments that individuals make of their own lives (Diener et al. 2009).

The relationship between engagement and well-being is viewed as dynamic and reciprocal (Datu and King 2018). According to Kahu and Nelson (2018), well-being is a mediating mechanism in the relationship between contextual variables (e.g. living far from campus) and student engagement. Alternatively, Upadyaya and Salmela-Aro (2021) propose that high engagement in studies fosters positive development and overall well-being. The meta-analysis conducted by Wong et al. (2024) revealed a substantial correlation between student engagement and SWB (r = .35). The affective, cognitive, and behavioral dimensions were strongly associated with SWB, in that respective order.

Measurement of student engagement

Self-report measures have become the most widely used method to assess student engagement, as they allow the assessment of students’ perceptions rather than solely relying on observable behaviors. Certain dimensions of engagement, such as affective and cognitive, can only be effectively assessed through subjective experiences. However, some concerns about this method are related to variations in conceptualizations of student engagement, particularly its multidimensionality. Consequently, there are several self-report questionnaires developed from different taxonomies (Fredricks and McColskey 2012).

In the last decade, several scales measuring student engagement have emerged for the higher education population. Trolian's (2024) chapter offers a comprehensive overview of these various instruments. Among the most recent scales, the Higher Education Student Engagement Scale (HESES; Zhoc et al. 2019) is, to our knowledge, the only one that captures the growing integration of technology into the learning experience. Additionally, it is one of the few scales that encompass the social dimension of engagement.

Purpose of the study

In the search for a valid and reliable instrument that comprehensively assesses engagement in higher education, this study presents the initial validation of an instrument measuring seven dimensions and sub-dimensions of student engagement, considering as validity criteria the relationships with academic achievement and SWB.

The Engagement in Higher Education Scale (EiHES) was adapted from previous scales (Marôco et al. 2016; Zhoc et al. 2019) and is theoretically based on the engagement model proposed by Finn and Zimmer (2012). Psychometric assessments of the scale will include factor structure, internal consistency, measurement invariance across gender, convergent, and concurrent validity. We hypothesize that there will be:

H1 - Positive and significant associations between engagement dimensions with each other;

H2 - Positive and significant associations between all dimensions of engagement and academic achievement;

H3 - Positive and significant associations between all dimensions of engagement and subjective well-being.

Method

Participants and procedures

A convenience sample of 760 college students from several Portuguese universities participated in the study. The entire sample was randomly split into two independent subsamples, allowing for the execution of exploratory factor analysis (EFA) on sample 1 (n = 378) and confirmatory factor analysis (CFA) on sample 2 (n = 382).

In sample 1, approximately 63% of the participants were female, 35% male, and 2% non-binary, with ages ranging from 18 to 53 years (M = 29.29, SD = 11.22). About 72% were undergraduates, 24% master's students, and 4% Ph.D. students. The participants were enrolled in courses across the following areas: Social Sciences (32%), Engineering and Technology (18%), Humanities (16%), Entrepreneur Sciences (9%), Services (8%), Arts (6%), Natural Sciences (5%), Health Sciences (3%), Physical Sciences (2%), and Education (1%).

Sample 2 comprised approximately 67% females, 32% males, and 1% non-binary. Ages range from 18 to 53 years (M = 29.31, SD = 10.65). About 65% were undergraduate students, 26% master's students, and 9% Ph.D. students. In this sample, students were enrolled in the following fields of study: Social Sciences (28%), Engineering and Technology (15%), Humanities (12%), Entrepreneurial Sciences (12%), Services (11%), Arts (10%), Natural Sciences (6%), Education (3%), Health Sciences (2%), and Physical Sciences (1%). These field of study are based on the classification of the Portuguese Ministry of Higher Education.

The study was approved by the Ethics Committee of the Faculty of Psychology of the University of Lisbon. We contacted program coordinators from 73 higher education institutions across the country and asked for their assistance in distributing the online survey link via email. The survey was completed using Qualtrics. After obtaining informed consent, the students voluntarily and anonymously completed the questionnaires, which took approximately 10 minutes.

Instruments

Engagement in higher education scale (EiHES)

The Engagement in Higher Education Scale (EiHES) was developed based on two other scales. It contains six subscales from the Higher Education Student Engagement Scale (HESES; Zhoc et al. 2019), developed in Hong Kong, and one subscale from the University Student Engagement Inventory (USEI; Marôco et al. 2016), developed in Portugal.

The six subscales adapted from the HESES were: Academic Learning (4 items), Online Engagement (4 items), Social Engagement with Teachers (4 items), Peer Engagement (4 items), Beyond-Class Engagement (4 items), and Affective Engagement (4 items). Additionally, we replaced the Cognitive Engagement Subscale from the HESES for the corresponding subscale from the USEI (5 items). Thus, the full version of the EiHES comprises seven subscales and 29 items, rated on a five-point Likert scale format (1 = Totally disagree, 5 = Totally agree). All the items were in the positive form, so higher scores reflect greater levels of engagement.

In the validation studies of the HESES and the USEI, evidence of construct validity (factorial, convergent, discriminant, concurrent, and criterion) was found, along with internal consistency estimates exceeding .70 (Marôco et al. 2016; Zhoc et al. 2019).

The HESES’ items (Zhoc et al. 2019) were translated into Portuguese by a team of four researchers in the educational psychology field. Each team member initially translated the items individually, after which a consensus approach was employed. During this phase, the team met to decide on the best translation for each item through discussion and agreement.

Flourishing scale

The Flourishing Scale (FS; Diener et al. 2009) comprises eight items designed to assess various facets of positive functioning, including positive relationships, feelings of competence, and a sense of meaning and purpose in life (e.g. ‘I am engaged and interested in my daily activities’). It is a unidimensional measure aimed at capturing the overall construct of subjective well-being. Respondents rate each item on a seven-point Likert scale (1 – Strongly disagree; 7 – Strongly agree).

In Portuguese samples, data from the FS demonstrated robust internal consistency (α ≥ 0.78) and strong convergent validity with similar scales (e.g. Silva and Caetano 2013). In the present sample, the reliability estimate was ω = .91.

Sociodemographic questionnaire

In this study, a sociodemographic questionnaire was developed, which included questions on gender, age, current academic degree, enrolled course, and grade point average (GPA). This latter question was presented as an ordinal variable comprising three response options (1 – below 11.9, 2 – between 12 and 13.9, 3 – above 14). In Portugal, the GPA system ranges between 0 and 20.

Data analysis

First, we conducted an EFA on Sample 1 using the principal axis factoring method and direct oblimin rotation in SPSS, version 29.0. The assumptions of EFA were verified, including the Kaiser – Meyer – Olkin (KMO) measure of sampling adequacy and Bartlett’s test of sphericity. Kaiser's criterion was applied to extract factors with eigenvalues greater than 1. According to Child (2006), we retained items with communalities exceeding .30 and factor loadings higher than .40. Subsequently, parallel analysis was performed to confirm the adequacy of the extracted number of factors, using the ‘paran’ package in R Studio.

On Sample 2, we conducted a CFA to validate the factor structure obtained in the EFA and to compare it with a unidimensional model of engagement. This analysis was performed in R Studio using the ‘lavaan’ package with the Maximum Likelihood Robust (MLR) estimator. To analyze the model fit, the following fit indices were employed: Satorra-Bentler (S-B) χ²/df, comparative fit index (CFI), the Tucker-Lewis index (TLI), the root mean square error of approximation (RMSEA), and the standardized root mean square residual (SRMR). The cut-off for an acceptable model fit is < 5 for χ²/df (Arbuckle 2009), > .90 for CFI and TLI (Bentler and Dudgeon 1996), < .10 for RMSEA and < .08 for SRMR (Hu and Bentler 1999).

The subsequent analyses were conducted using the total sample, encompassing descriptive analyses of the scale (range, means, standard deviations, skewness, and kurtosis), internal consistency, average variance extracted, measurement invariance, and correlations. Internal consistency was assessed using Omega (ω) coefficients (Reise, Bonifay, and Haviland 2013) and convergent validity through the percentage of average variance extracted (AVE), which represents the proportion of variance explained by the factors. AVE values greater than 50% are considered acceptable indicators of convergent validity (Fornell and Larcker 1981). Another indicator of convergent validity was obtained through Pearson correlations between the subscales of engagement.

We assessed measurement invariance by testing and comparing the fits of configural (M0), metric (M1), and scalar (M2) models across genders. Only female and male genders were considered, as the prevalence of non-binary individuals in this study was limited. Equivalence between models was determined through the significance of S-B chi-square (S-Bχ²) differences.

Concurrent validity was evaluated through the strength of correlations between engagement and GPA (Spearman coefficients), and between engagement and SWB (Pearson coefficients).

Results

Exploratory factor analysis

In the initial EFA, eight factors with eigenvalues greater than 1 were extracted, accounting for 58.82% of the variance. However, factor 7 consisted of only one item (Item 3, ‘I rarely skip classes’) and factor 8 comprised two items (Item 19, ‘I am actively involved in university extra-curricular activities’; and Item 20, ‘I am interested in the extra-curricular activities or facilities provided by the university’). Following the recommendation of Costello and Osborne (2005) that three indicators are the minimum required for effective measurement of a latent construct, these three items were consequently removed, and EFA was rerun.

The final solution comprises six factors, collectively accounting for approximately 65.34% of the variance. The KMO measure of sampling adequacy yielded a value of .86, suggesting that factor analysis can provide reliable and distinct factors, as suggested by Hutcheson and Sofroniou (1999). Furthermore, Bartlett’s test of sphericity showed significance (χ² = 5038.43, df = 325, p < .001), indicating that the correlations between variables are significantly different from zero.

The results of parallel analysis using 1000 replications suggested the extraction of six factors, confirming the results from the EFA.

Given the expected interrelatedness of engagement dimensions, crossloadings were considered and items were retained within the factor where higher loadings were observed. All the retained factor loadings were higher than .40.

Factor 1 encompasses four items (Items 9, 10, 11, and 12) and represents the dimension of social engagement with teachers. Factor 2 comprises seven items (Items 6, 13, 14, 15, 16, 17, and 18) reflecting the dimension of social engagement with peers. It’s noteworthy that Item 6 (‘I regularly use email and/or other electronic means, such as WhatsApp, WeChat, and Facebook, to contact friends in my course’) was supposed to belong to the online engagement subscale, but loaded more strongly on this factor due to its content being closely related to a typical activity students engage in with their peers. Factor 3 consists of five items (Items 25, 26, 27, 28, and 29) representing cognitive engagement. Factor 4 includes three items (Items 1, 2, and 4) pertaining to academic learning engagement. Factor 5 comprises four items (Items 21, 22, 23, and 24) representing the affective engagement. Lastly, Factor 6 is composed of three items (Items 5, 6, and 7) representing the online engagement dimension. (Table 1)

Table 1. Exploratory factor analysis results – pattern matrix and variance explained.

	Factor
Items	1	2	3	4	5	6
1. I regularly study on the weekends	−.08	−.00	−.05	.79	−.12	.08
2. I spend a lot of time to study on my own	−.10	.07	.04	.83	−.04	.09
4. I usually come to class having completed readings or assignments	.27	−.00	.17	.47	−.01	−.05
5. Regularly use web-based resources and information designed specifically for the course	.17	−.06	−.00	.32	.07	.62
6. I regularly use email and/or other electronic means (such as WhatsApp, WeChat and Facebook) to contact friends in my course	.04	.63	−.04	−.05	.09	.45
7. I regularly use the internet for study purpose	−.14	.06	.13	.10	.01	.54
8. Online resources (e.g. course notes, free software and materials on the web) are very useful for me	.05	−.02	.15	−.10	−.17	.73
9. Teachers make a real effort to understand difficulties in my work	.86	−.06	.01	.04	−.10	.01
10. Academic staff take an interest in my progress	.87	−.03	.02	.01	−.10	−.05
11. Teachers give helpful feedback on my progress	.88	−.05	−.04	.04	−.08	.01
12. Teachers are usually available to discuss my work	.79	.14	.04	−.17	−.01	.06
13. I regularly work with other students on course areas I have problems	.08	.80	−.01	.07	−.05	.01
14. I regularly get together with other students to discuss courses	.04	.85	.07	−.07	−.01	.01
15. I regularly study with other students	−.05	.75	−.01	.06	−.10	−.13
16. I feel part of a group of students committed to learning	.10	.61	.06	.30	−.01	.00
17. I tend to mix with other students at university	−.12	.75	.02	−.12	−.16	−.05
18. I have made at least one or two close friends at university	−.01	.80	−.07	−.02	.01	.01
21. I really like being a university student	−.06	.01	.06	.12	−.84	−.04
22. University has lived up to my expectations	.17	−.02	.00	−.02	−.80	.01
23. I feel that I belong to the university community	.03	.18	−.02	−.04	−.77	.04
24. I really like being on my campus	.16	.01	−.00	.04	−.82	−.03
25. When I read a book, I question myself to make sure I understand the subject I’m reading about	.04	−.02	.52	.23	−.01	−.03
26. I talk to people outside the school on matters that I learned in class	.10	.26	.66	.05	.08	−.24
27. If I do not understand the meaning of a word, I try to solve the problem, for example by consulting a dictionary or asking someone else	−.04	−.05	.67	−.16	−.04	.20
28. I try to integrate the acquired knowledge in solving new problems	.05	−.10	.82	−.06	−.04	.11
29. I try to integrate subjects from different disciplines into my general knowledge	−.07	−.07	.74	.06	−.15	.14
% of variance	26.26	13.48	10.34	5.91	4.98	4.36

Note. Factor loadings in bold identify the items that belong to the factor.

Confirmatory factor analysis

To assess the validity of the first-order model derived from the EFA (Model A) a CFA was performed. Subsequently, Model A was compared with an alternative unidimensional model, where all items loaded on a single factor (Model B). The results indicated that Model A adequately fits the data, while Model B exhibited poor adjustment, suggesting that the engagement dimensions, although related, were distinct from one another (Table 2).

Table 2. Summary of Goodness-of-Fit Statistics for the CFA.

	S-Bχ^2 (df)	S-Bχ^2/df	CFI	TLI	RMSEA [90%CI]	SRMR
Model A	714.53 (284)	2.52	.90	.88	.07 [.06−.07]	.08
Model B	2628.55 (299)	8.79	.44	.39	.15 [.15−.16]	.14

The factor loadings were higher than .50 for all items (Table 3), except for Item 4. For this item, further modifications may be needed.

Table 3. Standardized Factor Loadings of the Items in the Model A.

	Factor
Items	ALE	OE	CE	SET	SEP	AE
1	.65
2	.90
4	.32
5		.53
6					.67
7		.76
8		.74
9				.87
10				.91
11				.85
12				.73
13					.81
14					.84
15					.72
16					.57
17					.62
18					.59
21						.73
22						.86
23						.82
24						.85
25			.54
26			.53
27			.61
28			.82
29			.87

Note. ALE = Academic Learning Engagement, OE = Online Engagement, CE = Cognitive Engagement, SET = Social Engagement with Teachers, SEP = Social Engagement with Peers, AE = Affective Engagement.

Descriptive statistics of the subscales

Range scores displayed variability in participants’ responses. Mean values indicate a positive trend toward engagement among higher education students, with higher means observed in online and cognitive engagement subscales (Table 4). Skewness and kurtosis values fall within acceptable ranges (Byrne 2010).

Table 4. Descriptives, Reliability and average variance extracted for engagement subscales.

Subscale	N items	Range	M	SD	Sk	Ku	Ꞷ	AVE
Academic learning engagement	3	1–5	3.40	.95	−.27	−.41	.69	44%
Online engagement	3	1–5	4.39	.68	−1.35	2.15	.64	37%
Cognitive engagement	5	1–5	4.09	.68	−.66	.28	.75	40%
Social engagement – teachers	4	1–5	3.22	1.05	−.18	−.63	.91	73%
Social engagement – peers	7	1–5	3.48	.98	−.50	−.47	.87	50%
Affective engagement	4	1–5	3.48	1.07	−.44	−.60	.89	67%

Reliability and convergent validity

The Ꞷ coefficients exhibited a range from .64 to .91, indicating a level of internal consistency from adequate to good. Concerning AVE, values ranged between 40% and 73% (Table 4). Although three subscales presented values below the recommended threshold of 50%, AVE serves as a more conservative indicator. Consequently, the satisfactory Ꞷ values indicate that convergent validity could be considered acceptable.

As a second indicator of convergent validity, the coefficients between engagement subscales were all significant and positive (Table 5).

Table 5. Correlations between study variables.

Subscale	1	2	3	4	5	6	7	8
1. Academic learning engagement	1
2. Online engagement	.30***	1
3. Cognitive engagement	.34***	.40***	1
4. Social engagement – teachers	.24***	.19***	.30***	1
5. Social engagement – peers	.16***	.20***	.24***	.18***	1
6. Affective engagement	.28***	.23***	.37***	.54***	.34***	1
7. GPA	.12***	.04	.08*	.10**	.03	.14***	1
8. Subjective well-being	.31***	.31***	.43***	.34***	.28***	.47***	.03	1

Note. *** p < .001, ** p < .01, * p < .05.

Measurement invariance

To investigate the measurement equivalence across gender, we conducted invariance tests considering the configural (M0), metric (M1), and scalar (M2) models. The configural model (M0) demonstrated a satisfactory fit to the data (S-B χ^{2 (568)} = 1402.74, CFI = .90, RMSEA = .07). Next, the metric invariance models (M1) were tested by constraining all factor loadings to be equal. This model also provided an adequate fit (S-B χ^{2 (588)} = 1414.01, CFI = .90, RMSEA = .07). The results of the Santorra-Bentler chi-square differences between the configural and metric models indicated equivalence (ΔS-B χ² = 12.05, Δdf = 20), p > .05. Finally, the scalar invariance model (M2) was evaluated by constraining the item intercepts to be equal, resulting in an acceptable fit model (S-B χ^{2 (608)} = 1462.81, CFI = .90, RMSEA = .07 However, differences between the metric and scalar models were significant (ΔS-Bχ² = 48.90, Δdf = 20), p < .001), indicating non-equivalence.

Concurrent validity

Table 5 display the bivariate correlations between the study’s variables. Correlations between achievement (GPA) and engagement subscales (academic learning, cognitive, social with teachers, and affective) were significant. These results suggest a positive association between some dimensions of student engagement and achievement, albeit of low magnitude.

Correlations between the engagement subscales and the FS were all significant and moderate, indicating a positive association between students’ engagement and their levels of SWB.

Discussion

In order to identify factors associated with success and well-being in higher education, this study conducted the initial validation of the EiHES in a sample of Portuguese university students. The data suggest favorable psychometric properties of the measure, which can provide valuable information for researchers, decision-makers and practitioners, particularly as an assessment tool in interventions. Results from EFA and CFA consistently indicate a structure comprising six dimensions of engagement: academic learning, online, cognitive, social with teachers, social with peers, and affective. However, the beyond-class engagement subscale from the HESES (Zhoc et al. 2019) disappeared, a pattern also observed in the Italian validation study (Marcionetti and Zammitti 2023). All subscales demonstrated satisfactory to high levels of internal consistency and convergent validity. Regarding measurement invariance across gender, while it was established up to the level of metric invariance, it was not observed when comparing the metric and scalar models. This suggests that males and females may perceive certain items of the scale differently (Schmitt and Kuljanin 2008). Taken together, this data tends to provide empirical support for the engagement model proposed by Finn and Zimmer (2012) and further refined by Zhoc et al. (2019).

The data also confirm the hypotheses concerning the indicators of convergent and concurrent validity. Engagement dimensions were positively related to each other, with correlations typically falling within the moderate range. This confirms the first hypothesis and provides another evidence of convergent validity, suggesting that engagement subtypes belong to the same construct, while also being different enough to be considered distinct dimensions. This pattern of associations is consistent with prior research (e.g. Marcionetti and Zammitti 2023).

The data partially supported the second hypothesis. The achievement of university students was significantly correlated with only certain dimensions of engagement, namely academic learning, cognitive engagement, social engagement with teachers, and affective engagement. However, all the associations were found to be low. According to the literature, associations between engagement and achievement tend to be moderate to high (Wong et al. 2024). One possible explanation for the lower correlations is that GPA scores were assessed as ordinal variables rather than numeric values, limiting the capture of the full variability of grades and consequently reducing the magnitude of associations with related variables (Kampen and Swyngedouw 2000). Additionally, the scale may benefit from further refinement to better capture aspects of engagement more closely linked to academic achievement. For example, Zhoc et al. (2019) found that only cognitive and social engagement contributed to explaining the variance in students’ GPA. Given the expectation that academic engagement subscales would also predict achievement, this could suggest the need to review the content of their items.

The third hypothesis was also supported. All subtypes of engagement showed moderate associations with SWB. Existing evidence suggests a positive association between the engagement of higher education students and their reported happiness (Boulton et al. 2019). This study extends previous findings by demonstrating that the newly components of online learning, along with social interactions with teachers and peers, are also related to the well-being of university students. Moreover, the strength of associations with engagement components is consistent with the literature (Wong et al. 2024). Affective engagement is the dimension most strongly related to SWB, followed by cognitive engagement, social interactions with teachers, academic engagement, and finally social interaction with peers.

Limitations and directions for future research

Some limitations are recognized. Firstly, the sample does not fully represent the higher education student population, although it encompasses individuals from different areas of study. Future research should aim to validate the scale across a more diverse range of higher education institutions, including community colleges and vocational schools, to enhance generalizability.

Another limitation concerns the cross-sectional design of the study and the use of correlational analysis, which prevents us from drawing causal conclusions. While the current study establishes the scale's concurrent validity, longitudinal studies could elucidate the causal relationships between engagement and a range of academic, cognitive, social, and affective outcomes.

A third limitation pertains to the reduced number of items (3 items) in the academic learning and online engagement subscales. Furthermore, the dimension of beyond-class engagement disappeared, as an insufficient number of indicators were retained. Further investigation into the dimensions with fewer items or lower factor loadings, such as academic learning and online engagement, could strengthen the scale. Additionally, exploring ways to reintegrate or redefine the beyond-class engagement dimension may provide a more holistic view of student engagement. This dimension constitutes a significant aspect of university students’ lives and contributes to their academic success and employability (Ribeiro et al. 2023).

Future studies could also use the EiHES to analyze student engagement within the current educational paradigm of inclusion. Additionally, the EiHES could serve as a valuable tool for developing and assessing the effectiveness of psychoeducational interventions aimed at enhancing students’ engagement in their studies. Moreover, future research could explore the cross-cultural applicability of the EiHES, enabling researchers in different countries to assess its sensitivity to diverse cultural contexts.

Conclusion

Our findings suggest that the EiHES is a psychometrically suitable instrument for assessing student engagement in higher education. We underscore the associations of student engagement with GPA and SWB. Employing this newly adapted instrument could offer valuable predictive insights into key academic and personal variables. Beyond the typical dimensions of engagement assessed in most instruments, this scale also encompasses dimensions related to the digital learning experience and social interactions with peers and teachers. Therefore, we believe it provides a richer and broader perspective on the academic, psychological, and social experiences of higher education students. Given its brevity yet comprehensiveness, psychologists and counselors could utilize the EiHES to identify students’ strengths and areas of difficulty. Such utilization allows for the design of interventions targeting areas in need and the evaluation of their effectiveness. Furthermore, the EiHES demonstrates potential for advancing research on factors contributing to enhancing students’ experiences of success in higher education.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

This work received Portuguese national funding from FCT – Fundação para a Ciência e a Tecnologia, I.P, through the Research Center for Psychological Science of the Faculty of Psychology, University of Lisbon (UIDB/04527/2020; UIDP/04527/2020). The work was also supported by the FCT – Fundação para a Ciência e a Tecnologia, I. P, under a Ph.D. grant (2020.06562.BD) assigned to the first author. For the purpose of Open Access, the author has applied a CC-BY public copyright license to any Author’s Accepted Manuscript (AAM) version arising from this submission.

Appendix 1

– Portuguese translation of EiHES’ items

1. Estudo regularmente nos fins-de-semana [I regularly study on the weekends]

2. Dedico muito tempo a estudar por minha iniciativa [I spend a lot of time to study on my own]

3. Normalmente faço as tarefas ou leituras preparatórias para as aulas [I usually come to class having completed readings or assignments]

4. Utilizo regularmente recursos digitais concebidos especificamente para o meu curso [Regularly use web-based resources and information designed specifically for the course]

5. Utilizo regularmente o e-mail e/ou outras aplicações (como o WhatsApp ou o Instagram) para contactar os meus amigos de curso [I regularly use email and/or other electronic means (such as WhatsApp, WeChat and Facebook) to contact friends in my course]

6. Utilizo regularmente a Internet para estudar [I regularly use the internet for study purpose]

7. Os recursos online (ex. materiais do moodle, softwares gratuitos) são muitos úteis para mim [Online resources (e.g., course notes, free software and materials on the web) are very useful for me]

8. Os professores procuram verdadeiramente compreender as dificuldades do meu trabalho académico [Teachers make a real effort to understand difficulties in my work]

9. Os professores demonstram interesse nos meus progressos [Academic staff take an interest in my progress]

10. Os professores dão feedback útil sobre os meus progressos [Teachers give helpful feedback on my progress]

11. Geralmente os professores estão disponíveis para discutir os meus trabalhos [Teachers are usually available to discuss my work]

12. Trabalho regularmente com outros colegas as matérias em que tenho dificuldades [I regularly work with other students on course areas I have problems]

13. Frequentemente discuto assuntos do curso com os meus colegas [I regularly get together with other students to discuss courses]

14. Regularmente, estudo com os meus colegas [I regularly study with other students]

15. Sinto-me parte do grupo de alunos empenhados na aprendizagem [I feel part of a group of students committed to learning]

16. Costumo conviver com outros alunos da universidade [I tend to mix with other students at university]

17. Fiz, pelo menos, um ou dois amigos próximos na universidade [I have made at least one or two close friends at university]

18. Gosto mesmo de ser estudante universitário [I really like being a university student]

19. A universidade tem correspondido às minhas expectativas [University has lived up to my expectations]

20. Sinto que pertenço à comunidade universitária [I feel that I belong to the university community]

21. Gosto mesmo de estar na minha universidade [I really like being on my campus]

22. Quando leio um livro/artigo, questiono-me a mim próprio para ter certeza que entendo o assunto que estou a ler [When I read a book, I question myself to make sure I understand the subject I’m reading about]

23. Eu converso com outras pessoas fora da faculdade sobre as matérias que aprendo nas aulas [I talk to people outside the school on matters that I learned in class]

24. Se não compreendo o significado de uma palavra, eu tento resolver o problema, por exemplo, consultando um dicionário ou perguntando a outra pessoa [If I do not understand the meaning of a word, I try to solve the problem, for example by consulting a dictionary or asking someone else]

25. Tento integrar os conhecimentos adquiridos para resolver problemas novos [I try to integrate the acquired knowledge in solving new problems]

26. Tento integrar as matérias das diferentes disciplinas no meu conhecimento geral [I try to integrate subjects from different disciplines into my general knowledge]