Psychometric evaluation of the culturally adapted interprofessional socialisation and valuing scale (ISVS)-19 for health practitioners and students in Indonesia

ABSTRACT

We aimed to develop a culturally appropriate psychometrically robust measure for assessing interprofessional socialization for health practitioners and students in Indonesia. The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) were used as guidelines. Our study was organized in three phases (a) translation, (b) cross-cultural validation by evaluating the content validity and internal structure of the translated instrument (i.e. structural validity, internal consistency reliability, and measurement invariances), and (c) hypotheses testing for construct validity. A total of 266 health practitioners and 206 students from various professional backgrounds participated. The Indonesian ISVS-19 was confirmed unidimensional. Content validity evaluation confirmed the inclusion of relevant, understandable items and was comprehensive. Factor analysis supported removal of two items. Configural, metric, and scalar tests confirmed the invariance of the 1-Factor 19-Items model in practitioner and student cohorts. Age was a differentiating factor in both cohorts; length of work was only significant for practitioners, and educational background was significant for students (80% of assumptions were accepted, fulfilling COSMIN requirement for construct validity). The Indonesian ISVS-19 has good psychometric properties regarding content validity, internal structure, and construct validity and, therefore, is a psychometrically robust measure for assessing interprofessional socialization for health practitioners and students in Indonesia.

KEYWORDS:

• Confirmatory factor analysis
• interprofessional education
• interprofessional socialization
• measurement invariance
• psychometric evaluation
• unidimensional

Introduction

A gap exists between health professionals’ education at the pre-qualification stage of training and the practice demands placed on healthcare practitioners when they enter the workplace. Contemporary pre-qualification training focuses on uniprofessional learning experiences for students from the same professional background (Lapkin et al., 2013; Tong et al., 2016; Zwarenstein et al., 2001). In contrast, practice settings require practitioners to be able to work collaboratively with colleagues from a range of different professional backgrounds (Barr & Coyle, 2013; Barr et al., 2000). Thus, to strengthen the global healthcare system, there is an urgent need to promote interprofessional education that prepares the current and future health workforce for interprofessional collaborative practice (World Health Organization, 2010).

To prepare the future healthcare workforce, pre-qualifying students in health professional programs must develop collaborative competencies early in their training (Barr & Coyle, 2013). Providing opportunities for students from different health backgrounds to work collaboratively during their education can promote future collaboration in the clinical phase (during clinical placements) and the workplace (Brewer & Flavell, 2018; Mink et al., 2019; Reeves & Freeth, 2002). To be proactive, interprofessional research has shifted from focusing on qualified health practitioners to studies targeting pre-qualifying students (Findyartini et al., 2019; Soemantri et al., 2020; Tyastuti et al., 2014). Hence, to support greater integration between interprofessional education (IPE) and interprofessional collaborative practice (IPCP), it is important to have an outcome measure that can be used for both practitioners and students.

Background

Three outcome measures have been validated in Bahasa Indonesia: the Collaborative Practice Assessment Tool (CPAT), validated for practitioners (Findyartini et al., 2019); and the Readiness for Interprofessional Learning Scale (RIPLS) and Chiba Interprofessional Competency Scale (CICS29), validated for students (Soemantri et al., 2020; Tyastuti et al., 2014). However, to our knowledge, few equivalent interprofessional outcome measures validated for use by both practitioners and students exist. Moreover, no such instruments are available in Indonesian.

The Interprofessional Socialisation and Valuing Scale (ISVS)-21 (King et al., 2016), a revised version of the ISVS-24 (King et al., 2010), is considered a potentially valuable measure for Indonesia due to its reliability in measuring interprofessional socialization in practitioners and students. In a detailed analysis of IPE measures, the ISVS-24 was the only measure that met the standardized criteria to measure IPE outcomes at three levels of Kirkpatrick’s adapted model (Barr et al., 2000): Level 2a attitudes/perceptions, Level 2b knowledge/skills, and Level 3 behaviour (Oates & Davidson, 2015).

The original ISVS-21, a self-rated measure that includes 21 positively worded items in English, was developed in Canada. The ISVS-21 is unidimensional with a Cronbach’s α of .99 (95% CI). The unidimensionality of ISVS-21 was confirmed with the first principal factor eigenvalue containing 57% of the variance (King et al., 2016). ISVS-21 showed a factoring agreement for the practitioner and student data with intraclass correlation coefficient = .99, 95% CI (King et al., 2016).

Assumptions related to interprofessional socialisation

Age and length of experience

Age and length of experience are relatively complex to discuss as two separate issues, as they are highly correlated. Nevertheless, the two aspects are acknowledged as differing for interprofessional collaboration and practice (Anderson & Thorpe, 2008; Van et al., 2007). For instance, researchers have suggested that older generations of physicians were less engaged in collaborative practice due to their lack of exposure to adopting an interprofessional approach during training (Van et al., 2007). The current older generation of practitioners was generally trained in a uniprofessional manner and has a propensity to ignore the importance of interprofessional collaboration in patient care (Oandasan & Reeves, 2005; Thannhauser et al., 2010). In addition, several programs, such as medicine and dentistry, traditionally trained their students to be self-reliant in delivering care (Oandasan & Reeves, 2005).

Length of work experience affects interprofessional capacity more at initial exposure to practice (Legault et al., 2012). As experience accumulates over time, it reinforces the ability to understand the scope of practice (Fletcher et al., 2007; Horrocks et al., 2002; Legault et al., 2012). A period of 6 months is claimed to be adequate to develop a moderate collaborative relationship (Bradby, 1990). However, increased length of work experience displays two different possibilities. Researchers have found that physicians with extended experience were less likely to share information, expressed limited interaction, and were less willing to collaborate (Lalonde et al., 2011; Van et al., 2007).

Medical dominance

The medical profession has long been perceived as having the highest position in the health system hierarchy, denying the legitimacy of other professions’ evaluations, and having the privilege of controlling their work and managing the work of other professionals (Bollen et al., 2019; Clarin, 2007; Mian et al., 2012). Collaboration between medicine and other professions has been reported as having issues related to lacking trust, perceiving hierarchy when delivering care, delegating responsibilities to those perceived as being at the lower end of the hierarchy, and not adopting ideal collaborative practices in many different settings due to an unwillingness to collaborate (Bollen et al., 2019; Clarin, 2007; Legault et al., 2012; Mian et al., 2012).

To address the gap in IPE-IPCP outcome measurement in Indonesia, we aimed to (a) translate the ISVS-21 into Indonesian, (b) perform cross-cultural validation by evaluating content validity and internal structure of the translated instrument (i.e., structural validity, internal consistency reliability, and measurement invariances), and (c) conduct hypotheses testing based on predetermined assumptions related to the construct of interprofessional socialization (age, length of work, and professional or educational background).

Methods

Procedures

Permission to translate and use the instrument was obtained from the original ISVS-21 developer (King et al., 2016). Following translation, data were collected to evaluate the following psychometric properties: content validity, internal structure (structural validity, internal consistency reliability, and measurement invariances), and hypothesis testing. The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) taxonomy and standards of psychometric properties were used to guide this study (Mokkink, De Vet et al., 2018; Mokkink, Prinsen et al., 2018; Prinsen et al., 2018; Yoon et al., 2021). COSMIN requires instrument validation in the intended users’ context or population to ensure the instrument remains invariant across tested groups. The requirements indicate that if a translation instrument is to be used for practitioners or students, it must be validated on a cohort of practitioners and students (in the new setting). Developing such a measure requires careful consideration as some items may be less relevant for health practitioners yet highly suitable for students (De Vries et al., 2016). Figure 1 provides an overview of the processes adopted in this study.

Figure 1. Study procedures.

Participants

Participants were purposively sampled, with the inclusion criteria of Indonesian practitioners and students from any health professional/educational background, with experience in health-team collaboration with another practitioner/student of different professional/academic backgrounds from their own. The same inclusion criteria were used for the pilot and validation studies. Participants self-identified whether they had previous experience in healthcare-team collaboration.

Participation in the pilot and validation studies was voluntary; all responses were anonymous. Potential participants were identified through online advertisements to health professional/student associations social media (e.g., professional, or discipline-based Facebook groups), and through direct invitations via e-mail to health practitioners and students who were currently in the clinical phase of their study. The survey link was provided online using Qualtrics (Qualtrics, 2022). As the participants were required to have consented before completing the survey, their consent was assumed based on the survey completion. This study’s ethical approval was gained from the Curtin University Australia Human Research Ethics Committee (HREC2021–0274) and Faculty Medicine of Hasanuddin University Ethics Board (170/UN4.6.4.5.31/PP36/2023).

Sample

The suitability of the dataset for factor analysis was assessed according to the Kaiser–Meyer–Olkin (KMO) and Bartlett’s Test of Sphericity (Williams et al., 2010). For the pilot study, COSMIN adequate size was expected with a minimum of 30 participants for each cohort, while for the validation study, COSMIN very good criteria was expected with a minimum of 7 × 21 items = 147 participants, and n ≥ 100 for each cohort (Mokkink et al., 2018).

Translation

Four translators were involved in the translation process; two forward translators who translated the instrument from English to Indonesian and two backward translators who translated the instrument from Indonesian back to English. COSMIN and World Health Organization standards were followed in the translation procedure (Mokkink et al., 2018; World Health Organization, 2012). Both forward translators were native Indonesian speakers who were fluent in English. One was a health professional (who holds a postgraduate degree from an English-speaking country) and was familiar with the questionnaire’s content and terminology. The other was a naive Indonesian-speaking translator who did not have a health professional background. The second translator was unaware of the instrument’s objective, holds a postgraduate degree in English literature from an English-speaking country, is a nationally certified translator, and is a member of the Fédération Internationale des Traducteurs. The two backward translators were native English speakers, fluent in Indonesian, and naïve to the construct to be measured. Both backward translators had doctoral degrees from an Indonesian higher educational institution, one of which was in English Education. The translators were encouraged to emphasize conceptual equivalence rather than a literal verbatim translation of each item.

The translation process began with the forward translators working independently, and then the results of the two translations were compared for agreement. The agreed Indonesian translation was sent to the backward translators, who worked independently, and then the results were compared for agreement. A review of the back-translation involving the researcher and the four translators identified several items requiring revisions to better represent the Indonesian culture without changing the context of the original construct. Eight items: ISVS1, ISVS2, ISVS3, ISVS5, ISVS8, ISVS9, ISVS10, and ISVS19 were returned to the forward translators, and the translation process continued as before for these items. For final verification, an online meeting was held involving the researcher, translators, and six Indonesian health practitioners from five different health professional backgrounds. Several items (ISVS1, ISVS5, and ISVS19) received special attention in this panel meeting, but no wording was changed.

Pilot: content validity evaluation

Content validity reflects the extent to which the instrument’s contents adequately represent the measured construct (Mokkink et al., 2018). In particular, COSMIN guidelines were used in the pilot testing to assess relevance of each item to the purpose of the measurement, comprehensibility of each item, and comprehensiveness of the items in representing the overarching construct (Mokkink et al., 2018; Prinsen et al., 2018). Participants were presented with the Indonesian ISVS and requested to rate each item on relevance and comprehensibility using a 5-point Likert-type scale (5 = strongly agree, 1 = strongly disagree). To further explore the participants’ opinions, those selecting disagree or strongly disagree for comprehensibility were directed to an open-ended question to provide their reasoning for the rating and requested to suggest alternative wording. Comprehensibility was assessed quantitatively with descriptive statistics, and qualitatively using content analysis (Vaismoradi et al., 2013). The results were used as the basis for item revision. To assess comprehensiveness of the measure, participants were asked to propose any topics/items they felt were missing in the instrument.

Validation: internal structure evaluation

In the validation study, participants were presented with the Indonesian ISVS and asked to rate each item using the original instrument’s 7-point Likert-type scale descriptors (6 = to a very great extent, 0 = not at all). The Internal structure evaluation included the following three analytical steps (a) structural validity, analyzed using Exploratory Factor Analysis (EFA), Confirmatory Factor Analysis (CFA), and Multi-group Confirmatory Factor Analysis (MG-CFA); (b) Internal consistency reliability, calculated using Cronbach’s α and McDonald`s Omega, and inter-item correlation; and (c) measurement invariance, analyzed for configural, metric, and scalar invariances. Data analysis was estimated using SPSS v26.0, and AMOS v24.0 was used to conduct confirmatory factor analysis and measurement invariance (SPSS, 2019). The Mahalanobis distance was used to identify outliers (Zijlstra et al., 2011). Missing data were treated with listwise deletion, in which the analysis was run only on observations with complete datasets.

We performed EFA using principal component analysis with varimax rotation (Williams et al., 2010), and to the maximum extent possible, equated the extraction method in the original measure factorial analysis. To perform a systematic procedure and minimize subjectivity in interpreting the results, we applied multiple-decision rules, including Kaiser’s criteria and Scree test (Williams et al., 2010). Kaiser’s criteria define factorial numbers based on the cumulative percentage of variance, represented by components with an eigenvalue >1; the scree test analyses a plot developed from a plotline of eigenvalues. Unidimensionality is further confirmed when the first factor’s loading explains >40% of the variance (Reckase, 1979) or when the eigenvalue in the first factor is five times as high as that in the second factor (Rubio et al., 2007).

The CFA was started by independently testing the initial model (EFA results) against the practitioner and student datasets, which were then verified through MG-CFA. To obtain the optimal model fit, the error-term covariance was created based on the modification index (MI) >20, and, if required, the item with the lowest factor loading was removed with due consideration. There is no definitive cut-off to the application of model fit indices; for our study, COSMIN good fit criteria were used ,i.e., the standard root mean square of the residual (SRMR) <.08, and as a complement, the root mean square error of approximation, RMSEA, cut-off <.06, was also reported (Prinsen et al., 2018). The chi-square minimum difference function (CMIN/df) is expected to be between 1 and 3, with a score <5 regarded as acceptable.

Internal consistency reliability refers to the interrelationship of the observed variables and how well these variables are correlated in measuring the same general concept (Prinsen et al., 2018). The common measurements of correlation reliability are represented with a Cronbach’s α and McDonalds’s Omega to confirm unidimensionality and further confirmed with the inter-item correlations (Hayes & Coutts, 2020; Prinsen et al., 2018). The higher the correlation between items, the higher Cronbach’s α and McDonalds’s Omega scores. A value of .7 is acceptable, with a value ≥ .80 considered high; a value ≥ .95 is undesirable, as a very high value may suggest item redundancy rather than homogeneity (Hayes & Coutts, 2020; Prinsen et al., 2018; Yoon et al., 2021). Inter-item correlation examines the degree to which the score on one item relates to the scores on all other items on a respective scale (Mokkink et al., 2018). The inter-item correlation of a scale is expected to be between .30 and .50.

The final Indonesian ISVS model was tested in stages for configural, metric, and scalar invariances to assess model equivalence for the practitioner and student cohorts. Configural invariance tests a model without constraints, metric invariance applied constraint to observable variables’ factor loadings to be equal across the groups, and scalar invariance applied additional constraint to the metric model with the intercepts set to be equal across the groups (Putnick & Bornstein, 2016). A good model fit is required from each test as confirmation to proceed to the next invariance test. Differences in comparative fit index (∆CFI) of the respective configural, metric, or scalar models were compared to confirmed invariances (Cheung & Rensvold, 2002). The application of restraints to the model is expected to cause a decrease in the fit indices. Thus, a reduction in the value of the comparative fit index (CFI) can be expected, but this decrease should be ≤ .01 to confirm invariances (Cheung & Rensvold, 2002).

Hypotheses testing: construct validity evaluation

Based on the theoretical conceptualization of interprofessional socialization, several elements are believed to influence the construct, such as age, length of work, and medical dominance in patient care (Anderson & Thorpe, 2008; Bollen et al., 2019; Van et al., 2007). Therefore, these influential elements were used as the underlying theoretical concept for the construct validation of this study. Hypotheses were tested using analysis of variance with post-hoc multiple comparisons to identify individual differences between groups on significant results.

H1a.

There is a significant difference in interprofessional socialization for health practitioners across different age ranges (21–30 years; 31–40 years; 41–50 years; 51–60 years; 61–70 years).

H1b.

There is a significant difference in interprofessional socialization for health practitioners across different length of work ranges (1–2 years; 3–5 years; 6–10 years; 11–20 years; 20–30 years).

H1c.

There is a significant difference in interprofessional socialization for health practitioners across different professional backgrounds (dentist, nurse, pharmacist, physician, physiotherapist, public health expert, radiographer).

H2a.

There is a significant difference in interprofessional socialization for health students across different age ranges (16–20 years; 21–25 years; 26–30 years; 30–40 years).

H2b.

There is a significant difference in interprofessional socialization for health student across different educational backgrounds (dentistry, dietetics, health promotion, medicine, nursing, pharmacy).

Results

Pilot: content validity

Thirty-four health practitioners and 22 students from various health backgrounds completed the pilot study, fulfilling the COSMIN adequate requirement for instrument pilot testing for the practitioners, however not for the students (Mokkink et al., 2018; Prinsen et al., 2018). The practitioners and students scored significantly different on relevance of items (mean practitioner = 92.4, SD = 7.6; mean student = 85.1, SD = 7.8; t = 3.5; df = 54; p ≤ .001). No participants responded disagree or strongly disagree, suggesting all items should be retained in the questionnaire.

Comprehensibility of items was perceived as equally clear by both practitioners and students (mean practitioner = 88.9, SD = 9.3; mean student = 84.5, SD = 7.8; t = 1.8; df = 54; p = .073). ISVS1 (aware of preconceived ideas) received negative ratings (disagree or strongly disagree) from 14 practitioners and 11 students, and ISVS9 (sharing research evidence) and ISVS15 (comfortable clarifying misconception) from 6 practitioners. These three items were submitted to a translation panel meeting for review, where a consensus was reached to reword the items. The terms “misconception,” “profession,” and “professional,” were words of English origin and should be replaced with the words “miskonsepsi,” “profesi,” and “professional,” and defined according to The Great Dictionary of Bahasa Indonesia (Ministry of Education and Culture Indonesia, 2016). No equivalent word for “interprofessional” was found in the dictionary, so the panel agreed to use the word “interprofesi” as the equivalent in Bahasa Indonesia. No responses were given related to the comprehensiveness of the instrument. However, one pilot test was considered adequate due to the minimum concern identified by participants in the pilot study.

Validation: internal structure

A total of 266 health practitioners and 206 students participated in the validation study. The sample was above the 10 to 1 ratio of respondents to the number of tested items, fulfilling COSMIN very good criteria, with the number of respondents exceeding 147 for each cohort (Mokkink et al., 2018).

Practitioners’ ages ranged between 21 and 70 years (Mean = 45.9, SD = 8.6) and having worked professionally between 1 and 30 years (Mean = 10.3, SD = 6.6). Most participants were female. The largest group of practitioners was physicians. Students’ ages ranged between 16 and 40 years (Mean = 22.6, SD = 4.0) and a range in length of study between 3 and 10 years (Mean = 4.2, SD = 1.4). Again, most participants were female, with nursing students comprising the largest cohort (see Table 1). The suitability of the two datasets for factor analysis was confirmed with Kaiser–Meyer–Olkin (KMO) indices of .90 (practitioners) and .91 (students), respectively, and Bartlett’s Test of Sphericity for both datasets indicated values of p ≤ .01. No missing data was identified in both datasets.

Table 1. Validation study participant Characteristics.

Practitioner (n = 266)				Student (n = 206)
Characteristics	n (%)	Mean	SD	Characteristics	n (%)	Mean	SD
Gender				Gender
Female	187 (70.3%)	97.2	7.5	Female	153 (74.3%)	93.0	10.1
Male	79 (29.7%)	97.8	7.7	Male	53 (25.7%)	91.7	10.4
Age				Age
21–30 years	6 (2.3%)	100.5	7.8	16–20 years	55 (26.7%)	97.6	9.8
31–40 years	63 (23.7%)	95.8	7.5	21–25 years	127 (61.7%)	90.8	9.5
41–50 years	123 (46.2%)	97.1	7.2	26–30 years	13 (6.3%)	92.4	10.1
51–60 years	62 (23.3%)	98.2	7.8	30–40 years	11 (5.3%)	89.6	12.3
61–70 years	12 (4.5%)	102.3	8.6
Length of work experience				Length of Study
1–2 years	37 (13.9%)	93.5	6.7	3–4 years	151 (73.3%)	93.8	9.7
3–5 years	41 (15.4%)	97.5	7.8	5–6 years	39 (18.9%)	88.0	11.2
6–10 years	79 (29.7%)	97.0	6.9	7–8 years	8 (3.9%)	90.9	10.7
11–20 years	89 (33.5%)	98.5	7.8	9–10 years	8 (3.9%)	95.8	7.6
20–30 years	20 (7.5%)	100.5	7.6
Professional backgrounds				Educational backgrounds
Dentist	9 (3.4%)	97.2	8.0	Dentistry	16 (7.8%)	92.6	5.3
Nurse	39 (14.7%)	99.2	6.3	Dietetics	30 (14.6%)	88.5	11.7
Pharmacist	14 (5.3%)	100.7	8.3	Health promotion	25 (12.1%)	95.8	6.3
Physician	172 (64.7%)	96.9	7.5	Medicine	50 (24.3%)	89.1	9.5
Physiotherapist	19 (7.1%)	97.6	10.5	Nursing	67 (32.5%)	96.4	10.8
Public health expert	7 (2.6%)	93.9	4.3	Pharmacy	18 (8.7%)	91.0	8.9
Radiographer	6 (2.3%)	95.2	5.3

EFA provided 4-factor results for the practitioners’ and students’ datasets. The total variance explained was 54% and 60.7%, respectively. A scree plot generated for each dataset demonstrated one point above the break for both datasets, thus confirming the unidimensionality in both datasets. Unidimensionality was further confirmed in the student dataset, with the first-factor eigenvalues explaining 42.1% of the variance (Reckase, 1979). Although the first-factor eigenvalues of the practitioner dataset explained 35.1% of the variance, both cohorts met the unidimensionality requirement, with the eigenvalue of the first factor being about five times higher than that of the second factor (Rubio et al., 2007). Detailed results on the variance distribution are presented in Table 2.

Table 2. Variance distribution of initial eigenvalues.

Practitioner				Student
Component #	Total	% variance	Cumulative %	Component #	Total	% variance	Cumulative %
1	7.36	35.0	35.0	1	8.84	42.1	42.1
2	1.53	7.3	42.3	2	1.54	7.3	49.4
3	1.44	6.8	49.2	3	1.29	6.2	55.6
4	1.01	4.8	54.0	4	1.08	5.2	60.7

Initial CFA modeling showed that no items were deemed necessary to be deleted for either cohort; all 21 items demonstrated critical ratio (CR) >1.96 at p  < .05, indicating that each item met the validity requirements and reflected the construct. However, some items appeared to have low regression weights; sequentially, the three items with the lowest factor loading in the practitioner dataset were ISVS6, ISVS1, and ISVS7. In the student dataset, they were ISVS7, ISVS1, and ISVS2. Two items, ISVS1 (aware of preconceived ideas) and ISVS7 (comfortable in speaking out), were among the three items least endorsed by both cohorts (see detailed CFA results in Table 3).

Table 3. Confirmatory factor analysis results.

Item#	Practitioner			Student
	Estimate*	CR**	p	Estimate*	CR**	p
ISVS1	.32	4.76	<.001	.41	5.47	<.001
ISVS2	.54	7.51	<.001	.46	6.09	<.001
ISVS3	.62	8.37	<.001	.71	9.05	<.001
ISVS4	.58	7.96	<.001	.69	8.77	<.001
ISVS5	.45	6.44	<.001	.62	8.07	<.001
ISVS6	.31	4.59	<.001	.51	6.71	<.001
ISVS7	.32	4.78	<.001	.40	5.39	<.001
ISVS8	.37	5.39	<.001	.59	7.65	<.001
ISVS9	.60	8.13	<.001	.73	9.24	<.001
ISVS10	.54	7.52	<.001	.68	8.66	<.001
ISVS11	.68	8.95	<.001	.60	7.85	<.001
ISVS12	.65	8.66	<.001	.68	8.68	<.001
ISVS13	.56	7.77	<.001	.64	8.20	<.001
ISVS14	.70	9.14	<.001	.65	8.32	<.001
ISVS15	.52	7.26	<.001	.56	7.31	<.001
ISVS16	.65	8.63	<.001	.60	7.77	<.001
ISVS17	.70	9.09	<.001	.72	9.16	<.001
ISVS18	.63	8.46	<.001	.70	8.95	<.001
ISVS19	.69	8.98	<.001	.76	9.53	<.001
ISVS20	.57	7.82	<.001	.65	8.37	<.001
ISVS21	.59	-	-	.65	-	-

*Standardised estimates; **CR = Critical ratio.

The initial 1-Factor 21-items solution met the COSMIN requirement for a good model fit with SRMR in practitioner and student cohorts < .08 (SRMR = .069 and SRMR = .072, respectively); and acceptable CMIN/df in both datasets (χ2 (189) = 592.66, CMIN/df = 3.14 for practitioners, and χ2 (189) = 620.15, CMIN/df = 3.28 for students; see Figure 2A). Because it is predicted that the measurement invariance tests will erode the model fit indices at each stage due to the applied constraints (Putnick & Bornstein, 2016), improvements on the other indices were deemed necessary to optimize both datasets for invariance tests. There were five covariances with MI > 20, three associated with ISVS18 (comfortable initiating discussion), and two associated with ISVS15 (comfortable clarifying misconception).

Figure 2. Initial and final models comparison.

Based on these findings, several iterations of applying covariates and removing items were conducted to obtain the best-fit indices while retaining as many items as possible. This was done by performing alternative modeling of all or some of the five covariances between error terms, combined with deleting some or all of the four items: ISVS1 and ISVS7 (which had the lowest factor loadings), and ISVS15 and ISVS18 (which had the most error terms of correlations). The most improved model fit was obtained by removing ISVS1 and ISVS15 (retaining 19 items) and generating covariances between three correlated error terms (see Figure 2b). Improvements in the fit index profiles for both datasets are provided in Figure 2. The final 1-Factor 19-items model was confirmed as the best factorial solution of Indonesian ISVS-19 and used for further analysis.

CFA for the 1-Factor 19-items model in the practitioner dataset indicated a good model fit with SRMR = .061, χ² (149) = 429.73, and CMIN/df = 2.88. A similar model was then applied to the student dataset and provided a good model fit with SRMR = .067, χ² (149) = 459.94, and acceptable CMIN/df = 3.09. These good fit indices confirmed that conducting an MG-CFA is appropriate (Putnick & Bornstein, 2016).

Using the final model, MG-CFA was performed to confirm the model’s fit across the two groups. The model fit was good with SRMR = .061, RMSEA = .065, χ² (298) = 889.74, CMIN/df = 2.99. These good fit indices confirmed that evaluating measurement invariance is appropriate (Putnick & Bornstein, 2016). The Cronbach’s α for practitioners was .89 and .92 for students (95% CI), whilst the McDonald’s Omega was .88 and .92, respectively. The inter-item correlation was within the expected range (practitioner, r = .32; student, r = .40).

Measurement invariances were tested for the practitioner and student groups simultaneously, and therefore, the resulting fit indices referred to the group data/not individual datasets and will be reported accordingly. Configural model showed a good model fit with SRMR = .061, RMSEA = .065, χ² (298) = 889.74, CMIN/df = 2.99. The results indicated that the items tested do not differ across groups in terms of the structural modeling of 1-Factor 19-items. As configural invariance was achieved, the requirement for testing metric invariance was met (Cheung & Rensvold, 2002).

Metric model demonstrated a good model fit, with SRMR = .067, RMSEA = .065, χ² (316) = 936.95 (CMIN/df = 2.97). The ∆CFI between the configural and metric models = .008, confirming metric invariance (Cheung & Rensvold, 2002). The results indicated that the items tested do not differ across groups in terms of factor loadings and supported the testing of scalar invariance.

Scalar model showed a good model fit with SRMR = .069, RMSEA = .065 and χ² (335) = 992.67 (CMIN/df = 2.96). The ∆CFI between the metric and scalar models = .010, indicating no significant differences in the item factor loadings and intercepts across groups; scalar invariance was achieved (Cheung & Rensvold, 2002). As predicted, imposing constraints on factor loadings and intercepts can cause a decrease in the fit indices, which was demonstrated by SRMR slightly increasing from .061 to .069. The measurement invariance indices are presented in Table 4.

Table 4. Measurement invariance results.

Full Model Comparison	CMIN/df	CFI	∆CFI	SRMR	RMSEA	Invariance
Unconstrained	889.74(298) = 2.99	.841	-	.061	.065	Yes
Metric Invariance (measurement weights)	936.95(316) = 2.97	.833	.008	.067	.065	Yes
Scalar Variance (measurement intercepts)	992.67(335) = 2.96	.823	.010	.069	.065	Yes

CMIN/df = Chi-square minimum discrepancy function, df = Degree of Freedom; CFI = comparative fit index, ∆CFI = Differences of CFI, SRMR = standard root mean square of the residual, RMSEA = root mean square error of approximation.

Hypotheses testing: construct validity

No outliers were identified in the practitioner data set; two significant outliers were removed from the student dataset and not included for hypotheses testing (practitioner, n = 266; student, n = 204). As both practitioner and student datasets were normally distributed, a parametric analysis of variance test was performed for construct validity. The mean differences were set to be significant at .05 level.

Significant differences were found in the practitioner dataset for age-related variables, F(4, 261) = 2.53, p = .04 (H1a accepted), and for variables related to length of work experience, F(4, 261) = 4.07, p < .01 (H1b accepted). The mean scores of practitioners aged 61–70 years were significantly different from those aged 31–40 years and 41–50 years; and the mean score of practitioners with 1–2 years of work experience was significantly different from those with work experience of 3–5 years, 6–10 years, 11–20 years, and 20–30 years. No significant difference was found in the practitioners’ dataset for variables related to professional backgrounds, F(6, 259) = 1.29, p = .26 (H1c rejected).

Significant differences were found in the students’ dataset for variables related to age, F(3, 200) = 6.6, p < .001 (H2a accepted), and for variables related to educational backgrounds, F(5, 198) = 5.84, p < .01 (H2b accepted). The mean score of students aged 16–20 years was significantly different from those aged 21–25 years; the mean score of health promotion students was significantly different from that of dietetics, medicine, and pharmacy students; and the mean score of nursing students was significantly different from that of dietetics and medicine students. Detailed post-hoc results with LSD tests are provided in Supplementary Table . Four of the five proposed hypotheses were accepted (80%), fulfilling the COSMIN requirement of >75% acceptance of priori hypotheses.

Discussion

We aimed to translate and conduct a cross-cultural validation of ISVS-21 in Bahasa Indonesia for use with both practitioners and students by examining the psychometric properties of the culturally adapted measure. The 1-Factor 19-Items solution was considered the best model for Indonesian ISVS. Two items, ISVS1 and ISVS15, were identified in internal structure analyses as the sources of discrepancy across cohorts and thus removed from the instrument. Possible factorial differences between healthcare practitioners and students were recognized in the earlier version of ISVS-21 (King et al., 2016), and in another study (De Vries et al., 2016), which supported the removal or rewriting of these two items.

The development of these two items requires a deeper cultural and linguistic understanding. ISVS1 (aware of preconceived ideas) was identified in our pilot study as confusing items (not comprehensible) and in need of rewording. However, our ISVS1 rewording did not make this item more acceptable to practitioners and students in the validation study; ISVS1 was identified as the item with the second lowest factor loading in both datasets; see Table 3. This finding was similar to that reported in the original ISVS-21, where ISVS1 had the second lowest mean loading (King et al., 2016). Not only was it problematic in our study but also in other ISVS-21 cross-cultural studies in Germany (Mahler et al., 2022), Spain (González-Pascual et al., 2022), and Australia (Vari et al., 2021). Participants’ disagreement over the importance of including ISVS1 reflects difficulties with the concepts implied in this item (aware of preconceived ideas).

A practitioner’s professional identity is believed to begin to form at least 6 months after clinical exposure (Bradby, 1990). Our study involved participants with at least 1 year of experience working in a team environment with different healthcare professionals. Over this time, the participants’ perceptions, attitudes, and beliefs regarding teamwork have shaped their preconceived ideas. Many previous studies confirm that prejudice is evident in healthcare and mostly expressed concerning professional role stereotypes and medical dominance in care (Bollen et al., 2019; Clarin, 2007; Legault et al., 2012; Luetsch & Rowett, 2016; Mian et al., 2012).

Conversely, ISVS15 was discarded because the item correlated with many error terms with MI > 20 due to significant differences in the responses to this item given by participants in the two cohorts. However, the two error terms correlated with ISVS15 were only detected in the practitioners’ dataset, indicating construct overlaps between related items in this cohort and not in the students’ dataset. The findings in our study regarding ISVS15 were consistent with results from one earlier study where this item was suggested for rewording to improve the value of the instrument (De Vries et al., 2016).

Several attempted modeling alternatives indicated that more than the application of correlation between error terms with MI > 20 was needed to make the 1-Factor 21-item model pass the eroded constraints that would impact subsequent measurement invariance tests. Removing ISVS1 and ISVS15 significantly improved the model fit indices for MG-CFA and subsequent tests (see Figure 2).

The Indonesian ISVS-19 was also confirmed to have a unidimensional structure, which is consistent with the original instrument (King et al., 2016) and with previous studies involving cross-cultural validation of the ISVS-21 (González-Pascual et al., 2022; Mahler et al., 2022). The unidimensionality of Indonesian ISVS-19 was confirmed by combining results of exploratory and confirmatory factor analyses, which were then corroborated by excellent Cronbach’s α, McDonalds omega, and inter-item correlation scores for both groups (Cronbach’s α was .89 and .92, McDonald’s Omega was .88 and .92, and r were .32 and .40 for practitioner and student datasets, respectively). These scores reaffirmed the instrument’s unidimensionality and rejected the possibility of redundancy (Hayes & Coutts, 2020; Prinsen et al., 2018). Collectively, these results suggest that the 19 items included in the instrument were strongly related to one construct.

Measurement invariance analyses indicated that the Indonesian ISVS-19 met configural, metric, and scalar invariances, indicating that both practitioners and students agreed with the factorial structure of the Indonesian ISVS-19, the unidimensionality, the number of items included, and the meaning of the constructs underlying the Indonesian ISVS-19. The mean scores of both cohorts were expected to be comparable when assessed using the exact model, thereby enabling the identification and comparison of the student level of interprofessional development in their training (IPE) to the practitioners’ level of collaborative practice in the workplace (IPCP). However, to our knowledge, there has yet to be a study on the psychometric properties of the adapted ISVS-21 using invariance testing; thus, we have no study to use as a comparator.

Variables related to age and educational background are determinants for interprofessional socialization among students. Health practitioners confirmed similar assumptions regarding age and length of work experience. However, professional background was a non-determinant factor for practitioners. COSMIN requirement for hypotheses testing was met, with 80% of the tested assumptions being accepted (Prinsen et al., 2018).

Limitations

The series of analyses performed for the exploratory, confirmatory, and multi-group confirmatory factor analysis used the same datasets; thus, it may not fully ensure the independence of the tests. Despite this limitation, the CFA and MG-CFA provided valuable psychometric data of the ISVS that can be tested in future studies with independent samples and by independent researchers.

Physicians dominated the practitioner sample (64.7%). Moreover, most practitioners involved in this research were trained in a uniprofessional environment. However, understanding and experience regarding interprofessional collaborative practice varied widely among practitioners, considering the number of years working in healthcare. As for students, those involved had an average length of study of 3–4 years (73.3%) but were in the first year of their clinical placement program. This limited understanding and experience of interprofessional socialization may have led to biased responses.

Conclusion

These findings suggest that the Indonesian ISVS-19 has good psychometric properties regarding content validity, structural validity, internal consistency reliability, measurement invariance (i.e., the internal structure), and hypotheses testing. TheISVS-19 has demonstrated robust psychometric qualities for assessing interprofessional socialization for health practitioners and students in Indonesia.

Disclosure statement

The authors have no relevant financial or non-financial competing interests to report.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/13561820.2023.2285020

Additional information

Funding

This study is funded by the Australia Awards Scholarship and Curtin University Higher Degree Research Scholarship.

Notes on contributors

Bau Dilam Ardyansyah

Bau Dilam Ardyansyah, MD., MBSc., MHPE, is a lecturer at the Medical Faculty of Hasanuddin University and a PhD candidate at Curtin University Australia. She obtained her Master of Health Profession Education from Maastricht University, the Netherlands. Her areas of Interest include interprofessional education and collaborative practice, instrument psychometric evaluation, student learning and assessments, and curriculum development.

Reinie Cordier

Reinie Cordier is a Professor at the Faculty of Health & Life Sciences, Northumbria University, UK, the School of Allied Health, Curtin University, and the University of Cape Town, South Africa. He has a track record of conceptualizing and testing the efficacy of psychosocial interventions for children with behavioural and emotional disorders. Theoretically, his main research interests lie at the intersections between health and social care and phenomena specific to different population groups. The overarching theme of his research is promoting the social inclusion of children with developmental disorders, developing evidenced-based psychosocial interventions, measurement, and instrument development.

Margo L Brewer

Margo Brewer is an Associate Professor at the School of Allied Health, Curtin University, Australia. She has held several academic leadership positions at Curtin University since 2002 and is currently the Director of Strategic Projects at the Curtin School of Allied Health. Her vital focal areas are interprofessional education, clinical education, student resilience and wellbeing, assessment, and academic leadership. She is an inaugural member of the Curtin Academy, the inaugural Interprofessional Education Lead for the Australian and New Zealand Association for Health Professional Educators, and the current Chair of the Australasian Interprofessional Practice and Education Network.

Dave Parsons

Dave Parsons, PhD is a current Lecturer and Early Career Researcher in the School of Allied Health at Curtin University. He is also the Allied Health Research Lead at St John of God Public and Private Hospital, Midland. The role is responsible for building research skills, capacity, and culture within the Allied Health professions. Responsibilities include consulting and supporting allied health providers on their research projects, assisting with student supervision, providing advice and input into the hospital research strategy, and building relationships between professions (including medical and nursing staff) to drive high-impact research.