A differential item functioning (DIF) analysis of the mobile phone problem use scale in Indonesian schools with and without smartphone banned policy

Abstract

This study aimed to validate an Indonesian version of the Mobile Phone Problem Use Scale (MPPUS) in a sample of high school students from South Tangerang City, Banten Province,, Indonesia. Throughout this process, we evaluated the scale’s psychometric properties and examined the differential item functioning (DIF) across schools with and without a smartphone ban policy. The study used a cross-sectional survey design with a sample of 622 high school students (Mage = 16.10; SDage = 1.41). There were 312 respondents from schools with a smartphone ban and 310 from schools with no smartphone ban. We used confirmatory factor analysis (CFA), multiple indicators multiple causes (MIMIC) model, and Rasch Rating Scale Model (RSM) to analyze the data. The CFA and MIMIC model results indicated a good fit for the overall model using the pooled sample. Based on MIMIC model, one item was flagged as DIF. Furthermore, RSM analysis demonstrated that the Indonesian MPPUS has adequate psychometric properties. RSM-based DIF analysis identified four DIF items. Therefore, 23 of 27 items on the scale have the same meaning across schools with two different policies. Our findings provide empirical evidence that MPPUS performed well across students from schools with and without smartphone restrictions, despite the fact that the MPPUS was originally designed for the general population. The findings of this study can be used to develop an evidence-based policy for implementing smartphone policies in Indonesian schools.

Keywords:

• Problematic smartphone use
• rasch model
• smartphone ban
• validation

1. Background of the study

It goes without doubt that the growth of mobile phones or smartphones has recently been incredibly vast. According to a report made in 2020, the number has reached up to 3.5 billion (Statista, 2020a). As a huge country in Southeast Asia with 210 million users, Indonesia ranks number one as smartphone users (Statista, 2020b). Furthermore, approximately 21% of smartphone users are between the ages of 12 and 19 or of school age (Digital Influence Lab, 2020). However, the constant availability of smartphones combined with the addictiveness of popular applications can become problematic when people use them in unproductive or even harmful ways (Van Velthoven et al., 2018). Also, high school students were found to be very vulnerable to smartphone use because they spend a lot of time with smartphones (Cha & Seo, 2018; Fischer-Grote et al., 2021; Rachmayani et al., 2020).

When new technologies, such as smartphones, are introduced into schools, they pose a number of challenges. As a result, some governments and schools enact various smartphone ban policies and regulations (Gao et al., 2014). In Indonesia, for example, some schools have policies prohibiting students from using smartphones in school (Baginda et al., 2020; Machmud, 2018a, 2018b). Previous findings from Mollman (2015) stated that the smartphone ban policy in Indonesia was implemented to allow students to focus on their learning while providing ample time for students to interact directly with their peers and teachers. Furthermore, the arguments for a potential ban on smartphones in schools in Indonesia came primarily from teachers, parents, and school administrators who were concerned about the risks that students faced as a result of problematic smartphone use (Machmud, 2018a).

Previous findings indicating that a large number of Indonesian high school students are classified as problematic users have emphasized the negative consequences of problematic smartphone use in Indonesian schools (Nufus et al., 2020; Rachmayani et al., 2020). Another study of Indonesian high school students discovered that roughly 60% of respondents were moderate to severely addicted to their smartphones (Taufik et al., 2020). According to the most recent research, problematic smartphone use has a negative impact and will have serious psychological consequences (see, Busch & McCarthy, 2021). Unfortunately, few studies have been conducted to investigate the smartphone ban policy and its direct link to problematic smartphone use. Current research on smartphone ban policies and problematic smartphone use is frequently conducted in isolation rather than as part of an integrated study.

To name a few, one study investigated students’ perceptions of the smartphone ban using descriptive methods to explain the phenomenon (Machmud, 2018a). Another is a mixed-method study that investigated school principals’ and teachers’ perceptions of the smartphone ban policy, which discovered that the policy’s implementation was not driven by empirical evidence and was inconsistent across schools and areas (Machmud, 2018b). The most recent study by Baginda et al. (2020), suggested that the equality of such a policy across schools should be promoted. Furthermore, the majority, if not all, of the preceding studies were conducted after the smartphone ban was implemented. When this is the case, such studies cannot provide adequate evidence to support smartphone policy in schools.

In contrast to Indonesia, other countries (e.g., China) conducted research before enacting and widely implementing smartphone bans. A comprehensive survey was conducted to compare two school clusters, one of which had a smartphone ban policy and the other did not. The samples were drawn from a variety of school levels, including elementary and secondary schools (Gao et al., 2014). In addition, related research that underpinned the implementation of the smartphone ban policy was also carried out (for example, Zuo, 2021).

One reason for implementing the smartphone policy was to keep students from becoming addicted to harmful smartphone-based activities (Machmud, 2018a, 2018b). In support of this claim, a study conducted by the Australian state of Victoria revealed that the policy’s implementation effectively created a safe and conducive learning environment free of disruptions caused by inappropriate smartphone use (Baker, 2018; Victoria State Government, 2020). However, the study on the implementation of the smartphone ban policy should ideally be preceded by a measurement of the condition and severity of problematic smartphone use (Lepkowska, 2019). In other words, research into problematic smartphone use is critical for strengthening the conceptual framework for enacting the smartphone-ban policy (Wiederhold, 2019). Unfortunately, there have been no such studies conducted in Indonesia to date.

The smartphone-ban policy has recently received increased attention as a key construct in the field of psychology (Bianchi & Phillips, 2005; Busch & McCarthy, 2021; Lepkowska, 2019; Mach et al., 2020). A systematic review of problematic smartphone use can be conducted to decide on a smartphone ban policy (Lepkowska, 2019). In other words, a study like this is critical for solidifying the conceptual framework for enforcing a smartphone ban policy (Wiederhold, 2019).

2. Literature review

2.1. Problematic smartphone use: theoretical framework and measures

Problematic smartphone use is defined as a psychological and behavioral dependence on smartphones characterized by an inability to control one’s mobile phone use, which eventually leads to negative consequences in daily life (Billieux, 2012). Since the late 1990s, this emerging public health issue has been growing (Harris et al., 2020). Furthermore, problematic smartphone use addressed issues such as intolerance, escape from other problems, withdrawal, craving, and negative life consequences in various contexts (Bianchi & Phillips, 2005).

The construct of problematic smartphone use has several conceptual similarities with addiction, including overuse, failure of impulse control, dependence, use in risky and dangerous situations, and potential negative effects of not using smartphones (Harris et al., 2020). However, smartphone addiction falls within a broader category of behavioral addiction. As a relatively new conceptual addition to American psychiatry, behavioral addiction was first recognized by the Diagnostic and Statistical Manual of Mental Disorders in 2010 (DSM-V; Rosenberg & Feder, 2014). Like substance addiction, behavioral addiction is characterized by excessive salience, withdrawal symptoms, conflict, and relapse behaviors (Allred & Atkin, 2020).

Some researchers believe that technological addictions are a subset of behavioral addictions and operationally define technological addictions as non-chemical behavioral addictions involving human-machine interaction. However, regardless of whether excessive use of various forms of technology can or should be labeled an “addiction,” researchers have discovered some evidence for excessive use of technology that can be deemed problematic (Bianchi & Phillips, 2005; Griffiths, 1996; 1998).

Busch and McCarthy (2021) conducted a systematic literature review from 2008 to 2019 on the antecedents and consequences of problematic smartphone use and proposed seven key research questions for the future agenda of research on this topic. One of the questions is “how do different demographics experience problematic smartphone use?”. In parallel, in the context of measurement instruments, a recent systematic review has identified 78 instruments and their underlying conceptual framework on excessive or problematic smartphone use (Harris et al., 2020). Nonetheless, almost none of the instruments measuring problematic smartphone use in Indonesia have undergone adequate reviews (Baginda et al., 2020; Machmud, 2018a, 2018b). The dangers of this behavior are most likely to occur in Indonesia. Unfortunately, such research reports are hard to find, especially with well-established instruments to measure problematic smartphone use. Therefore, the problematic use of smartphones is an important construct to be studied in Indonesia’s context of the smartphone ban policy. This will provide information on whether the use of smartphones by Indonesian students is problematic or not.

In recent years, various instruments for problematic mobile phone use have been developed, and related research was conducted across continents resulting, as noted, in 78 instruments (Harris et al., 2020; Vally & Hichami, 2019). The Mobile Phone Problem Use Scale (MPPUS; Bianchi & Phillips, 2005) is one of the prevailing instruments that are most widely used. Henceforth, research related to adaptation as well as modification of MPPUS has been done in many countries across the globe, such as in China (Hongjuan et al., 2017), Germany (Foerster et al., 2015), Iran (Kalhori et al., 2015), Lebanon (Nahas et al., 2018), Poland (Mach et al., 2020), Spain (de-Sola et al., 2017), and United Arab Emirates (Vally & Hichami, 2019). Unfortunately, none has been made in Indonesia, although this country is categorized as the third of the top three smartphone users in Asia and first in Southeast Asia. In addition, it is important to check whether MPPUS can function well when administered to a specific sample, such as students from schools with a smartphone-banned policy.

2.2. Differential item functioning (DIF) analysis

When validating an adapted instrument, it is critical to consider the absence of bias. Assume bias exists; the item performs differently in different groups with the same latent trait level. In other words, people from different groups will respond to the items differently due to factors other than the latent trait being measured, such as group identity (Baylor et al., 2014; Saggino et al., 2020).

In this context, item bias was referred to as differential item functioning (DIF). Items marked as DIF have the potential to jeopardize the validity of scores if they are not investigated further (Martinkova et al., 2017). Different smartphone policies at school, for example, can be a source of bias. According to the review of literature, no study on this topic has ever been conducted, particularly in Indonesia.

There are currently several methods for detecting DIF. DIF detection methods are classified into two philosophical views: statistical (e.g., factor analysis-based) and measurement models (e.g., Rasch model-based), with each model having different assumptions (Hayat et al., 2021). However, the Rasch model has uniqueness in terms of specific objectivity, additivity, and parameter separation (Wu et al., 2016), where these features assume an absence of DIF (Mesbah & Kreiner, 2012). Given the importance of DIF testing for validating measuring instruments, we believe that in the context of the current study, the DIF analysis procedure should be performed as part of testing the psychometric properties to determine whether the measurement instrument of problematic smartphone use can function well across different groups (school with vs. without smartphone banned policy), even though the instrument was developed for the general population. DIF analysis findings will provide empirical evidence of measurement invariance across different groups. This means that the Indonesian MPPUS is valid to be used regardless of different smartphone-banned policy implementations.

2.3. Rationale of the study

As previously stated, research on problematic smartphone use in the Indonesian context, which underpins the policy’s implementation, is limited. Furthermore, no studies have been conducted on the adaptation of the problematic smartphone use scale. This study reports on the use and adaptation of the problematic smartphone use instrument, the findings of which are expected to benefit future researchers on smartphone use in Indonesia significantly. MPPUS is preferred over all other commonly used scales for measuring smartphone use, as noted previously. The significance of this study stems from the following factors: (1) it is the first validation of MPPUS conducted in Indonesia, (2) it is the first contribution of the Rasch measurement model to validating MPPUS, and (3) it employs DIF analysis.

If the adapted MPPUS is proven to be free from DIF, of course, it can be used for further research to examine differences in the use of problematic smartphones in the sample with and without the smartphone ban policy. This can be done because the instrument has been shown to function well in populations with different group members.

3. Methods

3.1. General background

This study employed a cross-sectional survey research design, with data collected on a single occasion. This design adheres to the fundamental principles of explanatory research by employing a confirmatory approach to test whether differences in school policy affect the measurement of problematic smartphone use. We examined whether MPPUS functions consistently across different School policies regarding smartphone bans. Quota sampling, a non-probability sampling technique, was used to select participants for this study. The ethical approval was granted by the Institute for Research and Community Services (LPPM), Universitas Negeri Jakarta, Jakarta, Indonesia (No. 676/UN39.14/PJ.00/2020). Participants agreed to take part in the study and to have the results published in accordance with ethical approval. Permission was granted from the principal of the school to obtain ethical clearance from the relevant institution.

3.2. Participants

The respondents of this research comprised 622 students (356 male, 266 female) from 20 secondary schools in South Tangerang City, Banten Province, Indonesia. The respondents’ ages ranged from 14 to 18 years (mean age = 16.10; SD = 1.41). There were 312 respondents from schools that banned smartphones and 310 from schools that did not. We expected a balanced sample of 350 respondents from schools with smartphone restrictions and 350 from schools without. However, only 622 people volunteered to participate in this study, with an 82.93% response rate.

3.3. Instrument

In this study, the Mobile Phone Problem Use Scale (MPPUS; Bianchi & Phillips, 2005) was used as the main instrument consisting of 27 items abbreviated as MPPUS-27. The scale was, as noted, initially developed in response to a comprehensive picture of the levels of problematic smartphone use and to understand the factors that influence it (Bianchi & Phillips, 2005; Harris et al., 2020). The main instrument in this study was the Mobile Phone Problem Use Scale (MPPUS; Bianchi & Phillips, 2005), which consisted of 27 items abbreviated as MPPUS-27. A professional translator translated MPPUS-27 into Indonesian for this study. In addition, lecturers in psychology thoroughly reviewed the translated version of MPPUS-27 (henceforth MPPUS-27 INA) to ensure its readability and content accuracy. In contrast to the original version, which used a 10-point Likert scale, MPPUS-27 INA used a five-point Likert scale (“strongly disagree” to’strongly agree’). The five-point Likert scale was also used in an Iranian version of the MPPUS-27 (Kalhori et al., 2015). Cronbach’s alpha (α) of MPPUS-27 INA in this study is 0.88, indicating high internal consistency.

3.4. Data analyses procedure

3.4.1. Confirmatory factor analysis (CFA)

The data analyses of this study were conducted in a two-step process: (1) Confirmatory Factor Analysis (CFA) and (2) Rasch analysis. CFA was used to investigate whether items are unidimensional and appropriate for calibration using a unidimensional IRT model (Hayat et al., 2021). CFA was performed in Mplus 8.4 on the hypothesized 1-factor model. Following that, we conducted DIF testing using the multiple indicators multiple causes model (MIMIC; Cheng et al., 2016; Muthén, 1989).

The robust maximum likelihood estimator was used to estimate the parameters of the MIMIC models. Model fit was evaluated in several ways, including the chi-square (${\chi }^2$) statistics, comparative fit index (CFI), Tucker-Lewis Index (TLI), and root mean square error of approximation (RMSEA). The following cut-offs were used to analyse goodness-of-fit of the model: a CFI > 0.90, TLI > 0.90, SRMR < 0.08, and values of RMSEA less than 0.05, respectively (Schumaker & Lomax, 2016). Because chi-square (${\chi }^2$) statistics are susceptible to sample size; small samples might present significant results even when the fit is acceptable in another fit statistic (i.e., RMSEA and SRMR; Wang & Wang, 2019).

3.4.2. Rasch model

Rasch measurement model (Rasch, 1960) recently received more attention from researchers in various fields dealing with test designs, construct validation, and addressing item bias (Hayat et al., 2021). The basic concept of the Rasch models is that psychological constructs are not directly observable and can only be observed through the manifest responses of persons to a set of items that define the latent construct (DiStefano et al., 2019). Several models in the family of Rasch measurement can accommodate different types of response formats, such as polytomous, dichotomous, or mixed formats (Suryadi et al., 2020). Since the response data of MPPUS-27 INA was in the form of a Likert scale, the rating scale model (RSM; Andrich, 1978) procedure was appropriate for this study. RSM estimates the probability that a person endorses a specific category of the rating scale, given both person’s trait level ($\theta$) and the difficulty of the item (DiStefano et al., 2019).

RSM shares three assumptions: (a) unidimensionality, which refers to the assumption that a single trait underlies a collection of items of a measuring instrument; (b) local independence, which means that the response to one item will be independent of the response to another item; and (c) parallel ICCs, which states that the discrimination is uniform across items (Mair, 2018). The RSM was used to analyze the items of the MPPUS-27 INA using the WINSTEPS statistical program. Person and item parameters were estimated using an unconditional maximum likelihood (UCON) estimator.

To examine the psychometric properties of the scales, the following procedures were performed: (1) unidimensionality assessment using the principal component analysis of standardized residual; (2) an analysis of the local independence using the standardized residual correlation; (3) inspecting item properties, including item location, item-fit statistics using infit and outfit MNSQ statistics, item-to-total correlation, and DIF analysis, and (4) rating scale diagnostics of MPPUS.

4. Results

4.1. Factor analysis

Based on CFA, we found that 1-factor model fitted the data [χ² (324) = 772.327, p < 0.001; RMSEA = 0.047 (90% CI = 0.043, 0.051), CFI = 0.929, TLI = 0.923, SRMR = 0.035] with five fit statistics met the predefined criteria. Although the p-value of the χ² for the model was significant (p < 0.01), this likely happened due to the large sample size. We concluded that the goodness-of-fit for the unidimensional factor model of MMPUS was satisfactory. Loadings on the Indonesian MPPUS ranged from 0.506 to 0.727 at a p < 0.01 significance level. After the 1-factor CFA model fitted the data, we regressed one dichotomous covariate (score 1 for students from schools with restriction and 0 for school without restriction) on the latent factor using MIMIC model.

Results of the MIMIC model indicated that the model fit the data reasonably well [χ² (324) = 796.923, p < 0.001; RMSEA = 0.045 (90% CI = 0.041, 0.050), CFI = 0.930, TLI = 0.924, SRMR = 0.035]. We found improvement in RMSEA, which is smaller than the 1-factor model. Loadings on MIMIC model ranged from 0.496 to 0.727 at a p < 0.01 significance level. Smartphone-banned policy as a covariate has a significant impact on the problematic smartphone use ($\beta$ = −0.269, p < 0.01). Considering that the significant impact has a negative direction, students from schools with code 0 (school without restriction) have higher problematic smartphone use than those with code 1 (school with restriction). Furthermore, the study results showed a significant difference in the mean score of problematic smartphone use of students from schools with or without smartphone restrictions, where students without restrictions (code 0) had a higher mean.

Taking a closer look at the results of the DIF analysis of the items, we found one of the 27 items (item 24) flagged as DIF. Item 24 “Saya telah diperingatkan bahwa saya menghabiskan waktu terlalu banyak untuk bermain handphone. [I have been told that I spend too much time on my mobile phone.]” has a significantly associated with smartphone-banned policy in negative direction ($\beta$ = −0.120, p < 0.01). This implies that participants from schools without smartphone restriction policy (coded as 0) get scores higher on Item 24 than those from schools with smartphone restriction (coded as 1), controlling for the factor of problematic smartphone use.

4.2. Rasch analysis

4.2.1. Unidimensionality

In this study, we used Principal Component Analysis on Standardised Residual (PCAR; Linacre, 1998) to test the unidimensionality of MPPUS-27 INA. Based on this analysis, we found evidence of unidimensionality since the minimum variance explained by the measure was 45.1% (22.1 in eigenvalues unit) above the minimum recommendation of 40% (Holster & Lake, 2016). In addition, we also found that the eigenvalue in the first contrast is 1.6 (in items unit), less than the predefined criteria of 2.0. The eigenvalue less than 2.0 indicates the presence of secondary factors with no more than two items. Based on this evidence, the MPPUS-27 INA showed to be unidimensional.

4.2.2. Local independence

The assumption of local independence states that any set of items does not significantly correlate except by latent trait measured by the items (Edwards et al., 2018). This study used the standardized residual correlation (Linacre, 1998) to examine possible locally dependent items. When using the residual correlation between item pairs, it is specified that the standardized residual correlation between item pairs is never > 0.30 (Saggino et al., 2020). With this criteria, we found one item pair exhibits a local dependence (Item 12 with Item 26) with a standardized residual correlation of 0.35. Considering that only one pair of items is greater than the criteria, the locally dependent item is not strong enough to violate unidimensionality. This is in line with the PCAR result that the secondary factor (multidimensionality) is insignificant (below two items).

4.2.3. Rasch item fit statistics, location, and DIF effect size

To determine how well each item fits the Rasch model, the “infit” and “outfit” mean-squares statistics (MNSQ) were calculated to investigate the extent to which each item follows the model expectation. An infit or outfit MNSQ values in the range of 0.6–1.4 are categorized as acceptable for the Likert scale format (Bond & Fox, 2015). In addition to Rasch fit statistics, an item-to-total correlation was used, which is similar to CTT’s item discrimination indices, with positive values indicating that the items behave as expected (Suryadi et al., 2020).

Using “infit” and “outfit” MNSQ statistics, all MPPUS-27 items were found to be in the acceptable range (0.6–1.4). Furthermore, as shown in Table 1, the item-to-total correlation was found to be in the range of 0.51 to 0.69, indicating that all of the items had positive correlations, as expected. Based on item measure (location), item 9: “Kesibukan saya pada percakapan di handphone menyebabkan saya tidak memperhatikan apa yang sedang saya kerjakan [Sometimes, when I am on the mobile phone and I am doing other things, I get carried away with the conversation and I don’t pay attention to what I am doing.]” was the hardest item to be endorsed (most “difficult”). In contrast, item 24 was the easiest item to be endorsed (most “easy”). The mean person measure was −1.33 logit (SD = 1.18), suggesting that the mean of students’ problematic smartphone use was lower than the mean of item difficulty (zero). Furthermore, the person score distribution of problematic smartphone use ranged from −4.07 to 4.63 logit, exceeding the item difficulty range (−0.84 to 0.82 logit). This indicates a wide range of problematic smartphone use levels among students, ranging from the least problematic to the most problematic.

Table 1. Item parameter and fit statistics of MPPUS-INA

Item	Measure	Infit MNSQ	Outfit MNSQ	Item-to-total correlation	DIF Contrast (N-Y)
Item 9	0.82	1.29	1.38	0.57	0.27*
Item 25	0.59	1.10	1.05	0.66	−0.18
Item 10	0.42	1.15	1.35	0.63	−0.07
Item 4	0.41	1.24	1.12	0.63	−0.05
Item 27	0.32	1.20	1.15	0.60	0.09
Item 8	0.25	1.02	1.26	0.61	0.09
Item 6	0.22	0.92	1.02	0.64	0.07
Item 20	0.14	0.71	0.82	0.66	−0.11
Item 23	0.11	0.77	0.87	0.63	0.19*
Item 12	0.11	1.28	1.18	0.59	−0.05
Item 7	0.10	1.09	1.00	0.63	−0.16
Item 15	0.09	0.83	0.94	0.60	0.17*
Item 11	0.07	0.64	0.76	0.69	0.00
Item 18	0.05	1.02	1.04	0.57	0.00
Item 16	0.03	0.89	0.94	0.61	0.00
Item 13	−0.01	1.34	1.28	0.57	−0.02
Item 21	−0.09	0.75	0.76	0.69	−0.09
Item 22	−0.13	1.15	1.36	0.56	−0.02
Item 17	−0.14	0.80	0.78	0.64	0.00
Item 2	−0.17	0.68	0.74	0.65	0.00
Item 3	−0.19	1.15	1.05	0.59	0.00
Item 1	−0.20	0.71	0.74	0.62	0.07
Item 19	−0.39	0.72	0.83	0.60	0.05
Item 26	−0.39	1.18	1.13	0.53	0.00
Item 5	−0.40	1.24	1.34	0.51	0.13
Item 14	−0.76	1.25	1.16	0.53	−0.13
Item 24	−0.84	1.23	1.12	0.54	−0.27*

DIF analysis was used to determine whether MPPUS is free of item bias. Despite the fact that different groups (e.g., the school with or without a smartphone ban) in the context of this study are at the same levels of the underlying trait, they may respond to an item differently, indicating the presence of an item bias between groups (Saggino et al., 2020). To detect the presence of item bias, the item-trait chi-square was used. A chi-square significance difference between groups (with and without smartphone restriction) and a DIF effect size greater than 0.50 indicate that there is item bias (Yan & Mok, 2012).

Using DIF analysis, we discovered four items with DIF: items 9, 15, 23, and 24. Despite being statistically significant, none of the DIF effect sizes exceeded 0.50. The largest DIF effect sizes are found in item 24 (DIF contrast = −0.27). This means that students with smartphone restrictions will find it more difficult to support item 24 (coded as 1). This finding was consistent with the MIMIC model results, which showed that students who were not restricted from using their smartphones scored higher.

4.2.4. Category endorsement

For each item, the number of endorsements, the observed averages, and the MNSQ statistics of each response for the items of MPPUS-27 INA were inspected (see, Table 2). To interpret the information in Table 2, we use criteria where for each response category, the MNSQ Infit and Outfit values are between 0.5 to 2.0 were acceptable, and no response category which not selected by the respondent indicates well functioning response category (Linacre, 1999).

Table 2. Response categories functioning

Categories	Thresholds	Observed count (%)	Observed average	Infit MNSQ	Outfit MNSQ
0 (Strongly Disagree)	None	8013 (48)	−1.94	0.99	1.03
1	−1.47	2671 (16)	−1.47	0.93	0.84
2	−1.06	2151 (13)	−1.14	0.90	0.86
3	−0.61	3464 (21)	−0.45	1.05	1.18
4 (Strongly Agree)	3.12	432 (3)	2.76	1.10	1.22

The observed response frequencies had positively skewed distributions, as shown in Table 2, with only about 3% of total endorsements falling into the’strongly agree’ category. However, because none of the “infit” and “outfit” MNSQ statistics were greater than 2.0, all response categories fit the model, indicating that the MPPUS-27 INA response set functioned well based on the Rasch measurement model. In other words, the assumption of a uniform threshold range across items is maintained when using RSM.

5. Discussion

The primary goal of this study was to assess the psychometric properties of the MPPUS-27 INA, followed by DIF analysis across schools with and without smartphone restrictions to see if MPPUS worked well across different groups. CFA was also used as a separate analysis from the RSM application for construct validation. We concluded that the MPPUS-27 INA was unidimensional because the 1-factor CFA model fited the data well. This finding was consistent with the original factor structure of MPPUS (see, Bianchi & Phillips, 2005) and the Spanish version of MPPUS (de-Sola et al., 2017), both of which discovered a unidimensional factor structure of MPPUS, despite the fact that other studies discovered a multidimensional structure of MPPUS (e.g., Foerster et al., 2015). We conducted an RSM analysis because the unidimensionality requirement of RSM was met. We discovered significant evidence of a single latent trait measuring problematic smartphone use by using PCAR to test unidimensionality. Based on the Rasch model, we confirm the unidimensionality of the Indonesian version of MPPUS.

Based on local independence testing, we discovered one pair of items with a standardized residual correlation greater than 0.30 (e.g., Saggino et al., 2020), which is Item 12 “Saya telah berusaha mengurangi waktu untuk bermain handphone tetapi gagal” [I have attempted to spend less time on my mobile phone but am unable to] with Item 26 ““Teman-teman saya tidak suka kalau handphone saya mati” [My friends don’t like it when my mobile phone is switched off] (residual correlation = 0.35) According to this finding, the highest correlation (0.35) indicated that those two items shared only about 12% (0.35 0.35) of their variance. As a result, no critical evidence of a violation of the assumption of local independence was discovered (Yan & Mok, 2012).

The Rasch analysis of the MPPUS-27 has several advantages over previous psychometric validation methods based on classical test theory (CTT). The current analysis, in particular, provides item-level information that supplements information from the CTT (Hayat et al., 2021). In addition, we discovered that all of the items fitted the RSM and had a positive item-to-total correlation. As a result, MPPUS-27 INA has a Rasch measurement advantage, such as allowing a non-specialist researcher to use the total MPPUS score because the raw score is a sufficient statistic for the Rasch models (Saggino et al., 2020).

The highest item measure (0.82 logit) is item 9 MPPUS-INA. Because of its content, these items are difficult for high school students to endorse. This is consistent with previous research findings that smartphone use can impact various aspects of daily life, including educational and learning activities (eg, Misra & Stokols, 2012; Shin et al., 2011).

For MPPUS-27 INA, item 24 had the lowest item measure (−0.84 logit). Because teachers, parents, and friends frequently remind students about spending too much time on their smartphones, the item-measured behavior was simple to endorse based on the item content. According to previous research, 42.45 percent of Indonesian students used a smartphone 1–5 hours per day, while 33.06 percent used it 6 hours per day. According to the data, the most significant percentage of smartphone users spent 1–5 hours per day on their smartphones (Machmud, 2018a).

We discovered DIF in four items using Rasch-based DIF analysis: items 9, 15, 23, and 24. They were all statistically significant, and none had DIF effect sizes greater than 0.50. The largest DIF effect sizes were found in item 24 (DIF contrast = −0.27). This implies that Item 24 is more difficult to endorse by students who do not have smartphone restrictions (coded 0). This finding was linked to the MIMIC model based on factor analysis results, which indicated that students who did not have smartphone restrictions would score higher.

This study’s findings add to the theoretical and practical implications of the validated MPPUS-27 INA. First, a smartphone-ban policy can be implemented following a thorough investigation into whether smartphone use is problematic. Second, prior to the implementation of the smartphone-ban policy, a comparative analysis should be conducted. The comparison can be made based on student achievement, grade level, and gender (e.g., Zuo et al., 2021). This suggestion is consistent with the findings of recent studies (Baginda et al., 2020; Machmud, 2018a, 2018b). Finally, the findings of this study can help to strengthen the underlying framework for enacting and enforcing the smartphone ban policy.

As previously stated, research on problematic smartphone use has been conducted in many countries, with various limitations. This study also has several limitations. First, we make no claim that the samples are representative based on probability sampling. Second, no convergent or discriminant validity was demonstrated in the current study. Future research can overcome the limitations before using the 27-item version of the MPPUS. Third, it should be noted that the problematic use is not limited to smartphones but also includes other devices such as iPads and laptop computers. As a result, our findings cannot be applied to the problematic use of other devices. Fourth, although most previous studies using MPPUS included the measurement of user intensity of smartphone use and the purpose of use, this study did not. Another limitation of this study is that the sample was limited to senior high school students. As a consequence, it cannot be generalized beyond the sampled group. This issue is hoped to be considered in future research.

6. Conclusion

In conclusion, applying the RSM, factor analysis, and the MIMIC model to the MPPUS helped confirm the validity and utility of the MPPUS-27 INA. This method provided detailed information about the scale’s psychometric properties. Some DIF items account for a small proportion of total items. The findings of this study should help non-specialists (e.g., students, teachers, and psychologists) in future research to examine the relationship between problematic smartphone use and other psychological constructs. However, future research can look into the MPPUS’s test-retest reliability and convergent validity. Although the MPPUS-27 INA was the focus of this study, the same Rasch analysis procedures that produced sample-free measures could be used to evaluate other psychological measurement scales. Overall, this is Indonesia’s first study comparing samples of students from schools with and without a smartphone ban policy. Moreover, the findings of this study show that, despite being designed for the general population, MPPUS can function well on a specific sample of students from schools with a smartphone ban policy.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The authors received no direct funding for this research.The author(s) reported there is no funding associated with the work featured in this article.

Notes on contributors

Muchlas Suseno

Muchlas Suseno, Senior lecturer of English for Master Program of English Language Education, State University of Jakarta, Jakarta, Indonesia [PhD, Educational Research and Evaluation, State University of Jakarta, 2010].

Bahrul Hayat

Bahrul Hayat, Lecturer at the Faculty of Psychology, Syarif Hidayatullah State Islamic University, Jakarta, Indonesia [PhD, Measurement, Evaluation and Statistical Analysis (MESA), University of Chicago, United States, 1992].

Muhammad Dwirifqi Kharisma Putra

Muhammad Dwirifqi Kharisma Putra, Doctoral student at the Faculty of Psychology, Gadjah Mada University, Yogyakarta, Indonesia [Master’s degree in Psychometrics, Syarif Hidayatullah State Islamic University Jakarta, Jakarta, 2016].

Jonna Karla Bien

Jonna Karla Bien, Director, Office of International Affairs, Polytechnic University of the Philippines, Manila, Philippines [Master’s degree in Clinical Psychology, Polytechnic University of the Philippines, Manila, 2013].

R. Rachmawati

R. Rachmawati, Lecturer at the Faculty of Medicine, Sriwijaya University, Palembang, Indonesia [Master’s degree in Psychology, Gadjah Mada University, Yogyakarta, 2011].

H. Hartanto

H. Hartanto, Lecturer at the Faculty of Psychology, Widya Dharma University, Klaten, Indonesia [Master’s degree in Psychology, Gadjah Mada University, Yogyakarta, 2012].