The construct validity and reliability of the Motor Development List for the assessment of motor skills in children with profound intellectual and multiple disabilities: The next step?

ABSTRACT

Background: There are no psychometrically validated instruments available to measure motor skills of children with profound intellectual and multiple disabilities (PIMD). In this study, the construct validity and reliability (Rho) of the Motor Development List for children with PIMD (MDL-PIMD), a proxy-reported measure, was investigated. Methods: The MDL-PIMD was filled in by parents of 52 children with PIMD every six months over a period of two years. The items of the original MDL were categorised into subscales. A Mokken scale analysis for polytomous items was conducted for five subscales (Posture, Locomotion, Reaching, Grabbing and Facial Actions). Results: Several items were deleted to improve scalability. All scales showed good scalability (range scale H .66-.78) and reliability (range Rho .92-.98). Conclusions: The MDL-PIMD seems to be a valid and reliable instrument for assessing the motor skills of children with PIMD, although more research is needed to confirm the factorial structure.

KEYWORDS:

• Profound intellectual and multiple disabilities
• assessment
• motor skills
• Mokken scale analysis
• validity
• reliability

Motor behaviour consists of all movements of the body, both involuntary and goal-directed. The development of these movements occurs across the entire lifespan (Adolph & Franchak, 2017). Several studies have shown that in children with and without disabilities, motor development is linked to development in other areas, such as perception, cognition, verbal performance, social-emotional development and language (Adolph & Joh, 2007; Houwen et al., 2019; Iverson, 2010). For example, learning to crawl allows children to explore on their own, resulting in more emotional independence (Adolph & Joh, 2007). In addition, several studies have shown that impaired motor development may have negative consequences in all other developmental areas. For example, Houwen and colleagues (2016) found that developmental domains are strongly related in children with disabilities. Pennington et al. (2004) and Yamauchi et al. (2019) found a relationship between motor development and other developmental domains, such as cognition or language skills, in children with disabilities such as Down syndrome or cerebral palsy. This relationship between developmental domains is possibly even stronger when impairments are more severe (Houwen et al., 2016). According to Adolph and Franchak (2017), motor skills are usually acquired during infancy and can be categorised into four closely related action systems: posture, locomotion, manual actions and facial actions. Postural control comprises all motor skills focused on obtaining and maintaining the balance of the body. Locomotion refers to motor skills used for mobility, such as crawling and walking. Manual actions comprise all movements by the hands and arms. Typically, development of manual action starts with spontaneous movement and develops towards more complex behaviour, such as reaching, grasping, exploring objects, and using tools. Finally, facial actions are movements that are driven by auditory and/or visual input. They comprise basic motor actions such as swallowing, sucking and chewing, as well as more complex activities such as conveying emotions with facial gestures (Adolph & Franchak, 2017). In typically developing children, motor development proceeds through multiple stages resulting in the achievement of several motor milestones (e.g., lifting the head, crawling, walking). Although for most children motor development proceeds in a similar pattern, the tempo and order of motor development can vary between children; moreover, children can master motor skills in various orders, skip skills, or revert to earlier skills (Adolph & Franchak, 2017; Hadders-Algra, 2010, 2018). Children with profound intellectual and multiple disabilities (PIMD) are characterised by severe or profound motor disabilities and profound intellectual disabilities. Many of these children also have other impairments and general health problems, such as reflux disease, respiratory problems or epilepsy (Van Timmeren et al., 2016). Profound intellectual disability is often defined as an intelligence quotient (IQ) below 20–25 points or a developmental age below 24 months. The assessment of intellectual disabilities in children with PIMD is challenging as there are no standardised tests to provide a valid estimation of IQ in this group (Maes et al., 2021). Moreover, for very young children, a developmental age below 24 months does not provide sufficient information about the severity of the disability, and little is known about how these children will develop in the future. Although all children with PIMD are characterised by complex and intertwined disabilities and are fully dependent on others in all aspects of daily life, they still form a heterogeneous group (Nakken & Vlaskamp, 2007; Van der Putten et al., 2017; Van Timmeren et al., 2016). Motor skills interventions could be particularly beneficial for children with PIMD (Houwen et al., 2016; Van Alphen et al., 2019), and in this respect, a valid and reliable assessment of their motor skills is paramount to be able to implement the appropriate intervention at the right time and evaluate its effects. Psychometrically validated assessment instruments that are adapted to the disabilities and needs of children with PIMD are scarce (Carnaby, 2007; Wessels et al., 2021). To our knowledge, there are no instruments specifically focused on measuring motor skills in children with PIMD. However, the use of instruments developed for other target groups to monitor the development of children with PIMD may be problematic and also result in the provision of inappropriate interventions (Mensch et al., 2015; Van der Putten, 2011; Visser et al., 2013; Visser et al., 2017; Wessels et al., 2021). For example, most instruments measuring motor ability use relatively large increments in difficulty of the items and do not allow for the fine-grained measurement needed for the usually small increments in the motor development of children with PIMD. Furthermore, many instruments assume the presence of certain abilities (e.g., visual or cognitive abilities) that are often impaired in people with PIMD. Moreover, many tests (e.g., Gross Motor Development-2; Ulrich, 2013) assume an order of development in skills that is typically found in children without disabilities. While it is unclear whether motor development proceeds in a similar order and pace in children with PIMD, several studies have suggested that the tempo and order of development in multiple developmental domains may be different in children with PIMD compared to people without disabilities (Van Braeckel et al., 2010; Visser et al., 2017). Thus, to identify delayed or diverging developmental pathways of motor development in people with PIMD, psychometrically validated instruments are needed. An instrument measuring motor ability that might be applicable in children with PIMD is the Motor Development List (MDL). The MDL is a proxy-reported measure that is derived from a checklist of motor skills from the Portage Program, which is an early childhood intervention program for children from 0 to 6 years old with special educational needs (Hoekstra et al., 2011). The MDL has several characteristics that are considered important in assessing motor ability in children with PIMD. The list is filled in by someone who knows the child well (e.g., a parent) and it allows for fine-grained measurement, as the increments in difficulty are relatively small. Moreover, the scoring options are “yes” ( =  2, child has obtained the skill), “sometimes” ( =  1, child shows the skill sometimes or with support) or “no” ( =  0, child has not obtained the skill). This allows for the identification of small improvements in motor ability. The original MDL consisted of one scale with 145 items that could be summed into a total score, with a maximum score of 290, which meant the child had mastered all 145 motor skills. Hoekstra et al. (2010) also established an order of difficulty in the 145 items of the MDL and concluded that the MDL was a reasonably fitting scale. In the current study, an adapted version of the MDL was devised for children with PIMD (MDL-PIMD). The aim of the current study is to assess the reliability and construct validity of the MDL-PIMD. The results of this study could support the more adequate assessment of the motor development of children with PIMD. Ultimately, this could lead to support that can be adapted to the abilities and needs of these children.

Method

Design

The data analysed in this study were collected as part of the project “Opvolging Jonge Kinderen met een Ontwikkelingsvertraging” (OJKO; Monitoring Young Children with Developmental Delays) (Dhondt et al., 2022; Van keer et al., 2017). The project used a single group repeated measures design to monitor the motor development of young children with PIMD. Using the MDL-PIMD, motor skills were measured every six months, with a maximum of five repeated measures. This study used the repeated measures data of the OJKO project to assess the construct validity and reliability of the MDL-PIMD for the assessment of motor skills in young children with PIMD.

Participants

Children were recruited for this study through various institutions (e.g., hospitals, early intervention teams, parent groups). They were included in the study if they were expected to meet the criteria of PIMD later in life, as described by Nakken and Vlaskamp (2007). The inclusion criteria were:

• Children aged between six months and five years.

• Children with a severe or profound intellectual disability, defined as an estimation of cognitive functioning as half or below half of the chronological age based on the Tandem List (Dutch: Tandemlijst). This list provides a rough estimation of children’s developmental age and is specifically designed for young children with developmental delays (Stadeus et al., 1994; Vig & Sanders, 2007).

• A severe or profound motor disability, defined with the use of the Gross Motor Functioning Classification System (GMFCS). Children were included if they functioned on levels III, IV or V on the GMFCS (Palisano, 2007). We used the GMFCS to identify gross motor functioning by conducting a GMFCS classification of the corresponding age group.

• Written informed consent from the parent or legal representative of the child with PIMD.

• Children with additional impairments or health problems were included in the study.

The aetiology of the disabilities was diverse and included acquired brain injury, severe seizure disorders, genetic syndromes or metabolic diseases. There were 52 children included in this study. The sample consisted of 25 boys (48.1%) and 27 girls (51.9%). The median age at the first measurement was 3.1 years (sd = 0.9). The GMFCS levels, additional impairments and the health problems of children are described in Table 1.

Table 1. Characteristics of the participants at the first measurement (n = 52).

GMFCS level
III	6 (11.5)
IV	12 (23.1)
V	34 (65.4)
	Yes n (%)	No n (%)	Unknown n (%)
Auditory impairment	3 (5.8)	47 (90.4)	2 (3.8)
Health problems
Seizure disorder	30 (57.7)	21 (40.4)	1 (1.9)
Gastrointestinal problems	23 (44.2)	29 (55.8)	0 (0.0)
Respiratory problems	15 (28.8)	37 (71.2)	0 (0.0)
Heart problems	3 (5.8)	49 (94.2)	0 (0.0)
Feeding tube	22 (42.3)	30 (57.7)	0 (0.0)

Instrument: the Motor Development List – profound intellectual and multiple disabilities

The MDL-PIMD is an adapted version of the MDL checklist from the Portage Program. Hoekstra et al. (2010) showed that the validity and reliability of the original MDL for typically developing children and children with special educational needs was acceptable. All items are scored as 0 (the child has not yet acquired the skill), 1 (child has partly or sometimes exhibits the skill), or 2 (the child has acquired the skill). The MDL-PIMD consists of 145 items that can be summed to calculate a total score for motor ability (range 0–290), and it can be filled in by a proxy such as a parent or a care professional. Several adaptations were made to the MDL to make it applicable to children with PIMD. The original MDL used a stopping rule that meant scoring ceased after 15 items were scored negatively. As it is unclear whether the progression of motor development in children with PIMD is similar to that of typically developing children, these stopping rules were removed and the proxies were asked to score all items (Van der Putten, 2011). As children with PIMD have considerable impairments, items related to complex skills were removed (e.g., Item 220, “Can change direction when running”) or adapted. In addition, the specification of the time to respond was also removed (e.g., Item 5, “Can raise the head within five seconds when lying on belly”), as well as references to distance (e.g., Item 17, “Eyes focus quickly on an object held at a twenty centimetre distance”) and the manner of performing a skill (e.g., Item 66, “Holds a cube with a radial palmar grasp”). Some items were rephrased to ensure they were applicable to children with sensory impairments. One example is the item, “Follows light with the eyes by turning their head,” which was adapted to “Follows light or sounds by turning their head”. Moreover, we aimed to enhance the applicability of the MDL-PIMD for the group of children with a visual impairment by defining subscales based on Adolph and Franchak (2017), classifying the items that depend on visual skills under the subscale of “Facial Actions”. In this way, children who have a severe visual disability will possibly score low on the subscale that aims to measure facial action, but they are able to score higher on other subscales. Filling in the MDL-PIMD takes approximately 30 min.

Procedure

Data were collected as part of a larger longitudinal study in Belgium (Flanders) and the Netherlands on developmental patterns in young children with PIMD. Children with PIMD and their parents were recruited through hospitals, diagnostic centres, specialised day care centres and early intervention teams. Professionals were contacted by email, telephone or mail (with a flyer) and were asked to inform potential participants about the study. Data were collected between May 2014 and August 2019. The MDL-PIMD was filled in by the primary caregivers of a child with PIMD. The caregivers could choose who filled in the MDL-PIMD, and usually this was the parent who was most involved in the support of the child. This study was approved by the Social and Societal Ethics Committee of the University of Leuven and the Ethics Committee of the University of Groningen, Pedagogical and Educational Sciences.

Analysis

In this study, we used theory and Mokken scale analysis iteratively to identify subdomains of motor development. Firstly, items were categorised into subscales based on the theoretical framework of Adolph and Franchak (2017). Secondly, we performed a Mokken scale analysis on these subscales. When the assumptions of the Mokken model were violated, we qualitatively examined the content of the items that did not fit the model.

Subdomains of motor skills

To begin with, subscales were defined based on the division of motor skills into different domains according to the definitions of Adolph and Franchak (2017): Posture, Locomotion, Manual Action and Facial Action (see Table 2).

Table 2. Definitions for subscales of the MDL based on Adolph and Franchak (2017).

Subscale	Definition	Example item
Posture	Posture refers to all motor actions that are focused at obtaining or maintaining balance of the body.	My child can stand with support.
Locomotion	Locomotion are all motor actions that are focused on changes in children’s mastery of mobility, starting with spontaneous leg and arm movements, to rolling, crawling, and walking.	My child can move forward when lying on the belly.
Manual action	Manual actions are all movements of the hands and arms.	My child grabs an object with both hands.
Facial action	Facial actions are all motor actions that are driven by visual or auditory input.	When my child is lying on the back, he or she can follow a moving object.

The items were categorised into these subdomains by two researchers independently. Several items could be allocated to multiple subdomains. For example, the item, “My child can move an object from one hand to another while sitting”, relies both on posture (sitting) and manual skills (manipulating an object). Items that could be categorised into multiple subdomains were allocated to the most suitable subdomain based on the focus of the item. Moreover, items that focused on behaviour that is a precursor to behaviour that belongs to one of the four motor action systems were classified into that motor action system. For example, the item, “My child pulls up their legs when lying on their back”, focuses on movements that are a precursor to walking and was therefore categorised in the subdomain of locomotion. Cohen’s kappa was calculated to assess the interrater reliability of the assignment of items to subdomains. Items that were assigned to different subscales were discussed by all authors until full agreement was reached about the assignment of items to subscales.

Missing data analysis

Before assessing the reliability and validity of the MDL-PIMD, a missing data analysis was conducted. The number of missing items was negligible (< 3%), and this was addressed by imputing the mean of the item on that specific measurement occasion. In six cases, the stopping rule of the Portage Program Checklist was used erroneously at the first measurement. An analysis of complete cases showed that only a small number of items were scored after 15 consecutive zero scores. Eight participants scored one item after 15 consecutive zero scores, one participant scored three items and one participant scored four items. Therefore, we decided to impute scores that were missing due to the application of the stopping rule as zero. The amount of drop-out at baseline was negligible (1.9%), but increased at the second (11.5%), third (25.0%), fourth (28.8%) and fifth (44.2%) measurement occasions. The MDL-PIMD scores of groups that dropped out at different measurements were compared to assess the missing data mechanism. Based on our analysis, we found that children with lower scores on the MDL-PIMD were likely to drop out earlier. This implies that missingness is likely to be dependent on motor ability scores of earlier measurements. Therefore, we assumed that data were Missing at Random (Van Buuren, 2012).

Mokken scale analysis for polytomous items

To assess the construct validity and reliability of the MDL-PIMD a Mokken scale analysis for polytomous items was conducted (Sijtsma & Molenaar, 2002). The Mokken model is a nonparametric item-response model, where the relationship between responses to items and the latent trait are modelled using item-response functions. In our analysis, we used the double monotonicity model, assuming that:

1. The constructs used in the MDL-PIMD (Posture, Locomotion, Manual Action, and Facial Action) are unidimensional. All items are assumed to be indicators of this trait.

2. The responses on items are locally independent. This implies that – given the motor ability of the child – the child’s responses on one item does not depend on responses on other items.

3. Item-response functions are monotonous and non-decreasing. This implies that for every item the probability of a child mastering a task (e.g., crawling) is higher for a child with a high level of motor ability than for a child with a lower level of motor ability.

4. Item-response functions are non-intersecting. This implies that, based on the item-response functions, the ordering of items should be the same for children of all levels of motor development. For example, if crawling is harder than rolling over for children with poor motor development, it should also be harder for children with less severe motor disabilities.

H coefficients were used to assess whether the scales satisfied the assumptions of the Mokken model. Scales were considered satisfactory when H$\ge$.40, with .40 ≤ H < .50 indicating a medium strong scale and H$\ge$.50 indicating a strong scale. Furthermore, we used crit values to assess whether the model violated the double monotonicity assumptions. Items with a crit value > 80 were considered to violate the assumptions of the double monotonicity model (Sijtsma & Molenaar, 2002). If the assumptions of the double monotonicity model hold, the reliability of the scale can be estimated using Rho. The Rho statistic is an adaptation of Cronbach’s alpha for Mokken scale analysis and estimates the lower boundary of reliability based on the covariance of items (for more information on Rho see Sijtsma & Molenaar, 1987). A Rho coefficient of > .90 indicates an excellent scale (George & Mallery, 2003). We used the statistical software MSP (Molenaar & Sijtsma, 2000) to estimate our model. For each subdomain we first entered all items included based on theory in the default search procedure. This procedure removed items with H < .30. Secondly, items were removed manually when they had a crit > 80 or H < .40. Items were removed step-wise by first removing those with the most serious violations and estimating the model again. This procedure was repeated until all items in the scale adhered to the criteria. As the data consisted of repeated measures nested within subjects, a subgroup analysis was performed to check whether there was sample independence between moments of measurement (i.e., if items were ordered similarly across measurements). This check was done by using the test option in MSP and evaluating the crit values for item invariance between time points. The ordering of the items in the original MDL was based on the study by Hoekstra et al. (2010) on typically developing children. To assess whether the ordering of items in the MDL-PIMD was similar to the ordering found in typically developing children, we compared the rank order of items in our study with the order of items in the original MDL. Because we did not have access to the original data of Hoekstra et al. (2010), we could not empirically compare rank order in both samples. Therefore, we only included some remarks about items that strongly differed in rank order (i.e., > 30% difference in rank order¹) between studies, as a first indicative result.

Results

Categorisation of items into subscales

Based on the categories of Adolph and Franchak (2017), all items could be assigned to one of the categories. The interrater reliability of the categorisation of two researchers was Kappa = .88. There was agreement in the assignment of 91.3% of the items. The remaining items (k = 12) were discussed by the entire research team and assigned to a category after full agreement was achieved. Most items were assigned to the Manual Actions scale (k = 54). The second largest scale was Locomotion (k = 41), followed by Posture (k = 39) and then Facial Actions (k = 9).

Mokken scale analysis

Posture

A total of 39 items were assigned to the Posture scale. Four items were removed in the Mokken scale analysis (see Table 3) due to violation of the non-intersection assumption. All remaining items were scalable, had an H value > .40 and were retained in the final scale. Therefore, the final scale consisted of 35 items. The ordering of the items in the final scale was invariant across measurement occasions, and the scalability (scale H = .77) and reliability were excellent (Rho = .98). The scale adhered to the assumptions of the double monotonicity model (highest crit-value = 50). This indicates that the item-response function is non-decreasing (i.e., the probability of a child mastering a skill is higher for children with better posture) and that item-response functions are non-intersecting (i.e., the ordering of items is similar for children with different levels of postural control). The ordering of items was generally similar to the ordering of items in the original MDL list. Only item 62 (“My child turns his/her head freely when the body is supported”) was ranked considerably harder for typically developing children. Analysis of data of separate measurement occasions also showed good scalability and reliability and showed no major differences in item ordering. Detailed results of the Mokken scale analysis of the Posture scale across measurement occasion, and for each measurement occasion separately, are provided in Appendix A.

Table 3. Mokken scale summaries for MDL subscales in children with PIMD.

Scale	k original	k retained	Rho	Scale H
Posture	39	35	.98	.77
Locomotion	41	40	.98	.78
Manual actions	54	32	.98	.66
Reaching	16	15	.95	.67
Grabbing	38	30	.98	.72
Facial actions	9	9	.92	.66

Note: k = number of items; Rho = reliability; Scale H = scalability.

Locomotion

A total of 41 items were assigned to the Locomotion scale. In the Mokken scale analysis (see Table 3), one item (“My child roles over and over to move around”) was removed, as the crit value for non-intersection was higher than 80. All remaining items were scalable and had an H value > .40. There were no violations of other assumptions of the Mokken scale model. Therefore, the final scale consisted of 40 items. The scale adhered to the assumptions of the double monotonicity model (highest crit-value = 71). The reliability (Rho = .98) and scalability (H = .78) of the scale were excellent. Item ordering was invariant across measurement moments. Moreover, the ordering of the items was similar to the ordering for typically developing children (Hoekstra et al., 2010), starting with items about stretching, then rolling, sitting, lifting, stabilising, crawling, standing, walking and climbing. Only one item (“In prone position my child moves backwards by pushing with the arms”) was ranked higher (more difficult) for children with PIMD than for typically developing children.² Scalability and reliability were also good in the separate analysis of measurement occasions. Furthermore, no major differences in item ordering were observed across measurement occasions. Detailed results of the Mokken scale analysis of the Locomotion scale across measurement occasion, and for each measurement occasion separately, are provided in Appendix B.

Manual action

A total of 54 items of the MDL-PIMD were categorised as manual actions. Next, based on the Mokken scale analysis (see Table 3), one item was removed because it was scored zero by all participants (“My child can make scoop movements, for example with a spoon or shovel”), indicating that no child had acquired the skill. Moreover, one item formed a negative item pair and was removed (“My child mostly has his/her hands open”). In total, 20 items had to be removed in the Manual Actions scale due to a crit value > 80. The scalability and reliability of the remaining items was good (H = .66, Rho = .98). Further inspection of the content of the items showed that the majority of those that violated the double monotonicity assumption were focused on grabbing. Therefore, based on the theoretical framework of Adolph and Franchak (2017), we split the original 54 items of this scale into two subscales for further analysis: reaching (k = 16) and grabbing (k = 38). By dividing the scale into two subscales, more items could be retained compared to the original Manual Action Scale.

Reaching. The subscale of Reaching consists of items about stretching out the arms and hands, from stretching hands out in front of the body to consciously reaching towards objects. In total, 16 items of the MDL-PIMD were categorised as reaching. One item was removed because no child had mastered this item (see above). None of the other items violated the assumptions (Item-H > .40 and crit < 80). Therefore, the definitive scale consisted of 15 items (see Table 3). The scale adhered to the assumptions of the double monotonicity model (highest crit-value = 79). Reliability (Rho = .95) and Scalability (H = .67) of the Reaching subscale were good. Comparison of item-step ordering across measurements showed items were ordered similarly at different measurements (crit < 80). Item ordering was similar to the ordering in typically developing children as found in Hoekstra et al. (2010). Only one item (“My child puts one or both hands in front of him/herself while lying down”) was ranked considerably harder for children with PIMD than typically developing children. Scalability and reliability were also good in separate analyses of measurement occasions. No considerable differences in item ordering were observed across measurement occasions. Detailed results of the Mokken scale analysis of the Reaching subscale across measurement occasion, and for each measurement occasion separately, are provided in Appendix C.

Grabbing. The subscale of Grabbing contains items about grasping and holding objects. In total, 38 items of the MDL-PIMD were categorised as grabbing. Eight items were removed because the crit value was > 80. All remaining items did not violate the assumptions of the Mokken model (Item-H > .40 and crit < 80). The final subscale consisted of 30 items (see Table 3). The scale adhered to the assumptions of the double monotonicity model (highest crit-value = 73). Item ordering was similar across measurements. Reliability (Rho = .97) and scalability (H = .69) of the Grabbing scale were high. Item ordering was similar to the ordering of items in typically developing children (Hoekstra et al., 2010). Only one item (“My child turns the pages of a book; a few at the same time”) was ranked easier for children with PIMD than in typically developing children. Scalability and reliability were also good in separate analyses of measurement occasions. No considerable differences in item ordering were observed across measurement occasions. Detailed results of the Mokken scale analysis of the Grabbing subscale across measurement occasion, and for each measurement occasion separately, are provided in Appendix D.

Facial action

In total, nine items of the MDL-PIMD were categorised as facial action, based on the theoretical framework of Adolph and Franchak (2017). All items of the Facial Action scale adhered to the assumptions of the Mokken model (see Table 3). Therefore, no items were removed based on Item-H values, crit values or negative item pairs. Hence, the final scale consisted of nine items. The scale adhered to the assumptions of the double monotonicity model (highest crit-value = 37). The reliability (Rho = .92) and scalability (H = 0.66) of the Facial Actions scale was good and items were ordered similarly at different moments in time (repeated measures). Items that focused on basic visual behaviour (e.g., “My child follows light or sound by turning his/her head”) were mastered more often than items focused on visual behaviour that involves a stimulus that is at a distance (e.g., “My child follows a moving object at some distance that possibly makes sound”). Most items were ordered similarly to the study by Hoekstra et al. (2010). However, three items were ordered differently. Two items (“My child looks at his/her own hands”; “The eyes of my child move conjunctly and in focus”) were ranked as more difficult for children with PIMD than in typically developing children. One item (“My child follows an object at some distance that moves and makes sound”) was ranked more difficult for typically developing children. Scalability and reliability were also good in separate analyses of measurement occasions. No considerable differences in item ordering were observed across measurement occasions. Detailed results of the Mokken scale analysis of the Facial Action scale across measurement occasion, and for each measurement occasion separately, are provided in Appendix E.

Discussion

Findings

This study investigated the construct validity and reliability (Rho) of the Motor Development List for children with PIMD (MDL-PIMD), a proxy-reported measure. The MDL-PIMD was filled in by parents every six months for 52 children with PIMD. Although the original checklist only consisted of one scale, the items were divided into four subscales based on the theoretical framework of Adolph and Franchak (2017). Mokken scale analyses showed that the scales of Posture, Locomotion and Facial Action were reliable and had high scalability. Because many items were not scalable in the Manual Action scale, it was divided into two – Reaching and Grabbing – based on theory. The reliability and scalability of these two scales were good. As a result, the final version of the MDL-PIMD consisted of five scales (Posture, Locomotion, Reaching, Grabbing, Facial Action). These findings support the construct validity and the reliability of the MDL-PIMD for the assessment of motor development in children with PIMD. Within each subscale, the ordering of items was similar to their ordering in the original MDL for typically developing children (Hoekstra et al., 2010). Similar to typically developing children (Adolph & Franchak, 2017; Hadders-Algra, 2010), there was some variability in the ordering of motor milestones between individual children. However, only a few items diverged more systematically in rank order from the original MDL. Most notably, several items related to hand movements diverged in ordering from the typically developing population. For example, the item “My child puts one or both hands in front of him/herself while lying down”, was ranked higher (i.e., as more difficult) for children with PIMD than typically developing children. Although our findings suggest that the order of difficulty of items (i.e., the ordering of difficulty of motor milestones) was similar across cases, trajectories of motor development over time may vary. The analysis of trajectories of motor development is the topic of a forthcoming study. Because the motor development of children with PIMD may vary across subdomains of motor skills, it may be informative to measure domains of motor development separately, rather than measuring motor ability as one construct. One advantage of this approach is that the subscales are well grounded in the theory of motor development. This approach is especially relevant to scale development for children with PIMD for several reasons. Because the size of the population of children with PIMD is small and they experience health issues that hinder their participation in research studies, sample sizes are usually small (Maes et al., 2021). Moreover, this approach is suitable, as children with PIMD have profound disabilities that may affect different aspects of motor development to a different extent. Although different aspects of motor and general development are usually closely linked (Adolph & Joh, 2007), it may be useful to identify specific areas of motor development where problems occur. As the developmental trajectories of children with PIMD may vary across different motor domains due to their disabilities (e.g., visual impairments may impede the development of grabbing), the measurement of subdomains of motor development may be especially relevant for these children. Moreover, the division into subscales is relevant in practice, as it provides information about the functioning of a person in each subdomain. For example, a person may need more or different support for their functioning in locomotion than manual action. Thus, the scale provides information about the abilities of a person on specific motor domains.

Methodological reflection

We used a top-down approach in our Mokken scale analysis. This allowed for an analysis of theoretically informed domains of motor development. Moreover, the use of Mokken scale analysis for polytomous items requires a considerable sample size that is often not feasible in research on children with PIMD (Maes et al., 2021). As larger samples are needed when there is an increase in the number of items, the subdivision into multiple scales allowed for an analysis with a smaller sample size. In this study, we also used an iterative process applying theory and Mokken scale analysis to identify subdomains of motor development. For example, based on a qualitative analysis of items identified in the theory of Adolph and Franchak (2017), the subscale of Manual Actions was further divided into two subscales. After dividing this scale based on theory, a new Mokken scale analysis was performed for these new subscales. One challenge in this division into subscales was that the domains of motor development are closely linked and some items could be assigned to multiple categories. For example, the item “My child carries objects while crawling”, relates to both locomotion (i.e., crawling) and manual action (i.e., grabbing). Nonetheless, the assessment of interrater reliability showed most items could be reliably assigned to the subcategories. Furthermore, the good scalability and high reliability of the scales used in this study support the categorisation of the subscales based on the domains described by Adolph and Franchak (2017). Nevertheless, it would have been interesting to assess whether an exploratory factor analysis resulted in the same scales as proposed by the theory of Adolph and Franchak. In the context of Mokken scale analysis, this could have been done by identifying subscales through a search procedure and checking whether these were similar to the theoretically informed subscales. However, with the number of items in the MDL, this procedure requires a considerable sample size. As the sample size was too small given the number of items, we first formed subscales based on theory and then checked whether these scales adhered to the assumptions of the double monotonicity model. As mentioned above, a small sample size is common in research with people with PIMD (Maes et al., 2021). There are several reasons for this. Firstly, there is a low prevalence rate of PIMD (Hatton et al., 2016; Vugteveen et al., 2014). Secondly, there are difficulties in the demarcation of the group (i.e., it is often unknown how young children will develop and whether they will eventually meet the PIMD criteria of Nakken & Vlaskamp, 2007). Thirdly, the children experience significant health problems that hinder participation in research (Maes et al., 2021). However, using the combination of theory and Mokken scale analysis we were able to assess the construct validity and reliability of the MDL-PIMD, despite the methodological challenges of studying the target group. To ensure we had sufficient data to conduct a Mokken scale analysis for polytomous items, we included data from five repeated measurements (n = 191). However, the use of multiple observations nested within subjects resulted in dependent observations. The use of nested observation usually results in an underestimation of standard errors and liberal testing results. Nevertheless, we assumed that the estimation of scalability, item ordering, monotonicity and non-intersection were valid because our focus was on scale construction. In our results, we focused on the interpretation of these aspects and did not report results concerning the variance of items, as these can be expected to be biased. Because item scores are assumed to be correlated across measurements within individuals, the reliability estimates of the scales are likely to be inflated. To further assess the impact of the nested observations, we tested for sample independence between measurement occasions and performed analyses on the data of separate measurement occasions. The results of separate measurement occasions were similar to the findings based on the repeated measures data and the test for sample independence did not indicate an effect of the dependencies on our results. Furthermore, this study was characterised by considerable drop-out in the follow-up measurements. High drop-out is common in studies that focus on children with PIMD. For example, children may drop out because of health issues, due to time constraints of families or emotional burden (Maes et al., 2021). An analysis of missing data showed that children with poorer motor development were more likely to drop out. As this implies that missingness is dependent on earlier observations of motor development, we assumed data to be mostly Missing at Random (MAR; Van Buuren, 2010). A separate assessment of data showed no significant differences in item ordering at baseline or follow-up. This provides a first indication that item ordering was not affected by the drop-out of participants. In this study, we found that the order of difficulty of items was similar at different measurement occasions. However, missing data on follow-up measurements may have affected the results regarding item-ordering. More research is needed to determine with more certainty whether item-ordering is invariant over time.

Recommendations

The findings of this study provide a first indication of the construct validity and reliability of the MDL-PIMD for measuring the motor ability of children with PIMD. The lack of assessment instruments that have had their psychometric properties studied has been recognised as highly problematic, both in research and in practice by professionals (Maes et al., 2021; Wessels et al., 2021). A reliable and valid assessment of motor skills can provide valuable information with which to determine appropriate interventions for children with PIMD and thus provide high-quality support. As children with PIMD have considerable motor disabilities that are linked to other developmental areas, it is advisable for practitioners to use instruments such as the MDL-PIMD to monitor motor skills. The MDL-PIMD does not currently contain norms. Because the target group of children with PIMD is very heterogeneous, comparison with a norm that is based on group averages may not be clinically relevant. The current version of the MDL-PIMD provides insight into motor functioning in the assessment of children with PIMD. The MDL-PIMD could be used to assess the level of motor development, as items increase in difficulty. Based on what items (skills) a child masters, goals can be formulated in support to develop their motor skills. The knowledge that is gained by using the MDL-PIMD can be integrated into the general assessment of the needs, wishes and abilities of a child with PIMD to be better able to adapt support (Schalock et al., 2021). This is in accordance with a person-centred, interdisciplinary approach, in which information about a person in all the different domains and from all those involved is collected in order to establish a personal profile (Lyons et al., 2017; Vlaskamp, 2005). Moreover, high-quality research is dependent on the psychometric quality of the instruments used to collect data (Maes et al., 2021). Further studies should focus on different aspects of validity and reliability, such as interrater reliability and convergent validity. In addition, further studies should compare how different informant groups (such as parents and various professionals, e.g., direct support persons or physical therapists) score the MDL-PIMD, as different informants could have different perspectives on the abilities of the child (Maes et al., 2021). Moreover, studies could also focus on the psychometric properties of the MDL for the group of adults with PIMD, as the current study focused specifically on children. The current version of the MDL-PIMD consists of 129 items. It would also be interesting to analyse whether a stopping rule could apply for this group to decrease the time needed to fill in the list. It is important to analyse the sensitivity of the MDL-PIMD to changes in motor ability. If the MDL-PIMD is sensitive in this regard, it could then be used to investigate the motor development of children with PIMD over time. Using a longitudinal design, changes in motor abilities could be analysed and it would be possible to study whether subgroups of children with PIMD can be defined based on their motor development pattern. Moreover, using a longitudinal design, the effect of an intervention could be studied and it would be possible to examine whether different subgroups of children with PIMD benefit from different types of interventions. In addition, the MDL-PIMD scores and scores based on the coding of videorecorded observations could be compared to assess convergent validity. This will allow for a comparison of changes in motor skill development as seen in the observations and observed changes in the MDL-PIMD scores. Finally, it would be relevant to integrate the instrument into a dynamic assessment approach, in which the focus is not on what skills a person has at that specific moment but on their potential for development. One possible way of doing this is by focusing on the items a child scores positively sometimes, or can perform with support (Kulesza, 2015; Poehner, 2008). A forthcoming study aims to model patterns of motor development in children with PIMD using the MDL-PIMD. If there is more knowledge about motor development patterns, this could be a starting point for analysing the relations between different developmental domains and motor development. The use of valid and reliable instruments such as the MDL-PIMD may also provide valuable information for the identification and/or treatment of problematic motor development of children with PIMD. This could lead to support that is better adapted to the needs of children with PIMD and stimulate the development of their motor skills.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Johanna Kinderfonds; ZonMw, Stichting Rotterdams Kinderrevalidatie Fonds Adriaanstichting and Fonds Wetenschappelijk Onderzoek.

Notes

1 For example, in a scale with ten items, an item ordering was considered to be strongly different when an item rank order differed by more than three places (e.g. the fourth item in the study by Hoekstra et al. (2010) and the eighth item in this study).

2 The Posture scale can also be divided into two subscales: Postural Control and Head Balance. These scales for Postural Control (k = 18, Rho = .98) and Head Balance (k = 18, Rho = .97) showed high reliability and good scalability. However, as both constructs were closely related and the total scale for Posture was reliable and scalable as well, the total scale was included in the final version of the MDL-PIMD.

Appendices

Appendix A. Scale posture

Table

	Total					T1		T2		T3		T4		T5
	0	1	2	M	Item H	M	Item H	M	Item H	M	Item H	M	Item H	M	Item H	Crit invariant ordering
Item 2	5	15	182	1.88	0.71	1.92	0.57	1.93	0.81	1.57	0.64	1.81	0.77	1.83	0.82	0
Item 7	11	27	164	1.76	0.55	1.70	0.39	1.80	0.60	1.88	0.47	1.68	0.61	1.73	0.72	0
Item 1	25	29	148	1.61	0.67	1.66	0.66	1.67	0.73	1.57	0.64	1.51	0.67	1.60	0.65	7
Item 3	24	40	138	1.56	0.79	1.50	0.78	1.51	0.81	1.57	0.77	1.62	0.80	1.67	0.78	10
Item 26	24	44	134	1.54	0.69	1.44	0.68	1.49	0.62	1.65	0.73	1.54	0.68	1.67	0.78	13
Item 6a	37	25	140	1.51	0.71	1.56	0.78	1.47	0.72	1.62	0.79	1.38	0.62	1.50	0.65	26
Item 8	30	41	131	1.50	0.77	1.50	0.75	1.53	0.83	1.52	0.80	1.41	0.71	1.53	0.81	0
Item 5	33	39	130	1.48	0.75	1.54	0.76	1.51	0.75	1.50	0.81	1.41	0.72	1.40	0.72	12
Item62	43	24	135	1.46	0.79	1.40	0.73	1.51	0.80	1.57	0.79	1.32	0.83	1.47	0.79	11
Item 18	55	21	126	1.35	0.74	1.40	0.77	1.36	0.74	1.35	0.80	1.30	0.68	1.33	0.70	12
Item 6c	53	30	119	1.33	0.72	1.36	0.78	1.29	0.78	1.43	0.68	1.22	0.69	1.33	0.62	14
Item 10	46	47	109	1.31	0.79	1.28	0.76	1.38	0.82	1.35	0.85	1.19	0.73	1.37	0.82	0
Item 36	55	34	113	1.29	0.79	1.20	0.77	1.33	0.78	1.35	0.85	1.27	0.75	1.30	0.82	5
Item 40	62	26	114	1.26	0.75	1.12	0.67	1.24	0.80	1.25	0.80	1.35	0.69	1.40	0.79	36
Item 28	66	30	106	1.20	0.81	1.12	0.77	1.11	0.79	1.23	0.88	1.24	0.78	1.37	0.82	10
Item 15	60	42	100	1.20	0.80	1.12	0.71	1.18	0.82	1.27	0.87	1.16	0.78	1.30	0.82	2
Item 41	73	22	107	1.17	0.77	1.12	0.76	1.13	0.76	1.18	0.82	1.16	0.72	1.30	0.77	3
Item 61	77	26	99	1.11	0.82	1.08	0.80	1.09	0.82	1.15	0.84	1.08	0.82	1.17	0.84	0
Item 68	89	20	93	1.02	0.74	0.90	0.71	0.96	0.81	0.97	0.82	1.11	0.61	1.27	0.75	39*
Item 107	95	14	93	0.99	0.76	0.92	0.72	0.93	0.78	0.93	0.78	1.08	0.75	1.17	0.80	13
Item 24	88	28	86	0.99	0.76	0.96	0.71	0.98	0.80	0.97	0.75	1.03	0.75	1.03	0.83	31
Item 30	99	34	69	0.85	0.71	0.78	0.61	0.82	0.68	0.88	0.76	0.86	0.73	0.97	0.79	0
Item 80	112	10	80	0.84	0.76	0.76	0.76	0.71	0.83	0.90	0.83	0.84	0.74	1.10	0.60	8
Item 79	111	12	79	0.84	0.82	0.70	0.75	0.82	0.86	0.90	0.83	0.89	0.83	0.97	0.79	0
Item 65	110	24	68	0.79	0.81	0.70	0.83	0.80	0.84	0.90	0.82	0.76	0.76	0.83	0.79	2
Item 106	119	20	63	0.72	0.80	0.56	0.75	0.64	0.74	0.78	0.84	0.84	0.82	0.90	0.85	0
Item 48	126	18	58	0.66	0.78	0.66	0.73	0.62	0.81	0.70	0.78	0.59	0.76	0.77	0.82	1
Item 96	129	16	57	0.64	0.82	0.62	0.82	0.51	0.81	0.72	0.88	0.62	0.80	0.80	0.76	1
Item 55	130	17	55	0.63	0.77	0.52	0.69	0.60	0.78	0.65	0.75	0.70	0.85	0.73	0.79	0
Item76	124	34	44	0.60	0.76	0.48	0.69	0.49	0.73	0.62	0.77	0.70	0.83	0.83	0.74	1
Item 88	137	18	47	0.55	0.76	0.38	0.63	0.60	0.75	0.65	0.79	0.59	0.68	0.67	0.88	6
Item 117	148	9	45	0.49	0.77	0.38	0.63	0.40	0.87	0.55	0.80	0.49	0.80	0.73	0.74	0
Item 140	162	12	28	0.34	0.78	0.24	0.81	0.29	0.83	0.45	0.84	0.30	0.90	0.47	0.51	0
Item 131	166	16	20	0.28	0.77	0.12	0.73	0.18	0.91	0.35	0.92	0.35	0.85	0.50	0.47	0
Item 121	176	7	19	0.22	0.84	0.10	0.78	0.09	0.93	0.28	0.91	0.24	0.89	0.53	0.74	0
Scale H					0.77		0.73		0.78		0.80		0.75		0.76
Rho					0.98		0.98		0.98		0.98		0.98		0.98
n					202		50		45		40		37		30
Mdn					36.0		30.5		34.0		38.5		36.0		45.0
SD					20.7		19.4		20.1		21.3		21.7		22.1
Range					2–68		4–65		2–66		3–68		2–68		2–68

Appendix B. Scale Locomotion

Table

	Total					T1		T2		T3		T4		T5
	0	1	2	M	Item H	M	Item H	M	Item H	M	Item H	M	Item H	M	Item H	Crit invariant ordering
Item 9b	31	28	141	1.55	0.67	1.59	0.40	1.53	0.68	1.62	0.84	1.41	0.69	1.60	0.73	0
Item 33	32	33	135	1.51	0.76	1.57	0.57	1.56	0.81	1.44	0.74	1.46	0.83	1.53	0.67	0
Item 54a	46	22	132	1.43	0.83	1.53	0.84	1.40	0.80	1.44	0.88	1.32	0.81	1.43	0.81	0
Item 12	44	29	127	1.42	0.74	1.43	0.74	1.47	0.79	1.41	0.67	1.27	0.73	1.50	0.78	0
Item35	51	35	114	1.31	0.78	1.37	0.70	1.31	0.87	1.36	0.83	1.27	0.80	1.23	0.71	6
Item 57	78	19	103	1.12	0.77	1.00	0.71	1.13	0.80	1.28	0.76	1.05	0.76	1.20	0.76	7
Item 58	81	22	97	1.08	0.79	1.10	0.71	1.07	0.77	1.13	0.87	1.05	0.83	1.03	0.75	27
Item 29	79	39	82	1.01	0.74	0.90	0.67	1.07	0.73	1.03	0.82	1.00	0.81	1.13	0.63	5
Item 60	91	19	90	0.99	0.80	0.92	0.77	1.02	0.82	1.08	0.83	1.00	0.83	0.97	0.75	35
Item 34	94	21	85	0.95	0.63	0.88	0.69	0.96	0.64	0.87	0.52	1.00	0.71	1.13	0.59	20
Item 52	110	12	78	0.84	0.69	0.65	0.65	0.82	0.69	0.90	0.75	0.84	0.63	1.10	0.69	44
Item 85	119	10	71	0.76	0.79	0.63	0.85	0.76	0.73	0.87	0.86	0.65	0.79	0.97	0.69	12
Item 84	120	14	66	0.73	0.80	0.63	0.80	0.67	0.85	0.79	0.84	0.68	0.76	0.97	0.71	14
Item 53	120	15	65	0.72	0.78	0.71	0.76	0.67	0.78	0.79	0.81	0.73	0.83	0.73	0.72	25
Item 56	122	16	62	0.70	0.80	0.59	0.83	0.67	0.75	0.85	0.81	0.65	0.80	0.80	0.79	34
Item 78	130	4	66	0.68	0.83	0.61	0.84	0.62	0.87	0.74	0.87	0.68	0.78	0.80	0.78	8
Item 87b	136	8	56	0.60	0.85	0.57	0.86	0.51	0.80	0.64	0.88	0.65	0.87	0.67	0.82	13
Item 86	135	11	54	0.59	0.78	0.59	0.80	0.49	0.75	0.62	0.78	0.59	0.82	0.73	0.77	9
Item 103	135	12	53	0.59	0.84	0.53	0.84	0.58	0.81	0.64	0.48	0.54	0.86	0.70	0.80	5
Item 87a	139	6	55	0.58	0.85	0.47	0.83	0.56	0.84	0.69	0.85	0.57	0.88	0.67	0.82	6
Item 105	139	8	53	0.57	0.84	0.49	0.82	0.51	0.84	0.67	0.87	0.57	0.86	0.67	0.79	5
Item 73	137	13	50	0.56	0.70	0.61	0.65	0.49	0.70	0.59	0.73	0.54	0.68	0.60	0.74	44
Item 124	138	14	48	0.55	0.71	0.35	0.69	0.56	0.70	0.56	0.79	0.57	0.75	0.83	0.59	31
Item 118	139	14	47	0.54	0.76	0.39	0.77	0.42	0.77	0.59	0.77	0.54	0.79	0.90	0.67	51*
Item 109	139	15	46	0.54	0.81	0.39	0.80	0.44	0.81	0.62	0.85	0.54	0.87	0.80	0.69	10
Item 63	139	18	43	0.52	0.75	0.41	0.69	0.51	0.73	0.56	0.67	0.57	0.87	0.60	0.77	1
Item 119	151	3	46	0.47	0.86	0.37	0.79	0.40	0.86	0.49	0.88	0.54	0.90	0.67	0.83	0
Item 104	151	13	36	0.42	0.74	0.24	0.69	0.33	0.64	0.56	0.79	0.54	0.82	0.53	0.70	18
Item 102	154	16	30	0.38	0.61	0.24	0.58	0.40	0.74	0.38	0.48	0.43	0.62	0.50	0.52	22
Item 122	160	9	31	0.35	0.83	0.24	0.84	0.29	0.80	0.36	0.78	0.49	0.88	0.47	0.85	0
Item 130	164	9	27	0.32	0.84	0.12	0.74	0.24	0.83	0.38	0.85	0.41	0.89	0.53	0.82	0
Item 139	167	9	24	0.28	0.72	0.12	0.65	0.29	0.85	0.33	0.63	0.30	0.75	0.47	0.64	10
Item 134	170	3	27	0.28	0.74	0.12	0.65	0.31	0.74	0.41	0.73	0.30	0.69	0.33	0.81	18
Item 144	173	3	24	0.26	0.77	0.12	0.74	0.18	0.69	0.31	0.84	0.30	0.81	0.47	0.71	8
Item 142a	174	4	22	0.24	0.82	0.12	0.72	0.16	0.85	0.26	0.87	0.32	0.90	0.43	0.70	0
Item 135	175	3	22	0.23	0.84	0.10	0.78	0.13	0.81	0.31	0.87	0.32	0.86	0.40	0.82	0
Item 143	175	5	20	0.23	0.78	0.10	0.68	0.18	0.79	0.26	0.86	0.27	0.88	0.40	0.64	0
Item 141	175	7	18	0.21	0.83	0.10	0.74	0.18	0.80	0.31	0.85	0.32	0.88	0.20	0.80	5
Item142b	178	3	19	0.20	0.79	0.08	0.70	0.11	0.84	0.23	0.81	0.32	0.89	0.37	0.66	0
Item 145	181	3	16	0.17	0.77	0.06	0.75	0.13	0.82	0.21	0.85	0.22	0.84	0.33	0.60	0
Scale H					0.78		0.75		0.78		0.80		0.81		0.73
Rho					0.98		0.98		0.98		0.99		0.99		0.99
n					200		49		45		39		37		30
Mdn					17.0		17.5		16.0		15.0		17.0		25.5
sd					23.8		20.2		22.8		25.5		26.3		26.1
Range					0–78		0–70		0–76		0–76		0–78		1–77

Appendix C. Scale reaching

Table

	Total					T1		T2		T3		T4		T5
	0	1	2	M	Item H	M	Item H	M	Item H	M	Item H	M	Item H	M	Item H	Crit invariant ordering
Item 23a	48	28	126	1.39	0.72	1.48	0.73	1.58	0.76	1.43	0.71	1.46	0.79	1.43	0.61	10
Item 23b	34	37	131	1.48	0.72	1.32	0.64	1.38	0.76	1.40	0.80	1.41	0.79	1.47	0.59	5
Item 42	66	19	117	1.25	0.75	1.12	0.71	1.16	0.76	1.40	0.80	1.24	0.72	1.43	0.73	19
Item 32	54	47	101	1.23	0.52	1.36	0.64	1.29	0.45	1.27	0.54	0.92	0.46	1.27	0.49	43*
Item 21b	68	21	113	1.22	0.72	1.20	0.64	1.20	0.75	1.30	0.77	1.16	0.73	1.27	0.71	11
Item 21a	67	28	107	1.20	0.66	1.10	0.56	1.18	0.65	1.25	0.69	1.19	0.74	1.33	0.69	14
Item 46	77	31	94	1.08	0.59	1.24	0.57	0.96	0.65	1.12	0.68	0.89	0.59	1.20	0.43	40
Item 75	89	21	92	1.01	0.65	0.92	0.61	0.91	0.64	1.05	0.70	1.08	0.68	1.20	0.62	27
Item 111	115	10	77	0.81	0.66	0.56	0.61	0.82	0.68	0.85	0.78	0.92	0.57	1.03	0.61	23
Item 77b	115	9	78	0.82	0.70	0.64	0.69	0.71	0.77	0.85	0.75	0.95	0.69	1.07	0.54	23
Item 95	121	27	54	0.67	0.60	0.64	0.62	0.69	0.64	0.62	0.64	0.62	0.62	0.80	0.43	14
Item 114	133	15	54	0.61	0.71	0.52	0.67	0.49	0.79	0.55	0.84	0.68	0.66	0.93	0.62	7
Item 97	146	8	48	0.51	0.72	0.48	0.66	0.44	0.73	0.55	0.80	0.51	0.75	0.63	0.65	0
Item 138	160	29	13	0.27	0.70	0.20	0.66	0.29	0.68	0.25	0.74	0.27	0.72	0.40	0.68	0
Item 115	184	9	9	0.13	0.50	0.04	0.14	0.02	0.82	0.07	0.69	0.24	0.52	0.40	0.41	6
Scale H					0.67		0.64		0.69		0.73		0.67		0.59
Rho					0.95		0.93		0.94		0.95		0.95		0.94
n					202		50		45		40		37		30
Mdn					14.0		13.0		12.0		14.5		14.0		17.0
SD					9.0		8.6		9.0		9.3		9.6		9.2
Range					0–29		0–27		0–28		0–28		0–28		0–29

Appendix D. Scale grabbing

Table

	Total					T1		T2		T3		T4		T5
	0	1	2	M	Item H	M	Item H	M	Item H	M	Item H	M	Item H	M	Item H	Crit invariant ordering
Item 37	41	54	107	1.33	0.72	1.26	0.58	1.29	0.83	1.38	0.74	1.30	0.85	1.47	0.55	5
Item 43	71	25	106	1.17	0.82	1.06	0.78	1.07	0.83	1.35	0.92	1.19	0.84	1.27	0.71	17
Item 44b	67	40	95	1.14	0.73	1.04	0.72	1.02	0.74	1.20	0.71	1.19	0.70	1.33	0.79	8
Item 45	78	21	103	1.12	0.71	0.98	0.61	1.16	0.88	1.05	0.61	1.14	0.70	1.40	0.77	18
Item 50	84	16	102	1.09	0.79	1.04	0.75	0.93	0.81	1.20	0.83	1.05	0.83	1.30	0.69	11
Item 44a	85	29	88	1.01	0.60	1.04	0.73	0.89	0.61	1.02	0.59	1.03	0.67	1.13	0.55	26
Item 82	103	8	91	0.94	0.80	0.86	0.75	0.93	0.81	0.97	0.84	0.89	0.80	1.10	0.77	22
Item 77a	94	34	74	0.90	0.73	0.82	0.73	0.82	0.68	0.97	0.71	0.81	0.72	1.17	0.78	14
Item 92	106	13	83	0.89	0.69	0.80	0.75	0.80	0.73	1.00	0.74	0.81	0.79	1.10	0.75	8
Item 74	102	21	79	0.89	0.76	0.78	0.68	0.84	0.70	0.97	0.65	0.89	0.76	1.00	0.65	24
Item 93	104	24	74	0.85	0.71	0.86	0.70	0.69	0.61	0.85	0.70	0.78	0.79	1.17	0.75	29
Item 91	108	19	75	0.84	0.71	0.78	0.67	0.80	0.64	0.78	0.65	0.78	0.71	1.13	0.67	29
Item 71	105	30	67	0.81	0.67	0.76	0.61	0.71	0.74	0.80	0.74	0.84	0.73	1.03	0.74	16
Item 94	114	31	57	0.72	0.73	0.70	0.67	0.60	0.58	0.68	0.71	0.70	0.70	1.00	0.62	37
Item 70	11	37	54	0.72	0.66	0.58	0.76	0.62	0.76	0.78	0.77	0.76	0.73	0.97	0.74	11
Item 90	125	12	65	0.70	0.74	0.58	0.76	0.53	0.69	0.93	0.77	0.70	0.73	0.87	0.71	23
Item 81	127	16	59	0.66	0.72	0.54	0.69	0.58	0.72	0.78	0.70	0.68	0.73	0.83	0.73	13
Item 89	135	20	47	0.56	0.59	0.50	0.53	0.64	0.58	0.65	0.68	0.46	0.60	0.57	0.49	42
Item 116	133	24	45	0.56	0.58	0.42	0.48	0.47	0.47	0.55	0.63	0.59	0.67	0.93	0.65	36
Item 108	144	7	51	0.54	0.68	0.46	0.68	0.42	0.65	0.60	0.66	0.54	0.72	0.77	0.65	11
Item 132	146	19	37	0.46	0.64	0.36	0.60	0.56	0.58	0.40	0.75	0.43	0.69	0.60	0.60	43
Item 123	144	24	34	0.46	0.62	0.32	0.62	0.44	0.63	0.53	0.73	0.41	0.55	0.67	0.50	18
Item 101	154	18	30	0.39	0.71	0.42	0.73	0.29	0.73	0.42	0.75	0.35	0.63	0.47	0.66	22
Item 112	164	9	29	0.33	0.68	0.22	0.64	0.20	0.76	0.35	0.66	0.43	0.69	0.57	0.63	37
Item 98	156	26	20	0.33	0.59	0.20	0.54	0.20	0.61	0.35	0.69	0.49	0.60	0.50	0.44	32
Item 113	167	9	26	0.30	0.56	0.24	0.65	0.20	0.77	0.23	0.79	0.43	0.68	0.50	0.63	32
Item 120	163	20	19	0.29	0.70	0.28	0.53	0.29	0.42	0.28	0.64	0.19	0.72	0.43	0.52	44*
Item 99	178	17	7	0.15	0.48	0.12	0.43	0.13	0.61	0.17	0.69	0.19	0.22	0.17	0.46	12
Item 110	184	10	8	0.13	0.65	0.06	0.55	0.07	0.83	0.10	0.78	0.14	0.64	0.37	0.51	10
Item 133	189	7	6	0.09	0.77	0.04	0.72	0.09	0.74	0.15	0.85	0.08	0.69	0.13	0.73	0
Scale H					0.69		0.66		0.69		0.72		0.71		0.66
Rho					0.97		0.97		0.97		0.98		0.98		0.97
n					202		50		45		40		37		30
Mdn					15.0		13.0		13.0		20.5		12.0		27.5
SD					17.4		16.4		17.1		17.9		18.0		18.0
Range					0–56		0–47		0–54		0–56		0–55		0–54

Appendix E. Scale facial actions

Table

	Total					T1		T2		T3		T4		T5
	0	1	2	M	Item H	M	Item H	M	Item H	M	Item H	M	Item H	M	Item H	Crit invariant ordering
Item 13	78	31	94	1.08	0.61	0.96	0.61	1.16	0.55	1.15	0.62	1.00	0.60	1.17	0.63	0
Item 16	53	77	73	1.10	0.66	1.02	0.66	1.11	0.62	1.25	0.67	1.03	0.60	1.10	0.71	0
Item 27	69	37	97	1.14	0.61	1.22	0.61	1.16	0.55	1.15	0.62	1.03	0.62	1.10	0.65	30
Item 19	33	62	108	1.37	0.66	1.31	0.66	1.29	0.67	1.45	0.68	1.49	0.68	1.33	0.73	9
Item 22	32	43	128	1.47	0.71	1.31	0.71	1.33	0.70	1.60	0.73	1.65	0.64	1.57	0.78	0
Item 17	19	56	128	1.54	0.68	1.53	0.68	1.49	0.70	1.55	0.73	1.62	0.67	1.50	0.68	32
Item 4	23	40	140	1.58	0.73	1.57	0.73	1.56	0.72	1.62	0.69	1.62	0.77	1.50	0.69	7
Item 14	18	49	136	1.58	0.74	1.51	0.74	1.49	0.71	1.68	0.71	1.62	0.76	1.67	0.81	49*
Item 11	12	48	143	1.65	0.62	1.57	0.62	1.64	0.63	1.73	0.60	1.62	0.52	1.70	0.79	49*
Scale H					0.66		0.66		0.65		0.67		0.66		0.71
Rho					0.92		0.92		0.92		0.92		0.92		0.93
n					203		51		45		40		37		30
Mdn					14.0		14.0		14.0		14.5		14.0		14.0
SD					5.2		5.5		5.3		4.9		5.0		5.5
Range					0–18		0–18		1–18		1–18		2–18		0–18