A gesture-based virtual art program for children with severe motor impairments – development and pilot study

Abstract

Art plays a vital role in developing a child's communication, problem solving, social and emotional skills as well as motor control, creativity and self-expression. For children with severe impairments that limit their access to traditional art processes, it is important to find alternative methods to enable these children to express themselves creatively. Advantages of a virtual art program include its ability to compensate for specific physical impairments, flexibility of incorporating sensory feedback, such as audio, to improve engagement, the avoidance of the untidiness often associated with children's arts activities, and the absence of physical parts conducive to accidental ingestion. The Kinect Virtual Art Program (KVAP) uses the Microsoft Kinect gesture recognition technology that facilitates a new method of engaging children in therapeutic recreation. The program was designed to allow the creation of art through non-contact ‘virtual’ button activation. A pilot study was performed with five children with severe impairments to determine the level of physical engagement that these children could attain while using the KVAP over five sessions. The results indicated that the participants enjoyed using the KVAP and increasingly engaged with it over the sessions. The KVAP encouraged physical activity and enabled children to create their own works of art, an activity that was previously inaccessible to them using traditional approaches. The KVAP may offer a potential new avenue for therapy, play and exploration.

• virtual art
• gesture recognition
• Microsoft Kinect
• physical engagement
• sensory feedback

Most children with severe impairments experience limited exposure to and engagement with leisure activities due to their disability. Designing an intuitive virtual art program that is easy to manipulate may assist in child development and therapy, and provide a leisure activity for children with severe impairments who lack fine motor control or muscle strength. The Kinect Virtual Art Program (KVAP) was designed by engineers in consultation with therapists, special needs teachers and disability professionals to best meet the needs of the target population. The program has no rules or instructions, encouraging experimentation and exploration in order to engage the participant cognitively as well as physically.

Art plays a vital role in a child's development in areas of communication, problem solving, social and emotional skills, as well as motor control, creativity and self-expression (1). Because of difficulties in gripping objects such as utensils, many children with physical or cognitive impairments often cannot participate in traditional art practices, reducing opportunities to experiment with art. This can hinder their development as well as exclude them from mainstream class activities.

The hypothesis of this study was that the KVAP would successfully engage children with severe impairments, providing individuals whose participation in art activities was previously limited with a means of engaging physically and expressing themselves creatively through art. It was expected that participants would enjoy the activity and be able to participate in the activity for at least 5 minutes.

The Microsoft Kinect technology

Microsoft's Xbox Kinect allows gesture recognition using the infrared projector and camera and a microchip that can track the movement of objects and individuals along the x, y and z axes using 3D image-based reconstruction (2).

In 2011 Microsoft released a Kinect software development kit (SDK) for Windows 7. It enabled skeletal tracking, which tracked up to 20 joints on the user. The SDK could also distinguish players and their movements even when they were partially hidden; extrapolating what the rest of a body was doing if it could only detect some parts of it. This SDK was used to design the KVAP.

A virtual art program to encourage physical engagement

Unlike computer games, visual art allows for visual communication of abstract ideas and encourages self-expression and creativity (1). Able-bodied children are encouraged to draw, paint and sculpt to improve development and to enjoy leisure. These media are typically inaccessible to children with poor hand control. Despite the growing number of ways to engage children with disabilities in art, there is always a search for new ideas. Children learn different skills through varied artistic opportunities. In the same way, providing multiple techniques for children with impairments to experiment creatively encourages a range of skills.

Kinect technology facilitates non-contact ‘virtual’ button activation that enables a child to more easily participate in art-making compared to traditional art methods. Advantages of a child using a virtual art medium include: no parts to accidentally ingest; no mess created; programmable to compensate for impairments; and the system can incorporate additional sensory feedback, such as audio, to improve engagement. Visual and auditory feedback is better for brain development than movement without the sensory feedback (3). A virtual art program that uses the latest commercial technology is more likely to appeal to children with impairments as it allows them to use a popular mainstream product like their able-bodied peers.

The multisensory aspect of the KVAP was designed to promote physical engagement, which is important in developing dexterity, muscle strength, flexibility and endurance. Many children with disabilities regularly see occupational therapists and physiotherapists who prescribe daily exercises to build and maintain muscle tone and flexibility. However, research suggests that many with motor disabilities do not perform the exercises as recommended by their therapists (4).

Adequate physical activity reduces the risk of cardiovascular disease, type 2 diabetes, certain cancers, osteoporotic fractures, and depression. It also improves physical function, cognitive function and quality of life (5). Many children with severe impairments are wheelchair users and have very low levels of physical activity (6). Creating a fun program that encourages activity may lead to increased fitness and health.

Market analysis

Programs such as Kinerehab (4), Mitii (7) and KiDnect (8) have been designed to encourage physical engagement by providing daily exercises and an easy way for the instructor to monitor a user's progress. The aim of the KVAP was to achieve physical engagement through a fun, leisure activity that encourages creativity and exploration.

Kinect Paint is a free, downloadable virtual painting application that is targeted at the mainstream market. The program is fun and engaging but difficult to manipulate (9). Children with severe impairments often do not have enough control of both arms to operate this complex system. No virtual art programs were identified that enabled severely impaired children to create art. This gap in the market requires addressing because of the benefits a virtual art program can provide for the target population.

Program design and setup

The KVAP was designed and developed collaboratively by Holland Bloorview Kids Rehabilitation Hospital in Toronto, Canada, and Flinders University in Adelaide, Australia. Health professionals within the field of disability were involved in the design of the program, including an art therapist, a speech pathologist, an occupational therapist and a special education arts teacher. Engineers from the PRISM Lab in Toronto were also consulted. Consulting a multi-disciplinary team enabled a broad view of what functions would be best in a virtual art program to suit the needs of children with disabilities and to make the program as universally accessible as possible.

The KVAP was programmed in C ++ in Visual Studio 2010. The system consists of an unmodified Microsoft Kinect motion sensor system, a computer running custom-designed software that allows tracking of the extremities, a visual display (captured by the Kinect camera), and corresponding sound effects.

An initial prototype was created and set up at Holland Bloorview Kids Rehabilitation Hospital. Children engaged with the KVAP and appeared to enjoy using it, but they found the program unintuitive and slow. The focus of the program shifted from creating a visual output to focusing on the experience, encouraging movement through both visual and auditory feedback to provide additional sensory feedback.

The program was further developed at Flinders University. The body of the program is held within a display loop. The display loop then calls the other functions such as ‘track skeleton’ or ‘paint/erase’ at the appropriate times. Graphics are performed using the Open GL 32 library and the irrKlang library is used for audio.

Custom software overlays coloured ‘virtual’ buttons on the live video image. Each time the tracked limb passes through a button, a new colour is painted, following the path of the moving limb. Keyboard and mouse controls allow the therapist or user to easily change settings or move the virtual palette buttons around.

The program is set to automatic mode to be simple and intuitive for the participant and the therapist or teacher but it can be switched to manual mode, putting the menus on the edge of the screen so that people with higher cognitive ability or motor skills can enjoy and create more advanced art works than those with severe impairments. The layouts of each mode are shown in Figure 1.

Figure 1 Layout of the KVAP screen.

The KVAP's automatic mode

The KVAP functions include paint, erase, zoom and provision for a white background canvas rather than a live video. The KVAP can be set to track the left or right hand, head, or the head, hands and feet all at once. Different effects such as shapes and sparkles are activated by different gestures and limb speeds. Each effect has a corresponding instrumental sound and each button plays a different piano chord of the C major scale to ensure the music progression sounds musical and cheerful. A screen shot of the KVAP in automatic mode is shown in Figure 2.

Figure 2 Screen shot of the KVAP's automatic mode.

Zoom level and selection of which limb to track can be chosen to encourage movement of specific limbs. This is particularly useful for encouraging movement of impaired limbs. Artwork is saved in bitmap form. This function removes the camera view and buttons on the screen so only the actual artwork is saved.

Pilot study methodology

The objective of performing a pilot study with children with severe impairments was to determine the extent to which the KVAP successfully engages its target population. Children had unstructured time with the KVAP to encourage them to engage physically, facilitating creative exploration while minimising frustration and fatigue. Quantitative data on the children's head and limb movements were recorded for the analysis of the extent to which each child physically engaged with the KVAP. Before completing a full study of the KVAP's benefits and further extending it for use in physical or recreational therapy, a pilot study was required to show that the program achieved its primary functions of being an enjoyable leisure activity, engaging children with impairments and encouraging movement of limbs.

The pilot study was conducted at a local primary school. It included participants with severe impairments between the ages of five and ten years, with enough visual acuity to see the screen at a distance of two metres and the ability to move either a hand or their head intentionally with movements of greater than 10 cm displacement. Participants were not required to have fine motor control. Ethics approval was granted by the Flinders University Social and Behavioural Research Ethics Committee (SBREC) and the Department for Education and Child Development (DECD) in South Australia before commencing the study.

Five participants between the ages of five and ten were recruited for the pilot study from a pool of 12 eligible students. The other participants were excluded from the study due to absence or unsigned permission forms. All had severe impairments of varying types. Four out of the five participants had cerebral palsy, while one had severe autism. All five participants were female. Four of the participants were Level V on the Gross Motor Function Classification System (GMFCS) and one was level IV. All participants were level IV on the Manual Ability Classification System (MACS) and all participants were non-verbal; two were level IV and three were level V on the Communication Function Classification System (CFCS).

Sessions were held weekly at the primary school. Sessions were video recorded to enable the collection of qualitative responses. Five sessions were held for each participant over a two-month period. Due to participant absences for health reasons, an additional catch-up session was held. Participant 5 started the study late because of absences and even with the additional session, only completed four sessions in total. Session lengths were a minimum of three minutes, with the session terminating if the participant communicated that they wanted it to end, became distracted, fatigued or frustrated, or after 15 minutes if none of these signs occurred.

Data collection and analysis

The first session demonstrated the KVAP to the participants, building familiarity with the system and environment, and establishing the best settings for each participant. Specific characteristics of each child were determined by comparing the child's responses to the form pre-filled by parents, to ascertain inter-rater reliability on positive, negative and fatigue responses.

First session observations suggested that the participants were nervous or confused by the new program and did not understand what was expected of them. Participants showed little response to visuals but some response to audio, and an attention span of approximately three minutes.

Four subsequent sessions allowed participants unstructured time using the KVAP, with the system in automatic mode and set to paint. People acquire knowledge and skills more successfully when they are free to move and engage in self-directed activities within their learning environment (10). Verbal encouragement to explore the different functions was provided during early sessions and gradually reduced.

The KVAP recorded quantitative results from each participant's second to fifth sessions except for Participant 5, who had no initial setup and training session. Hence, her data were collected from her first to fourth sessions instead. The researcher also collected qualitative responses of each participant throughout the sessions. The type of data collected is listed in Table I.

Table I. Data recorded and collected

Quantitative (KVAP)	Qualitative (Researcher)
• Limb position on x, y and z axes and the corresponding time when movements occurred • Number of button activations • Which buttons were activated • Number of times special effects were activated Image stored as. bmp at 3-minute intervals • Session length	• Signs of:  - Enjoyment  - Frustration  - Fatigue  - Other emotions • Activities corresponding to positive and negative responses • Frequency and type of prompting required

The x, y and z coordinates were recorded for each limb at approximately five Hertz. The vector displacement was calculated from the x, y and z changes for each unit of time. Average amount of movement per second was calculated so comparisons could be made across sessions. The qualitative data were extracted retrospectively using video recordings.

Results

Each limb's range of movement in x, y and z planes was calculated to track changes in range of motion over the sessions. Typically the volume of space utilised by each limb increased in later sessions, with a decreased range in the z direction for participants’ hands (towards the screen). Each session was graphed minute by minute on an x, y scatter graph to show which areas of the screen were covered by each limb. Four participants showed less movement at the start and end of the sessions but greater movement in the middle. Figure 3 shows the average amount of movement per second for each participant over the entire session for each session. The data showed an increase in average displacement per second over the sessions for all but Participant 3, whose amount of movement remained consistent.

Figure 3. Stacked graphs showing total average displacement for all five participants over their second to fifth sessions.

The trend lines in Figure 4 show that each participant roughly doubled the time they spent using the KVAP by the final session.

Figure 4. Session lengths and trends for all five participants over the five sessions.

Two participants demonstrated obvious signs of enjoyment from the first session but all participants showed signs of enjoyment by the final session, with four out of the five participants increasing the percentage of time showing enjoyment. By the later sessions, two participants demonstrated obvious signs of anticipation before the KVAP was turned on. Another two participants showed signs of happiness before later sessions that may have been manifestations of anticipation. The emotional responses changed over the sessions, with more concentration, more enjoyment and more anticipation showing in later sessions, but while the KVAP did not appear to cause frustration, two participants showed signs of fatigue.

Discussion

The pilot study shows that participants increasingly engaged physically with the KVAP over the five sessions. Their average velocity and volume of movement space increased as well as their attention span and enjoyment. Greater participant enjoyment is likely to have occurred because of increased familiarity with the virtual environment and its exploration. Most children like familiarity and routine, particularly those on the autism spectrum (11). Discovery of new KVAP features may have contributed to the prolonged engagement and suggests participant exploration and motivation.

Increase in the volume of space utilised by each limb and decreased range in the z direction for participants’ hands may indicate greater awareness of the tracking of movements in the x-y plane, as this plane provided the greatest visual feedback in response to movements. The general increase in velocity and range of motion of limbs shows that the KVAP encourages limb activity, which suggests that the program may be beneficial as a physiotherapy tool. These results are in line with Raghavendra and Ahonen-Eerikäinen's studies demonstrating that virtual music, which also provides a creative outlet through movement, encouraged activity for children with severe and multiple disabilities and fostered exploration and engagement (12, 13). The decreased velocity of movement at the start and end of the sessions may demonstrate that participants required time to ‘warm up’ to the KVAP and that fatigue or distraction increased towards the end of a session. Physical activity for children with motor impairments reduces the risks of fatigue and decreased attention (14). This suggests that regular use of the KVAP may increase a user's attention span and the participants may build endurance with extended use.

The observed positive anticipation demonstrates cognitive recognition and expectation, suggesting that the early experiences with the KVAP were enjoyable and satisfying for participants. Anticipation is a sign of neural processing connected to perception, motor and cognitive control, and decision-making (15). The study results correspond to Reid's study on the influence of virtual reality on children with cerebral palsy, which concluded that environments that allowed creativity, expression and control provided the greatest increase in enjoyment, motivation and satisfaction for children with cerebral palsy (16).

Conclusion

The results suggest that with familiarity, the KVAP can provide a fun and engaging activity for children with severe physical impairments. The attention span of participants increased as they explored the program and created their own works of art. A larger study is required to systematically gauge the potential of the KVAP for facilitating therapy, play and exploration.

Key areas for further research include expanding the program to respond to multiple users simultaneously and creating an option to choose which people within the camera view are tracked. This will encourage social engagement and collaboration. As a therapeutic device, the KVAP could possibly be extended to help increase the range of motion of an impaired limb, as well as being used to increase physical activity for wheelchair users.

Conflict of interest and funding

We have no conflict of interest in this study with suppliers or others.

Acknowledgements

The authors gratefully acknowledge Tom Chau (Bloorview Research Institute, Toronto, Canada) for his professional guidance as well as Pierre Duez (Bloorview Research Institute, Toronto, Canada) and Brad Wesson (Flinders University, Adelaide, Australia) for their programming assistance.

Notes

This project was presented at the 7th International Conference on Rehabilitation Engineering and Assistive Technology in Gyeonggi, Korea, from 29 to 31 August 2013.