Quantifying patient experiences with therapeutic neurorehabilitation technologies: a scoping review

Abstract

Purpose

Neurorehabilitation technologies are a novel approach to providing rehabilitation for patients with neurological conditions. There is a need to explore patient experiences. This study aimed; 1) To identify available questionnaires that assess patients’ experiences with neurorehabilitation technologies, and 2) where reported, to document the psychometric properties of the identified questionnaires.

Materials and Methods

Four databases were searched (Medline, Embase, Emcare and PsycInfo). The inclusion criteria were all types of primary data collection that included neurological patients of all ages who had experienced therapy with neurorehabilitation technologies and completed questionnaires to assess these experiences.

Results

Eighty-eight publications were included. Fifteen different questionnaires along with many self-developed scales were identified. These were categorised as; 1) self-developed tools, 2) specific questionnaire for a particular technology, and 3) generic questionnaires originally developed for a different purpose. The questionnaires were used to assess various technologies, including virtual reality, robotics, and gaming systems. Most studies did not report any psychometric properties.

Conclusion

Many tools have been used to evaluate patient experiences, but few were specifically developed for neurorehabilitation technologies and psychometric data was limited. A preliminary recommendation would be use of the User Satisfaction Evaluation Questionnaire to evaluate patient experience with virtual reality systems.

IMPLICATIONS FOR REHABILITATION

• Fifteen unique tools evaluating patient experiences with neurorehabilitation technology were identified

• The User Satisfaction Evaluation and ArmAssist Usability Assessment were designed specifically for therapeutic neurorehabilitation technology

• For all identified tools, psychometric data were poorly reported or not available

• A preliminary recommendation is to use the User Satisfaction Evaluation Questionnaire for evaluating virtual reality systems

Keywords:

• Rehabilitation
• technology
• questionnaire
• patient experience
• survey
• robotics
• virtual reality

Introduction

Neurological conditions are the leading cause of disability and the second leading cause of death globally [1]. Fortunately, survival prospects for patients with neurological disorders have increased due to advances in treatment provision and acute therapies [2]. Despite improvement in medical management, patients often have persisting impairments, resulting in limitations with many occupations, including activities of daily living, leisure activities, work, and recreation [2]. As populations steadily increase, health service providers will likely face an increasing demand for rehabilitation services [3].

Neurorehabilitation technologies, defined as therapeutic technology to deliver or enhance rehabilitation, are emerging as a novel, engaging and potentially beneficial approach to providing rehabilitation [4]. The specific therapeutic technologies continue to evolve, but can include robotics, non-invasive brain stimulation, virtual reality, augmented reality, gaming devices, and bodyweight support treadmill training [5–8]. Therapeutic benefits of neurorehabilitation technologies appear to be underpinned by ability to deliver increased repetitions, duration and intensity of training, reduce physical demands for the therapists, improved patient safety and better cost efficiency [9]. Evidence indicates that neurorehabilitation technologies are likely to be as effective, or superior, compared to traditional forms of therapy in supporting recovery [5]. As such, technology is likely to play an increasing role in neurorehabilitation.

Despite potential therapeutic benefits, it remains important to consider the patient’s perspective when introducing novel therapies. Client-centred practice is vital to support a therapeutic process that is suited to each patient by being responsive to each individual’s values, needs, and preferences [10]. Evaluating patients’ experiences with neurorehabilitation technologies using questionnaires and surveys is one way to gain insight to the patient experience with these emerging therapies [11]. A previous systematic review provided evidence that there were several questionnaires available to assess patient motivation and satisfaction during technology-assisted rehabilitation [12]. Nine different questionnaires were identified; however, their search was limited to robotics, virtual reality, and serious games, and did not include other types of neurorehabilitation technology [12]. Furthermore, it was not reported whether the identified tools had psychometric properties, which could prove important for selecting appropriate measures, or alternatively, may guide future areas of work to evaluate the reliability and validity of the identified questionnaires. Further work is required to collate all available measures to assess patients’ experiences for a wide range of neurorehabilitation technologies and their psychometric properties.

The primary aim of this study was to identify all available questionnaires that assess patients’ experiences with neurorehabilitation technologies. The secondary aim was to extract the psychometric data of the questionnaires where reported. With the increasing use of neurorehabilitation technologies and the continued development of new technology-based therapies, there is value in a comprehensive summary of questionnaires, and their psychometric data, to understand patient experiences and perspectives.

Methods

Design

A scoping review was selected to address the aims of this study. By definition, a scoping review maps the available body of literature on a topic area [13]. It assesses the scope and size of the available literature to identify the extent, quantity and type of research on a given topic [13]. This scoping review was conducted and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses Extension for Scoping Reviews (PRISMA-ScR) [14]. This approach included: 1) defining the research objective; 2) developing a protocol; 3) searching for relevant studies; 4) screening relevant studies; 5) extracting data; 6) summarising and reporting the results [14]. The final protocol was registered with Open Science Framework on 28 January 2022 (https://osf.io/8ay5j/).

Inclusion & exclusion criteria

The review was guided by the population, concept, context framework. The population for this review included people with a neurological condition who have used therapeutic neurorehabilitation technologies. Neurological pathologies included, but were not limited to, stroke, traumatic brain injury, spinal cord injury, Parkinson’s Disease, or multiple sclerosis. There were no other limits on the population regarding age, sex, impairment, or chronicity, as the aim was to seek as much information as possible. The overarching concept was patients’ experiences with neurorehabilitation technologies. Neurorehabilitation technology refers to therapeutic-based technology used to help patients recover. This equipment included upper and lower limb robotics (including resistance-based, accelerometer-based or dynamic systems), exoskeletons, non-invasive brain stimulation, virtual reality, augmented reality, gaming devices, or bodyweight support treadmill training. Assistive technologies such as text-to-speech systems or wheelchairs that were compensatory rather than therapeutic were not included. A secondary element of interest included the reported validity and reliability of the questionnaires, where available. The context for this review was specifically using questionnaires or surveys. Qualitative studies that included interviews or focus groups to assess patients’ experiences were excluded.

Search strategy

Identification of studies was achieved by searching Medline, Embase, Emcare, and PsycInfo databases. The search strategy was developed with the support of an academic librarian and carried out on 28 April 2022. An example search for Medline is provided in Supplementary Appendix 1, and this strategy was adapted for each database. There was no limit set on time or language. Search results were exported into EndNote and Covidence, and duplicates were removed respectively by both software.

Study selection

The screening process was carried out in Covidence independently by two reviewers (CMN and BH), with discrepancies were resolved by discussion, or in consultation with a third reviewer (JU). The first step involved reviewers eliminating irrelevant articles by examining their titles and abstracts. The second step involved reviewers selecting and eliminating articles based on full-text examination. Details of articles included in the review and reasons why articles were excluded were stated by each reviewer.

Data extraction

Two reviewers (CMN and BH) jointly developed a data extraction table to determine the critical pieces of information to chart from each article. One reviewer extracted the data, with a second reviewer checking and confirming the extractions. Discrepancies were resolved by discussion. Data extracted included the author/s, year of publication, study aim, population, intervention type and details, type of questionnaire used, and psychometric data including reliability and validity. Key themes were summarised in a tabular format to integrate and synthesise the information found. Key information that was synthesised included the number of studies that used a specific type of questionnaire, the neurorehabilitation technology for which different questionnaires were used, the psychometric data available for each questionnaire, and the number of different neurological conditions. A quality appraisal was not conducted, as scoping reviews aim to show an overview of existing evidence regardless of the quality of the evidence [14].

Results

Search results

The search identified 2919 articles, which reduced to 2163 once duplicates were removed. After undergoing the two screening processes, 88 studies were included in the final review (Figure 1).

Figure 1. PRISMA flow chart of search results and included studies.

Summary of articles identified through database searching, screening for eligibility, reasons for exclusion and final number of articles included in the review. The search identified 2919 articles and 88 were included in the review.

Description of included studies

The 88 included studies were all published between 2003 and 2022. A broad range of neurological conditions were identified, including people with stroke (n = 50), spinal cord injury (n = 11), multiple sclerosis (n = 11), and cerebral palsy (n = 6). Most studies were an observational design (n = 66), followed by randomised controlled trials (n = 15), case studies (n = 3), non-randomised controlled trials (n = 3), and a pseudo-randomised controlled trial (n = 1). A range of neurorehabilitation technologies were used, including virtual reality (n = 40), robotics (n = 20), exoskeletons (n = 14), video games (n = 11), treadmill training (n = 1), accelerometers and surface electromyography used as a game controller (n = 1), and transcranial direct current stimulation (n = 1). Fifteen different questionnaires were identified, along with many self-developed tools based on likert scales, visual analogue scales, or numeric rating scales, to assess patient experiences with neurorehabilitation technologies. These questionnaires are presented in three categories; 1) Self-developed measures, 2) Tools designed specifically for neurorehabilitation technology, and 3) Generic assessments that were used to assess patient experiences with neurorehabilitation technology but were developed for a different purpose. A summary of the identified tools, the neurological populations which have used each tool and types of neurorehabilitation technology are provided in table 1. Reported psychometric properties for the identified tools are summarised in table 2.

Table 1. Summary of questionnaires identified to evaluate patient experiences with neurorehabilitation technologies.

Measure	Population	Technology
Self-developed tools
Self-developed	Stroke [15–38] CP [39–45] MS [46–51] SCI [16,18,24, 52-60] TBI [18] ABI [61] PD [62] Congenital hemiplegia [63] Tetraplegia [64]	Virtual reality [15,17,20–23,32–35,40,43,44, 46,47,49–51,59,63] Robotics/Exoskeletons [16,18,19,24–29,31,37–39,45,48, 52,53,55–60,64] Treadmill [54] Accelerometers [41] Video games [30,42,61,62] tDCS [36]
Designed specifically for neurorehabilitation technology
USEQ	Balance disorders [11] Stroke [65] MS [66,67]	Virtual reality [11,65–67]
ArmAssist Usability Assessment Questionnaire	Stroke [68]	Robotics [68]
Generic Assessments
SUS	PD [69,70] Stroke [68,71–79] SCI [80] Mixed neurological [81]	Virtual reality [69,70,72,74,75,77,79,80] Robotics/Exoskeleton [68,71,72,76,78] Video games [73,81]
QUEST	MS [82] Stroke [68,83–85] SCI [80]	Robotics/Exoskeleton [68,82,83,85] Virtual reality [80,84]
PIADS	SCI [80]	Virtual reality [80]
Short Feedback Questionnaire	Stroke [79,86–88] ABI [89]	Virtual reality [79,86,88,89] Robotics [87]
IMI	Stroke [68,90–92]	Robotics [68,90,91] Gaming system [92]
TAM tool	Stroke [93–96]	Gaming system [93,96] Virtual reality [94,95]
Usefulness, Satisfaction and Ease of Use Questionnaire	ABI [97]	Virtual reality [97]
Tele-healthcare Satisfaction Questionnaire	PD [98]	Gaming system [98]
Task-Specific Feedback Questionnaire	Stroke [99]	Virtual reality [99]
NASA Task Load Index	Stroke [65,78]	Virtual reality [65] Robotics/Exoskeleton [78]
Questionnaire for User Interface Satisfaction	Stroke [100]	Virtual reality [100]
GEQ	General neurological impairments [101] Stroke [77]	Video games [101] Virtual reality [77]
CSQ-8	Stroke [92]	Gaming system [92]

Measures are summarised in three categories; 1) Self-developed tools, 2) Designed specifically for neurorehabilitation technology, and 3) Generic assessments originally developed for a different purpose. For each tool, there is a summary of the population for which it has been used and the type of technology.

ABI: acquired brain injury; CP: cerebral palsy; CSQ-8: client satisfaction questionnaire 8 item; GEQ: game experience questionnaire; IMI: intrinsic motivation inventory; MS: multiple sclerosis; PD: Parkinson’s disease; PIADS: psychosocial impact of assistive devices scale; QUEST: Quebec user evaluation of satisfaction with assistive technology; SCI: spinal cord injury; SUS: system usability scale; TAM tool: technology acceptance model tool; TBI: traumatic brain injury; tDCS: transcranial direct current stimulation; USEQ: user satisfaction evaluation questionnaire.

Table 2. Summary of psychometric data for identified tools to evaluate patient experiences with neurorehabilitation technologies.

Green solid tick indicates psychometric property reported in study identified in this review, outlined hollow tick indicates psychometric property is reported in literature, but not identified by studies included in this review, red cross indicates psychometric property was not reported.

Self-developed questionnaires

Fifty studies reported use of self-developed assessments to quantify patient experiences. Often these questionnaires were adapted to suit the technology and population of interest. Twenty studies chose to use Likert scales [15–26,39–42,52–55, nine used visual analogue scales [27–31,46,56,57,62], and two used numeric rating scales to assess patient experiences [47,48]. Nineteen of the 50 studies that used self-developed assessments did not detail the features of these questionnaires or how responses were quantified [32–38,43–45,49–51,58–61,63,64].

For self-developed tools that were well described, some clearly reported the questionnaire features. For example, Aguilera-Rubio, et al. [15] created a Likert scale questionnaire comprising nine items that assessed experiences of the usefulness of their virtual reality platform, the degree of patients’ motivation, technical problems encountered by patients, pain levels, and the importance of therapists’ support during the intervention. Each item was scored from one to four, with a maximum score of 36. All questions were directly proportional, meaning the higher the score, the better the patient’s perception of the intervention. An example of a visual analogue scale can be seen in Gagnon, et al. [57], where the self-developed questionnaire contained 41 statements pertaining to seven key areas: overall satisfaction with the robotic exoskeleton training program, satisfaction related to the exoskeleton itself, patients’ perceived learnability, patients’ satisfaction relating to the program, and patients’ perceived health benefits gained from the program. To rate each statement, patients positioned an electronic cursor on a 100 mm visual analogue scale [57]. Many studies produced questionnaires with similar features to Gagnon, et al. [57] and Aguilera-Rubio, et al. [15], such as using Likert scales to assess satisfaction, motivation, and experience [39,45,52,54–56]. As these questionnaires were self-developed, most have not been tested for reliability and validity. Only Gagnon, et al. [57] reported that the content validity of their questionnaire was considered acceptable since it was developed following a review of the literature, along with consultations with therapists and feedback provided by individuals in their target population group.

Specifically developed tools for technology in neurorehabilitation

The User satisfaction evaluation questionnaire (USEQ)

Four studies used the USEQ, a questionnaire specifically designed to evaluate patient satisfaction with virtual reality systems used for rehabilitation [11,65–67]. The original study that created and introduced the USEQ was also identified during the search [17]. This study composed a set of questions evaluating satisfaction derived from the Suitability Evaluation Questionnaire [17]. The USEQ has six questions on a five-point Likert scale, with the total score ranging from six (indicating poor satisfaction) to 30 (indicating excellent satisfaction) [17]. The USEQ has a Cronbach’s alpha of 0.716, indicating adequate internal consistency [17].

The ArmAssist Usability Assessment Questionnaire

One study used this questionnaire and reported that it consisted of a 17-item survey designed for the ArmAssist device used in the study’s robotic system [68]. Patients rated the questions from ‘one’, meaning ‘strongly agree’ to ‘seven’, meaning ‘strongly disagree’ to evaluate satisfaction with the system [68]. The questionnaire also included three open-ended questions about the patient’s subjective opinion, such as aspects of the system they most and least liked, and any ideas for improving the system [68]. No psychometric properties were reported in this study [68].

Generic assessments

The system usability scale

The SUS was originally developed to assess usability of electronic office systems but was used by 14 studies to evaluate patients’ experiences with VR, exoskeletons, robotics, and gaming technologies [68–81]. One study reported that the SUS consisted of ten statements, such as “I thought the system was easy to use”, and patients rated their level of agreement from “strongly agree” to “strongly disagree” on a five-point Likert scale [73]. The total score ranged from 0 to 100 points, with higher scores equating to greater usability [73]. No other study provided further information on the SUS, as evaluating patients’ experiences was a minor aim. None of the included studies reported on the reliability and validity of this questionnaire.

The Quebec User evaluation of satisfaction with assistive technology (QUEST)

The QUEST is composed of 27 items that evaluate satisfaction with assistive devices and related services [83]. Six studies used the QUEST for evaluating patient experiences with therapeutic neurorehabilitation technologies, with one study only using the assistive device subscale of the questionnaire [68,80,82–85]. The therapeutic technologies included robotics, exoskeletons, and virtual reality [68,80,82–85]. None of the included studies reported the psychometric properties of the QUEST.

The psychosocial impact of assistive devices scale (PIADS)

The PIADS was also originally developed for assistive devices rather than therapeutic rehabilitation technology. It measures user perception and psychological factors associated with device use. One study used the PIADS to evaluate patient experiences with a VR system [80]. The authors described the PIADS as a self-reported, 26-item questionnaire that assessed the effects of an assistive device on independence, well-being, and quality of life. The tool is reported to have good internal consistency, test-retest reliability, and construct validity [80].

The short feedback questionnaire (SFQ)

The SFQ was originally developed to evaluate virtual environment tasks by quantifying the sense of presence, perceived difficulty and discomfort. Although not originally designed for health and rehabilitation systems, four virtual reality studies and one robotic study used the SFQ to evaluate patient expereinces with rehabilitation technology [79,86–89]. Demers, et al. [86] reported that the SFQ assessed success, control, enjoyment, realism, immersion, and understandability of computer feedback on a five-point Likert scale, with a rating of ‘one’ meaning “not at all” and ‘five’ meaning “to a great extent”, for a total score of 30 points. Two studies adapted the SFQ and prepared their own questionnaire to fit their virtual reality or robotic system [87,89]. None of the studies reported on the psychometric properties of the SFQ.

The intrinsic motivation inventory (IMI)

The IMI was developed to evaluate subjective experiences and motivation for targeted activities. Four studies used the IMI and reported on features of this questionnaire [68,90–92]. The IMI consisted of six subscales and measured interest/enjoyment, perceived competence, effort, pressure/tension, value and relatedness, which are rated on seven-point Likert scales from ‘one’ being “not at all true” to ‘seven’ being “very true”. None of the included studies reported the psychometric properties of this questionnaire.

The technology acceptance model (TAM) tool

The TAM tool is a generic questionnaire that measures user acceptance and usage of technological devices regarding patients’ perceived ease of use, perceived usefulness, and attitude toward using the technology. Four studies used the TAM tool [93–96] and reported that it contained 24 items rated on a seven-point Likert scale, where a score of ‘one’ means “I do not agree at all” and a score of ‘seven’ means “I agree entirely”. No study reported the questionnaire’s psychometric properties.

The usefulness, satisfaction, and ease of use questionnaire

The usefulness, satisfaction, and ease of use questionnaire measures subjective usability of a product or service. One study used this questionnaire for a virtual reality system and did not report any information or psychometric properties about it, as evaluating experiences was a minor aim [97].

The tele-healthcare satisfaction questionnaire

The tele-healthcare satisfaction questionnaire measures patient satisfaction with telemedicine, but was used by one study to evaluate a gaming system [98]. Authors of that study reported that that the tele-healthcare satisfaction questionnaire measured six different areas, including patients’ perceived benefit, usability, self-concept, privacy and loss of control, quality of life, and wearing comfort [98]. The questionnaire included five statements rated by patients on a five-point Likert scale between ‘zero’, meaning “strongly disagree with the statement”, and ‘four’, meaning “strongly agree with the statement” [98]. This study only administered the benefit, usability, and wearing comfort components of the questionnaire and did not report on psychometric properties [98].

The Task-Specific feedback questionnaire

One study used this questionnaire for a virtual reality system and reported that it was composed of six questions that patients could respond to on a Likert scale from one to five, giving a total score between one and 30 [99]. No information on psychometric properties was reported [99].

NASA Task Load Index

The NASA task load index was originally developed to obtain workload estimates for operators working with various human-machine interfaces. Two studies used this questionnaire and reported that it tested six domains of patients’ self-perception of tool usability [65,78]. The studies that used the NASA Task Load Index did not report on psychometric properties.

The questionnaire for User interface satisfaction

Originally developed to evaluate user acceptance of computer interfaces, this questionnaire was adapted by one study to evaluate a virtual reality system [100]. The study reported that this questionnaire had Cronbach’s alpha of 0.94, which indicates excellent internal consistency [100]. No other information on the features of this questionnaire was reported.

The game experience questionnaire (GEQ)

The GEQ captures the game and playing experience. Two studies used or adapted the GEQ to develop evaluate patients’ experiences with a gaming system [77,101]. The first one, referred to as the “In-Game questionnaire”, was filled in by patients during the intervention and was used to investigate patients’ perceived competence, patients’ immersion in the game, the game’s flow, and the negative and positive effects of the game on patients [101]. The second questionnaire in the study, referred to as the “Post-Game questionnaire”, was filled in at the end of the intervention and was composed of four groups of questions aimed to evaluate patients’ “return back to reality”, patients’ energy levels at the end of the game, and negative or positive experiences [101]. Patients were asked to respond to the questions in both questionnaires on a five-point Likert scale. No psychometric properties were reported in this study.

Client satisfaction questionnaire – 8 (CSQ-8)

The CSQ-8 was originally developed to quantify satisfaction with delivery of services. It comprises 8-items, scored from on a scale from one to four, with higher scores indicating greater satisfaction. One study adopted the CSQ-8 to evaluate satisfaction with a virtual reality system [92], but did not provide any psychometric properties of the tool.

Discussion

This scoping review identified 88 studies that used different questionnaires to assess patients’ experiences with therapeutic neurorehabilitation technologies. Patient experiences were evaluated for a wide range of technologies, including virtual reality, robotic systems, and computer-based games. However, few studies provided psychometric properties of the tools, raising questions about the validity and reliability of these measures. Together, this review provides a comprehensive summary of available tools to evaluate patient experiences with neurorehabilitation technologies and identifies the need to conduct further work to evaluate psychometric properties.

This review identified that self-developed questionnaires were most common, as 50 of the 88 studies included developed their own tool for evaluation. Likert scales were frequently used. An advantage of using Likert scales is that they are universally used in different fields, making them easy to understand [102]. Furthermore, as it is a scale, patients can express their opinions more accurately as opposed to choosing an “either-or” option [102]. However, the disadvantages are low test-retest reliability, statistical limitations, and they can be time consuming to complete [103]. As Likert scales are easily customisable, researchers can specifically curate questionnaires to evaluate patients’ experiences with their particular technology. However, this makes it difficult to compare questionnaires across different studies. It also means that the questionnaires are not often proven to be reliable and valid and therefore may be biased. The review also did not find any information on whether these questionnaires accurately measure patients’ experiences and whether they are an adequate evaluation method.

Our findings identified that several general questionnaires aimed at evaluating technological systems were used, including the SUS, TAM tool and SFQ [73,86,93], with the SUS being the most common questionnaire used. The SUS also uses the Likert scale, as it consists of ten statements rated on a 5-point scale [104]. Some examples of these statements include: “I found the system unnecessarily complex”, “I thought the system was easy to use”, “I felt very confident using the system”, and “I needed to learn a lot of things before I could get going with this system” [104]. Evidently, these statements apply to neurorehabilitation technology in terms of the technological aspect. However, none of the statements capture patients’ experiences with the therapeutic aspect, such as whether the technology is beneficial for their rehabilitation or not. This is because the SUS was originally developed for assessing the usability of electronic office items, not neurorehabilitation technologies [93].

Contrastingly, our review found two questionnaires specifically designed to evaluate patients’ experiences with therapeutic neurorehabilitation technologies. These questionnaires were the USEQ [59], designed to evaluate experience with VR, and the ArmAssist Usability Assessment Questionnaire [68], designed to evaluate a specific robotic system. The USEQ was developed in 2017 and has been used in subsequent work [65,66]. Like the SUS, the USEQ has six statements that patients can rate on a 5-point scale [11]. Statements in the USEQ also specifically capture the therapeutic aspect of the technology, such as “do you think that this system will be helpful for your rehabilitation”, “did you feel discomfort during your experience with the system” and “were you able to control the system” [11]. However, it should be acknowledged that there may be potential bias in response to these questions. For example, responses may be overly positive by those fearful that a negative response might lead to loss of access to the technology being trialled. This could be mitigated by clarifying that responses have no bearing on future access to the technology. Similar to the USEQ, the ArmAssist Usability Assessment Questionnaire is based on Likert scales (7-point scale), but consists of 15 items to evaluate ease of use, comfort, pain, fatigue, enjoyment, benefits, desire for further use, possible difficulties and recommendations. However, psychometric properties do not appear to have been reported for this tool. Given the USEQ was designed specifically for therapeutic VR systems, and that the USEQ has adequate internal consistency [11], a preliminary observation would be that this tool is appropriate for use in evaluating patient experiences with therapeutic VR systems, while acknowledging and mitigating potential bias of overly positive responses. The development of the USEQ demonstrates that questionnaires which capture both technological and therapeutic aspects of experiences with neurorehabilitation technologies can be developed. Since client-centred practice is important in rehabilitation, it is worth questionnaires considering the therapeutic factors of the technology, not just its generic usability. Therefore, this review proposes that there is a need to develop new tools specific to neurorehabilitation technologies.

The vast majority of identified studies did not report on the psychometric properties of the questionnaires. However, this of course does not mean that psychometric evaluation has not been performed. For example, the SUS is reported to have a Cronbach’s alpha of 0.92 [104], indicating excellent internal consistency. Furthermore, it is reported to have strong face validity [104], content validity and concurrent validity [105]. It was also found to have fair agreement between raters when evaluating a medical software platform [106]. The QUEST was found to have good internal consistency (Cronbach alpha 0.82) and test-retest reliably when evaluated with adults who have multiple sclerosis and used assistive technology [107,108]. The QUEST was also developed with subject matter experts (face validity), with content validity evaluated by 12 experts [108,109]. A systematic review on psychometric properties of the PIADS identified high content validity, moderate criterion validity, and concurrent validity reported across a range of studies [110]. Further studies also reported excellent internal consistency and face validity for the Usefulness, Satisfaction, and Ease of Use Questionnaire [111], good internal consistency and construct validity for the telehealth satisfaction questionnaire [112], and good internal consistency and test-retest reliability in older adults for the NASA Task-Load Index [113,114]. Finally, the CSQ-8 was reported to have excellent internal consistency when tested in a community mental health centre (Cronbach alpha 0.92) [115], and internal consistency with a measure for treatment perceptions evaluated in patients with substance abuse [116]. Although these studies were not identified in the search as they did not include neurorehabilitation technology, or were not conducted in a neurological population, there is value in recognising psychometric evaluations performed on these tools. A key outcome of this review is that there are a range of measures available, but few have psychometric properties for patients with a neurological condition who have used neurorehabilitation technologies.

Future directions

A variety of questionnaires were identified in this review, but only two were explicitly made for therapeutic neurorehabilitation technology. As such, there exists a need to develop new reliable and validated questionnaires designed specifically for neurorehabilitation technologies, as generic tools such as the SUS do not cover therapeutic aspects of the technology. We have also identified a need for further research on the use of generic and self-developed questionnaires for patient evaluation. It is critical that the psychometric properties of these types of questionnaires are established to provide evidence that they can truly assess patient experiences within the context of using neurorehabilitation technologies. This is important for the assessment of patient outcomes and treatment planning of the patient’s rehabilitation. Finally, in developing and testing future questionnaires designed specifically for neurorehabilitation technologies, there is need to consider cognitive capacity of potential participants, ensuring questions are free of technical terminology for accurate assessment.

Limitations

Our review was limited to papers published only in English and did not include grey literature. Furthermore, the search was completed in Medline, Emcare, Embase and PsycInfo, and thus studies in other databases may not have been found through the search. Moreover, we only extracted psychometric data if they were reported in the included studies. While we are aware of studies that do report psychometric data for these measures, these evaluations may have been performed in different clinical populations or settings that were not included in this review.

Conclusion

Our review identified a variety of questionnaires to evaluate patient experiences with neurorehabilitation technologies. These tools included self-developed measures, specific questionnaires made for therapeutic technologies and generic questionnaires originally developed for a different purpose. This review is a useful summary of available measures for neurorehabilitation technologies along with reported psychometric data. A preliminary recommendation from our review is to consider use of the USEQ for studies evaluating virtual reality systems as it was specifically developed for this therapeutic technology and has adequate internal consistency, but we suggest caution regarding potential bias for overly positive responses. This review advocates for more research into producing specific questionnaires for neurorehabilitation technologies along with further work to evaluate psychometric properties.

Disclosure statement

BH holds a paid consultancy role for Recovery VR and has a clinical partnership with Fourier Intelligence. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.