Boosters of the metaverse: a review of augmented reality-based brain-computer interface

Abstract

Introduction

The ultimate objective of the metaverse is to create a new digital life system using brain-computer interface (BCI) technology that integrates and even enhances reality and virtuality, among which the efficient and natural interaction technology between the virtual and real world is one of the key technologies of the metaverse. Augmented reality-based BCI (AR-BCI) technology provides a channel for direct interaction between the brain and the virtual world by decoding the human brain's response to AR stimuli in real-time, thus the development of the metaverse requires the advancement of AR-BCI technology.

Method

This paper began with the concepts and interrelationships between the metaverse, BCI, and AR technologies. Then the technical characteristics and potential applications of AR-BCI were introduced, and the research progress of three typical AR-BCI paradigms, including SSVEP, P300, and MI, were then summarized.

Conclusion

The survey results indicated that the use of optical see-through glasses to present the SSVEP paradigm has become the key breakthrough direction of AR-BCI research at present. Medical care and robot control are currently the main application areas of AR-BCI. And in the future, AR-BCI will be an important booster to promote the rapid evolution of the metaverse.

Keywords:

• Metaverse
• augmented reality
• brain-computer interface
• SSVEP
• brain–apparatus conversation

1. The development stage of the metaverse

The metaverse is the most eye-catching latest technological innovation in recent years [1]. With the help of augmented reality (AR) technology, the metaverse transcends the limitations of the traditional physical world by building an immersive data mirror world using the real world as a mapping template [2,3]. This virtual world can not only be self-contained, self-circulating, and self-renewing, but also interact with the real world in a new type of time and space, and build a brand-new digital life system that integrates and even enhances reality and virtuality [4]. As shown in the Figure 1, the metaverse integrates several key technologies, each with its specific role and function. Augmented Reality (AR) technology overlays virtual information onto the real world, providing a richer, immersive user experience. Virtual Reality (VR) technology creates entirely virtual environments, immersing users in a new digital world. Brain-Computer Interface (BCI) technology allows users to interact and control using brain signals, enabling direct communication between brain and the external world. Blockchain ensures data security, transparency, and tamper-proof transactions, forming a secure foundation for interactions within the metaverse. The 5 G communication system offers high-speed, low-latency network connectivity, facilitating instant communication and connectivity among various devices in the metaverse. The integration of these technologies creates a multidimensional, highly interactive, and feature-rich digital space, offering unprecedented experiences and possibilities for users [1, 5]. At present, there is no established boundary for the metaverse, and its concept is constantly evolving. In particular, professionals from a variety of sectors are experimenting and attempting new ideas in the fields of medicine, video games, urban planning [6], and online education [7]. The fundamental tenet of the metaverse is the hypothesis of virtual reality compensation: individuals would want to make up for what they lack in the real world in the virtual one, making fiction the driving force behind human civilization. As seen in Figure 2, virtual objects range from historical 2D static objects like calligraphy and paintings to contemporary 2D dynamic movies and well-known 3D dynamic virtual games. According to experts’ predictions, people will be freed from the restrictions of space and time, and our intelligence will carry out deeper evolution with an efficiency that cannot be calculated if the metaverse achieves its ultimate state [8].

Figure 1. Basic composition of the metaverse.

Figure 2. Information complexity of the virtual objects at different times.

The concept of the metaverse evolves across three stages: the virtual metaverse, the brain-connected metaverse, and the consciousness metaverse [2]. Initially, hardware limitations prevented immediate integration into the metaverse. Specific equipment could accurately simulate only a few visual or auditory sensations, making it challenging to replicate other sensory experiences. In the virtual metaverse, VR or AR devices are usually used as display carriers. The BCI is the key technological advancement in the brain-connected world and humans have started to enter the brain-connected metaverse with its help. At the final stage, we can directly experience the sensory information provided by the outside environment through electrical signals instead of using external devices. We are currently in the transient period between the first and second stages of the technical reserve, from the first stage’s basic technology of AR to BCI. At this point, it is inevitable that AR and BCI technologies will be merged, and this will have a favorable impact on how the metaverse develops.

As shown in the Figure 3(a and c), traditional modes of interaction in the metaverse include familiar tools such as keyboards, mice, gaze-based interactions, gesture controls, and voice commands. These methods are user-friendly and easily adaptable. However, they rely on external devices or input methods to facilitate the interaction with the virtual environment, thus this way of interaction is indirect. As shown in the Figure 3(b), AR-BCI provide direct connection between the user’s brain and the virtual environment through decoding brainwave signals, and it brings a deeper sense of immersion and personalized experiences, enabling users to interact more naturally with the virtual environment. For users with physical abilities or communication barriers, AR-BCI could also offer expanded possibilities. Therefore, the emergence of AR-BCI holds significant potential in transforming human-machine interaction mode and user experiences.

Figure 3. Comparison between traditional modes of interaction in the metaverse and AR-BCI. (a) Depicts the process of interacting with the metaverse using voice; (b) illustrates the process of interacting with the metaverse using a brain-computer interface (BCI); (c) demonstrates the process of interacting with the metaverse using gesture controls.

In recent years, AR technology which creates a new virtual environment, and BCI technology which offers a new way for the human brain to communicate with the outside world, have been developed particularly fast. A pioneering endeavor to integrate these two exciting technologies is currently underway, holding immense significance for advancing the evolution of the metaverse. This review aims to provide an overview of the current state of BCI and AR combined systems. The second section briefly introduces the origin and development history of BCI. The third section provides the principle of AR technology and its applications. Subsequently, the fourth section delves into the potential applications arising from the fusion of AR and BCI technologies, outlining the research progress of three exemplary AR-BCI systems. Finally, our discoveries and insights are deliberated upon in the concluding section.

2. History of brain-computer interface

Professor Hawking, a renowned physicist who passed away in 2018, battled amyotrophic lateral sclerosis (ALS). This disease deteriorates peripheral and central motor systems but minimally impacts sensory or cognitive functions. Consequently, he could only generate computer output at a rate of one word per minute by utilizing eye movements. His remarkable research contributions were significantly hindered by his slow typing speed, posing a substantial loss to astrophysics. Apart from ALS, paralysis resulting from brain injury, cerebral infarction, epilepsy [9], and traumatic car accidents can sever the connection between a patient’s brain or volition and their limbs or organs [10]. Even after such illnesses, many patients retain healthy nerve cells and cerebral cortex activity. Their challenge lies in the disruption of signals sent by the cerebral cortex, impeding instructions from reaching their limbs or organs. BCI technology stands as a promising solution to address these issues. It captures “human thought” from the brain using acquisition devices, decodes these signals via a computer, and subsequently transmits them to control external effectors such as spelling devices, manipulators, and intelligent wheelchairs [11]. Initially developed to aid individuals with physical impairments, including robotic arms [12], smart home appliances [13], and mobile phone dialing [14], BCI applications have now expanded. Non-invasive BCI technology enables even non-afflicted individuals to operate drones and games, marking an evolution in its use. In the past few decades, the great potential of BCI has attracted a large number of researchers to study in different directions. In 1988, Farwell et al. developed the P300-BCI system for the first time to communicate with the computer by means of the P300 potential, which could realize the spelling task of words [15]. At the first international BCI meeting held in 1999, the concept of BCI was clearly defined firstly: BCI created a unique pathway for communication between brain activity and external devices, enabling direct brain control of devices without the use of muscles or peripheral nerves [16,17]. Since then, BCI technology based on the classical paradigm has developed rapidly [18]. Gao et al. conducted a series of research on the paradigm designs, recognition algorithms, and application scenarios of SSVEP-BCI [19]. Graz-BCI team explored the prospects of MI-BCI in clinical applications [20]. Serby et al. improved the traditional P300-BCI system to make it achieve a high-speed communication rate while maintaining a lower error rate [21]. In 2004, Schalk et al. released a general BCI research platform, BCI2000, which greatly reduced the workload of building a BCI system and attracted researchers in related fields to devote themselves to BCI research [22]. Hochberg et al. build a BCI-controlled robotic arm system in 2012, and their results showed that two people with long-standing tetraplegia could use it to feed themselves [23]. In fact, the brain not only communicates with computers or equipment but also interacts with the external environment and organs in the body. In order to describe the interaction of the brain more comprehensively and accurately, in 2010, Yao proposed the definition of the brain–apparatus interaction at the first Chinese brain-computer interface competition workshop for the first time and formally proposed the concept of the brain–apparatus conversation (BAC) in 2020 based on the development of interaction technology between the brain and apparatus [24]. In 2011, Koji Takano first applied the experimental paradigm of P300 to AR displays [25]. Our study also showed that the AR-BCI system running with transparent head-mounted displays may help to build an advanced intelligent environment [26,27]. These prior studies opened the prelude to the combination of BCI and AR technologies.

3. Principle of AR technology

The fundamental concept of an AR display involves presenting users with translucent images that adjust as they move [28]. In essence, an object’s reflected light generates a two-dimensional image on the observer’s retina. AR technology manipulates the light beam produced by the light source, modifying it based on image information following a specific path and sequence over time [29]. The optical projection system projects the predetermined outgoing light onto the retina of the human eye [30]. If we simultaneously project a suitable two-dimensional image and a real-world image onto the observer’s retina, then the two-dimensional virtual image will transform into a “three-dimensional object” in reality after the aberration processing of the human cerebral cortex [31,32]. We can create the illusion of seeing three-dimensional objects in the real world from the perspective of the observer [33].

AR technology integrates many features that flat-panel display technology does not have. It is possible to add virtual information that does not exist in reality [34], integrate real scenes and virtual information by building a new large-world coordinate system [35], and virtual and real information can enter human eyes simultaneously without any delay. AR enhances human perception of the world, if AR technology is combined with 5 G technology, ultra-low latency and extremely high data communication rate of the system can be achieved [36]. In recent years, AR technology has developed rapidly, and some application scenarios have been realized in the fields of clinical medicine [37], health monitoring [38], navigation [39], and engineering [40]. Furthermore, with the explosive growth of powerful and affordable mobile devices, and the construction of advanced communication infrastructure, wearable AR devices are becoming more and more popular [41].

At this stage, the metaverse is still gradually migrating from traditional fixed devices to mobile and even wearable devices [42]. The video game interactive devices have been upgraded from computers to mobile phones, and are now evolving into AR/VR glasses. If AR technology achieves another breakthrough, it will greatly improve the efficiency and speed of human-computer interaction [43]. According to the idea that the metaverse is developing towards mobile display, AR equipment is the future entrance of the metaverse.

4. Current research progress of AR-BCI

Compared with traditional display technology, AR display has many advantages. If BCI technology is combined with AR technology, a new research field can be developed, which can even revolutionize many industries including healthcare and robotics [44]. Users can use neural commands to control controllers in AR to improve their lives. Traditional BCI applications use bulky computer screens to present stimulation paradigms, but the large size and weight of the display are inconvenient and obscure the user’s vision.

The eyes are an important window for human beings to express emotions and transmit information. Eyes contact allows us to better express our own emotions and receive the emotions of others, which promotes effective communication [45]. In the traditional BCI systems based on visual evoked EEG signals, computer screens and VR glasses are usually used to present visual stimuli, which will hinder the direct communication between users and the outside world, resulting in the loss of some information, but using AR glasses can avoid this situation. On the other hand, BCI technology provides an efficient interaction channel for head-mounted AR devices. For example, in surgery based on AR technology, the doctor’s hands must be kept sterile and cannot operate AR glasses, so BCI technology is the best solution to replace hands [46]. Many research fields are being promoted by the combination of AR and BCI technology. Arquissandas et al. developed an AR-BCI system that can change the environment to perform exposure therapy for users, and experiments showed that it can adjust the mood of patients and contribute to the recovery of depression [47]. Barresi et al. employed AR-BCI technology to combine the user’s attention with the control of the surgical robot, and created the surgeon’s perception of the real surgical environment based on AR technology. The length of the laser scalpel depend on the user’s level of attention, making it convenient to observe and receive feedback in real time [48]. Zeng et al. proposed a technique for giving consumers visual feedback by utilizing AR devices, which combined eye tracking and EEG-based BCI to enable intuitive and efficient control of the robot arm when perform object manipulation tasks while avoiding obstacles in the workspace [49]. The above research indicated that AR-BCI has great advantages compared with traditional BCI. Next, the existing AR-BCI research are summarized according to the category of stimulus paradigms.

4.1. AR-BCI based on P300

As shown in the Figure 4(a), the Oddball stimulation paradigm, which alternates two events with different probabilities to induce the P300 in the brain waves, is the most well-known experimental design in P300-BCI [50]. As shown in the Figure 4(e), P300 is a type of event-related potential that is named because its positive peak arises 300 ms after stimulation [51], it is highly responsive to the target stimulus, and the lower the probability of the target stimulus, the greater the amplitude of P300 evoked [52]. Using the generation rules of P300, direct communication between the brain and external devices can be realized, such as the P300 speller [53].

Figure 4. Three common paradigms used in designing AR-BCI. (a–c) represent the experimental paradigms for P300, MI, and SSVEP, respectively. (d) Displays the collected EEG signals, (e) shows the time-domain waveform of P300, (f) exhibits the energy topography of imagined left and right-hand movements, and (g) depicts the frequency-domain waveform of SSVEP.

In the initial stage of applying AR technology to the BCI system, many studies were conducted based on P300. In 2010, Lenhardt et al. combined AR and asynchronous P300-based BCI to control a robotic actuator, and experiment results showed that subject could move a specific object on the table among several objects by concentrating on it [54]. Takano et al. added AR to P300-based BCI in 2011, and demonstrated that it could help build advanced intelligent environments [25]. In 2017, Kerous designed an AR-BCI system based on P300, discussed the design options, and provided the direction of future work [55]. In order to compare the performance of P300-BCI in AR and VR, Kim et al. created a BCI application in 2021 that can be used to control aircraft in VR and AR [56]. Although P300-based AR-BCI started earlier, with the gradual rise of other paradigms, research and papers related to P300-based AR-BCI have shown a downward trend in recent years.

4.2. AR-BCI based on MI

As shown in the Figure 4(b and f), it has been discovered that the EEG rhythmic energy of the contralateral cortex of the brain dramatically decreases when the brain controls or imagines the movement of one side of the limb, and such phenomenon is called event-related desynchronization (ERD) [57]. After limb movement or imagination, the energy of the contralateral EEG rhythm returns to the previous level, which is event-related synchronization (ERS) phenomenon. The different spatial characteristics and frequency band characteristics of ERD/ERS of different body parts are the theoretical basis of MI-BCI decoding. On this basis, the human body’s motion intention can be decoded in real time by utilizing the appropriate classification algorithms [58].

In 2010, Chin et al. designed an AR 3D virtual hand which superimposed on the real hand for MI-BCI feedback, and subjects could receive continuous visual feedback from the MI-BCI through the virtual hand movements. After the experiment, subjects reported the AR feedback is more engaging and motivating compared with the conventional horizontal bar feedback [59]. In 2020, Choi et al. combined MI- and SSVEP-based BCI with AR equipment, designed a new asynchronous hybrid AR-BCI system for navigation of quadcopter flight [60]. According to the survey of Al-Qaysi et al. in 2021, the application of MI-BCI is more integrated with VR environment [61].

4.3. AR-BCI based on SSVEP

As shown in the Figure 4(c and g), the electrical activity of neurons in the human cerebral cortex will be modulated when the retina is stimulated by flickering at a fixed frequency, resulting in continuous feedback on the stimulation frequency (under the stimulation frequency or its doubling frequency). This EEG signal, which has a periodic rhythm similar to visual stimuli, is called SSVEP [62]. The spectral peak at the particular stimulus frequency and harmonic is the primary characteristic of SSVEP [63]. Although some noises are unavoidable, SSVEP is still widely used because of its stable spectrum, high signal-noise ratio (SNR), no training required, and less affected by EMG noise [64].

Combining AR technology with SSVEP-BCI is a good choice to improve the practicability of the BCI system. Since 2013, Microsoft, Google and other companies have successively launched various types of AR glasses, which greatly increased the wearability of AR display devices. AR technology facilitates more natural information interaction and simplifies experimental equipment, offering a novel perspective on how to develop the SSVEP-BCI system. The extended virtual flickering stimuli and the real environment appear in the eyes at the same time, increasing the amount of information perceived by the human brain, and enabling users to perform SSVEP experimental tasks in a more realistic environment.

Horri et al. extended the SSVEP experiment to the AR display and verified that SSVEP can be induced in the subject’s brain by rhythmically altering the contrast and brightness of the stimulus in the AR setting [65]. In order to make the interaction between the AR environment and the actual world more natural, Faller et al. employed four stimulus blocks of different frequencies in the AR environment as command and achieved the control of the robot’s route in the real world [66]. Si-Mohammed et al. compared the AR-SSVEP experiment with the traditional CS-SSVEP experiment, their results showed that when the stimulus window length is greater than 2s, there was no significant difference between them in the recognition accuracy, which indicated that it is feasible to conduct SSVEP-BCI experiments in AR devices, and SSVEP-BCI has the portability from computer screen to AR glasses [67]. Ke et al. designed an eight-class BCI designed in an OST-AR headset, and evaluated the accuracies, information transfer rates (ITRs), and SSVEP signal characteristics of the AR-BCI and CS-BCI, which provided methodological guidelines for developing more wearable BCIs in OST-AR environments [68]. Recently, Zhang et al. explored the influence of the layout, quantity, color and other parameters of SSVEP stimulus in AR glasses on the accuracy and ITR of the BCI system, and proved that the experience and rules obtained by computer screen cannot be completely copied to design the SSVEP-BCI system in AR [26,27]. It is worth noting that when SSVEP-BCI is deployed in AR, the advantage of its large instruction set can still be preserved, but longer stimulus presentation time and more efficient decoding algorithm are required [26]. In addition, using AR devices to present flashing stimuli has the advantage of direct interaction with the real environment, but it will also cause the induced SSVEP signal to be interfered by ambient light, which requires special attention when designing and using SSVEP-based AR-BCI [27].

SSVEP-based AR-BCI has been explored in the fields of robot control and surgery. Zhang et al. developed a humanoid robot controlled by SSVEP-BCI based on AR stimulation, and the experiments results showed that 12 subjects could successfully complete robot walking tasks in the maze, which verified the applicability of the AR-BCI system in complex environments and helped to promote the robot control technology [69]. Chen et al. proposed an asynchronous control system of robot arm made of SSVEP-based AR-BCI, and all 8 subjects in the experiment could control the robot arm to complete a jigsaw puzzle [70]. Arpaia et al. integrated medical detection equipment, EEG acquisition device, and AR glasses to allow anesthesiologists to observe the patients’ vital signs gathered from electronic medical equipment in real time, thus complete anesthetic work more efficiently [38]. In medical practice, we look forward to the rapid landing and progress of SSVEP-based AR-BCI technology, which can free doctors’ hands and eyes from the keyboard and screen [71].

5. Future prospects of AR-BCI

Table 1 lists papers on AR-BCI published between 2019 and 2022. The following research trends can be found when compared it with the review summarized by Si-Mohammed et al. [85]. As shown in Figure 5, (1) In the early days, half of the studies used P300 and MI as the BCI paradigm, while most of the recent AR-BCI studies were based on the SSVEP paradigm. (2) Regarding to the display mode of AR devices, 75% of them chose the VST type in the early studies, but in recent years, OST-based AR devices have dominated. (3) Half of the early AR-BCI studies required the use of computers or mobile phones to achieve enhanced display, while recent studies mainly choose head-mounted Hololens glasses as the display device. This trend shows that AR-BCI is developing in the direction of immersion, efficient interaction and portability, which is exactly the demand for BCI technology in the actual application environment and is indispensable for the development of the metaverse.

Figure 5. The pie charts of paradigms and application scenarios of the AR-BCI.

Table 1. Overview of previous systems combining AR and BCIs.

Years	Author	AR type	BCI paradigm	Application
2019	Liu et al. [72]	OST/Hololens 1	SSVEP	Feasibility analysis
2019	Angrisani et al. [73]	OST/Moverio BT-200	SSVEP	Industrial applications
2020	Chen et al. [74]	OST/Hololens 1	SSVEP	Mechanical arm
2020	Ke et al. [68]	OST/Hololens 1	SSVEP	Feasibility analysis
2020	Park et al. [75]	OST/Hololens 1	SSVEP	Home Appliance Control
2020	Zhao et.al. [70]	OST/Hololens 1	SSVEP	Paradigm improvement
2021	Arpaia et al. [38]	OST/Moverio BT-350	SSVEP	Medical health
2021	Kosmyna et al. [76]	OST/Hololens 1	Attention	Attention state
2021	Kim et al. [56]	VST	P300	Quadcopter control
2021	Arpaia et al. [77]	OST	SSVEP	Algorithm decoding
2021	Ravi et al. [78]	OST	SSMVEP	Algorithm improvement
2021	Chen et al. [79]	OST/Hololens 1	SSVEP	Robot control
2022	Zhao et.al. [80]	OST/Hololens 1	SSVEP	Algorithm improvement
2022	Zhang et al. [81]	OST/Hololens 1	SSVEP	Rehabilitation
2022	Zhang et al. [26]	OST/Hololens 1	SSVEP	Paradigm improvement
2022	Si-Mohammed et al. [82]	OST/Hololens 1	SSVEP	Robot control
2022	Andrews et al. [83]	OST/Hololens 1	SSVEP	Medical health
2022	Andrews et al. [84]	OST/Hololens 1	SSVEP	Science education

AR-BCI technology is a technology that is admired by a few people and feared by most people, but it is destined to be integrated into everyone’s life in the future. Every revolutionary technological innovation conforms to the historical law that a few promote it and the majority gradually accept it, but in the end everyone will accept it in an active, passive or half-push way, which is the unstoppable movement of the wheel of history. The metaverse, which is based on the BCI and AR technology, is still very cutting-edge. As an ordinary person, if you don’t deliberately understand it, you may still be at the level of artificial exoskeletons helping paralyzed patients move their fingers. But the brain-computer interconnection in the AR environment will eventually be a revolution that changes the definition of the entire human species and the organizational structure of human society.

While AR-BCI technology holds promise, it faces certain limitations and technical constraints. Presently, challenges persist in terms of accuracy and stability due to environmental interference, individual variability, and inherent noise during brain signal acquisition and processing. Additionally, the computational demand for real-time processing of extensive brain signal data poses constraints, particularly in resource-constrained or mobile device applications. The need for personalization and adaptation also remains a hurdle, as developing universally applicable AR-BCI systems requires tailored training and adaptability to individual brain signal patterns. Addressing these limitations may require advancements in algorithmic sophistication and computational power. Furthermore, there is significant room for advancement in the collection equipment for EEG signals, such as more accurate and convenient dry-electrode EEG caps, as well as ear canal electrodes that can be inserted into the ear, both of which can enhance the usability of EEG collection systems

6. Conclusion

This review is guided by the AR-BCI technology that helps to promote the evolution of the metaverse. Firstly, the composition of the metaverse and its future development direction are concluded. Then, AR-BCI, which combines the advantages of the two, is introduced on the basis of summarizing the characteristics and development history of BCI and AR technologies. The principles and application scenarios of three important AR-BCI paradigms, involving P300, MI, and SSVEP, are reviewed next. Finally, the future research directions for AR-BCI are proposed.

Acknowledgements

This work was supported by the STI 2030-Major Project (2022ZD0208500), and the Technology Project of Henan Province (222102310031).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.