Boosters of the metaverse: a review of augmented reality-based brain-computer interface

Figures & data

Figure 1. Basic composition of the metaverse.

Figure 2. Information complexity of the virtual objects at different times.

Figure 3. Comparison between traditional modes of interaction in the metaverse and AR-BCI. (a) Depicts the process of interacting with the metaverse using voice; (b) illustrates the process of interacting with the metaverse using a brain-computer interface (BCI); (c) demonstrates the process of interacting with the metaverse using gesture controls.

Figure 4. Three common paradigms used in designing AR-BCI. (a–c) represent the experimental paradigms for P300, MI, and SSVEP, respectively. (d) Displays the collected EEG signals, (e) shows the time-domain waveform of P300, (f) exhibits the energy topography of imagined left and right-hand movements, and (g) depicts the frequency-domain waveform of SSVEP.

Figure 5. The pie charts of paradigms and application scenarios of the AR-BCI.

Table 1. Overview of previous systems combining AR and BCIs.

Years	Author	AR type	BCI paradigm	Application
2019	Liu et al. [72]	OST/Hololens 1	SSVEP	Feasibility analysis
2019	Angrisani et al. [73]	OST/Moverio BT-200	SSVEP	Industrial applications
2020	Chen et al. [74]	OST/Hololens 1	SSVEP	Mechanical arm
2020	Ke et al. [68]	OST/Hololens 1	SSVEP	Feasibility analysis
2020	Park et al. [75]	OST/Hololens 1	SSVEP	Home Appliance Control
2020	Zhao et.al. [70]	OST/Hololens 1	SSVEP	Paradigm improvement
2021	Arpaia et al. [38]	OST/Moverio BT-350	SSVEP	Medical health
2021	Kosmyna et al. [76]	OST/Hololens 1	Attention	Attention state
2021	Kim et al. [56]	VST	P300	Quadcopter control
2021	Arpaia et al. [77]	OST	SSVEP	Algorithm decoding
2021	Ravi et al. [78]	OST	SSMVEP	Algorithm improvement
2021	Chen et al. [79]	OST/Hololens 1	SSVEP	Robot control
2022	Zhao et.al. [80]	OST/Hololens 1	SSVEP	Algorithm improvement
2022	Zhang et al. [81]	OST/Hololens 1	SSVEP	Rehabilitation
2022	Zhang et al. [26]	OST/Hololens 1	SSVEP	Paradigm improvement
2022	Si-Mohammed et al. [82]	OST/Hololens 1	SSVEP	Robot control
2022	Andrews et al. [83]	OST/Hololens 1	SSVEP	Medical health
2022	Andrews et al. [84]	OST/Hololens 1	SSVEP	Science education

Liu P, et al. An SSVEP-BCI in augmented reality. Int Conf IEEE Eng Med Biol Soc EMBC. 2019;:4. 41st Annu Oct. 2019. doi: 10.1109/embc.2019.8857859.

 Google Scholar

Angrisani L, Arpaia P, Esposito A, et al. A wearable brain–computer interface instrument for augmented reality-based inspection in industry 4.0. IEEE Trans Instrum Meas. 2020;69(4):1530–1539. doi: 10.1109/TIM.2019.2914712.

 Web of Science ®Google Scholar

Kerous B, Liarokapis F. BrainChat - a collaborative augmented reality brain interface for message communication. 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), Nantes, France, Oct. 2017, pp. 279–283. doi: 10.1109/ISMAR-Adjunct.2017.91.

 Google Scholar

Ke Y, Liu P, An X, et al. An online SSVEP-BCI system in an optical see-through augmented reality environment. J Neural Eng. 2020;17(1):016066. doi: 10.1088/1741-2552/ab4dc6.

 PubMed Web of Science ®Google Scholar

Park S, Cha H-S, Im C-H. Development of an online home appliance control system using augmented reality and an SSVEP-Based brain–computer interface. IEEE Access. 2019;7:163604–163614. doi: 10.1109/ACCESS.2019.2952613.

 Web of Science ®Google Scholar

Zhao X, Liu C, Xu Z, et al. SSVEP stimulus layout effect on accuracy of brain-computer interfaces in augmented reality glasses. IEEE Access. 2020;8:5990–5998. doi: 10.1109/ACCESS.2019.2963442.

 Web of Science ®Google Scholar

Arpaia P, De Benedetto E, Duraccio L. Design, implementation, and metrological characterization of a wearable, integrated AR-BCI hands-free system for health 4.0 monitoring. Measurement. 2021;177:109280. doi: 10.1016/j.measurement.2021.109280.

 Web of Science ®Google Scholar

Tang Z, Sun S, Zhang S, et al. A brain-machine interface based on ERD/ERS for an upper-limb exoskeleton control. Sensors. 2016;16(12):2050. doi: 10.3390/s16122050.

 PubMed Web of Science ®Google Scholar

Kim S, Lee S, Kang H, et al. P300 brain–computer interface-based drone control in virtual and augmented reality. Sensors. 2021;21(17):5765. doi: 10.3390/s21175765.

 PubMed Web of Science ®Google Scholar

Arpaia P, De Benedetto E, Donato N, et al. A wearable SSVEP BCI for AR-based, real-time monitoring applications. 2021 IEEE International Symposium on Medical Measurements and Applications (MeMeA), 2021, pp. 1–6. doi: 10.1109/MeMeA52024.2021.9478593.

 Google Scholar

Ravi A, Lu J, Pearce S, et al. Asynchronous SSMVEP BCI and influence of dynamic background in augmented reality. 2021 27th International Conference on Mechatronics and Machine Vision in Practice (M2VIP), Nov. 2021, pp. 340–343. doi: 10.1109/M2VIP49856.2021.9665078.

 Google Scholar

Chen L, Chen P, Zhao S, et al. Adaptive asynchronous control system of robotic arm based on augmented reality-assisted brain–computer interface. J. Neural Eng. 2021;18(6):066005. Dec doi: 10.1088/1741-2552/ac3044.

 Web of Science ®Google Scholar

Zhao X, Du Y, Zhang R. A CNN-based multi-target fast classification method for AR-SSVEP. Comput Biol Med. 2022;141:105042. doi: 10.1016/j.compbiomed.2021.105042.

 PubMed Web of Science ®Google Scholar

Zhang S, Chen Y, Zhang L, et al. Study on robot grasping system of SSVEP-BCI based on augmented reality stimulus. Tsinghua Sci Technol. 2023;28(2):322–329. doi: 10.26599/TST.2021.9010085.

 Web of Science ®Google Scholar

Zhang R, Xu Z, Zhang L, et al. The effect of stimulus number on the recognition accuracy and information transfer rate of SSVEP–BCI in augmented reality. J Neural Eng. 2022;19(3):036010. doi: 10.1088/1741-2552/ac6ae5.

 Web of Science ®Google Scholar

Si-Mohammed H, Haumont C, Sanchez A. Designing functional prototypes combining bci and ar for home automation. Virtual Real Mix Real. p. 3–21.

 Google Scholar

Andrews A. Integration of augmented reality and brain-computer interface technologies for health care applications: exploratory and prototyping study. JMIR Form Res. 2022;6(4):e18222. Apr doi: 10.2196/18222.

 PubMedGoogle Scholar

Andrews A. Mind power: thought-controlled augmented reality for basic science education. Med Sci Educ. 2022;32(6):1571–1573. doi: 10.1007/s40670-022-01659-x.

 PubMedGoogle Scholar