Boosters of the metaverse: a review of augmented reality-based brain-computer interface

References

• Mystakidis S. Metaverse. Encyclopedia. 2022;2(1):486–497. doi: 10.3390/encyclopedia2010031.
   Google Scholar
• Lee S-H, Lee Y-E, Lee S-W. “Toward imagined speech based smart communication system: potential applications on metaverse conditions.” arXiv, Feb. 07, 2022. [cited 2022 Nov 2]. Available from: http://arxiv.org/abs/2112.08569.
   Google Scholar
• Wang H, Ning H, Lin Y, et al. A survey on the metaverse: the state-of-the-Art, technologies, applications, and challenges. IEEE Internet Things J. 2023;10(16):14671–14688. doi: 10.1109/JIOT.2023.3278329.
   Web of Science ®Google Scholar
• Dionisio JDN, Iii WGB, Gilbert R. 3D virtual worlds and the metaverse: current status and future possibilities. ACM Comput. Surv. 2013;45(3):1–38. doi: 10.1145/2480741.2480751.
   Web of Science ®Google Scholar
• Yang Q, Zhao Y, Huang H, et al. Fusing blockchain and AI with metaverse: a survey. IEEE Open J Comput Soc. 2022;3:122–136. doi: 10.1109/OJCS.2022.3188249.
   Google Scholar
• Bibri SE, Allam Z, Krogstie J. The metaverse as a virtual form of data-driven smart urbanism: platformization and its underlying processes, institutional dimensions, and disruptive impacts. Comput Urban Sci. 2022;2(1):24. doi: 10.1007/s43762-022-00051-0.
   PubMedGoogle Scholar
• Tlili A, Huang R, Shehata B, et al. Is metaverse in education a blessing or a curse: a combined content and bibliometric analysis. Smart Learn. Environ. 2022;9(1):24. doi: 10.1186/s40561-022-00205-x.
   Google Scholar
• van der Merwe DF. The metaverse as virtual heterotopia . Proceedings of the. 3rd World Conference on Research in Social Sciences, Oct 2021. doi: 10.33422/3rd.socialsciencesconf.2021.10.61.
   Google Scholar
• Huang L, van G. Brain computer interface for epilepsy treatment. In: Fazel-Rezai, R, editor. Brain-computer interface systems - recent progress and future prospects. InTech; 2013. doi: 10.5772/55800.
   Google Scholar
• Birbaumer N. Breaking the silence: brain?computer interfaces (BCI) for communication and motor control. Psychophysiology. 2006;43(6):517–532. doi: 10.1111/j.1469-8986.2006.00456.x.
   PubMed Web of Science ®Google Scholar
• Tang J, Liu Y, Hu D, et al. Towards BCI-actuated smart wheelchair system. Biomed Eng Online. 2018;17(1):111. doi: 10.1186/s12938-018-0545-x.
   PubMedGoogle Scholar
• Chen X, Zhao B, Wang Y, et al. Combination of high-frequency SSVEP-based BCI and computer vision for controlling a robotic arm. J Neural Eng. 2019;16(2):026012. doi: 10.1088/1741-2552/aaf594.
   PubMed Web of Science ®Google Scholar
• Chai X, Zhang Z, Guan K, et al. A hybrid BCI-controlled smart home system combining SSVEP and EMG for individuals with paralysis. Biomed Signal Process Control. 2020;56:101687. doi: 10.1016/j.bspc.2019.101687.
   Web of Science ®Google Scholar
• Lin JS, Wang M, Lia PY, et al. An SSVEP-based BCI system for SMS in a mobile phone. AMM. 2014;513-517:412–415. doi: 10.4028/www.scientific.net/AMM.513-517.412.
   Google Scholar
• Farwell LA, Donchin E. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr Clin Neurophysiol. 1988;70(6):510–523. doi: 10.1016/0013-4694(88)90149-6.
   PubMedGoogle Scholar
• Wolpaw JR, Birbaumer N, McFarland DJ, et al. Brain–computer interfaces for communication and control. Clin Neurophysiol. 2002;113(6):767–791.
   PubMed Web of Science ®Google Scholar
• Wolpaw JR, McFarland DJ, Vaughan TM. Brain-computer interface research at the wadsworth center. IEEE Trans Rehabil Eng. 2000;8(2):222–226. doi: 10.1109/86.847823.
   PubMedGoogle Scholar
• Wolpaw JR, Birbaumer N, Heetderks WJ, et al. Brain-computer interface technology: a review of the first international meeting. IEEE Trans Rehabil Eng. 2000;8(2):164–173. doi: 10.1109/TRE.2000.847807.
   PubMedGoogle Scholar
• Gao X, Xu D, Cheng M, et al. A BCI-based environmental controller for the motion-disabled. IEEE Trans Neural Syst Rehabil Eng. 2003;11(2):137–140. doi: 10.1109/TNSRE.2003.814449.
   PubMed Web of Science ®Google Scholar
• Pfurtscheller G, Neuper C, Muller GR, et al. Graz-BCI: state of the art and clinical applications. IEEE Trans Neural Syst Rehabil Eng. 2003;11(2):1–4. doi: 10.1109/TNSRE.2003.814454.
   PubMed Web of Science ®Google Scholar
• Serby H, Yom-Tov E, Inbar GF. An improved P300-based brain-computer interface. IEEE Trans Neural Syst Rehabil Eng. 2005;13(1):89–98. doi: 10.1109/TNSRE.2004.841878.
   PubMed Web of Science ®Google Scholar
• Schalk G, McFarland DJ, Hinterberger T, et al. BCI2000: a general-purpose brain-computer interface (BCI) system. IEEE Trans Biomed Eng. 2004;51(6):1034–1043. doi: 10.1109/TBME.2004.827072.
   PubMed Web of Science ®Google Scholar
• Hochberg LR, Bacher D, Jarosiewicz B, et al. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature. 2012;485(7398):372–375. doi: 10.1038/nature11076.
   PubMed Web of Science ®Google Scholar
• Yao D, Zhang Y, Liu T, et al. Bacomics: a comprehensive cross area originating in the studies of various brain–apparatus conversations. Cogn Neurodyn. 2020;14(4):425–442. doi: 10.1007/s11571-020-09577-7.
   PubMed Web of Science ®Google Scholar
• Takano K, Hata N, Kansaku K. Towards intelligent environments: an augmented reality–brain–machine interface operated with a see-through head-mount display. Front Neurosci. 2011;5:60. doi: 10.3389/fnins.2011.00060.
   PubMed Web of Science ®Google Scholar
• Zhang R, Xu Z, Zhang L, et al. The effect of stimulus number on the recognition accuracy and information transfer rate of SSVEP–BCI in augmented reality. J Neural Eng. 2022;19(3):036010. doi: 10.1088/1741-2552/ac6ae5.
   Web of Science ®Google Scholar
• Zhang R, Cao L, Xu Z, et al. Improving AR-SSVEP recognition accuracy under high ambient brightness through iterative learning. IEEE Trans Neural Syst Rehabil Eng. 2023;31:1796–1806. doi: 10.1109/TNSRE.2023.3260842.
   PubMed Web of Science ®Google Scholar
• Carmigniani J, Furht B. Augmented reality: an overview. In: Furht B, editor. Handbook of augmented reality. New York, NY: Springer New York; 2011. pp. 3–46. doi: 10.1007/978-1-4614-0064-6_1.
   Google Scholar
• Lin J, Cheng D, Yao C, et al. Retinal projection head-mounted display. Front Optoelectron. 2017;10(1):1–8. doi: 10.1007/s12200-016-0662-8.
   Google Scholar
• Jang C, Bang K, Moon S, et al. Retinal 3D: augmented reality near-eye display via Pupil-Tracked light field projection on retina. ACM Trans Graph. 2017;36(6):1–13. doi: 10.1145/3130800.3130889.
   Web of Science ®Google Scholar
• Peillard E, Itoh Y, Moreau G, et al. Can retinal projection displays improve spatial perception in augmented reality?. 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2020, pp. 80–89. doi: 10.1109/ISMAR50242.2020.00028.
   Google Scholar
• Xia T, Lou Y, Hu J, et al. Simplified retinal 3D projection rendering method and system. Appl Opt. 2022;61(9):2382–2390. doi: 10.1364/AO.451482.
   PubMed Web of Science ®Google Scholar
• Belcher D, Billinghurst M, Hayes SE, et al. Using augmented reality for visualizing complex graphs in three dimensions . Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings, Tokyo, Japan, 2003, pp. 84–93. doi: 10.1109/ISMAR.2003.1240691.
   Google Scholar
• Carmigniani J, Furht B, Anisetti M, et al. Augmented reality technologies, systems and applications. Multimed Tools Appl. 2011;51(1):341–377. doi: 10.1007/s11042-010-0660-6.
   Web of Science ®Google Scholar
• Lee B, Jo Y, Yoo D, et al. Recent progresses of near-eye display for AR and VR. presented at the Proc.SPIE, Jun. 2021, vol. 11785, p. 1178503. doi: 10.1117/12.2596128.
   Google Scholar
• Siriwardhana Y, Porambage P, Liyanage M, et al. A survey on mobile augmented reality with 5G mobile edge computing: architectures, applications, and technical aspects. IEEE Commun Surv Tutorials. 2021;23(2):1160–1192. doi: 10.1109/COMST.2021.3061981.
   Web of Science ®Google Scholar
• Condino S, Montemurro N, Cattari N, et al. Evaluation of a wearable AR platform for guiding complex craniotomies in neurosurgery. Ann Biomed Eng. 2021;49(9):2590–2605. doi: 10.1007/s10439-021-02834-8.
   PubMed Web of Science ®Google Scholar
• Arpaia P, De Benedetto E, Duraccio L. Design, implementation, and metrological characterization of a wearable, integrated AR-BCI hands-free system for health 4.0 monitoring. Measurement. 2021;177:109280. doi: 10.1016/j.measurement.2021.109280.
   Web of Science ®Google Scholar
• Condino S, Fida B, Carbone M, et al. Wearable augmented reality platform for aiding complex 3D trajectory tracing. Sensors. 2020;20(6):1612. doi: 10.3390/s20061612.
   PubMed Web of Science ®Google Scholar
• Mladenov B, Damiani L, Giribone P, et al. A short review of the SDKs and wearable devices to be used for AR application for industrial working environment. Proceedings of the World Congress on Engineering and Computer Science 2018 Vol I WCECS 2018; 2018. p. 6. .
   Google Scholar
• Starner T, Mann S, Rhodes B, et al. Augmented reality through wearable computing. Presence: Teleoperators Virtual Environ. 1997;6(4):386–398. doi: 10.1162/pres.1997.6.4.386.
   Web of Science ®Google Scholar
• Cascio WF, Montealegre R. How technology is changing work and organizations. Annu Rev Organ Psychol Organ Behav. 2016;3(1):349–375. doi: 10.1146/annurev-orgpsych-041015-062352.
   Google Scholar
• Billinghurst M, Clark A, Lee G. A survey of augmented reality. FNT Human–Compute Interact. 2015;8(2-3):73–272. doi: 10.1561/1100000049.
   Google Scholar
• Nikitenko MS, Zhuravlev SS, Rudometov SV, et al. Walking support control system algorithms testing with brain-computer interface (BCI) and augmented reality (AR) technology integration. IOP Conf Ser: Earth Environ Sci 2018;206(1):012043. doi: 10.1088/1755-1315/206/1/012043.
   Google Scholar
• Bohannon LS, Herbert AM, Pelz JB, et al. Eye contact and video-mediated communication: a review. Displays. 2013;34(2):177–185. doi: 10.1016/j.displa.2012.10.009.
   Web of Science ®Google Scholar
• Blum T, Stauder R, Euler E, et al. Superman-like X-ray vision: towards brain-computer interfaces for medical augmented reality. 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Atlanta, GA, USA, Nov. 2012, pp. 271–272. doi: 10.1109/ISMAR.2012.6402569.
   Google Scholar
• Arquissandas P, Lamas D, Oliveira J. Augmented reality and sensory technology for treatment of anxiety disorders. 2019 14th Iberian Conference on Information Systems and Technologies (CISTI), Coimbra, Portugal, Jun. 2019, pp. 1–4. doi: 10.23919/CISTI.2019.8760859.
   Google Scholar
• Barresi G, Olivieri E, Caldwell DG, et al. Brain-controlled AR feedback design for user’s training in surgical HRI. 2015 IEEE International Conference on Systems, Man, and Cybernetics, Kowloon Tong, Hong Kong, Oct. 2015, pp. 1116–1121. doi: 10.1109/SMC.2015.200.
   Google Scholar
• Zeng H, Wang Y, Wu C, et al. Closed-Loop hybrid gaze brain-Machine interface based robotic arm control with augmented reality feedback. Front Neurorobot. 2017;11:60. doi: 10.3389/fnbot.2017.00060.
   PubMed Web of Science ®Google Scholar
• Picton T. The P300 wave of the human event-related potential. J Clin Neurophysiol. 1992;9(4):456–479. doi: 10.1097/00004691-199210000-00002.
   PubMed Web of Science ®Google Scholar
• Pritchard WS. Psychophysiology of P300. Psychol Bull. 1981;89(3):506–540. doi: 10.1037/0033-2909.89.3.506.
   PubMed Web of Science ®Google Scholar
• Thomas E, Fruitet J, Clerc M. Investigating brief motor imagery for an ERD/ERS based BCI. Annu Int Conf IEEE Eng Med Biol Soc. 2012;2012. p 2929–2932. doi: 10.1109/EMBC.2012.6346577.
   Google Scholar
• Xiao X, Xu M, Han J, et al. Enhancement for P300-speller classification using multi-window discriminative canonical pattern matching. J Neural Eng. 2021;18(4):046079. doi: 10.1088/1741-2552/ac028b.
   Web of Science ®Google Scholar
• Lenhardt A, Ritter H. An augmented-reality based brain-computer interface for robot control. In: Wong KW, Mendis BSU, and Bouzerdoum A, editors. Neural information processing. Models and applications. vol. 6444. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, pp. 58–65. doi: 10.1007/978-3-642-17534-3_8.
   Google Scholar
• Chen X, Huang X, Wang Y, et al. Combination of augmented reality based brain- computer interface and computer vision for high-level control of a robotic arm. IEEE Trans Neural Syst Rehabil Eng. 2020;28(12):3140–3147. Dec doi: 10.1109/TNSRE.2020.3038209.
   PubMed Web of Science ®Google Scholar
• Kim S, Lee S, Kang H, et al. P300 brain–computer interface-based drone control in virtual and augmented reality. Sensors. 2021;21(17):5765. doi: 10.3390/s21175765.
   PubMed Web of Science ®Google Scholar
• Kosmyna N, Wu Q, Hu C-Y, et al. Assessing internal and external attention in AR using brain computer interfaces: a pilot study. 2021 IEEE 17th International Conference on Wearable and Implantable Body Sensor Networks (BSN), Athens, Greece, Jul. 2021, pp. 1–6. doi: 10.1109/BSN51625.2021.9507034.
   Google Scholar
• Li P, Gao X, Li C, et al. Granger causal inference based on dual laplacian distribution and its application to MI-BCI classification. IEEE Trans Neural Netw. Learning Syst. 2023:1–15. doi: 10.1109/TNNLS.2023.3292179.
   Web of Science ®Google Scholar
• Chin ZY, Ang KK, Wang C, et al. Online performance evaluation of motor imagery BCI with augmented-reality virtual hand feedback. Annu Int Conf IEEE Eng Med Biol Soc. Aug. 2010;2010 . p. 3341–3344. doi: 10.1109/IEMBS.2010.5627911.
   Google Scholar
• Choi J, Jo S. Application of hybrid brain-computer interface with augmented reality on quadcopter control. 2020 8th International Winter Conference on Brain-Computer Interface (BCI), Feb. 2020, pp. 1–5. doi: 10.1109/BCI48061.2020.9061659.
   Google Scholar
• Al-Qaysi ZT, Ahmed MA, Hammash NM, et al. Systematic review of training environments with motor imagery brain–computer interface: coherent taxonomy, open issues and recommendation pathway solution. Health Technol. 2021;11(4):783–801. doi: 10.1007/s12553-021-00560-8.
   Web of Science ®Google Scholar
• Gao S, Wang Y, Gao X, et al. Visual and auditory brain–computer interfaces. IEEE Trans Biomed Eng. 2014;61(5):1436–1447. doi: 10.1109/TBME.2014.2300164.
   PubMed Web of Science ®Google Scholar
• Lin Z, Zhang C, Wu W, et al. Frequency recognition based on canonical correlation analysis for SSVEP-Based BCIs. IEEE Trans Biomed Eng. 2006;53(12 Pt 2):2610–2614. doi: 10.1109/TBME.2006.886577.
   PubMed Web of Science ®Google Scholar
• Müller-Putz GR, Pfurtscheller G. Control of an electrical prosthesis with an SSVEP-Based BCI. IEEE Trans Biomed Eng. 2008;55(1):361–364. doi: 10.1109/TBME.2007.897815.
   PubMed Web of Science ®Google Scholar
• Horii S, Nakauchi S, Kitazaki M. AR-SSVEP for brain-machine interface: e stimating user’s gaze in head-mounted display with USB camera . 2015 IEEE Virtual Reality (VR), Mar. 2015, pp. 193–194. doi: 10.1109/VR.2015.7223361.
   Google Scholar
• Faller J, Allison BZ, Brunner C, Scherer R, Schmalstieg D, Pfurtscheller G, and Neuper C. A feasibility study on SSVEP-based interaction with motivating and immersive virtual and augmented reality. 2017; arXiv preprint arXiv:1701.03981.
   Google Scholar
• Si-Mohammed H, Petit J, Jeunet C, et al. Towards BCI-Based interfaces for augmented reality: feasibility, design and evaluation. IEEE Trans Vis Comput Graph. 2020;26(3):1608–1621. doi: 10.1109/TVCG.2018.2873737.
   PubMed Web of Science ®Google Scholar
• Ke Y, Liu P, An X, et al. An online SSVEP-BCI system in an optical see-through augmented reality environment. J Neural Eng. 2020;17(1):016066. doi: 10.1088/1741-2552/ab4dc6.
   PubMed Web of Science ®Google Scholar
• Zhang S, Gao X, Chen X. Humanoid robot walking in maze controlled by SSVEP-BCI based on augmented reality stimulus. Front Hum Neurosci. 2022;16:908050. doi: 10.3389/fnhum.2022.908050.
   PubMed Web of Science ®Google Scholar
• Zhao X, Liu C, Xu Z, et al. SSVEP stimulus layout effect on accuracy of brain-computer interfaces in augmented reality glasses. IEEE Access. 2020;8:5990–5998. doi: 10.1109/ACCESS.2019.2963442.
   Web of Science ®Google Scholar
• Wang J, Suenaga H, Hoshi K, et al. Augmented reality navigation with automatic Marker-Free image registration using 3-D image overlay for dental surgery. IEEE Trans Biomed Eng. 2014;61(4):1295–1304. doi: 10.1109/TBME.2014.2301191.
   PubMed Web of Science ®Google Scholar
• Liu P, et al. An SSVEP-BCI in augmented reality. Int Conf IEEE Eng Med Biol Soc EMBC. 2019;:4. 41st Annu Oct. 2019. doi: 10.1109/embc.2019.8857859.
   Google Scholar
• Angrisani L, Arpaia P, Esposito A, et al. A wearable brain–computer interface instrument for augmented reality-based inspection in industry 4.0. IEEE Trans Instrum Meas. 2020;69(4):1530–1539. doi: 10.1109/TIM.2019.2914712.
   Web of Science ®Google Scholar
• Kerous B, Liarokapis F. BrainChat - a collaborative augmented reality brain interface for message communication. 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), Nantes, France, Oct. 2017, pp. 279–283. doi: 10.1109/ISMAR-Adjunct.2017.91.
   Google Scholar
• Park S, Cha H-S, Im C-H. Development of an online home appliance control system using augmented reality and an SSVEP-Based brain–computer interface. IEEE Access. 2019;7:163604–163614. doi: 10.1109/ACCESS.2019.2952613.
   Web of Science ®Google Scholar
• Tang Z, Sun S, Zhang S, et al. A brain-machine interface based on ERD/ERS for an upper-limb exoskeleton control. Sensors. 2016;16(12):2050. doi: 10.3390/s16122050.
   PubMed Web of Science ®Google Scholar
• Arpaia P, De Benedetto E, Donato N, et al. A wearable SSVEP BCI for AR-based, real-time monitoring applications. 2021 IEEE International Symposium on Medical Measurements and Applications (MeMeA), 2021, pp. 1–6. doi: 10.1109/MeMeA52024.2021.9478593.
   Google Scholar
• Ravi A, Lu J, Pearce S, et al. Asynchronous SSMVEP BCI and influence of dynamic background in augmented reality. 2021 27th International Conference on Mechatronics and Machine Vision in Practice (M2VIP), Nov. 2021, pp. 340–343. doi: 10.1109/M2VIP49856.2021.9665078.
   Google Scholar
• Chen L, Chen P, Zhao S, et al. Adaptive asynchronous control system of robotic arm based on augmented reality-assisted brain–computer interface. J. Neural Eng. 2021;18(6):066005. Dec doi: 10.1088/1741-2552/ac3044.
   Web of Science ®Google Scholar
• Zhao X, Du Y, Zhang R. A CNN-based multi-target fast classification method for AR-SSVEP. Comput Biol Med. 2022;141:105042. doi: 10.1016/j.compbiomed.2021.105042.
   PubMed Web of Science ®Google Scholar
• Zhang S, Chen Y, Zhang L, et al. Study on robot grasping system of SSVEP-BCI based on augmented reality stimulus. Tsinghua Sci Technol. 2023;28(2):322–329. doi: 10.26599/TST.2021.9010085.
   Web of Science ®Google Scholar
• Si-Mohammed H, Haumont C, Sanchez A. Designing functional prototypes combining bci and ar for home automation. Virtual Real Mix Real. p. 3–21.
   Google Scholar
• Andrews A. Integration of augmented reality and brain-computer interface technologies for health care applications: exploratory and prototyping study. JMIR Form Res. 2022;6(4):e18222. Apr doi: 10.2196/18222.
   PubMedGoogle Scholar
• Andrews A. Mind power: thought-controlled augmented reality for basic science education. Med Sci Educ. 2022;32(6):1571–1573. doi: 10.1007/s40670-022-01659-x.
   PubMedGoogle Scholar
• Si-Mohammed H, Argelaguet F, Casiez G, et al. Brain-computer interfaces and augmented reality: a state of the art. Graz Brain-Computer Interface Conference, 2017. Available from: https://api.semanticscholar.org/CorpusID:322089.
   Google Scholar