A bimodal registration and attention method for speed imagery brain-computer interface

References

• Chaudhary U, Birbaumer N, Ramos-Murguialday A. Brain–computer interfaces for communication and rehabilitation. Nat Rev Neurol. 2016;12(9):513–525. doi: 10.1038/nrneurol.2016.113.
   PubMed Web of Science ®Google Scholar
• Lebedev MA, Nicolelis MA. Brain-machine interfaces: from basic science to neuro-prostheses and neurorehabilitation. Physiol Rev. 2017;97(2):767–837. doi: 10.1152/physrev.00027.2016.
   PubMed Web of Science ®Google Scholar
• Velliste M, Perel S, Spalding MC, et al. Cortical control of a prosthetic arm for self-feeding. Nature. 2008;453(7198):1098–1101. doi: 10.1038/nature06996.
   PubMed Web of Science ®Google Scholar
• Pichiorri F, Morone G, Petti M, et al. Brain–computer interface boosts motor imagery practice during stroke recovery. Ann Neurol. 2015;77(5):851–865. doi: 10.1002/ana.24390.
   PubMed Web of Science ®Google Scholar
• Al-Qaysi Z, Zaidan B, Zaidan A, et al. A review of disability eeg based wheelchair control system: coherent taxonomy, open challenges and recommendations. Comput Methods Programs Biomed. 2018;164:221–237. doi: 10.1016/j.cmpb.2018.06.012.
   PubMed Web of Science ®Google Scholar
• Holz EM, Botrel L, Kaufmann T, et al. Long-term independent brain-computer interface home use improves quality of life of a patient in the locked-in state: a case study. Arch Phys Med Rehabil. 2015;96(3 Suppl):S16–S26. doi: 10.1016/j.apmr.2014.03.035.
   PubMed Web of Science ®Google Scholar
• Scherer R, Muller G, Neuper C, et al. An asynchronously controlled eeg-based virtual keyboard: improvement of the spelling rate. IEEE Trans Biomed Eng. 2004;51(6):979–984. doi: 10.1109/TBME.2004.827062.
   PubMed Web of Science ®Google Scholar
• Bang J-S, Lee M-H, Fazli S, et al. Spatio spectral feature representation for motor imagery classification using convolutional neural networks. IEEE Trans Neural Netw Learn Syst. 2022;33(7):3038–3049. doi: 10.1109/TNNLS.2020.3048385.
   PubMed Web of Science ®Google Scholar
• Sun B, Zhao X, Zhang H, et al. Eeg motor imagery classification with sparse spectro-temporal decomposition and deep learning. IEEE Trans. Automat. Sci. Eng. 2021;18(2):541–551. doi: 10.1109/TASE.2020.3021456.
   Web of Science ®Google Scholar
• Sun B, Zhang H, Wu Z, et al. Adaptive spatiotemporal graph convolutional networks for motor imagery classification. IEEE Signal Process. Lett. 2021;28:219–223. doi: 10.1109/LSP.2021.3049683.
   Web of Science ®Google Scholar
• Sun B, Wu Z, Hu Y, et al. Golden subject is everyone: a subject transfer neural network for motor imagery-based brain computer interfaces. Neural Netw. 2022;151:111–120. doi: 10.1016/j.neunet.2022.03.025.
   PubMed Web of Science ®Google Scholar
• Qi F, Li Y, Wu W. Rstfc: a novel algorithm for spatio-temporal filtering and classification of single-trial eeg. IEEE Trans Neural Netw Learn Syst. 2015;26(12):3070–3082. doi: 10.1109/TNNLS.2015.2402694.
   PubMed Web of Science ®Google Scholar
• Hou Y, Jia S, Lun X, et al. Gcns-net: a graph convolutional neural network approach for decoding time-resolved eeg motor imagery signals. IEEE Trans Neural Netw Learn Syst. 2022;pp:1–12. doi: 10.1109/TNNLS.2022.3202569.
   Google Scholar
• Ofner P, Schwarz A, Pereira J, et al. Attempted arm and hand movements can be decoded from low-frequency eeg from persons with spinal cord injury. Sci Rep. 2019;9(1):7134. doi: 10.1038/s41598-019-43594-9.
   PubMed Web of Science ®Google Scholar
• Shajil N, Mohan S, Srinivasan P, et al. Multiclass classification of spatially filtered motor imagery eeg signals using convolutional neural network for bci based applications. J. Med. Biol. Eng. 2020;40(5):663–672. doi: 10.1007/s40846-020-00538-3.
   Web of Science ®Google Scholar
• Ai Q, Chen A, Chen K, et al. Feature extraction of four-class motor imagery eeg signals based on functional brain network. J Neural Eng. 2019;16(2):026032. doi: 10.1088/1741-2552/ab0328.
   PubMed Web of Science ®Google Scholar
• Zhang H, Zhao X, Wu Z, et al. Motor imagery recognition with automatic eeg channel selection and deep learning. J Neural Eng. 2021;18(1):016004. doi: 10.1088/1741-2552/abca16.
   Web of Science ®Google Scholar
• Edelman BJ, Baxter B, He B. Eeg source imaging enhances the decoding of complex right-hand motor imagery tasks. IEEE Trans Biomed Eng. 2015;63(1):4–14. doi: 10.1109/TBME.2015.2467312.
   PubMed Web of Science ®Google Scholar
• Yuan H, Perdoni C, He B. Relationship between speed and eeg activity during imagined and executed hand movements. J Neural Eng. 2010;7(2):26001. doi: 10.1088/1741-2560/7/2/026001.
   PubMed Web of Science ®Google Scholar
• Yin X, Xu B, Jiang C, et al. A hybrid bci based on eeg and fnirs signals improves the performance of decoding motor imagery of both force and speed of hand clenching. J Neural Eng. 2015;12(3):036004. doi: 10.1088/1741-2560/12/3/036004.
   PubMed Web of Science ®Google Scholar
• Fu Y, Xiong X, Jiang C, et al. Imagined hand clenching force and speed modulate brain activity and are classified by nirs combined with eeg. IEEE Trans Neural Syst Rehabil Eng. 2016;25(9):1641–1652. doi: 10.1109/TNSRE.2016.2627809.
   PubMed Web of Science ®Google Scholar
• Edelman BJ, Meng J, Suma D, et al. Noninvasive neuroimaging enhances continuous neural tracking for robotic device control. Sci Robot. 2019;4(31):peaaw6844 doi: 10.1126/scirobotics.aaw6844.
   Web of Science ®Google Scholar
• Saha S, Baumert M. Intra-and inter-subject variability in eeg-based sensorimotor brain computer interface: a review. Front Comput Neurosci. 2019;13:87. doi: 10.3389/fncom.2019.00087.
   PubMed Web of Science ®Google Scholar
• Ma T, Wang S, Xia Y, et al. Cnn-based classification of fnirs signals in motor imagery bci system. J Neural Eng. 2021;18(5):056019. doi: 10.1088/1741-2552/abf187.
   Google Scholar
• Weiskopf N, Mathiak K, Bock SW, et al. Principles of a brain-computer interface (bci) based on real-time functional magnetic resonance imaging (fmri). IEEE Trans Biomed Eng. 2004;51(6):966–970. doi: 10.1109/TBME.2004.827063.
   PubMed Web of Science ®Google Scholar
• Lee M-H, Kwon O-Y, Kim Y-J, et al. Eeg dataset and openbmi toolbox for three bci paradigms: an investigation into bci illiteracy. GigaScience. 2019;8(5):giz002. doi: 10.1093/gigascience/giz002.
   PubMed Web of Science ®Google Scholar
• Lotte F, Bougrain L, Cichocki A, et al. A review of classification algorithms for eeg-based brain–computer interfaces: a 10 year update. J Neural Eng. 2018;15(3):031005. doi: 10.1088/1741-2552/aab2f2.
   PubMed Web of Science ®Google Scholar
• Zhao H, Zheng Q, Ma K, et al. Deep representation-based domain adaptation for nonstationary eeg classification. IEEE Trans Neural Netw Learn Syst. 2021;32(2):535–545. doi: 10.1109/TNNLS.2020.3010780.
   PubMed Web of Science ®Google Scholar
• Abiri R, Borhani S, Sellers EW, et al. A comprehensive review of eeg-based brain–computer interface paradigms. J Neural Eng. 2019;16(1):011001. doi: 10.1088/1741-2552/aaf12e.
   PubMed Web of Science ®Google Scholar
• Tariq M, Trivailo PM, Simic M. Eeg-based bci control schemes for lower-limb assistive-robots. Front Hum Neurosci. 2018;12:312. doi: 10.3389/fnhum.2018.00312.
   PubMed Web of Science ®Google Scholar
• Ju C, Guan C. Tensor-cspnet: a novel geometric deep learning framework for motor imagery classification. IEEE Trans Neural Netw Learn Syst. 2022;pp:1–15. doi: 10.1109/TNNLS.2022.3172108.
   Google Scholar
• Naseer N, Hong K-S. Fnirs-based brain-computer interfaces: a review. Front Hum Neurosci. 2015;9:3. doi: 10.3389/fnhum.2015.00003.
   PubMed Web of Science ®Google Scholar
• Khan MJ, Ghafoor U, Hong K-S. Early detection of hemodynamic responses using eeg: a hybrid eeg-fnirs study. Front Hum Neurosci. 2018;12:479. doi: 10.3389/fnhum.2018.00479.
   PubMed Web of Science ®Google Scholar
• Liu Z, Shore J, Wang M, et al. A systematic review on hybrid eeg/fnirs in brain-computer interface. Biomed Signal Process Control. 2021;68:102595. doi: 10.1016/j.bspc.2021.102595.
   Web of Science ®Google Scholar
• Shin J, von L¨uhmann A, Blankertz B, et al. Open access dataset for eeg + nirs single-trial classification. IEEE Trans Neural Syst Rehabil Eng. 2016;25(10):1735–1745. doi: 10.1109/TNSRE.2016.2628057.
   PubMed Web of Science ®Google Scholar
• Shin J, Kwon J, Im C-H. A ternary hybrid eeg-nirs brain-computer interface for the classification of brain activation patterns during mental arithmetic, motor imagery, and idle state. Front Neuroinform. 2018;12:5. doi: 10.3389/fninf.2018.00005.
   PubMed Web of Science ®Google Scholar
• Morioka H, Kanemura A, Morimoto S, et al. Decoding spatial attention by using cortical currents estimated from electroencephalography with near-infrared spectroscopy prior information. Neuroimage. 2014;90:128–139. doi: 10.1016/j.neuroimage.2013.12.035.
   PubMed Web of Science ®Google Scholar
• Liu S, Cai W, Liu S, et al. Multimodal neuroimaging computing: a review of the applications in neuropsychiatric disorders. Brain Inform. 2015;2(3):167–180. doi: 10.1007/s40708-015-0019-x.
   PubMedGoogle Scholar
• Ahn S, Nguyen T, Jang H, et al. Exploring neuro-physiological correlates of drivers’ mental fatigue caused by sleep deprivation using simultaneous eeg, ecg, and fnirs data. Front Hum Neurosci. 2016;10:219. doi: 10.3389/fnhum.2016.00219.
   PubMed Web of Science ®Google Scholar
• Ko L-W, Lu Y-C, Bustince H, et al. Multimodal fuzzy fusion for enhancing the motor-imagery-based brain computer interface. IEEE Comput. Intell. Mag. 2019;14(1):96–106. doi: 10.1109/MCI.2018.2881647.
   Web of Science ®Google Scholar
• Fazli S, Mehnert J, Steinbrink J, et al. “Using nirs as a predictor for eeg-based bci performance,” in. Annu Int Conf IEEE Eng Med Biol Soc. 2012;2012:4911–4914. doi: 10.1109/EMBC.2012.6347095.
   PubMedGoogle Scholar
• Han C-H, M¨uller K-R, Hwang H-J. Enhanced performance of a brain switch by simultaneous use of eeg and nirs data for asynchronous brain-computer interface. IEEE Trans Neural Syst Rehabil Eng. 2020;28(10):2102–2112. doi: 10.1109/TNSRE.2020.3017167.
   PubMed Web of Science ®Google Scholar
• Chiarelli AM, Croce P, Merla A, et al. Deep learning for hybrid eeg-fnirs brain–computer interface: application to motor imagery classification. J Neural Eng. 2018;15(3):036028. doi: 10.1088/1741-2552/aaaf82.
   PubMed Web of Science ®Google Scholar
• Sun Z, Huang Z, Duan F, et al. A novel multimodal approach for hybrid brain–computer interface. IEEE Access. 2020;8:89909–89918. doi: 10.1109/ACCESS.2020.2994226.
   Web of Science ®Google Scholar
• Kasabov NK. Neucube: a spiking neural network architecture for mapping, learning and understanding of spatio-temporal brain data. Neural Netw. 2014;52:62–76. doi: 10.1016/j.neunet.2014.01.006.
   PubMed Web of Science ®Google Scholar
• Hosni SM, Borgheai SB, McLinden J, et al. An fnirs-based motor imagery bci for als: a subject-specific data-driven approach. IEEE Trans Neural Syst Rehabil Eng. 2020;28(12):3063–3073. doi: 10.1109/TNSRE.2020.3038717.
   PubMed Web of Science ®Google Scholar
• Ghonchi H, Fateh M, Abolghasemi V, et al. Spatio-temporal deep learning for eeg-fnirs brain computer interface in 2020 42nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2020, p. 124–127.
   Google Scholar
• Snoek CG, Worring M, Smeulders AW. Early versus late fusion in semantic video analysis,” in proceedings of the. 13th annual ACM international conference on Multimedia, 2005, pp. 399–402. doi: 10.1145/1101149.1101236.
   Google Scholar
• Kamrani E, Sawan M. Fully integrated cmos avalanche photodiode and distributed-gain tia for cw-fnirs,” in. 2011 IEEE Biomedical Circuits and Systems Conference (BioCAS). IEEE, 2011, pp. 317–320. doi: 10.1109/BioCAS.2011.6107791.
   Google Scholar
• Zhang Z, Sun B, Gong H, et al. A fast neuronal signal-sensitive continuous-wave near-infrared imaging system. Rev Sci Instrum. 2012;83(9):094301. doi: 10.1063/1.4752021.
   PubMed Web of Science ®Google Scholar
• Gramfort A, Luessi M, Larson E, et al. Mne software for processing meg and eeg data. Neuroimage. 2014;86:446–460. doi: 10.1016/j.neuroimage.2013.10.027.
   PubMed Web of Science ®Google Scholar
• Szachewicz P. Classification of motor imagery for brain-computer interfaces. Poznan university of technology, institute of computing science., Poznán, Poland, 2013. [online] Available: http://www.cs.put.poznan.pl/wjaskowski/pub/theses/pszachewicz_msc.pdf
   Google Scholar
• Comon P. Independent component analysis, a new concept? Signal Process. 1994;36(3):287–314. doi: 10.1016/0165-1684(94)90029-9.
   Web of Science ®Google Scholar
• McFarland DJ, Wolpaw JR. Sensorimotor rhythm-based brain-computer interface (bci): feature selection by regression improves performance. IEEE Trans Neural Syst Rehabil Eng. 2005;13(3):372–379. doi: 10.1109/TNSRE.2005.848627.
   PubMed Web of Science ®Google Scholar
• Scholkmann F, Kleiser S, Metz AJ, et al. A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology. Neuroimage. 2014;85 Pt 1:6–27. doi: 10.1016/j.neuroimage.2013.05.004.
   PubMed Web of Science ®Google Scholar
• Cao S-Y, Shen H-L, Chen S-J, et al. Boosting structure consistency for multispectral and multimodal image registration. IEEE Trans. on Image Process. 2020;29:5147–5162. doi: 10.1109/TIP.2020.2980972.
   Web of Science ®Google Scholar
• Gholipour A, Kehtarnavaz N, Briggs R, et al. Brain functional localization: a survey of image registration techniques. IEEE Trans Med Imaging. 2007;26(4):427–451. doi: 10.1109/TMI.2007.892508.
   PubMed Web of Science ®Google Scholar
• Boyd S, Parikh N, Chu E, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. FNT in Machine Learning. 2010;3(1):1–122. doi: 10.1561/2200000016.
   Google Scholar
• Lu Z, Liu Y, Jin M, et al. Virtual-scanning light-field microscopy for robust snapshot high-resolution volumetric imaging. Nat Methods. 2023;20(5):735–746. doi: 10.1038/s41592-023-01839-6.
   PubMedGoogle Scholar
• Howard AG, Zhu M, Chen B, et al. “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861, 2017.
   Google Scholar
• Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. AdvNeural Inf Processing Syst. 2017;30:5998–6008.
   Google Scholar
• Lawhern VJ, Solon AJ, Waytowich NR, et al. Eegnet: a compact convolutional neural network for eeg-based brain–computer interfaces. J Neural Eng. 2018;15(5):056013. doi: 10.1088/1741-2552/aace8c.
   PubMed Web of Science ®Google Scholar
• Kwak Y, Song W-J, Kim S-E. Fganet: fnirs-guided attention network for hybrid eeg-fnirs brain-computer interfaces. IEEE Trans Neural Syst Rehabil Eng. 2022;30:329–339. doi: 10.1109/TNSRE.2022.3149899.
   PubMed Web of Science ®Google Scholar
• Zhang Y, Sun B, Jiang W, et al. WSQ-AdderNet: efficient weight standardization based quantized AdderNet FPGA accelerator design with high-density INT8 DSP-LUT Co-Packing optimization Proceedings of the 41st IEEE/ACM international conference on computer-aided Design, pp. p. 1–9. 2022.
   Google Scholar