Improved dilation CapsuleNet for motor imagery and mental arithmetic classification based on fNIRS

References

• Abougarair AJ, Gnan HM, Oun A, et al. Implementation of a brain-computer interface for robotic arm control. 2021 IEEE 1st International Maghreb Meeting of the Conference on Sciences and Techniques of Automatic Control and Computer Engineering MI-STA. 2021; 58–63. doi: 10.1109/MI-STA52233.2021.9464359.
   Google Scholar
• Zhou Z, Yin E, Liu Y, et al. A novel task-oriented optimal design for P300-based brain-computer interfaces. J Neural Eng. 2014;11(5):056003. doi: 10.1088/1741-2560/11/5/056003.
   PubMed Web of Science ®Google Scholar
• Villringer A, Dirnagl U. Coupling of brain activity and cerebral blood flow: basis of functional neuroimaging. Cerebrovasc Brain Metabol Rev. 1995;7(3):240–276.
   PubMedGoogle Scholar
• Jöbsis FF. Noninvasive, infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters. Science. 1977;198(4323):1264–1267. doi: 10.1126/science.929199.
   PubMed Web of Science ®Google Scholar
• Tinga AM, Clim M-A, de Back TT, et al. Measures of prefrontal functional near-infrared spectroscopy in visuomotor learning. Exp Brain Res. 2021;239(4):1061–1072. doi: 10.1007/s00221-021-06039-2.
   PubMed Web of Science ®Google Scholar
• Wang H, Tang C, Xu T, et al. An approach of one-vs-rest filter bank common spatial pattern and spiking neural networks for multiple motor imagery decoding. IEEE Access. 2020;8:86850–86861. doi: 10.1109/ACCESS.2020.2992631.
   Google Scholar
• Paulmurugan K, Vijayaragavan V, Ghosh S, et al. Brain-computer interfacing using functional near-infrared spectroscopy (fNIRS). Biosensors. 2021;11(10):389.). doi: 10.3390/bios11100389.
   PubMedGoogle Scholar
• Lazarou I, Nikolopoulos S, Petrantonakis PC, et al. EEG-based brain-computer interfaces for communication and rehabilitation of people with motor impairment: a novel approach of the 21st century. Front Hum Neurosci. 2018;12:14. doi: 10.3389/fnhum.2018.00014.
   PubMed Web of Science ®Google Scholar
• Gratton E, Toronov V, Wolf U, et al. Measurement of brain activity by near-infrared light. J Biomed Opt. 2005;10(1):11008. doi: 10.1117/1.1854673.
   PubMed Web of Science ®Google Scholar
• DeBoer T, Scott LS, Nelson CA. Methods for acquiring and analyzing infant event-related potentials. 2013;5–38.
   Google Scholar
• Shellock FG, Kanal E. SMRI safety committee. Policies, guidelines, and recommendations for MR imaging safety and patient management. J Magn Reson Imaging. 1991;1(1):97–101. doi: 10.1002/jmri.1880010114.
   PubMed Web of Science ®Google Scholar
• Kozel FA, Padgett TM, George MS. A replication study of the neural correlates of deception. Behav Neurosci. 2004;118(4):852–856. doi: 10.1037/0735-7044.118.4.852.
   PubMed Web of Science ®Google Scholar
• Kleinschmidt A, Obrig H, Requardt M, et al. Simultaneous recording of cerebral blood oxygenation changes during human brain activation by magnetic resonance imaging and near-infrared spectroscopy. J Cereb Blood Flow Metab. 1996;16(5):817–826. doi: 10.1097/00004647-199609000-00006.
   PubMed Web of Science ®Google Scholar
• Wang Z, Zhang J, Zhang X, et al. Transformer model for functional near-infrared spectroscopy classification. IEEE J Biomed Health Inform. 2022;26(6):2559–2569. doi: 10.1109/JBHI.2022.3140531.
   PubMed Web of Science ®Google Scholar
• Naseer N, Hong KS. fNIRS-based brain-computer interfaces: a review. Front Hum Neurosci. 2015;9:3. doi: 10.3389/fnhum.2015.00003.
   PubMed Web of Science ®Google Scholar
• Guillot A, Di Rienzo F, et al. The neurofunctional architecture of motor imagery. Advanced brain neuroimaging topics in health and disease-methods and applications. 2014; 433–456.
   Google Scholar
• Wang HT, Li T, et al. A motor imagery analysis algorithm based on spatio-temporal-frequency joint selection and relevance vector machine. Kongzhi Lilun Yu Yingyong/Control Theory Appl. 2017;34:1403–1408.
   Google Scholar
• Raichle ME. Behind the scenes of functional brain imaging: a historical and physiological perspective. Proc Natl Acad Sci U S A. 1998;95(3):765–772. doi: 10.1073/pnas.95.3.765.
   PubMed Web of Science ®Google Scholar
• Pfurtscheller G, Neuper C. Neuper motor imagery and direct brain-computer communication. Proc IEEE. 2001;89(7):1123–1134. doi: 10.1109/5.939829.
   Web of Science ®Google Scholar
• Wang H, Bezerianos A. Brain-controlled wheelchair controlled by sustained and brief motor imagery BCIs. Electron Lett. 2017;53(17):1178–1180. doi: 10.1049/el.2017.1637.
   Web of Science ®Google Scholar
• Wang H, Li T, Bezerianos A, et al. The control of a virtual automatic car based on multiple patterns of motor imagery BCI. Med Biol Eng Comput. 2019;57(1):299–309. doi: 10.1007/s11517-018-1883-3.
   PubMed Web of Science ®Google Scholar
• Hong KS, Naseer N, Kim YH. Classification of prefrontal and motor cortex signals for three-class fNIRS-BCI. Neurosci Lett. 2015;587:87–92. doi: 10.1016/j.neulet.2014.12.029.
   PubMed Web of Science ®Google Scholar
• Hsu WY, Sun YN. EEG-based motor imagery analysis using weighted wavelet transform features. J Neurosci Methods. 2009;176(2):310–318. doi: 10.1016/j.jneumeth.2008.09.014.
   PubMed Web of Science ®Google Scholar
• Ang KK, Guan C, Chua KSG, et al. A large clinical study on the ability of stroke patients to use an EEG-based motor imagery brain-computer interface. Clin EEG Neurosci. 2011;42(4):253–258. doi: 10.1177/155005941104200411.
   PubMed Web of Science ®Google Scholar
• Li C, Xu Y, He L, et al. Research on fNIRS recognition method of upper limb movement intention. Electronics. 2021;10(11):1239. doi: 10.3390/electronics10111239.
   Web of Science ®Google Scholar
• Gero JS, Milovanovic J. A framework for studying design thinking through measuring designers’ minds, bodies and brains. Des Sci. 2020;6:e19. doi: 10.1017/dsj.2020.15.
   Web of Science ®Google Scholar
• Ghaffar MSBA, Khan US, Iqbal J, et al. Improving classification performance of four class FNIRS-BCI using mel frequency cepstral coefficients (MFCC). Infrared Phys Technol. 2021;112:103589. doi: 10.1016/j.infrared.2020.103589.
   Web of Science ®Google Scholar
• Liu R, Walker E, Friedman L, et al. fNIRS-based classification of mind-wandering with personalized window selection for multimodal learning interfaces. J Multimodal User Interface. 2021;15(3):257–272. doi: 10.1007/s12193-020-00325-z.
   Web of Science ®Google Scholar
• Xu T, Zhou Z, Yang Y, et al. Motor imagery decoding enhancement based on hybrid EEG-fNIRS signals. IEEE Access. 2023;11:65277–65288. doi: 10.1109/ACCESS.2023.3289709.
   Web of Science ®Google Scholar
• Wang H, Xu T, Tang C, et al. Diverse feature blend based on filter-bank common spatial pattern and brain functional connectivity for multiple motor imagery detection. IEEE Access. 2020;8:155590–155601. doi: 10.1109/ACCESS.2020.3018962.
   Web of Science ®Google Scholar
• A J, M S, Chhabra H, et al. Investigation of deep convolutional neural network for classification of motor imagery fNIRS signals for BCI applications. Biomed Signal Process Control. 2020;62:102133. doi: 10.1016/j.bspc.2020.102133.
   Web of Science ®Google Scholar
• Schudlo LC, Chau T. Dynamic topographical pattern classification of multichannel prefrontal NIRS signals: II. Online differentiation of mental arithmetic and rest. J Neural Eng. 2013;11(1):016003. doi: 10.1088/1741-2560/11/1/016003.
   PubMed Web of Science ®Google Scholar
• Nazeer H, Naseer N, Khan RA, et al. Enhancing classification accuracy of fNIRS-BCI using features acquired from vector-based phase analysis. J Neural Eng. 2020;17(5):056025. doi: 10.1088/1741-2552/abb417.
   PubMed Web of Science ®Google Scholar
• Ho TKK, Gwak J, Park CM, et al. Discrimination of mental workload levels from multi-channel fNIRS using deep leaning-based approaches. IEEE Access. 2019;7:24392–24403. doi: 10.1109/ACCESS.2019.2900127.
   Web of Science ®Google Scholar
• Asgher U, Khalil K, Khan MJ, et al. Enhanced accuracy for multiclass mental workload detection using long short-term memory for brain-computer interface. Front Neurosci. 2020;14:584. doi: 10.3389/fnins.2020.00584.
   PubMed Web of Science ®Google Scholar
• Wang Z, Fang J, Zhang J, et al. Rethinking delayed hemodynamic responses for fNIRS classification. IEEE Trans Neural Syst Rehabil Eng. 2023;31:4528–4538. doi: 10.1109/TNSRE.2023.3330911.
   PubMed Web of Science ®Google Scholar
• Frederick B D, Nickerson LD, Tong Y, et al. Physiological denoising of BOLD fMRI data using regressor interpolation at progressive time delays (RIPTiDe) processing of concurrent fMRI and near-infrared spectroscopy (NIRS). Neuroimage. 2012;60(3):1913–1923. doi: 10.1016/j.neuroimage.2012.01.140.
   PubMed Web of Science ®Google Scholar
• Shibasaki H. Human brain mapping: hemodynamic response and electrophysiology. Clin Neurophysiol. 2008;119(4):731–743. doi: 10.1016/j.clinph.2007.10.026.
   PubMed Web of Science ®Google Scholar
• Guo M-H, Lu C-Z, Liu Z-N, et al. Visual attention network. Comp Visual Media. 2023;9(4):733–752. doi: 10.1007/s41095-023-0364-2.
   Google Scholar
• Van Quang N, Chun J, Tokuyama T. CapsuleNet for micro-expression recognition. 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). 2019; 1–7. doi: 10.1109/FG.2019.8756544.
   Google Scholar
• Bauernfeind G, Scherer R, Pfurtscheller G, et al. Single-trial classification of antagonistic oxyhemoglobin responses during mental arithmetic. Med Biol Eng Comput. 2011;49(9):979–984. doi: 10.1007/s11517-011-0792-5.
   PubMed Web of Science ®Google Scholar
• Pfurtscheller G, Bauernfeind G, Wriessnegger SC, et al. Focal frontal (de)oxyhemoglobin responses during simple arithmetic. Int J Psychophysiol. 2010;76(3):186–192. doi: 10.1016/j.ijpsycho.2010.03.013.
   PubMed Web of Science ®Google Scholar
• Bauernfeind G, Steyrl D, Brunner C, et al. Single trial classification of fNIRS-based brain-computer interface mental arithmetic data. Annu Int Conf IEEE Eng Med Biol Soc. 2014;2014:2004–2007. doi: 10.1109/EMBC.2014.6944008.
   PubMedGoogle Scholar
• Shin J, von Luhmann A, Blankertz B, et al. Open access dataset for EEG + NIRS single-trial classification. IEEE Trans Neural Syst Rehabil Eng. 2016;25(10):1735–1745. doi: 10.1109/TNSRE.2016.2628057.
   PubMed Web of Science ®Google Scholar
• Bak S, Park J, Shin J, et al. Open-access fNIRS dataset for classification of unilateral finger-and foot-tapping. Electronics. 2019;8(12):1486. doi: 10.3390/electronics8121486.
   Web of Science ®Google Scholar
• Ortega P, Zhao T, Faisal AA, et al. HYGRIP: full-stack characterization of neurobehavioral signals (fNIRS, EEG, EMG, force, and breathing) during a bimanual grip force control task. Front Neurosci. 2020;14:919. doi: 10.3389/fnins.2020.00919.
   PubMed Web of Science ®Google Scholar
• Cope M, Delpy DT. System for long-term measurement of cerebral blood and tissue oxygenation on newborn infants by near infra-red transillumination. Med Biol Eng Comput. 1988;26(3):289–294. doi: 10.1007/BF02447083.
   PubMed Web of Science ®Google Scholar
• Sabour S, Frosst N, et al. Dynamic routing between capsules. Adv Neural Inform Proces Syst. 2017;30:3856−3866.
   Google Scholar
• Yin J, Li S, Zhu H, et al. Hyperspectral image classification using CapsNet with well-initialized shallow layers. IEEE Geosci Remote Sensing Lett. 2019;16(7):1095–1099. doi: 10.1109/LGRS.2019.2891076.
   Web of Science ®Google Scholar
• Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision. Proceedings of the IEEE conference on computer vision and pattern recognition. 2016; p. 2818–2826.
   Google Scholar
• Müller R, Kornblith S, Hinton GE. When does label smoothing help. Adv Neural Inform Proces Syst. 2019;32:5.
   Google Scholar
• Trakoolwilaiwan T, Behboodi B, et al. Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a brain-computer interface: three-class classification of rest, right-, and left-hand motor execution. Neurophotonics. 2018;5(1):011008–011008.
   PubMed Web of Science ®Google Scholar
• Loshchilov I, Hutter F. Decoupled weight decay regularization. arXiv Preprint arXiv:1711.05101. 2017.
   Google Scholar
• Zhao X, Zhang H, Zhu G, et al. A multi-branch 3D convolutional neural network for EEG-based motor imagery classification. IEEE Trans Neural Syst Rehabil Eng. 2019;27(10):2164–2177. doi: 10.1109/TNSRE.2019.2938295.
   PubMed Web of Science ®Google Scholar
• Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions. arXiv Preprint arXiv:1511.07122. 2015.
   Google Scholar
• Wang Z, Chen C, Li J, et al. ST-CapsNet: linking spatial and temporal attention with capsule network for P300 detection improvement. IEEE Trans Neural Syst Rehabil Eng. 2023;31:991–1000. doi: 10.1109/TNSRE.2023.3237319.
   Web of Science ®Google Scholar
• Zhang Z, Sabuncu M. Generalized cross entropy loss for training deep neural networks with noisy labels. Adv Neural Inform Proces Syst. 2018;31:154.
   Google Scholar
• Villringer A, Planck J, Hock C, et al. Near infrared spectroscopy (NIRS): a new tool to study hemodynamic changes during activation of brain function in human adults. Neurosci Lett. 1993;154(1–2):101–104. doi: 10.1016/0304-3940(93)90181-j.
   PubMed Web of Science ®Google Scholar
• Menon V, Rivera SM, White CD, et al. Dissociating prefrontal and parietal cortex activation during arithmetic processing. Neuroimage. 2000;12(4):357–365. doi: 10.1006/nimg.2000.0613.
   PubMed Web of Science ®Google Scholar
• Ischebeck A, Zamarian L, Schocke M, et al. Flexible transfer of knowledge in mental arithmetic—an fMRI study. Neuroimage. 2009;44(3):1103–1112. doi: 10.1016/j.neuroimage.2008.10.025.
   PubMed Web of Science ®Google Scholar
• Hardwick RM, Caspers S, Eickhoff SB, et al. Neural correlates of action: comparing meta-analyses of imagery, observation, and execution. Neurosci Biobehav Rev. 2018;94:31–44. doi: 10.1016/j.neubiorev.2018.08.003.
   PubMed Web of Science ®Google Scholar
• Kilintari M, Narayana S, Babajani-Feremi A, et al. Brain activation profiles during kinesthetic and visual imagery: an fMRI study. Brain Res. 2016;1646:249–261. doi: 10.1016/j.brainres.2016.06.009.
   PubMed Web of Science ®Google Scholar
• Fulford J, Milton F, Salas D, et al. The neural correlates of visual imagery vividness-An fMRI study and literature review. Cortex. 2018;105:26–40. doi: 10.1016/j.cortex.2017.09.014.
   PubMed Web of Science ®Google Scholar