Improved dilation CapsuleNet for motor imagery and mental arithmetic classification based on fNIRS

Abstract

Purpose

This study aimed to improve the accuracy of brain-computer interface (BCI) systems based on motor imagery (MI) and mental arithmetic (MA) by utilizing functional near-infrared spectroscopy (fNIRS) and an improved dilation CapsuleNet (ID-CapsuleNet) model.

Methods

The study focused on the characteristics of fNIRS and employed large-kernel dilation convolution to extract hemodynamic features from fNIRS data. Inspired by CapsuleNet’s success in image classification, an ID-CapsuleNet model was designed, combining large-kernel dilation convolution and CapsuleNet. Four publicly available datasets (A, B, C, and D) were utilized for evaluating the proposed model. Datasets A and B were MA type, while datasets C and D were MI type. Ablation experiments were conducted to assess the usefulness of large-kernel convolution, dynamic routing, and dilation convolution.

Results

The average accuracies for each dataset were 95.01%, 76.88%, 74.03%, and 80.29% respectively. Cross-subject average accuracies were 88.72%, 75.80%, 75.78%, and 80.34%. Ablation experiments confirmed the importance of large-kernel convolution, dynamic routing, and dilation convolution in the ID-CapsuleNet model.

Conclusion

The developed ID-CapsuleNet model demonstrated promising potential for enhancing the performance of BCI systems based on MI and MA. The findings contribute to the advancement of BCI technology, offering improved assistive tools for disabled individuals.

Keywords:

• Brain-computer interface
• motor imagery
• mental arithmetic
• fNIRS
• CapsuleNet
• dilation convolution

1. Introduction

Brain-Computer Interface (BCI) technology is a cutting-edge field that enables direct communication with the brain for controlling external devices or performing various tasks [1,2]. Functional Near-Infrared Spectroscopy (fNIRS) is a non-invasive brain imaging technique that utilizes near-infrared light to measure changes in the concentrations of oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) in brain tissue, thereby detecting alterations in cerebral blood flow [3,4]. This allows for the study of brain activity, providing real-time information about brain function. In comparison to other brain imaging techniques such as electroencephalography (EEG) and functional magnetic resonance imaging (fMRI), fNIRS has several advantages. It is non-invasive, resistant to interference, and portable [5–9]. It is suitable for individuals of different ages, including infants and the elderly, because it does not require significant cooperation or immobility. In contrast, EEG may need additional attention and preparation for infants and young children [10], while fMRI for children and young participants may require the use of sedatives that may not be suitable for certain populations, such as pregnant women or individuals with metal implants [11,12]. Moreover, although fNIRS has lower spatial resolution compared to fMRI, fNIRS devices are more convenient and suitable for mobile scenarios [13]. In comparison to EEG, fNIRS shows an advantage of the higher spatial resolution. It is less susceptible to electrical noise and motion artifacts [14,15].

Motor Imagery (MI) is a brain activity in which neural signals related to actual movements are generated when individuals imagine performing a specific action [16,17]. This mental activity is accompanied by physiological changes in brain blood oxygenation levels [18]. Using fNIRS technology, it is possible to capture changes in blood flow in brain regions triggered by MI, allowing for control of external devices or task execution through mental imagery [19–21]. Mental Arithmetic (MA) is a cognitive activity unrelated to physical movements. BCI systems that combine fNIRS technology can identify different MA activities and use them to control external devices [22]. The fNIRS-based BCI has potential applications in various fields, including rehabilitation medicine, human-computer interaction, and neuroscience research [23]. In the field of rehabilitation medicine, fNIRS-based BCI for MI/MA can assist stroke, spinal cord injury, and other motor impairment patients in regaining their motor abilities [24,25]. Additionally, researchers can use fNIRS and BCI to explore cognitive processes and understand brain functions related to movement, decision-making, and concentration [26]. In the field of BCI based on fNIRS for MA, there have been advancements in both personalized adaptation and real-time feedback. Personalized BCI systems are under development, optimizing according to each user’s brain characteristics and the nature of the MA task. Real-time and effective feedback mechanisms are aiding users in better control and execution of MA tasks [14, 27].

fNIRS classification primarily involves both traditional machine learning and deep learning approaches. For instance, statistical features such as mean, peak, and slope extracted from fNIRS signals are utilized for training Support Vector Machines (SVM), Linear Discriminant Analysis (LDA), and Hidden Markov Models (HMM). Additionally [15, 22, 28–32], vectors like the changes in cerebral blood volume (+CBV) and vector magnitude can also be employed for the training [33]. However, traditional machine learning heavily relies on manually crafted feature extraction and prior knowledge. In recent years, deep learning has emerged as the mainstream in near-infrared spectroscopy classification research. Deep learning models can directly learn and extract diverse features from raw fNIRS signals, providing an end-to-end approach [34,35]. Specifically, Convolutional Neural Networks (CNNs) and Long Short-Term Memory networks (LSTMs) have been applied in multiple fNIRS-based Brain-Computer Interface (BCI) applications [31, 35]. Due to the image-like characteristics of multi-channel near-infrared signals, which can be organized into multiple matrices, they can be treated as image data for processing [14]. This approach, leveraging convolutional layers for spatial and channel-level feature extraction followed by classification using Transformer, has demonstrated high performance [14]. fNIRSNet, an efficient and concise knowledge-driven model, incorporates delayed hemodynamic responses as domain knowledge into the model design [36]. This streamlined model has achieved performance nearly comparable to larger models.

When individuals perform MI/MA tasks, different brain regions exhibit varying levels of activity, and relevant neural signals activate within the brain, resulting in changes in brain oxygenation, leading to the emergence of the cerebral hemodynamic response. Therefore, extracting features related to the hemodynamic response of fNIRS is crucial for fNIRS-based BCI for MI/MA [36]. Due to the inherent delay in the hemodynamic response of the brain, long-range dependencies are observed in the temporal dimension [37,38]. Convolution is typically used to extract image features, whereas large-kernel convolutions can capture these long-range dependencies but introduce parameter redundancy [39]. Dilation convolutions can address this issue by reducing parameters. CapsuleNet [40], a type of deep learning architecture, offer some interpretability and exhibit a degree of resistance to transformations compared to traditional convolutional neural networks (CNN). While this approach has not been applied to the field of fNIRS-based BCI, we aim to propose a network architecture that combines CapsuleNet structures with large dilation convolutions for classification tasks using fNIRS signals.

2. Datasets and participants

In this study, four datasets are adopted for the validation of the proposed method. The detail of the datasets is described below.

Dataset A was designed to investigate how the levels of HbR and HbO in the prefrontal cortex change when individuals perform MA [41–43]. Dataset B is a publicly accessible dataset that primarily concentrates on analyzing visual instructions related to MA tasks [44]. Dataset C is an open-access three-class classification dataset [45]. This dataset comprises three types of movements: right-hand finger-tapping (RHT), left-hand finger-tapping (LHT), and foot-tapping (FT). Dataset D is an open-access binary classification dataset that includes left and right hand grip data [46].

Table 1 lists the number of subjects, experimental paradigms, and task specifications for each dataset.

Table 1. Details of the four datasets.

Dataset	Subject	Paradigm	Task specification
Dataset A	8 (3males,5females, 26.0 ± 2.8 years old)	Mental arithmetic; Task period (12s); Rest period (28s)	In the instruction phase, participants memorized a subtraction problem on the screen. During the activity phase, they solved problems by subtracting a single-digit number sequentially. In the resting phase, participants took a break, following instructions to stare at a black screen.
Dataset B	29(14males,15females 28.5 ± 3.7 years old)	Mental arithmetic; Task period (10s) Rest period (15–17s)	The MA task in question shares similarities with Dataset D, while the BL task involved subjects being allowed to relax and fixate their gaze on a black cross that appeared before them. To mark the beginning and end of the task period, a brief 250 ms beep was played.
Dataset C	30 (17males,13females, 23.4 ± 2.5 years old)	Motor imagery; Task period (10s); Rest period (17–19s)	The task started with a brief sound, followed by on-screen instructions for movements with the right hand (RHT), left hand (LHT), and feet (FT). After completion, a rest period started with a sound and a “STOP” sign, signaling participants to relax. Volunteers performed assigned movements continuously throughout the task.
Dataset D	14	Motor imagery; Task period (21s); Rest period (15–21s)	Participants gripped with either hand in 10 cycles: 1.55s contraction, 0.55s relaxation, inducing rapid force changes. They followed instructions to exert force (25%–50% MVC), synchronized with the rhythm of cycles

3. Method

3.1. Signal preprocessing

According to the Beer-Lambert law [47], the concentration changes of cerebral hemoglobin (ΔHbR and ΔHbO) can be inferred by measuring the change in optical density (Δ$\mathit{\text{OD}}$) of light during a specific time interval (Δ$t$), along with known values for the absorption coefficient ($\epsilon$) and the path length (d). (1) $$\begin{array}{c}\left[\begin{array}{c}\Delta \mathit{\text{HbR}}\\ \Delta \mathit{\text{HbO}}\end{array}\right]=\frac{1}{d}{\left[\begin{array}{cc}{\epsilon }_{\mathit{\text{HbR}}}\left({\lambda }_1\right)& {\epsilon }_{\mathit{\text{HbO}}}\left({\lambda }_1\right)\\ {\epsilon }_{\mathit{\text{HbR}}}\left({\lambda }_2\right)& {\epsilon }_{\mathit{\text{HbO}}}\left({\lambda }_2\right)\end{array}\right]}^{-1}\left[\begin{array}{c}\frac{\mathrm{\Delta }\mathit{\text{OD}}\left(\mathrm{\Delta }t,{\lambda }_1\right)}{\mathit{\text{DPF}}\left({\lambda }_1\right)}\\ \frac{\mathrm{\Delta }\mathit{\text{OD}}\left(\mathrm{\Delta }t,{\lambda }_2\right)}{\mathit{\text{DPF}}\left({\lambda }_2\right)}\end{array}\right],\end{array}$$(1)

In this context, ${\lambda }_1$ and ${\lambda }_2$ represent different illumination wavelengths, and $\mathit{\text{DPF}}$ refers to the differential path length factor for $\lambda .$

To remove noise and motion artifacts, we applied filtering techniques to each of the four datasets based on previous research. Dataset A was filtered using a fourth-order low-pass Butterworth filter with a cutoff frequency of 0.009 Hz [41]. Dataset B utilized a bandpass filter with a frequency range of 0.01–0.1 Hz [42], while Dataset C employed a third-order Butterworth filter with a frequency range of 0.01–0.1 Hz [45]. Dataset D employed a seventh-order elliptical low-pass filter with a cutoff frequency of 0.25 Hz [46].

3.2. Feature extraction

Selecting features relevant to the task paradigm is crucial for effective classification in fNIRS. fNIRS data consists of information from both HbO and HbR, represented as a C × H × W matrix, where C represents the two chromophores (HbO and HbR), H is the number of channels, and W is the number of sampling points. Normalizing the fNIRS signals through z-score standardization is employed to ensure the preservation of fNIRS features effectively in this arrangement [14]. Taking into account the task paradigm and the characteristics of the brain’s hemodynamic response, we utilize large-kernel dilation convolution (Conv_dilation) to focus on the hemodynamic response features of brain regions, aiming to extract the long-range dependency features of fNIRS (Figure 1). The CapsuleNet we have introduced, incorporating Conv_dilation, is referred to as improved dilation CapsuleNet (ID-CapsuleNet).

Figure 1. The Conv_dilation extract hemodynamic response information from fNIRS.

3.3. Structure of ID-CapsuleNet

In the ID-CapsuleNet, we start by using two Conv_dilation to extract long-range dependencies from fNIRS data, with specific parameters dilation in Table 2. Following the first dilation convolution operation and relu activation, we proceed with the second dilation convolution to complete feature extraction. Next, we partition the extracted features into p segments, each of size 1 × 8. The number of segments, p, for each of the four datasets is specified in Table 2. Subsequently, we transform these feature segments into primary capsule vectors, each of length 8. The number of primary capsule vectors matches the number of feature segments. These primary capsule vectors are then fed into the dynamic routing [48], which produces q 16-dimensional output capsules. The size of each of the q output capsules is determined by calculating the L2 norm and then applying softmax activation. Here, q represents the number of categories for classification (Table 2). The sizes of these q output capsules correspond to the probabilities of belonging to various categories. The category associated with the highest probability among these q output capsules is selected as the class to which the input data belongs (Figure 2).

Figure 2. The ID-CapsuleNet includes two Conv_dilations for feature extraction. After dynamic routing, the size of the output vectors is computed.

Table 2. Parameter settings of the model.

Datasets	Conv_dilation1	Conv_dilation2	p	q
Dataset A	kernel_size=(1,26)	kernel_size=(1,8)	1664	2
Dataset B	kernel_size=(1,26)	kernel_size=(1,16)	1152	2
Dataset C	kernel_size=(1,26)	kernel_size=(1,22)	640	3
Dataset D	kernel_size=(1,64)	kernel_size=(1,24)	768	2

Dynamic routing is an iterative learning algorithm in CapsuleNet used to learn the routing logit $b_{\mathit{\text{ij}}}$ between PrimaryCaps and DigitCaps in order to obtain the representation of DigitCaps. In our capsule network, there are PrimaryCaps ${\overrightarrow{\mathrm{c}}}_{\mathrm{i}}^{\mathrm{l}}\in {\mathbb{R}}^{{\mathrm{N}}_{\mathrm{l}}\times 1},\mathrm{i}\in \{1,\dots ,\mathrm{m}\},$ and DigitCaps ${\overrightarrow{\mathrm{c}}}_{\mathrm{j}}^{\mathrm{h}}\in {\mathbb{R}}^{{\mathrm{N}}_{\mathrm{h}}\times 1},\mathrm{j}\in \{1,\dots ,\mathrm{n}\},$ where m and n represent the number of capsules, and ${\mathrm{N}}_{\mathrm{l}},$ ${\mathrm{N}}_{\mathrm{h}}$ represent the dimensions of the capsules. The routing logit $b_{\mathit{\text{ij}}}$ can be expressed as: (2) $$\begin{array}{c}{b}_{\mathit{\text{ij}}}={\left({\overrightarrow{c}}_j^h\right)}^T{S}_{\mathit{\text{ij}}}{\overrightarrow{c}}_i^l,\end{array}$$(2) where $S_{\mathit{\text{ij}}}\in {\mathbb{R}}^{{\mathrm{N}}_{\mathrm{h}}\times 1}$ is the transformation matrix used to calculate the candidate vector for high-level capsule $j$ as a weighted sum of all the PrimaryCaps, based on the routing logit ${ b}_{\mathit{\text{ij}}}.$ (3) $$\begin{array}{c}{\overrightarrow{z}}_j^h=\sum _{i=1}^m{w}_{\mathit{\text{ij}}}{S}_{\mathit{\text{ij}}}{\overrightarrow{c}}_i^l,\end{array}$$(3) the weights $w_{\mathit{\text{ij}}}$ are calculated by applying softmax to $b_{\mathit{\text{ij}}}.$ (4) $$\begin{array}{c}{w}_{\mathit{\text{ij}}}=\frac{\mathrm{\text{exp}}\hspace{0.17em}{b}_{\mathit{\text{ij}}}}{{\sum }_{k=1}^m\mathrm{\text{exp}}{b}_{\mathit{\text{ik}}}},\end{array}$$(4) the vector for the high-level capsule is obtained through a non-linear compression of ${\overrightarrow{z}}_j^h.$ (5) $$\begin{array}{c}{\overrightarrow{c}}_j^h=\frac{\parallel {\overrightarrow{z}}_j^h{\parallel }^2}{1+\parallel {\overrightarrow{z}}_j^h{\parallel }^2}\frac{{\overrightarrow{z}}_j^h}{\parallel {\overrightarrow{z}}_j^h\parallel },\end{array}$$(5)

The dynamic routing process is typically iterated three times to achieve convergence [48,49]; hence, we obtain the output vector after three rounds of dynamic routing.

The size of the capsule vectors (${\parallel x\parallel }_2$) is calculated by computing the L2 norm of the capsule vectors, where n represents the number of elements in the vector, in this case, n is the dimension of the output vectors, which is 16. L2 norm is as follow: (6) $$\begin{array}{c}{\parallel x\parallel }_2=\sqrt{\sum _{i=1}^n{x}_i^2},\end{array}$$(6)

3.4. Training strategy

Label Smoothing is a technique used to improve the classification performance and generalization of deep learning models [50,51]. When training deep learning models on fNIRS data, overfitting is often a concern. Therefore, we modify the loss function using label smoothing. For K labels, the probability of each label k is first calculated as $p_k.$ Wherein zi is the unnormalized log-probabilities. (7) $$\begin{array}{c}{p}_k=\frac{\mathrm{\text{exp}}\hspace{0.17em}\left(z_k\right)}{{\sum }_{\mathrm{\text{i = 1}}}^k\mathrm{\text{exp}}\left(z_{\mathrm{i}}\right)},\end{array}$$(7) the cross-entropy loss function is: (8) $$\begin{array}{c}L\left(y,p\right)=\sum _{k=1}^k-y_k\mathrm{\text{log}}\hspace{0.17em}\left(p_k\right),\end{array}$$(8)

When a classification is correct, the label $y_k$ equals 1. The smoothing parameter α is typically set to 0.1, and $y_k$ is smoothed to $y_k^{\mathit{\text{LS}}}.$ (9) $$\begin{array}{c}{y}_k^{\mathit{\text{LS}}}=\left(1-\alpha \right)y_k+\frac{\alpha }{k},\end{array}$$(9)

The label smoothing is written as: (10) $$\begin{array}{c}L\left(y^{\mathit{\text{LS}}},P\right)=\sum _{k=1}^k-y_k^{\mathit{\text{Ls}}}\mathrm{\text{log}}\hspace{0.17em}\left(p_k\right),\end{array}$$(10)

3.5. Classifiers for comparison

In our comparative experiments, we employed both traditional machine learning classifiers and deep neural networks (DNN). The traditional machine learning algorithms used in the experiments include the Artificial Neural Network (ANN). The ANN comprises 128 hidden layer units and is trained for a maximum of 1000 iterations [14]. In this study, we not only compared its performance with two existing deep neural network (DNN) models, CNN and LSTM, but also contrasted it with the original CapsuleNet and the efficient and concise knowledge-driven model, fNIRSNet [36]. The CNN model consists of three 1-dimensional convolutional layers, two fully connected layers, and a softmax layer. Each convolutional layer includes 32 filters with a kernel size of 3 [52]. The LSTM model consists of 4 LSTM layers and 2 fully connected layers, with each LSTM layer containing 64 hidden units, and the fully connected layers having 512 and 128 hidden nodes [35], respectively. CapsuleNet comprises three convolutional layers and dynamic routing, with each convolutional layer having a kernel size of 9. The primary capsules are 8-dimensional, and the output capsule vectors have a dimension of 16 [40]. fNIRSNet is a model proposed with delayed hemodynamic responses as domain knowledge, consisting only of three convolutional layers and one fully connected (FC) layer [36].

3.6. Training parameters

We set the batch size to 32, conducted training for 120 epochs, and maintained a learning rate of 0.0001. For two datasets representing different MA paradigms, two datasets representing MI paradigms, and considering the varying data sizes within these four datasets, we applied the AdamW [53] with a weight decay of 0.01. We utilized label smoothing to accelerate training and improve performance.

To assess the classification accuracy of the test set, we conduct 3 runs × 5-fold cross-validation. In order to evaluate the model’s generalization capability on individual subjects, we implemented a leave-one-subject-out cross-validation strategy (LOSO-CV) [54]. This method involves using one subject’s data as the evaluation set while utilizing data from other subjects as the training set. We selected multiple metrics to evaluate the performance of the classification outcomes, including accuracy, recall, F1 score (macro F1 score for Dataset C), and the Kappa coefficient.

4. Results

4.1. Classification results

To assess the classification accuracy of our model on MI and MA tasks based on fNIRS data, we conducted three rounds of 5-fold cross-validation on the MI and MA datasets (Table 3). It can be observed that ID-CapsuleNet shows the best classification performance, surpassing CNN, LSTM, and CapsuleNet with a significant margin. CNN and LSTM exhibit significantly better classification performance compared to the traditional classifier ANN. Our model achieved the highest average classification accuracy across the three datasets, outperforming the latest knowledge-driven model, fNIRSNet, with a noticeable advantage. These results indicate that our model possesses robust classification capabilities for fNIRS data based on MI and MA tasks, showing significant improvements over the original CapsuleNet and highlighting the effectiveness of our enhanced model.

Table 3. Average accuracy of the test sets.

Method	Dataset A	Dataset B	Dataset C	Dataset D
ANN	75.79 ± 4.32	58.66 ± 2.38	47.54 ± 2.07	56.28 ± 8.35
LSTM	88.45 ± 3.41	63.75 ± 1.65	61.94 ± 1.95	64.23 ± 3.46
CNN	87.00 ± 3.32	74.14 ± 1.78	72.89 ± 1.70	69.62 ± 5.04
CapsuleNet	60.81 ± 5.54	51.74 ± 1.19	36.92 ± 1.94	59.34 ± 3.13
fNIRSNet	95.59 ± 2.44	75.48 ± 1.64	73.16 ± 2.15	78.98 ± 4.41
ID-CapsuleNet	95.01 ± 2.41	76.88 ± 1.95	74.03 ± 2.26	80.29 ± 3.75

To further evaluate the model’s individual-level generalization ability, we employed a LOSO-CV strategy. Figure 3 displays the classification accuracy for each individual. It can be observed that our model achieves the highest accuracy for the majority of individuals across the four datasets. Table 4 lists the detailed results. From the results of F1 score and Kappa coefficient, it can be seen that our model exhibits the best generalization ability at the individual level.

Figure 3. Classification accuracy for individual subjects.

Table 4. Average classification results of all individual subjects.

Dataset	Method	Accuracy	Precision	Recall	F1-score	Kappa
Dataset A	ANN	68.84 ± 9.92	70.18 ± 12.05	67.88 ± 8.74	68.73	0.38
	LSTM	83.68 ± 12.42	84.88 ± 12.62	82.99 ± 13.41	83.55	0.67
	CNN	82.47 ± 11.64	93.96 ± 13.59	82.29 ± 10.66	82.70	0.65
	CapsuleNet	85.42 ± 10.96	84.34 ± 12.93	88.89 ± 8.65	86.25	0.71
	fNIRSNet	87.85 ± 11.30	86.87 ± 13.00	90.97 ± 8.65	88.60	0.76
	ID-CapsuleNet	88.72 ± 10.47	88.20 ± 12.41	90.97 ± 8.87	89.25	0.77
Dataset B	ANN	58.68 ± 7.61	59.25 ± 8.71	62.30 ± 12.13	59.80	0.17
	LSTM	66.78 ± 6.81	69.53 ± 9.51	63.45 ± 11.09	65.35	0.34
	CNN	70.43 ± 7.63	72.65 ± 10.59	71.30 ± 13.95	71.41	0.42
	CapsuleNet	73.97 ± 6.80	75.38 ± 10.45	74.71 ± 12.94	73.85	0.48
	fNIRSNet	75.72 ± 7.45	76.67 ± 9.49	75.40 ± 16.53	75.11	0.52
	ID-CapsuleNet	75.80 ± 8.16	76.96 ± 10.04	76.90 ± 14.54	75.59	0.52
Dataset C	ANN	55.68 ± 9.86	56.88 ± 10.56	54.68 ± 10.37	55.47	0.29
	LSTM	69.73 ± 15.75	71.04 ± 15.38	69.73 ± 15.75	69.13	0.55
	CNN	73.27 ± 17.19	75.25 ± 16.51	73.27 ± 17.19	72.85	0.59
	CapsuleNet	33.38 ± 4.18	23.80 ± 9.89	33.38 ± 4.18	23.68	0.00
	fNIRSNet	75.47 ± 17.61	76.89 ± 17.40	75.47 ± 17.61	74.98	0.63
	ID-CapsuleNet	75.78 ± 16.73	77.52 ± 15.83	75.78 ± 16.73	75.16	0.64
Dataset D	ANN	52.97 ± 4.70	38.66 ± 19.12	52.97 ± 4.70	41.07	0.06
	LSTM	56.90 ± 6.37	63.62 ± 17.68	56.90 ± 6.37	48.20	0.14
	CNN	58.61 ± 12.24	58.12 ± 20.26	58.61 ± 12.24	53.56	0.17
	CapsuleNet	66.32 ± 7.61	61.50 ± 19.46	63.96 ± 13.70	62.39	0.25
	fNIRSNet	80.51 ± 11.20	82.05 ± 14.14	82.45 ± 14.08	81.00	0.61
	ID-CapsuleNet	80.34 ± 7.04	81.81 ± 7.28	80.34 ± 7.04	80.11	0.61

4.2. Dilation convolution

To validate the effectiveness of dilation convolutions, we conducted classification experiments in ID-CapsuleNet using both dilation convolutions and normal convolutions while ensuring the same receptive field. The experimental results indicate that under the same receptive field conditions, dilation convolutions significantly outperform norm convolutions in terms of classification accuracy (Table 5), and dilation convolutions have approximately half the parameters of regular convolution (Table 6). Additionally, dilation convolutions can reduce parameters and improve processing speed [39, 55].

Table 5. Average classification results of different convolution.

Method	Dataset A	Dataset B	Dataset C	Dataset D
Convolution	94.96 ± 2.08	76.61 ± 1.37	73.42 ± 1.69	79.78 ± 2.85
Dilation Convolution	95.01 ± 2.41	76.88 ± 1.95	74.03 ± 2.26	80.29 ± 3.75

Table 6. The parameters of different convolution.

Method	Dataset A	Dataset B	Dataset C	Dataset D
Convolution	894784	927552	960320	991296
Dilation Convolution	560704	544320	527936	598336

The fNIRS measurements of brain hemodynamic responses based on MI and MA exhibit certain time-delay characteristics, indicating long-range dependencies in the data [37,38]. This long-range dependence has a long-time span, which may lead to data redundancy. Therefore, dilation convolutions can capture long-range dependencies while reducing feature redundancy, ultimately enhancing the network’s classification performance.

5. Discussion

In this study, we propose a new method that utilizes fNIRS signals for MI and MA classification, combining dilation convolution and CapsuleNet. Our results mainly focus on comparing the classification accuracy of our proposed method with other prevalent techniques. Furthermore, we compared the parameter differences and classification accuracy between using dilation convolution and regular convolution in our model.

In the following discussion, we will delve into the structural significance of ID-CapsuleNet and the possibilities for achieving a fast and robust BCI system in more detail.

5.1. Margin loss and reconstruction loss for ID-CapsuleNet

Margin loss encourages correct dynamic routing between DigitCaps, with its main goal being to ensure that the correct DigitCaps receive higher probabilities while incorrect ones receive lower probabilities [48]. On the other hand, reconstruction loss primarily aims to measure the difference between input data and the reconstructed data generated by the model [48], thus encouraging the model to learn effective data representations.

To encourage correct dynamic routing between DigitCaps and promote effective data representation learning, we used margin loss and reconstruction loss as the overall loss (CapsuleLoss) for model training [40, 49, 56]. However, the results indicated that using cross-entropy loss with label smoothing outperformed the individual effects of CapsuleLoss (Table 7). This suggests that cross-entropy loss with label smoothing is a better match for fNIRS data and ID-CapsuleNet.

Table 7. Average classification results of different loss.

Method	Dataset A	Dataset B	Dataset C	Dataset D
CapsuleLoss	91.39 ± 3.04	68.87 ± 2.03	67.24 ± 1.38	63.93 ± 4.13
LabelSmoothing	95.01 ± 2.41	76.88 ± 1.95	74.03 ± 2.26	80.29 ± 3.75

The difference in classification accuracy may be attributed to the fact that the primary functions of margin loss and reconstruction loss are not directly related to classification. Therefore, CapsuleLoss may not be well-suited for the specific task of classification. Cross-entropy loss with label smoothing [51, 57], on the other hand, is designed to optimize classification tasks and appears to be a more suitable choice for this particular scenario. In summary, the choice of loss function depends on the specific task and dataset. While margin loss and reconstruction loss have their advantages in certain contexts, they may not always be the best choice for classification tasks, as demonstrated in the results.

5.2. Ablation study

In order to validate the importance of Conv_dilation and dynamic routing in our model, we conducted ablation experiments on ID-CapsuleNet. ID-CNN replaced the dynamic routing in ID-CapsuleNet with pooling layers and fully connected layers while retaining the Conv_dilation. We compared ID-CapsuleNet with ID-CNN to assess the importance of dynamic routing. Additionally, since CapsuleNet includes normal convolutions, we compared ID-CapsuleNet with CapsuleNet to evaluate the effectiveness of Conv_dilation.

The experimental results showed that both Conv_dilation and dynamic routing algorithms are crucial for our model, and the absence of either component led to a decrease in model performance (Table 8). Conv_dilation extract long-range dependency features, while dynamic routing captures feature invariance, resulting in better feature representation and, consequently, improved classification performance. Furthermore, we compared ID-CNN with CNN to verify the performance gain achieved by Conv_dilation compared to convolutions with a kernel size of 3. The results demonstrated that Conv_dilation led to significant improvements on certain datasets, with Dataset A and Dataset D showing improvements of 6% and 10%, respectively. Some improvements were also observed on the other two datasets (Table 8), indicating that Conv_dilation is more effective at feature extraction. In summary, both Conv_dilation and dynamic routing are essential components of our model, and they work synergistically to enhance feature extraction and classification performance.

Table 8. Average accuracy of the test sets.

Method	Dataset A	Dataset B	Dataset C	Dataset D
CNN	87.00 ± 3.32	74.14 ± 1.78	72.89 ± 1.70	69.62 ± 5.04
CapsuleNet	60.81 ± 5.54	51.74 ± 1.19	36.92 ± 1.94	59.34 ± 3.13
ID-CNN	93.19 ± 2.56	74.56 ± 1.87	73.59 ± 2.23	79.25 ± 4.87
ID-CapsuleNet	95.01 ± 2.41	76.88 ± 1.95	74.03 ± 2.26	80.29 ± 3.75

In the context of MA and MI tasks, specific brain regions are known to become activated, and opposite changes in HbO and HbR are often observed [58]. When participants engage in MI, the concentration of HbO associated with active brain regions usually increases, while the concentration of HbR decreases. In MA tasks, studies have found that HbO responses are stronger in the left dorsolateral prefrontal cortex (DLPFC) and anterior prefrontal cortex (APFC), whereas HbR responses are relatively weaker and less pronounced [59,60]. In the context of MI, the brain activation regions can differ depending on the specific MI task. For example, left and right hand MI tasks are associated with activation in contralateral brain regions, whereas foot MI tasks activate both brain regions [61–63]. These active brain regions are closely related to cerebral hemodynamic responses, and the long-range dependencies extracted by our model represent the cerebral hemodynamic responses. Therefore, in some respects, this provides an approach to understanding neural physiological reactions in the brain. These findings have important practical implications in the field of fNIRS-based classification.

5.3. Limitation

The receptive field of Conv_dilation may vary for each dataset and may require adjustments for different data lengths, which can be time-consuming. Additionally, CapsuleNet have a high computational complexity, leading to longer training times, which may hinder their scalability and deployment in certain practical applications. Further research is needed to validate their performance in multi-class classification problems. In terms of interpretability, the internal workings of CapsuleNet are20 relatively complex, making them challenging to explain and visualize.

6. Conclusion

In this study, we utilized Conv_dilation for extracting features from fNIRS-based brain hemodynamic responses and employed dynamic routing to capture feature invariance. Consequently, Conv_dilation combined with dynamic routing were employed for MI and MA classification based on fNIRS. Our experiments were conducted on four publicly available datasets. Label smoothing was employed to address overfitting problem. The proposed network architecture demonstrated strong performance in terms of classification accuracy, and ablation studies confirmed the importance of Conv_dilation and dynamic routing. Therefore, the proposed network architecture not only offers an effective approach for classifying MI and MA based on fNIRS but also provides conceptions for designing network models specifically tailored to fNIRS. It serves as an inspiration for gaining insights into specific neural activities within the brain.

Disclosure statement

No potential competing interest was reported by the authors.

Additional information

Funding

This work was supported by the Special Projects in Key Fields Supported by the Technology Development Project of Guangdong Province; Scientific Research of High-level Talents of Wuyi University; Special Fund Project for Science and Technology Innovation Strategy of Guangdong Province.