A novel brain inception neural network model using EEG graphic structure for emotion recognition

Abstract

Purpose

EEG analysis of emotions is greatly significant for the diagnosis of psychological diseases and brain-computer interface (BCI) applications. However, the applications of EEG brain neural network for emotion classification are rarely reported and the accuracy of emotion recognition for cross-subject tasks remains a challenge. Thus, this paper proposes to design a domain invariant model for EEG-network based emotion identification.

Methods

A novel brain-inception-network based deep learning model is proposed to extract discriminative graph features from EEG brain networks. To verify its efficiency, we compared our proposed method with some commonly used methods and three types of brain networks. In addition, we also compared the performance difference between the EEG brain network and EEG energy distribution for emotion recognition.

Result

One public EEG-based emotion dataset (SEED) was utilized in this paper, and the classification accuracy of leave-one-subject-out cross-validation was adopted as the comparison index. The classification results show that the performance of the proposed method is superior to those of the other methods mentioned in this paper.

Conclusion

The proposed method can capture discriminative structural features from the EEG network, which improves the emotion classification performance of brain neural networks.

Keywords:

• Emotion recognition
• brain network
• EEG
• deep learning

1. Introduction

Emotions play important roles in our social interaction and daily life, and they also have a deep influence on other high-order cognitive functions such as decision-making, attention, and memory [1]. In recent years, emotion analysis is rising popularity within the field of neuroscience study [2,3], which could provide a reference for evaluating some psychiatric diseases such as depression and autism [4]. With the development of neuroimaging technologies such as electroencephalogram (EEG), magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI) [5,6], more and more available ways for deciphering neuroscience mechanisms have been provided. But how to effectively reveal the mechanism of emotions and accurately identify the emotional states remain unsolved. Recently, emotion recognition has become a hot topic in the field of emotion analysis, which plays a crucial role in developing effective human-computer interaction [7]. In recent years, there have been a series of studies putting forward novel strategies to achieve more efficient emotion recognition performance. In these methods, deep learning has made great achievements in the field of image segmentation [8], data classification [9], and feature analysis [10].

Due to its powerful capability of feature extraction and classification, it has earned much attention in emotion classification [11]. Elham et al. used the 3-dimension convolutional neural networks (3D-CNN) for emotion recognition. They extracted the 3D data representation from multi-channel EEG signals and achieved higher recognition accuracies along both the valence and arousal dimensions of the DEAP dataset [12]. Moreover, Zheng et al. compared the performance differences between power spectral density (PSD), differential entropy (DE), and energy distribution based asymmetric features, in which they observed activations of crucial brain regions for emotion processing [13]. Zheng et al. used deep confidence network for emotion recognition of DE features and identified key frequency bands [14]. Wang et al. proposed the DE-CNN model to further improve performance of DE features for emotion recognition [15]. In addition, Hwang et al. generated topology-preserving differential entropy features and used a convolutional neural network (CNN) to identify three-class emotional states [16]. Wang et al. utilized DE features as the input to their proposed broad dynamical graph learning system and obtained excellent classification results for emotion recognition [17].

Basically, the methods mentioned above mainly utilized the features of brain activation for emotion classification. Comparing with these energy-distribution based features, network analysis could reflect the coupling relationship among various brain regions [18], and reveal the synergistic effects of different regions for different cognitive states [19]. Generally, the methods for brain network estimation can be divided into two categories, i.e. the functional brain network analysis and effective brain network analysis. The functional brain network analysis focuses on revealing the cooperative oscillation of different brain regions. Among these methods, the phase-locking value is one of the commonly used technologies, which measures the phase difference between two signals at different time points [20]. Wang et al. used phase-locking value (PLV) as the features and designed graph convolutional neural networks (P-GCNN) to extract the implicitly emotional information more effectively [21]. Gonuguntla et al. measured the synchronicity between different regions of the brain during emotional perception tasks using the PLV method [22]. Different from functional brain network analysis. the effective brain network analysis focuses on revealing the driving relationship between any pair of brain regions. Among these methods, the Granger causality analysis is one of the most used. Chen et al. analyzed the causal relationship between EEG signals through Granger causality to determine the brain networks associated with specific emotional states. The results indicate that using this method can accurately identify the emotional state of participants with high accuracy [23]. Chen et al. Propose a method of(Sparse Granger Causality Analysis, SGCA)method and apply it to emotion recognition tasks, improving classification accuracy and robustness [24]. Gao et al. proposed a Student’s t-based Granger causality for brain network analysis. This method can improve the robustness of causal inference while considering outliers and noise [3].

Though these brain-connection-based methods have been successfully applied, one challenge unsolved well is that there has significant differences between subjects even under the same emotional states [25]. Therefore, some studies have been carried out to deal with this problem such as transfer learning strategies. In these methods, transfer component analysis (TCA) is one of the simplest methods for adaptive domain transfer [26], which achieves domain adaptation by constructing a kernel matrix to minimize the maximum mean discrepancy (MMD) between the source-domain and target-domain distributions. In essence, the performance of these methods depends on the selection of kernels and their parameters. However, in real applications, it is hard to offer the prior optimal kernels [27]. Alternatively, the deep learning-based domain invariant models have earned much attention in recent studies [28]. Luo et al. proposed a novel transfer strategy by using Wasserstein generative adversarial network (GAN) to handle the domain shift problem, which alleviates the differences between subjects to some degree [29]. Recently, Lin et al. found that the transform learning strategy showed excellent performance on classification problems in particular cross-subjects [30]. Furthermore, Li et al. utilized this feature in EEG-based classification and obtained more significant improvement than others [31]. In their study, they designed a multisource transfer learning method to reduce the EEG differences between the target and source domains.

Thus, in our current work, by considering the facts that 1) the directed brain network analysis has shown excellent performance in revealing the driving relationships among multiple brain regions [23]; and 2) the merit of inception strategy for multi-scale structural feature extraction [32], we designed a novel model for EEG based emotion recognition which consists of Granger causality analysis for directed brain network estimation and deep domain adaptation architecture for domain invariant network structural feature extraction.

2. Methods

In this study, we tried to design a novel deep learning framework with inception blocks to extract structural features from brain networks for emotion recognition. To evaluate its efficiency in capturing discriminative network features for emotion classification, both the functional and effective brain networks were estimated from EEG of emotions.

2.1. Functional brain network estimation

In this study, to characterize the synchronized activity of multiple brain regions, we adopted phase-locking value (PLV) which describes the non-linear phase synchronization to statistically measure the phase synchronization between two brain areas [33]. Suppose there are two channels of EEG data $x_i(t)$ and $x_k(t),$ and their corresponding instantaneous phase could be calculated via Hilbert transformation as [34] (1) $$\phi (t)=H(x(t))=\frac{1}{\pi }{\int }_{-\infty }^{+\infty }\frac{x(\tau )}{t-\tau }d\tau$$(1)

Subsequently, the functional brain network could be descripted as (2) $$PL{V}_{ik}=\frac{1}{N}\left|{{\sum }_{n=1}^{N}e}^{j(\Delta {\phi }_{ik}(n))}\right|$$(2) where N is the total number of time points. The $\Delta {\phi }_{ik}(t)={\phi }_i(t)-{\phi }_k(t)$ represents the instantaneous phase difference between time series i and time series k, and j represents the imaginary unit.

2.2. Effective brain network estimation

2.2.1. Granger analysis

In current study, we adopted the Granger causality analysis (GCA) to estimate effective brain network, of which the kernel is to solve the following multivariate autoregressive (MVAR) model as (3) $$\left[\begin{array}{c}{x}_{1,i}\\ ⋮\\ x_{D,i}\end{array}\right]=\sum _{r=1}^q{A}_r\left[\begin{array}{c}{x}_{1,i-r}\\ ⋮\\ x_{D,i-r}\end{array}\right]+\left[\begin{array}{c}{\epsilon }_{1,i}\\ ⋮\\ {\epsilon }_{D,i}\end{array}\right],A_r=\left[\begin{array}{ccc}{a}_{11,r}& \cdots & a_{1D,r}\\ ⋮& \ddots & ⋮\\ a_{D1,r}& \cdots & a_{DD,r}\end{array}\right]$$(3) where [x_1,i, $\cdots ,$x_D,i]^T (1$\le i\le$N) denotes D-dimensional joint stationary time series at time point i, q is the model order, and ${\epsilon }_{k,i}$ is the residual for the kth time series. $A_r\in R^{D\times D}$ is the coefficient matrix at time lag r, and N indicates the length of time series. For the k-th time series, Equation (3)(3) $$\left[\begin{array}{c}{x}_{1,i}\\ ⋮\\ x_{D,i}\end{array}\right]=\sum _{r=1}^q{A}_r\left[\begin{array}{c}{x}_{1,i-r}\\ ⋮\\ x_{D,i-r}\end{array}\right]+\left[\begin{array}{c}{\epsilon }_{1,i}\\ ⋮\\ {\epsilon }_{D,i}\end{array}\right],A_r=\left[\begin{array}{ccc}{a}_{11,r}& \cdots & a_{1D,r}\\ ⋮& \ddots & ⋮\\ a_{D1,r}& \cdots & a_{DD,r}\end{array}\right]$$(3) can be further rewritten as (4) $$Y_k=\mathit{u}{\mathit{g}}_k+E_k,\mathit{u}=\left[\begin{array}{cccc}\begin{array}{cccc}\begin{array}{c}{x}_{1,q}\\ ⋮\\ x_{1,N-1}\end{array}& \begin{array}{c}\cdots \\ \ddots \\ \cdots \end{array}& \begin{array}{c}{x}_{D,q}\\ ⋮\\ x_{D,N-1}\end{array}& \begin{array}{c}\cdots \\ \ddots \\ \cdots \end{array}\end{array}& \begin{array}{c}{x}_{1,1}\\ ⋮\\ x_{1,N-q}\end{array}& \begin{array}{c}\cdots \\ \ddots \\ \cdots \end{array}& \begin{array}{c}{x}_{D,1}\\ ⋮\\ x_{D,N-q}\end{array}\end{array}\right]\in R^{\left(N-q\right)\times \left(D\times q\right)}$$(4) where $\mathit{u}\in R^{(N-q)\times (D\times q)}$ is the transform matrix and $Y_k={\left[x_{k,q+1},\dots ,x_{k,N}\right]}^T\in R^{(N-q)\times 1}$ is the measurement value, ${\mathit{g}}_k={\left[a_{k1,1},\dots ,a_{kD,1},\dots ,a_{k1,q},\dots ,a_{kD,q}\right]}^T\in R^{(D\times q)\times 1}$ is the MVAR coefficient for k-th time series, $E_k={\left[{\epsilon }_{k,q+1},\dots ,{\epsilon }_{k,N}\right]}^T\in R^{(N-q)\times 1}$ represents the measurement noise. When the model coefficients in Equation (4)(4) $$Y_k=\mathit{u}{\mathit{g}}_k+E_k,\mathit{u}=\left[\begin{array}{cccc}\begin{array}{cccc}\begin{array}{c}{x}_{1,q}\\ ⋮\\ x_{1,N-1}\end{array}& \begin{array}{c}\cdots \\ \ddots \\ \cdots \end{array}& \begin{array}{c}{x}_{D,q}\\ ⋮\\ x_{D,N-1}\end{array}& \begin{array}{c}\cdots \\ \ddots \\ \cdots \end{array}\end{array}& \begin{array}{c}{x}_{1,1}\\ ⋮\\ x_{1,N-q}\end{array}& \begin{array}{c}\cdots \\ \ddots \\ \cdots \end{array}& \begin{array}{c}{x}_{D,1}\\ ⋮\\ x_{D,N-q}\end{array}\end{array}\right]\in R^{\left(N-q\right)\times \left(D\times q\right)}$$(4) have been estimated, the causal relationship between any pair of time series $Y_k$ and $Y_h$ can be further inferred. Theoretically, the causal interactions between $Y_k$ and $Y_h$ are measured in accordance with the following principle: if $Y_h$ improves the prediction performance when all the other processes are also included, we can state $Y_h$ G-cause $Y_k.$ Mathematically, this causal interaction can be further evaluated by the logarithmic ratio between unrestricted noise covariance and the restricted one as: (5) $${\mathcal{l}}_{h\to k|\Delta }=\hspace{0.17em}\ln \hspace{0.17em}\frac{{\Sigma }_{k,k}^{*}}{{\Sigma }_{k,k}},\left(\Delta =[1,\dots ,D]-[h,k],\Delta \in R^{D-2}\right)$$(5) where ${\Sigma }_{k,k}^{*}$ denotes the noise covariance of k-th equation from restricted model (which omits the influence of the k-th time series) and ${\Sigma }_{k,k}$ is the noise covariance from unrestricted model (considering the influence of all the time series). Practically, the statistical significance for Y_h to cause Y_k can be determined via the F-statistic [35,36] based on the evaluated ${\mathcal{l}}_{h\to k|\Delta }.$ In our current work, if the p-value is less than 0.05, we then judge that there is a significant causal interaction [37].

2.2.1.1. Traditional Granger causality analysis

Traditional Granger causality analysis is usually solved with the least square method, which holds the form as (6) $$\underset{g_k}{\text{argmin}}{f}_k\left(g_k\right)={\Vert Y_k-\mathit{u}{g}_k\Vert }_2^2$$(6) where ${\Vert ·\Vert }_2^{}$ denotes the $L_2$ norm space. In our current study, we name this method as LS-GCA to differentiate it with other GCAs.

2.2.1.2. Granger causality analysis solved by LASSO

In real applications, traditional Granger causality for EEG analysis is easily influenced by various noise and there are usually redundant linkages for network analysis. In fact, resent neuroscience research have revealed the high efficiency of brain for neural cognition, which also indicates the sparse brain network structure of information processing [38]. Thus, to alleviate the influence of EEG noise and characterize the sparse structure of brain network for cognition processing, the least absolute shrinkage and selection operator (Lasso) based Granger causality analysis models (Lasso-GCA) has been proposed [39]. Mathematically, Lasso-GCA characterizes the causal interactions among multiple time series in a sparse way, which effectively preserves the strong-causal data between EEG sensors and eliminates the data contain weak causality [24]. To attach this goal, we can impose the LASSO penalty on the coefficients [18]. Thus, Equation (6)(6) $$\underset{g_k}{\text{argmin}}{f}_k\left(g_k\right)={\Vert Y_k-\mathit{u}{g}_k\Vert }_2^2$$(6) can be further rewritten as (7) $$\underset{g_k}{\text{argmin}}{f}_k\left(g_k\right)={\Vert Y_k-\mathit{u}{g}_k\Vert }_2^2\text{+}\lambda {\Vert g_k\Vert }_1^1$$(7) where $\lambda \ge 0$ is the regularization parameter. Theoretically, the larger the $\lambda$ is, the sparser the network structure is. In fact, the above equation is a well-known convex optimization problem, which has the equability function as least angle repression’s method [40].

2.3. Brain inception neural network (BiNN)

To extract the structural features of brain networks, we designed a novel brain inception neural network. Specifically, we used 2-D convolution blocks to extract the features of brain network matrix and used the inception blocks to ensure the diversity of features. Figure 1 illustrates the structure of our model and Figure 2 illustrates the process of constructing brain networks and utilizing deep learning models to extract features, we directly select the edge connection matrix of each brain network as the feature matrix input to the deep learning model, and use the model to complete the feature extraction of the brain network connection matrix.

Figure 1. The structure of brain inception network.

Figure 2. The flow chart of deep learning model for brain network construction and brain network edge connection input.

In addition, we used three fully connected layers and SoftMax activation function to predict the probability of each emotional state, and the loss for emotional classification loss $L_p$ is calculated by cross entropy, that is (8) $$L_p=-\frac{1}{N}\sum _i\sum _{c=1}^M{y}_{ic}\hspace{0.17em}\log \hspace{0.17em}(p_{ic})$$(8) where $y_{ic}$ is the predefined emotional label of $\mathit{ith}$ segment and $p_{ic}$ is the probability of emotion prediction

The deep domain adaptation architecture is adapted to further alleviate the domain difference between participants [41]. Therefore, the loss function of Domain Discriminator could be represented as (9) $$L_d=\frac{1}{N}\sum _i-[y_{id}\cdot \hspace{0.17em}\log \hspace{0.17em}(p_{id})+(1-y_{id})\cdot \hspace{0.17em}\log \hspace{0.17em}(1-p_{id})]$$(9) where $y_{id}$ is the domain label of $\mathit{ith}$a segment and $p_{id}$ is the probability of domain prediction.

Generally, in our current study, we utilized the GRL layer for adversarial learning [41]. Thus, the loss of brain inception network could be represented as (10) $$L_f=-\frac{1}{N}\sum _i\sum _{c=1}^M{y}_{ic}\hspace{0.17em}\log \hspace{0.17em}(p_{ic})+\lambda \cdot \frac{1}{N}\sum _i-[y_{id}\cdot \hspace{0.17em}\log \hspace{0.17em}(p_{id})+(1-y_{id})\cdot \hspace{0.17em}\log \hspace{0.17em}(1-p_{id})]$$(10) and the learning rate of the network learning process would be decreasing with the increase of training steps, which could be expressed as (11) $${\eta }_p=\frac{{\eta }_0}{{(1+\alpha \cdot p)}^{\omega }}$$(11) where ${\eta }_p\in (0,0.0005)$ is the learning rate and ${\eta }_0$ = 0.0005, $\alpha$=10, $\omega$= 0.75, respectively. The detailed information of the Brain Inception network deep learning model is integrated in Table 1. The Table 2. Shows the parameters of the Inception block, and our model training hyper-parameters are shown in Table 3.

Table 1. Parameters of the BINN.

Layer	Operation	Kernel Size/Stride	Kernel number	Activation Function	Padding	BatchNormalization / Dropout
Conv1	Convolutional	3 × 3/1	32	SeLU	Same Padding	BatchNormalization
Pool1	Max-pooling	2 × 2/2	–	–	–	–
Conv2	Convolutional	3 × 3/1	64	SeLU	Same Padding	BatchNormalization
BN-Inception1	Inception block	–	–	ReLU	–	BatchNormalization
BN-Inception2	Inception block	–	–	ReLU	–	BatchNormalization
Flatten	–	–		–	–	–
FC	Liner	–	128	SeLU	–	BatchNormalization Dropout(0.2)

Table 2. Parameters of the Inception block.

Layer	Operation	Kernel Size/Stride	Kernel number	Activation Function	Padding	Connected(Input) Layer
Conv1	Convolutional	1 × 1/1	64	ReLU	Same Padding	Previous Layers feature
Conv2	Convolutional	1 × 1/1	32	ReLU	Same Padding	Previous Layers feature
Conv3	Convolutional	1 × 1/1	32	ReLU	Same Padding	Previous Layers feature
Pool1	Max-pooling	3 × 3/1	–	–	–	Previous Layers feature
Conv4	Convolutional	3 × 3/1	64	ReLU	–	Conv2
Conv5	Convolutional	5 × 5/1	64	ReLU	–	Conv3
Conv6	Convolutional	1 × 1/1	64	ReLU	–	Pool1
Channel Concat	–	–	–	–	–	Conv1,Conv4, Conv5,Conv6

Table 3. Model training hyper-parameters.

hyper-parameters	Value
BINN dropout rate	0.2
Classifier dropout rate	0.5
Discriminator dropout rate	0.3
Epochs	100
Optimizer	Adam(beta_1 = 0.9, beta_2 = 0.999)
Optimizer epsilon	1e^-7
Initial learning rate	0.0005
Batch size	32

2.4. EEG dataset in our current study

To verify the efficiency of our proposed model for EEG-based emotion recognition, we conduct an experiment on one public emotional database, i.e. SEED [13]. This database includes both facial and EEG signals recorded from 15 subjects (7 males and 8 females; age: MEAN: 23.27, STD: 2.37). During experiments, the subjects were asked to watch film clips with different emotional tags i.e. positive, neutral and negative. Each emotion tag has five film clips in one experiment. The duration of each film clip is about 4 min. In total, there were 15 trials for each experiment. To explore the stable pattern of different emotions, all participants were required to perform the experiments for three sessions. The time interval between two sessions was one week or longer. The EEG signal was recorded using an ESI NeuroScan System2 at a sampling rate of 1000 Hz from a 62-channel active AgCl electrode cap according to the international 10–20 system. More detail could be referred in https://bcmi.sjtu.edu.cn/∼seed/seed.html.

2.5. Emotion EEG preprocessing

The EEG preprocessing mainly consists of the following steps: 1) Down sampling the original dataset to 200 Hz, 2) Segmenting the EEGs into subsegments by a window of 1-s length without overlap, 3) Correcting the baseline of each subsegment, 4) Re-referencing each subsegment with common average reference, and 5) Filtering the data with a 5-order Butterworth band-pass filter of 1–50 Hz.

3. Results

In current study, we mainly utilized classification accuracy to evaluate the performance of our proposed method for emotion recognition. Specifically, we compared the performance difference between EEG activation and EEG network features, and the performance difference between our proposed model and other widely used classifiers. Totally, there are four types of features, i.e. the feature from EEG functional network (PLV), the features from EEG efficient network (GCA and Lasso-GCA) and the features from EEG energy distribution (DE) [13], and two types of traditional classifiers, i.e. the shallow classifiers including multilayer perceptron (MLP) [42], support vector machine (SVM) [43], deep learning models such as DANN [41] and our proposed brain inception neural network (BiNN) for comparison. For each method, the leave-one-subject-out cross-validation (LOSOCV) procedure was adopted to evaluate the classification performance. In general, Table 4 illustrates the classification results of different features and classifiers. It’s worth noting that the value with the bold symbol indicates the best result. "$\mathit{\psi }$", "$\xi$" and "$\tau$" denote the significant improvement of BiNN (Lasso-GCA), BiNN (PLV) and BiNN (GCA) when compared with others under p < 0.05.

Table 4. The cross-subject classification accuracy of models.

Model	Input	Accuracy(mean ± std)
MLP	Lasso-GCA	54.80 ± 7.46 $\mathit{\psi }\mathit{\text{ ξ τ}}$
SVM	Lasso-GCA	55.65 ± 8.58 $\mathit{\psi }\mathit{\text{ ξ τ}}$
MLP	GCA	53.57 ± 6.18 $\mathit{\psi }\mathit{\text{ ξ τ}}$
SVM	GCA	53.30 ± 6.82 $\mathit{\psi }\mathit{\text{ ξ τ}}$
MLP	PLV	50.80 ± 8.01 $\mathit{\psi }\mathit{\text{ ξ τ}}$
SVM	PLV	51.89 ± 10.45 $\mathit{\psi }\mathit{\text{ ξ τ}}$
MLP	DE	59.76 ± 12.13 $\mathit{\psi }\mathit{\text{ ξ τ}}$
SVM	DE	58.97 ± 11.28 $\mathit{\psi }\mathit{\text{ ξ τ}}$
DANN	PLV	65.74 ± 10.04 $\mathit{\psi } \tau$
DANN	GCA	68.80 ± 8.38 $\mathit{\psi }$
DANN	Lasso-GCA	69.56 ± 7.90 $\mathit{\psi }$
DANN	DE	71.24 ± 9.62
BiNN	PLV	67.30 ± 10.16 $\mathit{\psi }$
BiNN	GCA	69.98 ± 7.52
BiNN	Lasso-GCA	73.53 ± 7.38

In general, Figure 3 consistently illustrates that the performance of our proposed method for emotion recognition is superior to those of others, which verifies the powerful capability of our proposed method in capturing discriminative structural features from brain networks. As shown in the Figure 4, we validated the accuracy of three brain network extraction algorithms under different numbers of inception blocks. We found that for the three brain network extraction algorithms, the appropriate number of inception blocks can effectively improve the performance of emotion recognition. However, an excessive number of inception block stacking will lead to model overfitting and seriously reduce the recognition accuracy. Notably, the graphic features extracted from Lasso-GCA-based brain network by our proposed BiNN hold the best classification performance, which further indicates the efficiency of efficient brain networks in offering discriminant features for emotion recognition as illustrated in Figure 5. As shown in Figure 5, compared with BiNN (PLV), BiNN (GCA), DANN (DE), the features extracted by BiNN (Lasso-GCA) are easier to distinguish between different emotions. On the contrary, other methods don’t have ability to reveal the differences among emotions, especially within neutral and negative emotions. Figure 6 illustrates the capability of different methods in correctly identifying each emotional state through confusion matrices. Obviously, the results of effective brain-network features extracted by BiNN show the best performance than others, where the diagonal values in the confusion matrix is predominant. Furthermore, compared with traditional classifiers, the transfer learning strategies such as DANN and BiNN further improve the classification performance for emotion recognition across subjects.

Figure 3. Average accuracy of SEED dataset on different methods.

Figure 4. The accuracy of three brain networks under Different Inception Blocks.

Figure 5. t-SNE visualization of different emotion features from the last fully-connection layer before classification. (a) The features of effective brain network extracted by BiNN(Lasso-GCA); (b) The features of functional brain network extracted by BiNN(PLV); The features of effective brain network extracted by BiNN(Lasso-GCA); (d) The EEG activation features extracted by DANN(DE).

Figure 6. The confusion matrix of classification results for different methods.

4. Discussion

In recent years, the effectiveness of network in EEG-based classification or mechanisms analysis has been verified in many works [23]. For example, Wang et al. used PLV to construct a brain network to model multi-channel EEG features as graph signals for EEG emotion classification [21]. However, how to extract discriminative structural features from brain networks remains a big challenge. Thus, in this paper, a novel deep learning structure was proposed to extract discriminative features from the EEG based brain networks for emotion recognition. To evaluate its effectiveness, we applied it to a publicly available emotional EEG dataset, i.e. SEED and compared its performance difference with a series of commonly used methods such as SVM and DANN. Generally, the proposed method shows the best performance in terms of classification accuracy and feature distribution, which indicates that the discriminative structural features extracted from brain networks could further improve the classification capability of EEG for emotion recognition. Furthermore, we analyzed the working mechanism of the inception block and found that multi-scale convolution can more effectively extract effective information from brain networks. Focusing on the performance difference between different GCAs, we also found that the Lasso-GCA is superior to traditional GCA in emotion recognition. In fact, many previous studies have revealed that the Lasso-GCA can effectively alleviate the problem of redundant linkages in network construction [18]. which may result in discriminative network structures for emotion recognition.

Through the results of confusion matrix illustrated in Figure 6, we found that the ratio of misclassification between the neutral and negative emotions is higher than that between positive and neutral emotions and that between positive and negative emotions, which indicates that the network structures of negative and neutral emotions may be close to each other. In fact, this observation is consistent with previous studies [44,45]. In the work conducted by Gao et al. the authors revealed the structural similarity of brain networks between neutral and negative emotions by analyzing the network characteristics between these two emotions [3]. Zheng et al. characterized the average connection patterns of different emotions and found that the neural patterns between neutral and negative emotions are similar to each other [44]. Through Table 4 and Figure 3, we also observed that model difference has great influence on the classification accuracy. In fact, the traditional machine learning classifier such as SVM and MLP cannot achieve satisfying classification performance, which may be caused by the fact that they are not able to extract powerful discriminative information from the brain network matrix, such as representative subnetworks, and emotion processing related hub nodes [46]. By using CNN-based inception blocks the proposed model can effectively capture these discriminative structure through multiple spatial scales [32]. In our current study, we also observed that the number of inception blocks will influence the classification performance of different kinds of brain network. Basically, as shown in Figure 4, the optimal number of inception blocks for Lasso-GCA and LS-GCA is 2, while the optimal number of inception blocks for PLV is 3. One thing needs to note is that though the number of inception blocks influence the classification performance difference of different kinds of brain networks, the performance of GCAs for emotion recognition is superior to that of PLV, where the classification accuracies of brain networks estimated by Lasso-GCA are always the best. These results further verified the efficiency of Lasso-GCA in characterizing the crucial network structures for emotion processing. In addition to inception blocks, the transfer learning further makes our proposed method feasible to extract common features among subjects. Moreover, compared with the brain network based graphic features, traditional energy-based features such as PSD and DE could not directly reveal the coupling relationship among brain regions, and may not offer elaborate structural features to identify cognitive states.

Generally, through both Table 4 and Figures 3 and 4, we also observed that the classification performance of an effective brain network is better than that of a functional brain network. Focusing on the performance difference between Lasso-GCA and GCA, we observed that the Lasso-GCA is superior to GCA in the aspect of emotion recognition accuracy and feature separability. In essence, due to the constraints of the L1-norm-based regularization terms, Lasso-GCA characterizes the sparser network structures of cognition processing in the brain, which is also consistent with the sparse network structure of the human brain for cognitive processing. Focusing on Figure 5, we observed that the samples of the same class in the space spanned by the features extracted from an effective brain network are more gathered, while the samples of different classes get more separated, especially for negative and neutral emotions. This may indicate that there are more driving relationships between multiple brain regions for emotion processing, comparing with the synchronization relationships characterized by functional brain networks such as PLV [47].

5. Conclusion

This paper proposed a novel method for emotion recognition by designing a transform learning structure through the convolutional neural network, which efficiently extracts discriminant structural features from brain networks. The results based on the SEED dataset showed that our proposed BiNN can extract domain invariant discriminative features from the network matrix, so as to significantly improve the classification accuracy of cross-subject emotion recognition tasks. Furthermore, we also verified the superiority of an efficient brain network for EEG-based emotion analysis. Our proposed method may provide a novel insight into the EEG analysis of emotions.

Disclosure statement

All Authors agree that there is no relevant conflict of interest.

Additional information

Funding

This work was supported in part by the STI 2030-Major Projects under Grant 2022ZD0208500 and Grant 2022ZD02114000; in part by National Natural Science Foundation of China. (#61901077, #62171074, #U19A2082, and #61961160705); and in part by the Natural Science Foundation of Chongqing (CSTB2022NSCQ-MSX1171) and the Scientific and Technological Research Program of Chongqing Municipal Education Commission (KJQN202200640).This work was supported by National Science and Technology Innovation 2030 Major Project of China.