Lightweight multilayer interactive attention network for aspect-based sentiment analysis

Abstract

Aspect-based sentiment analysis (ABSA) aims to automatically identify the sentiment polarity of specific aspect words in a given sentence or document. Existing studies have recognised the value of interactive learning in ABSA and have developed various methods to precisely model aspect words and their contexts through interactive learning. However, these methods mostly take a shallow interactive way to model aspect words and their contexts, which may lead to the lack of complex sentiment information. To solve this issue, we propose a Lightweight Multilayer Interactive Attention Network (LMIAN) for ABSA. Specifically, we first employ a pre-trained language model to initialise word embedding vectors. Second, an interactive computational layer is designed to build correlations between aspect words and their contexts. Such correlation degree is calculated by multiple computational layers with neural attention models. Third, we use a parameter-sharing strategy among the computational layers. This allows the model to learn complex sentiment features with lower memory costs. Finally, LMIAN conducts instance validation on six publicly available sentiment analysis datasets. Extensive experiments show that LMIAN performs better than other advanced methods with relatively low memory consumption.

KEYWORDS:

• Sentiment analysis
• deep learning
• interactive learning
• attention networks

1. Introduction

Recently, aspect-based sentiment analysis (ABSA) (Brauwers & Frasincar, 2022; Nazir et al., 2020) has attracted wide attention as a fine-grained sentiment analysis task (Wei, Zhu, et al., 2022; Xu, Zhang, Zhu, et al., 2022). Unlike traditional sentiment analysis (Wei, Liu, et al., 2022; Zhang, Yu, et al., 2022), ABSA aims to determine the sentiment polarity (a.k.a., emotional tendencies) of a given sentence or document at the aspect level. As shown in Figure 1, given a sentence, “Despite him not being my designated server, he took over for our inexperienced waiter the rest of the dinner”. When the word “server” is viewed as an aspect, the polarity of the sentence at the aspect level is positive, as the sentence reflects a favourable attitude towards “server”. Considering the word “waiter” as an aspect, the expected sentiment polarity is negative. Similarly, the corresponding polarity of the word “dinner” is neutral.

Figure 1. An illustration of ABSA.

Regarding the research on ABSA tasks, existing methods mostly employ deep neural networks to construct sentiment classifiers. Long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997), as a mainstream neural network model, has been widely used in ABSA. For example, Ma et al. (2017) proposed a LSTM-based interactive attention network. Huang et al. (2018) built connections between aspects and contexts by introducing an attention-over-attention network based on the LSTM. However, LSTM-based networks are difficult to parallelise, and suffer from problems with truncated backpropagation and gradient disappearance. Another widely used strategy is to introduce convolutional neural networks (CNN), such as GCAE (Xue & Li, 2018) and UP-CNN (Wang et al., 2021). CNN-based ABSA methods are mostly simple and computationally efficient. However, fixed-size convolutional windows make them difficult to learn long-distance dependencies between words. Recently, with the development of Transformer (Vaswani et al., 2017), attention networks are becoming popular. For instance, Song et al. (2019) suggested an interactive learning model that avoids recurrence by utilising pure attention network for encoding. Yang et al. (2021) constructed an attention network with local context focus mechanism to identify aspect-specific sentiment polarity. Besides these methods mentioned above, various other types of neural networks are also present in ABSA tasks. Such as, CapsNet (Jiang et al., 2019) and KGAN (Zhong et al., 2022).

Although these methods are effective, we believe that aspect-based sentiment analysis remains a challenge: how to describe complex semantic relations between aspect words and their contexts in a lightweight way. This challenge motivated us to develop a powerful neural network that captures the complex correlations between aspect words and their contexts with less memory cost.

To achieve this, we developed a Lightweight Multilayer Interactive Attention Network (LMIAN) for ABSA. Our method is data-driven, parallelisable, and does not rely on external knowledge (e.g. sentiment lexicons). The core of the entire network is an interactive computational layer. Each computational layer contains two interactive attention models. To make LMIAN able to recognise the intricate semantic relationships between aspect words and their contexts with low memory consumption. We stack multiple computational layers with their parameters shared among the layers. The final layer’s computational results are regarded as sentiment features. Our method is an end-to-end model since each component is differentiable, with the advantage of efficiently learning complex sentiment information in a lightweight way. The disadvantage of our method is that the more layers, the longer the computation time. The following is the primary work of this study.

1. We propose a lightweight multilayer interactive attention network for ABSA. The network can learn intricate semantic relationships between aspect words and their contexts through multiple interactive computational layers.

2. We design two attention models for learning representations of specific-aspects and specific-contexts. Moreover, parameters are shared among computational layers to achieve a good balance between model performance and memory cost.

2. Related works

2.1. Neural networks for ABSA

RNN-based ABSA methods have recently achieved remarkable results due to the power of recurrent neural networks (RNN) (Funahashi & Nakamura, 1993) for mining temporal and semantic information from data (Zhang et al., 2018; Zhang, Li, et al., 2022). AdaRNN (Dong et al., 2014) applied a transferred target-dependent dependency tree to a standard recursive neural network for feature learning and achieved competitive performance. TC-LSTM (Tang, Qin, Feng, et al., 2016) significantly improved the model’s performance by capturing information about aspects in a sentence. Apart from these methods mentioned above, many LSTM-based attention models have been proposed recently. For instance, Wang et al. (2016) pioneered combining attention and LSTM for ABSA. RAM (Chen et al., 2017) computes long-distance sentiment features by encoding aspect words employing Bi-LSTM and attention mechanism. Xu, Zhang, Zhang, et al. (2022) proposed an attention-BiLSTM method with transfer learning ability. However, RNN-based methods are memory-intensive and cannot perform parallel computations.

As another dominant neural network, convolution neural networks (CNNs) are simple and efficient since they can be well-parallelised on graphics processing units (GPUs). For example, GCAE (Xue & Li, 2018) and UP-CNN (Wang et al., 2021). However, with a fixed-size kernel window, it is difficult to establish dependencies between distant words.

2.2. Attention networks for ABSA

Recently, with the continuous progress of Transformer, how to apply it to ABSA has attracted the interest of researchers (Tang et al., 2020; Wu et al., 2020; Wu & Ong, 2021). Multi-Head Attention (MHA) in Transformer can model the relationship between distant words in text. This powerful representation ability can help to make connections between aspect words and their contexts. Many MHA-based techniques for sentiment analysis have been proposed, facilitating the development of ABSA. For example, Xu et al. (2020) presented a novel sentiment analysis framework to overcome the time loss inherent in recursive structures. Zeng et al. (2019) proposed a novel ABSA framework with a local context focus. Lv et al. (2021) proposed CAMN with excellent results by introducing MHA to memory networks. However, these models ignore the memory cost caused by complex operations, which makes it challenging to balance model size and performance.

Different from the above-mentioned methods, our method achieves a good balance between performance and size. Specifically, we select attention networks that avoid recurrent structures as the model’s framework. And multiple attentional computational layers are stacked to capture complex sentiment information. In addition, we also introduce a parameter sharing strategy among the computational layers for better optimise the memory cost of the deep network.

3. Lightweight multilayer interactive attention network

We present a lightweight and effective ABSA structure, namely LMIAN. First, we provide a task definition. Second, we overview the method before introducing the interactive attention models in each computational layer. Finally, we show how the technique can be applied to the ABSA tasks.

3.1. Task definition

Given a sentence, as shown in Figure 1, the sentence contains n (e.g. n = 19 in Figure 1) words and m (e.g. m = 3 in Figure 1) aspects. Each aspect contains at least one word. Our task is to identify the emotional tendencies expressed by aspect words like “server”, “waiter”, and “dinner” in the sentence. When we process a text corpus, we first need to perform word embedding (Mikolov et al., 2013; Paccanaro & Hinton, 2001), i.e. map the word w_i into a computable continuous vector ${\mathit{e}}_{\mathit{i}}\in {\mathbb{R}}^{1\times d}$. We stack the vectors of each word in the sentence to form a word embedding matrix $\mathit{E}\in {\mathbb{R}}^{V\times d}$. The notation V is the vocabulary size, and the notation d is the word vector dimension. e_i is a row of the E.

In LMIAN, we utilise BERT (Devlin et al., 2019) to map context words $\left\{w_c^1,\dots ,w_c^n\right\}$ and aspect words $\left\{w_a^1,\dots ,w_a^n\right\}$ into a context embedding matrix $\mathit{C}=\left[{\mathit{e}}_{\mathit{c}}^1,\dots ,{\mathit{e}}_{\mathit{c}}^n\right]\in {\mathbb{R}}^{n\times d}$and an aspect embedding matrix $\mathit{A}=\left[{\mathit{e}}_{\mathit{a}}^1,\dots ,{\mathit{e}}_{\mathit{a}}^m\right]\in {\mathbb{R}}^{m\times d}$, respectively.

3.2. An overview of the method

The structure of LMIAN is shown in Figure 2, which draws inspiration from MHA in machine translation (Vaswani et al., 2017). Our method contains multiple computational layers, each of which consisting of two MHA modules and Point-wise Convolution Transformation (PCT) modules. In the first computational layer, the mapped aspect vector and context vector are used as the inputs of this layer, and the MHA modules within the layer adaptively compute the significant relationships between them. The PCT modules transform the outputs of the MHA modules, and the obtained results are used as inputs to the next computational layer. We stack a series of such computational layers and repeat these steps multiple times to allow LMIAN to learn a broader and more complex abstract relationships. The output hidden representations of the last computational layer are considered as representations containing context and aspect information in relation to each other. The next step will be considered as features for sentiment analysis and used for classification.

Figure 2. The structure of the proposed LMIAN. Share indicates the parameter sharing of BERT. Pool indicates the pooling operation.

Typically, the more layers of computation are stacked, the more parameters the model has, and the more memory is consumed. To decrease the difficulty of model training, we let the parameters of each computational layer be shared. Therefore, the total number of parameters is fixed regardless of how many computational layers the model has. This strategy lays the foundation for building a lightweight multilayer interactive attention network.

3.3. Interactive attention

The attention mechanism has the ability to focus on key information through weight assignment. In this work, we design two interactive attention modules. They can learn specific-aspect context hidden representations and specific-context aspect hidden representations, respectively. The intuition is that coordinating aspect words and their context helps facilitate the capture of complex emotional information. The specific details are described in Figure 3.

Figure 3. Interactive attention mechanism. Linear, T, and Concat denote linear projection, transpose operation, and concatenate operation.

Suppose there is a pair of sequences $\mathit{K}=\left\{{\mathit{k}}_1,\dots ,{\mathit{k}}_n\right\}\in {\mathbb{R}}^{n\times d}$ and $\mathit{Q}=\left\{{\mathit{q}}_1,\dots ,{\mathit{q}}_m\right\}\in {\mathbb{R}}^{m\times d}$, which we take as inputs to the attention modules. The corresponding output sequence is calculated using formula (1), where d_k is the dimension of the sequence. (1) $\text{Attention}(\mathit{Q},\mathit{K})=\text{softmax}\left(\frac{\mathit{Q}{\mathit{K}}^T}{\sqrt{d_k}}\right)\mathit{K}$(1) MHA can learn emotional information from h distinct subspaces, thus enabling the network to capture more nuanced emotional features. The MHA is calculated as follows. (2) $\begin{array}{rl}\text{MHA}\left(\mathit{Q},\mathit{K}\right)& =\left[\text{hea}{\text{d}}_1;\dots ;\text{hea}{\text{d}}_h\right]{\mathit{W}}^O\end{array}$(2) (3) $\begin{array}{rl}\text{hea}{\text{d}}_i& =\text{Attention}\left({\mathit{Q}\mathit{W}}_i^Q,{\mathit{K}\mathit{W}}_i^K\right)\end{array}$(3) where “;” denotes vector concatenation. ${\mathit{W}}_i^Q\in {\mathbb{R}}^{d\times d_k}$, ${\mathit{W}}_i^K\in {\mathbb{R}}^{d\times d_k}$, and ${\mathit{W}}^O\in {\mathbb{R}}^{h{d}_k\times d}$ are learnable weights. Vector sizes in subspaces are calculated as follows. (4) $d_k=d//h$(4) In practice, we take a pair of sequences of contexts and aspects to describe the process of attention calculation. We regard the context matrix C as Q and the aspect matrix A as K, and calculate the context-specific aspect hidden representation ${\mathit{A}}_{\text{c}}$ by: (5) ${\mathit{A}}_c=\text{MHA}(\mathit{C},\mathit{A})$(5) Next, we regard the aspect matrix A as Q and the context matrix C as K, and use formula (6) to compute the aspect-specific context hidden representation ${\mathit{C}}_a$. (6) ${\mathit{C}}_a=\text{MHA}(\mathit{A},\mathit{C})$(6)

Afterward, these hidden state representations are fed into the PCT modules for further transformation.

3.4. Point-wise convolution transformation

PCT is a tick mainly used to transform the information collected by MHA modules. According to the results of the ablation tests (Section 4.5.1), it is known that PCT is beneficial in enhancing the performance of LMIAN. The inputs of the PCT modules are the outputs of the MHA modules. The PCT is calculated as follows. (7) $\text{PCT}\left(\mathit{H}\right)=\sigma \left(\mathit{H}\ast {\mathit{W}}_1+b_1\right)\ast {\mathit{W}}_2+b_2$(7) where H is the MHA output result, * denotes the convolution operation. ${\mathit{W}}_1\in {\mathbb{R}}^{d\times d}$ and ${\mathit{W}}_2\in {\mathbb{R}}^{d\times d}$ are trainable model parameters, $b_1\in {\mathbb{R}}^d$ and $b_2\in {\mathbb{R}}^d$ are the corresponding bias terms. The symbol $\sigma$ refers to the ReLU (Nair & Hinton, 2010) activation function.

Given A_c and C_a, PCT modules are applied to obtain the final context-guided aspect hidden states representation ${\mathit{H}}^A=\left\{{\mathit{H}}_1^A,\dots ,{\mathit{H}}_m^A\right\}$ and aspect-guided context hidden states representation ${\mathit{H}}^C=\left\{{\mathit{H}}_1^C,\dots ,{\mathit{H}}_n^C\right\}$ by: (8) $\begin{array}{rl}{\mathit{H}}^A& =\mathrm{P}\mathrm{C}\mathrm{T}\left({\mathit{A}}_c\right)+\mathit{A}\end{array}$(8) (9) $\begin{array}{rl}{\mathit{H}}^C& =\mathrm{P}\mathrm{C}\mathrm{T}\left({\mathit{C}}_a\right)+\mathit{C}\end{array}$(9)

3.5. Output layer

We pool and concatenate H^A and H^C to obtain final sentiment features X. Finally, a softmax is employed to compute the probability distribution Y over each polarity category. (10) $\mathit{X}=\left[\mathrm{P}\mathrm{o}\mathrm{o}\mathrm{l}\left({\mathit{H}}^A\right);\mathrm{P}\mathrm{o}\mathrm{o}\mathrm{l}\left({\mathit{H}}^C\right)\right]$(10) (11) $Y=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\mathit{X}\right)$(11)

3.6. Why multilayer interactive attention is needed

In general, networks consisting of multiple computational layers have a more robust learning ability than single-layer networks (Chen et al., 2021; LeCun et al., 2015; Tang, Qin, and Liu, 2016). In LMIAN, the essence of single-layer interactive attention is a weight assignment process that may have difficulty learning complex information such as transitions, negation, and emphasis in text. Multi-layer interactive attention is capable of learning high-dimensional abstract text representations. This is because each computational layer focuses on the imported aspect words and their contexts and gradually transforms them into abstract text representations. Combining a sufficient number of such transformations makes it possible to learn highly complex functions of sentence representation for a given aspect.

3.7. Model training

In this work, the aspect input and context input of the LMIAN are reconstructed as “[CLS]” + aspect + “[SEP]” and “[CLS]” + context + “[SEP]”, respectively. We update LMIAN’s parameters via backpropagation and cross-entropy loss minimisation. The loss function is as follows. (12) $loss=-\frac{1}{m}\sum _{i=1}^m\sum _{j=1}^c{y}_{ij}\log \left(\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(z_{ij}\right)}{{\sum }_{k=1}^c\mathrm{e}\mathrm{x}\mathrm{p}\left(z_{ik}\right)}\right)$(12) where m refers the number of samples, c is the number of classes, $y_{ij}$ is the true value of the i-th sample on the j-th category, and $z_{ij}$ denotes the classifier’s output.

4. Experiments

4.1. Datasets

We prepare two types of sentiment analysis datasets, Chinese and English, to comprehensively evaluate the performance of LMIAN. Chinese datasets (Che et al., 2015; Peng et al., 2018) include four categories: Notebook, Phone, Car and Camera. English datasets consist of two categories: Laptop and Restaurant, from SemEval-2014 Task4 subtask2 (http://alt.qcri.org/semeval2014/task4). These datasets are collected from customer reviews of products or services. Each sample in the datasets contains three parts: review content, review object, and sentiment label. Table 1 shows details of these datasets.

Table 1. Statistical information on sentiment analysis datasets, including two English and four Chinese.

		Train			Test
Languages	Datasets	Positive	Neural	Negative	Positive	Neural	Negative
English	Laptop	994	464	870	341	169	128
	Restaurant	2164	637	807	728	196	196
Chinese	Notebook	328	–	88	168	–	35
	Phone	1319	–	341	667	–	156
	Car	708	–	164	213	–	66
	Camera	1197	–	322	541	–	112

4.2. Experimental settings

In experiments, we perform hyperparameter setting and weight parameter initialisation. For the word embedding vectors, we choose a pre-trained BERT-base with 768 embedding sizes for initialisation. For remaining updatable weight matrices are initialised by Glorot (Glorot & Bengio, 2010). See Table 2 for hyperparameter settings.

Table 2. Hyperparameter settings.

Hyperparameters	English/Chinese
Embedding size	768
Hidden size	768
Batch size	16
Max length	80/40
Learning rate	2e-05
L2	1e-05
Dropout	0.5
Optimiser	Adam (Kingma & Ba, 2015)
Evaluation metrics	Accuracy & F1 score

4.3. Model comparisons

We list eight excellent methods as comparison models. Most of them achieved most advanced results for ABSA at that time.

1. ATAE-LSTM differs from the traditional LSTM in that it utilises aspect information and finally focuses with attention (Wang et al., 2016).

2. IAN is a kind of interactive attention network. After LSTM encoding, aspect words and their context words interact with attention. This is the first work to introduce the concept of interactive learning in ABSA tasks (Ma et al., 2017).

3. AOA introduce attention-over-attention to capture the interactions between aspects and contexts (Huang et al., 2018).

4. GCAE is a CNN-based ABSA model. It selectively outputs aspect-specific sentiment features by combining convolutional neural networks and gating mechanisms (Xue & Li, 2018).

5. AEN eschews RNNs and is an entirely attention-based interactive network. Two attention mechanisms, intra- and inter-MHA, are proposed for contexts modelling and context-aware aspects modelling (Song et al., 2019).

6. LCF-BERT is a BERT-based, local context-focused attention model. LCF-BERT argues that local context can better emphasise sentence polarity, and proposes two focusing schemes (Zeng et al., 2019).

7. UP-CNN is a location-aware convolutional neural network ABSA model, which embeds location information into context vectors and extracts aspect-related features (Wang et al., 2021).

8. LCF-ATEPC is a multi-task learning model improved on LCF-BERT with aspect polarity classification ability (Yang et al., 2021).

4.4. Experimental results

LMIAN and comparison models’ prediction results on the six datasets are shown in Tables 3 and 4. From them, several conclusions can be drawn as follows.

1. On the Chinese dataset, LMIAN generally outperforms all comparative models overall, including LSTM-based methods and attention-based methods. For the English datasets, LMIAN also achieves highly competitive results, outperforming both CNN-based and LSTM-based methods.

2. Methods that initialise word vectors with pre-trained language models usually achieve better performance, in both Chinese and English datasets.

3. Further research on the result that LMIAN performs better on the Chinese datasets but slightly worse on the English datasets suggests that longer sentences contain more noise to the detriment of LMIAN.

Table 3. Experimental results of LMIAN and comparison models on four Chinese datasets (%).

	Notebook		Phone		Car		Camera
Models	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1
ATAE-LSTM*	83.47	82.14	85.77	83.87	81.90	76.88	85.54	84.09
IAN-BERT	94.19	93.17	95.87	95.23	97.19	96.47	95.69	94.72
AOA-BERT	93.55	92.46	96.63	96.13	98.23	97.77	96.03	95.21
AEN-BERT	94.19	93.00	96.17	95.59	97.19	96.47	96.72	96.05
LCF-BERT-CDM	92.90	91.56	96.17	95.57	96.49	95.48	95.52	94.70
LCF-BERT-CDW	92.90	91.65	96.63	96.10	97.54	96.90	95.86	95.14
LCF-ATEPC-CDW*	92.96	91.35	97.04	96.17	97.36	96.98	96.78	95.77
LCF-ATEPC-CDM*	93.50	92.34	96.98	96.48	96.52	95.79	96.70	95.76
LCF-ATEPC-Fusion*	93.50	94.45	97.05	96.57	96.81	96.14	96.63	95.60
LMIAN	95.48	94.62	96.63	96.08	98.60	98.22	97.06	96.48

refers to the optimal results in the comparison models. * refers to the results from (Yang et al., 2021). Bold font indicates the best results.

Table 4. Experimental results of LMIAN and comparison models on two English datasets (%).

	Laptop		Restaurant
Models	Acc.	F1	Acc.	F1
ATAE-LSTM	68.70	–	77.20	–
IAN	72.10	–	78.60	–
AOA	74.50	–	81.20	–
GCAE	69.14	–	77.28	–
AEN-GloVe	73.51	69.04	80.98	72.14
AEN-BERT	79.93	76.31	83.12	73.76
UP-CNN	73.73	–	80.05	–
UP-CNN-BERT	76.33	–	84.82	–
LCF-ATEPC-CDM	80.03	76.60	86.06	80.22
LCF-ATEPC-CDW	80.03	77.17	85.97	80.42
LCF-ATEPC-Fusion	80.97	77.86	86.77	80.54
LMIAN	80.95	77.90	85.88	80.06

With the above analysis, LMIAN is effective on ABSA tasks in Chinese and English. Moreover, LMIAN is more effective in predicting the polarity of short sentences.

4.5. Model analysis

4.5.1. Ablation tests

To verify each element in LMIAN is reasonable, several LMIAN variants are constructed, and they are described below.

1. LMIAN w/o PCT: The PCT modules are removed from each computational layer.

2. LMIAN w/o MHA: The MHA modules are removed from each computational layer.

3. LMIAN with MHSA: Replacing MHA modules in each computational layer with MHSA (Vaswani et al., 2017).

4. LMIAN with LSTM: Replacing MHA modules in each computational layer with LSTM.

The F1 scores of LMIAN are generally better than its variants according to the experimental results in Table 5. When MHA or PCT modules are removed from each computational layer, F1 scores have different degrees of decrease. This suggests that the existence of MHA and PCT modules are meaningful. When MHA was replaced by MHSA, LMIAN performs poorly, because the model loses its interactive capability. Interestingly, LMIAN with LSTM performs well on the Phone datasets. The reason for this phenomenon may be that the sentiment features in the Phone dataset show more obvious and do not need to be captured by complex computation. However, with the introduction of LSTM, the model loses the ability of parallel computation.

Table 5. Ablation results.

	Notebook	Phone	Car	Camera
Models	F1	F1	F1	F1
LMIAN w/o PCT	93.09	95.79	95.87	95.85
LMIAN w/o MHA	92.54	95.78	96.07	95.53
LMIAN with MHSA	92.37	95.73	96.07	95.48
LMIAN with LSTM	91.75	96.24	96.41	95.90
LMIAN	94.62	96.08	98.22	96.48

4.5.2. Effect of number of computational layers

According to the results in Figures 4 and 5, we can draw the following conclusions:

1. LMIAN with multiple computational layers performs better. This conclusion validates our idea that models learn complex sentiment information in a shallow interactive way is difficult.

2. With stacking of computational layers, LMIAN's performance may decrease. We can observe that LMIAN with two computational layers performs best on the Notebook and Camera datasets. LMIAN achieves optimal performance on the Car and Phone datasets when the number of computational layers is stacked to 3 and 5. We speculate that some relationship may exist between the size of the dataset and the complexity of the semantics. The more complex the dataset is, the more computational layers are required.

Figure 4. Accuracy for different stacking layers on the four datasets.

Figure 5. F1 score for different stacking layers on the four datasets.

4.5.3. Model size

LMIAN with parameter-sharing strategy takes less memory during training. A good balance is achieved between the computational ability and the size of the model. Results recorded in Table 6 confirm this. We reproduce part of models that do not perform weakly with the same device and code framework. IAN-BERT and AOA-BERT are LSTM-based attention models, and memory optimisation is more difficult since hidden states need to be kept in memory to calculate attention scores. AEN-BERT and LCF-ATEPC are pure attentional encoding networks that avoid recurrent and are more lightweight compared to LSTM-based models. Our LMIAN constructs computational layers with attention networks and shares parameters among the computational layers.

Table 6. Parameters and memory cost of the model training on the Car dataset.

	Size
Models	Params × 10^6	Memory
IAN-BERT	116.44	444.8
AOA-BERT	121.17	462.83
AEN-BERT	109.95	420.03
LCF-ATEPC	109.94	420.01
LMIAN	109.35	417.77

Memory is in MB.

4.5.4. Case study

Lastly, we explore to what extent multiple computational layers improve LMIAN’s predictive ability. Figure 6 presents the single- and multi-layer LMIAN prediction results for three different sentences.

1. We can find that model is easier to make correct predictions for sentences with explicit sentiment. For example, in the first sentence, both single- and multi-layer LMIAN successfully predict the polarity of aspect words.

2. For sentences with complex information like “transitions”, multi-layer LMIAN performers better. For instance, in the second sentence, single-layer LMIAN makes incorrect judgements due to without enough learning ability.

3. For sentences with implicit sentiment like the third example, both single- and multi-layer LMIAN fail to predict correctly. This situation may require us to combine other modals (e.g. images) to make a better judgment.

Figure 6. Prediction results of single- and multi-layer LMIAN on real samples. Samples extracted from the laptop dataset. The words in red are the aspect words in samples. “Gold” indicates true sentiment polarity. “√” denotes a correct prediction. “×” denotes an incorrect prediction. We use a three-layer LMIAN for testing.

5. Conclusions

To lightly and effectively learn complex sentiment information hidden in the text, we propose a lightweight multilayer interactive attention network, namely LMIAN. The conclusion of this paper focus on the following two points:

1. Multilayer interactions help improve model performance. We argue that the deep interaction of aspects and contexts is beneficial to facilitate the capture of complex emotional information. The experimental results in Section 4.5.2 show that multilayer interactions improve the model performance.

2. LMIAN has less memory consumption. In LMIAN, we make the parameters of each layer of the network shared, aiming at learning data representations with multiple levels of abstraction at lower complexity. The experimental results in Section 4.5.3 show that LMIAN has fewer parameters and less memory consumption.

Extensive experiments have proved that our LMIAN achieves a better balance between the model’s performance, size, and GPU memory consumption. In the future, we will further optimise our interactive attention model to have higher performance and lower GPU memory consumption. One of our ideas is to perform deep interaction of local features with aspect words to improve model performance by reducing interference information.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the National Natural Science Foundation of China [grant number 62076006], the University Synergy Innovation Programme of Anhui Province [grant number GXXT-2021-008], the Anhui Provincial Key R&D Programme [grant number 202004b11020029], and the Scientific Research Fund for Young Teachers of Anhui University of Science & Technology [grant number QNZD2021-02].