STSG: A Short Text Semantic Graph Model for Similarity Computing Based on Dependency Parsing and Pre-trained Language Models

ABSTRACT

Short text semantic similarity is a crucial research area in nature language processing, which is used to predict the similarity between two sentences. Due to the sparsity features of short texts, words are isolated in the sentence and the correlations of words are ignored, it is very difficult to calculate the global semantic information. Based on this, short text semantic graph (STSG) model based on dependency parsing and pre-trained language models is proposed in this paper. It utilizes the syntactic information to obtain word dependency relationships and incorporate it into pre-trained language models to enhance the global semantic information of sentences. So it can address the semantic sparsity more effectively. A text semantic graph layer based on the graph attention network (GAT) is also realized, which regards word vectors as node features and word dependency as edge features. The attention mechanism of GAT can identify the importance of different word correlations and solve the word dependency modeling effectively. On the challenging short text semantic benchmark dataset MRPC, the STSG model achieves an F1-score of .946, which is further improved 2.16% over previous SOTA approaches. At the time of writing, STSG has achieved a new SOTA performance on the MRPC dataset.

Introduction

Short text semantic similarity (STS) computation is a fundamental problem in natural language processing, which aims to predict the similarity between two sentences. STS computation has numerous applications, especially in the information retrieval. STS can calculate the relevance of a user’s question to the content, addressing the issue of information overload and improving search strategies and results (Chandrasekaran and Mago 2021; Mengting et al. 2021). For example, paraphrase recognition is used to determine the category scores of two sentences in text classification, and STS is used to assess the similarity between sentences in question and answer. However, it is important to note that the recognition tasks for the short text are different from long texts such as news articles and magazines. In addition, the content of short texts is often sparse (Mengting et al. 2021), it is challenging to accurately compute the semantic similarity between two sentences.

There are three primary methods for computing short text semantic similarity. Deep learning-based methods (Jonas and Aditya 2016; Weidong et al. 2022; Zhiguo, Haitao, and Abraham 2016) can generate vector representations of sentences and calculate their similarity using neural networks. However, these methods require a large amount of computing power and may struggle to accurately capture the semantics of a sentence. BERT (Devlin et al. 2018), T5 (Raffel et al. 2020), DeBERTav3 (Pengcheng, Jianfeng, and Weizhu 2023) and other approaches based on the pre-trained language model use a large-scale unlabeled corpus on upstream tasks, and then fine-tune the downstream task to achieve good results in text semantic similarity computing. SimCSE (Tianyu, Xingcheng, and Danqi 2021), DiffCSE (Yung-Sung et al. 2022), and other (Lingling et al. 2024) comparative learning-based approach distinguish between similar and dissimilar representations of data points. Although existing methods have achieved good results in computing text semantic similarity, there are two challenges for computational research on the semantic similarity of short texts:

1. Sparsity of Short Texts. Short texts are typically less than 200 words and contain sparse information, making it difficult to extract effective feature words. In addition, short texts often have unpredictable language variations and lack contextual information, such as emoticons, acronyms, and internet slang, which can result in a significant amount of extraneous information. For example, the sentence “I like the larger screen size of the iPhone 6 compared to its predecessor” does not mention that the iPhone 6 Plus, a similar product, also has a larger screen than its predecessor. Short texts may lack related words, making it difficult for models to reason about global semantic information. This can affect the accuracy of text similarity.

2. Word Relevance Modelling. One of the main challenges facing STS models is how to capture semantics and underlying context (Zhe et al. 2019; Ghafour, Jamshid Bagherzadeh, and Mohammad-Reza 2022). Deep embedding models have been studied for STS tasks with promising results (Jianguo et al. 2019). However, existing models usually assume that words are independent of each other and that all words and attributes are equally important for a sentence. This assumption is not always true. The semantic capacity of short texts is limited, and it is difficult to discriminate the importance of different relationships if the correlations between words are ignored.

Motivation: Using neural models in STS is not only simple and efficient, but also it can integrate a wider range of informative knowledge. Syntactic information and dependency trees have been used in numerous studies (Yuanhe et al. 2021). Dependency trees can establish long-distance connections between key words, facilitating the extraction of relationships between sentence pairs, which help the system to extract relations between sentence pairs more effectively. Graph neural networks have been used recently to combine grammatical knowledge with pre-trained language models, and it achieves good outcomes (Hironori et al. 2022; Liang, Chengsheng, and Yuan 2019; Yangfan et al. 2021). However, these methods tend to focus solely on the feature learning of local nodes in the graph neural network, while disregarding the feature learning of neighboring nodes. Graph Attention Networks (GAT) can learn distinct weights for neighboring nodes by using the attention mechanism, addressing the limitations of traditional graph neural networks like GCN (Velikovi et al. 2017).

To address the above challenges, we propose the short text semantic graph (STSG) model, which based on dependency parsing and pre-trained language models. The STSG model, which makes use of syntactic information, extracts the word dependency relationships and incorporates it into pre-trained language models to enhance the global semantic information of sentences. The STSG model can addresses the semantic sparsity more effectively. We also propose a text semantic graph layer based on the graph attention network (GAT) (Velikovi et al. 2017), which regards word vectors as node features and dependency parsing relationships of words as edge features. The attention mechanism of GAT enhances its ability to capture remote dependencies, which can identify the importance of different word correlations and solve the word dependency modeling effectively. Four layers are proposed in the STSG model. 1) The dependency parsing layer is used to extract the dependency parsing relations of a short text and obtain multiple relation triples. 2) The sentence encoding layer is used to encode the semantic information and extract word vectors of the short text by the DeBERTav3 model (Pengcheng, Jianfeng, and Weizhu 2023). 3) The text semantic graph layer is used to learn graph attention features of word vectors and syntactic dependencies. 4) The similarity computing layer is used to build a fine-tuning model to calculate the similarity for different sentences by the fully connected network.

The main contributions of this paper are summarized as follows:

1. A short text semantic graph (STSG) model by using syntactic information is proposed. It extracts the word dependency relationships and incorporates it into pre-trained language models to enhance the global semantic information of sentences.

2. A GAT-based text semantic graph layer is proposed to use word vectors as node features and words dependency as edge features. It improves word relevance modeling by identifying the importance of different words.

3. Extensive evaluation and analysis shows that our framework not only achieves new SOTA performance on MRPC, but also has strong generalization capabilities. It is also more effective on low-resource datasets.

There are still some limitations in the STSG model. The longer texts more than 512 tokens cannot be supported in the STSG model. The semantic similarity of English text can be calculated in the model, but the cross-lingual semantic similarity, such as Chinese, cannot be calculated. For the limited computational resources, we are unable to conduct experiments on recently pre-trained large language models, such as GPT4 (Achiam et al. 2023)

The remainder of this paper is organized as follows. The related work is reviewed in Section 2. The proposed model STSG are described in Section 3. The experimental settings are presented in Section 4. Finally, the work is concluded and prospects are given in Section 5.

Related Works

In this section, we first describe the related research on short text semantic similarity in Section 2.1, followed by related work on dependency parsing in Section 2.2. Finally, we present applications of short text semantic similarity in Section 2.3.

Short Text Semantic Similarity

For semantic similarity in short texts, deep learning-based approaches are an important approach. Zhiguo, Haitao, and Abraham (2016) focus on the importance of non-similar parts between two sentences, and use a two-channel convolutional neural network (CNN) to decompose similar and non-similar components, which improves the global semantic representation ability of sentences. Paul, Maarten, and Mihai (2016) use the bidirectional LSTM model to obtain information from two directions of input text, which allows them to capture the bidirectional semantic information of sentences. Weidong et al. (2022) propose Re-LSTM, a weighted word-embedded long- and short-term memory network that reduces model parameters and computation. Methods based on pre-trained language models are the other main approach to semantic similarity of short texts. BERT (Devlin et al. 2018) uses an existing unlabeled corpus to pre-train the Transformer model to obtain word embedding vectors, and then fine-tunes the model to complete the sentence similarity computation. Joshi et al. (2020) propose a pre-trained model (called SpanBERT) based on word segmentation, which adds a mask to the random adjacent word. Kevin et al. (2020) propose an efficient pre-training method called replaced token detection (RTD). During the pre-training process, the generator can replace the words in the sentence, and then the discriminator determines which words in the sentence have been replaced. T5 (Raffel et al. 2020) uses the unified pre-trained language models to learn multiple different NLP tasks. DeBERTa (He et al. 2021) and DeBERTav3 (Pengcheng, Jianfeng, and Weizhu 2023) use disentangled attention to improve BERT and further improve model performance. However, these methods lack a semantically enhanced representation of the text and do not take into account the dependencies between different words. Comparative learning-based approaches for short text semantic similarity have been extensively researched. Sentence-BERT (Nils and Iryna 2019) improves BERT by using siamese and triplet networks to update parameters and generate semantic embedding vectors. DiffCSE (Yung-Sung et al. 2022) presents an unsupervised contrast learning framework for sentence embeddings that is sensitive to the differences between original and edited sentences, enhancing their representations. SimCSE (Tianyu, Xingcheng, and Danqi 2021) proposes a straightforward contrast learning framework that predicts input sentences in the contrast target, using only standard dropout as noise. The article effectively promotes the use of state-of-the-art sentence embedding. Lingling et al. (2024) propose similarity and relative similarity strategies to identify false-negative samples in potential contrast learning, which can further improve performance significantly. Previous research methods are described in Table 1.

Table 1. Previous research methods.

	Methods	Year	Descriptions
DL-based Similarity	CNN (Zhiguo, Haitao, and Abraham 2016)	2016	This model used two-channel convolutional neural network (CNN) to decompose similar and non-similar components, which improves the global semantic representation ability of sentences.
	LSTM (Paul, Maarten, and Mihai 2016)	2016	This model used the bidirectional LSTM model to obtain information from two directions of input text, and can capture the bidirectional semantic information of sentences.
	Re-LSTM (Weidong et al. 2022)	2022	This model used a weighted word-embedded long- and short-term memory network that reduces model parameters and computation. Methods that rely on pre-trained language models
PLM-based Similarity	BERT (Devlin et al. 2018)	2018	This model used an existing unlabeled corpus to pre-train the Transformer model to obtain word embedding vectors, and then fine-tuned the model to complete the sentence similarity computing.
	DeBERTa (He et al. 2021)	2021	This model used the disentangled attention to improve BERT and further improve the performance of the model.
	ESM-2 (Zemin et al. 2023)	2022	This model used the sequence text data using large-scale pre-trained models.
	DeBERTav3 (Pengcheng, Jianfeng, and Weizhu 2023)	2023	This model is an efficient and simple pre-trained model to calculate text similarity.
CL-based Similarity	Sentence-BERT (Nils and Iryna 2019)	2019	This model improves BERT by using Siamese and Triplet Networks to update parameters and generate semantic embedding vectors
	DiffCSE (Yung-Sung et al. 2022)	2022	This model presents an unsupervised contrast learning framework for sentence embeddings that are sensitive to the differences between original and edited sentences, enhancing their representations.
	SimCSE (Tianyu, Xingcheng, and Danqi 2021)	2021	This model proposes a straightforward contrast learning framework that predicts input sentences in the contrast target, using only standard dropout as noise.
	Lingling et al. (2024)	2024	The method distinguishes between similar and dissimilar representations of data points.

Dependency Parsing

Recently, some methods have combined grammatical information in pre-trained language models and achieved good results in NLP tasks. Zhe et al. (2019) proposed the ACV tree, which combines word embedding and syntactic information. They also developed the ACVT kernel to calculate sentence similarity. Minh Hieu and Philip (2020) introduced syntax into the attention and considered partial speech embedding, dependency-based embedding, and contextualization embedding to improve the performance of the extractors. Yu et al. (2020) propose a continuous learning framework (called ERNIE) for deeply integrated knowledge to continuously learn vocabulary, syntactic, and semantic knowledge by introducing more tasks. Yuanhe et al. (2021) propose a dependent-driven relational traction method (called a-GCN), which applies graph convolutional network attention to different context words in the syntax dependency tree to distinguish the importance of different word dependencies. Guimin et al. (2021) propose a type-aware mapping memory (called TaMM) for relation extraction, which can encode sentence dependencies and dependency types. Binyuan et al. (2022) propose an S²SQL model for the text-to-SQL task, which jointly encodes the text syntax and the database schema. However, existing grammar-based methods do not consider feature learning of neighbor nodes, and cannot handle dependencies between words. Based on this, the proposed STSG uses the dependency parsing method to achieve an accurate understanding of the semantic relationships between different sentences. We utilize a dependency parsing layer to extract multiple relationship triples, and use the GAT to learn the attention score between nodes by triples.

Application of Short Text Semantic Similarity

Short text semantic similarity has various applications, including text classification, text clustering, sentiment analysis, information retrieval, social networks, academic plagiarism detection, and specific domains (Mengting et al. 2021; Song and Hai 2022). Successful approaches in the biomedical domain include ontology-based and neural network-based metrics. For example, Dazhi et al. (2023) utilized textual semantic similarity to identify cancer driver genes. They introduced GASN, a similarity network-driven gamma distribution test for gene identification, which combines machine learning and distributional statistical methods. Yan et al. (2024) proposed a multiple similarity deep graph network to deal with user clustering in human-computer interaction using textual semantic similarity. Wenjuan et al. (2024) also used textual semantic similarity to detect personality traits and proposed a PS-GCN model that integrates psychological knowledge and affective semantic features through graph convolutional networks. This model efficiently exploits textual similarity to extract affective features, resulting in a significant improvement in the classification accuracy of the personality detection task. Zhaorui et al. (2023) employed textual semantic similarity to address the task of visual language comprehension. They proposed a new metric model, Semantic Similarity Distance (SSD), to mitigate semantic inconsistencies and bridge the gap between text and image.

The Proposed Method

This section presents a detailed explanation of STSG. Firstly, a method for solving short text sparsity is introduced by using the dependency parsing layer and sentence coding layer. Next, the semantic graph layer is introduced to distinguish the relevance of different words, which improves word relevance modeling. Finally, the similarity computing layer is used for similarity computation, as shown in Figure 1.

Figure 1. Workflow of the proposed STSG.

Problem Definition

Definition 1.

(Short text semantic similarity) Given the two short sentences $S_1=\left(p_1,p_2,\dots ,p_m\right)$and$S_2=\left(q_1,q_2,\dots ,q_n\right)$, where $\mathit{p}$ and $\mathit{q}$ are the words in the sentences $S_1$and $S_2$, respectively. In additional, $\mathit{m}$ and $\mathit{n}$ represent the length of the sentences $S_1$ and $S_2$, respectively. A classifier was trained to accurately predict the similarity between $S_1$ and $S_2$. Lin (1998) derived a text similarity theorem based on information theory, as shown in Equation 1(1) $\mathit{Similarity}\left(S_1,S_2\right)=\frac{logP\left(c\mathrm{ommon}\left(S_1,S_2\right)\right)}{\mathit{log}\mathit{P}\left(\mathit{description}\left(S_1,S_2\right)\right)}$(1) :

(1) $\mathit{Similarity}\left(S_1,S_2\right)=\frac{logP\left(c\mathrm{ommon}\left(S_1,S_2\right)\right)}{\mathit{log}\mathit{P}\left(\mathit{description}\left(S_1,S_2\right)\right)}$(1)

where $\mathit{P}\left(\cdot \right)$ represents probability, $\mathit{common}\left(S_1,S_2\right)$ is the common information of $S_1$and $S_2$, and $\mathit{description}\left(S_1,S_2\right)$ is the descriptions of all information in $S_1$and $S_2$. The range of the similarity is between 0 and 1. The $\mathit{log}\left(\cdot \right)$simplifies the calculation of probabilities.

Dependency Parsing Layer

The syntax dependency between each word in sentence $S_1$and sentence $S_2$ can be extracted by the dependency parsing layer, which comprises the dependency parsing module and the dependency representing module. Dependency parsing technology is used to obtain syntax components and analyze the relationships between them. The attention of different dependencies between words in short text is calculated using dependency parsing features. In order to obtain the similarity of sentences $S_1$and $S_2$, the model jointly calculates the semantic representation between two sentences. In this paper, we constructed our dependency parsing module using the DDParser toolkit (Shuai et al. 2020) to generate dependency analysis results. DDParser is capable of processing text in both Chinese and English languages (Shuai et al. 2020).

Given a sentence $\mathit{S}\mathit{\hspace{0.17em}}=\left\{W_1,W_2,\dots ,W_n\right\}$, where $W_n$ is the $\mathrm{n}$-th word in S. According to the result of dependency parsing, we obtain the syntax dependency graph $\mathit{G}=\left(\mathit{V},\mathit{E}\right)$of $\mathit{S}$, where $\mathit{V}$ represents some nodes corresponding to each word in $\mathit{S}$, and $\mathit{E}$ represents directed edges, indicating the dependency relationship between words. Each $\mathit{E}$ includes a label to indicate the specific dependency. Dependency syntax allows for the mapping of the core word to its dependency syntax tree, as shown in Figure 2.

Figure 2. Dependency parsing layer.

Dependency parsing concentrates on the word and its dependencies, rather than its components. To make the model’s ability to learn dependency representation, we convert the syntax dependency graph G into a relation triplet R, as shown in Equation 2(2) $\mathit{R}=\left\{\left(W_1,W_3,r_1\right),\left(W_4,W_7,r_5\right)\left(W_6,W_8,r_3\right),\dots ,\left(W_n,W_m,r_i\right)\right\}$(2) :

(2) $\mathit{R}=\left\{\left(W_1,W_3,r_1\right),\left(W_4,W_7,r_5\right)\left(W_6,W_8,r_3\right),\dots ,\left(W_n,W_m,r_i\right)\right\}$(2)

where $r_i$ represents the $\mathit{i}$-th type of dependency between words.

The relation triplet $\mathit{R}$ is then converted into an adjacency matrix $\mathit{A}={\left({\mathit{a}}_{\mathit{ij}}\right)}_{\mathit{n}\times \mathit{n}}$, where $\mathit{n}$ represents the matrix size. In order to calculate the relational attention between words by using the text semantic graph layer, it is necessary to remove the noisy data in the dependencies. Therefore, it is necessary to ignore the specific direction of dependencies between words in the adjacency matrix A and focus only on their relations. A value in the adjacency matrix $\mathit{A}$ represents whether there is a dependency relation between the $\mathit{i}$-th word and the $\mathit{j}$-th word. The adjacency matrix $\mathit{A}$ is defined as Equation 3(3) $a_{\mathit{ij}}=\left\{\begin{array}{c}1,\mathit{is}\mathit{adependency}\mathit{relation}\\ 0,\mathit{is}\mathit{not}\mathit{adependency}\mathit{relation}\end{array}\right.$(3) .

(3) $a_{\mathit{ij}}=\left\{\begin{array}{c}1,\mathit{is}\mathit{adependency}\mathit{relation}\\ 0,\mathit{is}\mathit{not}\mathit{adependency}\mathit{relation}\end{array}\right.$(3)

Finally, the adjacency matrix $\mathit{A}$ is vectorized to obtain the representation $\mathit{D}$of dependency syntax, which constitutes the edge features of the text semantic graph. The process of the dependency parsing layer is shown in Algorithm 1.

Table

	Algorithm 1: Dependency Parsing Layer
	Input: Sentence $\mathit{S}=\left\{W_1,W_2,\dots ,W_n\right\}$ Output: Characterized Adjacency Matrix $\mathit{D}$ Progress: 1: $\mathit{InitializeD}\in R^{d\times d_q}$ 2: $\mathrm{G}\leftarrow \mathit{DDParser}\left(\mathit{S}\right)$ 3: for $\mathit{t}=1$ to G$.\mathrm{size}$ do 4: $\mathit{R}\leftarrow \left\{\left(W_1,W_3,r_1\right),\left(W_4,W_7,r_5\right)\left(W_6,W_8,r_3\right),\dots ,\left(W_n,W_m,r_i\right)\right\}$ 5: end for 6: for r$=1$ to $\mathit{R}.\mathit{size}$ do 7: adjacency matrix $\mathit{A}\leftarrow {\left({\mathit{a}}_{\mathit{ij}}\right)}_{\mathit{n}\times \mathit{n}}$ 8: $a_{\mathit{ij}}\leftarrow 0$ 9: if $\mathit{R}.r_i\mathit{\hspace{0.17em}}==\mathrm{true}$ do 10: $a_{\mathit{ij}}\leftarrow 1$ 11: end if 12: end for 13: $\mathit{D}\leftarrow \mathrm{tensor}\left(\mathit{A}\right)$

Algorithm 1

Algorithm 1 describes the computing process of dependency parsing layer. First, in step 1, the output variable $\mathit{D}$ is declared, and in step 2, the syntax dependency graph $\mathrm{G}$ of the sentence $\mathit{S}$ is obtained. In steps 3–5, according to the syntax dependency graph $\mathrm{G}$, we can obtain a relation triplet $\mathit{R}$. Simultaneously, in step 6–7, we convert the relation triplet $\mathit{R}$ into the adjacency matrix $\mathit{A}={\left({\mathit{a}}_{\mathit{ij}}\right)}_{\mathit{n}\times \mathit{n}}$, and initialize $a_{\mathit{ij}}$ to 0 (step 8). In steps 9–12, we judge whether two words contain dependency relations, and if they do, we set $a_{\mathit{ij}}$ to 1. Finally, we characterize the adjacency matrix into vectors (step 13).

Sentence Coding Layer

The sentence coding layer can encode a sentence into vectors. Following DeBERTav3 (Pengcheng, Jianfeng, and Weizhu 2023), we use a multi-layer bidirectional Transformer (Ashish et al. 2017; Mary and Marcus 2022) as the model backbone to encode the semantic information of short texts, as shown in Figure 3.

Figure 3. Sentence coding layer.

First, given two sentences $S_1=\left\{W_1^1,W_1^2,W_1^3,W_1^4,\dots ,W_1^n\right\}$ and $S_2=\left\{W_2^1,W_2^2,W_2^3,W_2^4,\dots ,W_2^n\right\}$, where $W_i^j$represents the $\mathit{j}$-th word in $S_i$. To learn the deep interaction features between sentences $S_1$ and $S_2$, we use tags $\left[\mathit{CLS}\right]$ and $\left[\mathit{SEP}\right]$ to concatenate the above sentences for joint representation. The tagged sequence $X_i$ is shown in Equation 4(4) $X_i=\left[\mathit{CLS}\right]W_1^1,W_1^2,W_1^3,W_1^4,\dots ,W_1^n\left[\mathit{SEP}\right]W_2^1,W_2^2,W_2^3,W_2^4,\dots ,W_2^n\left[\mathit{SEP}\right]$(4) .

(4) $X_i=\left[\mathit{CLS}\right]W_1^1,W_1^2,W_1^3,W_1^4,\dots ,W_1^n\left[\mathit{SEP}\right]W_2^1,W_2^2,W_2^3,W_2^4,\dots ,W_2^n\left[\mathit{SEP}\right]$(4)

Then we encode the sequence $X_i$. Specifically, the sentence coding layer takes a sequence $X_i$ as the input and applies $\mathit{N}$ transformer layers to generate a contextual representation $H^n$, as shown in Equation 5(5) $H^n=\mathit{Transforme}{r}_n\left(H^{\mathit{n}-1}\right)$(5) :

(5) $H^n=\mathit{Transforme}{r}_n\left(H^{\mathit{n}-1}\right)$(5)

where $\mathit{n}\in \left[1,\mathit{N}\right]$ represents the $\mathit{n}$-th layer, $H^0$ is the embedding of the sequence $X_i$, and each transformer layer contains an identical transformer block. The transformer block consists of disentangled attention (He et al. 2021) and a feed-forward neural network (FFN). Formally, the output of the transformer block is calculated as shown in Equation 6(6) $H^0=\left\{H_i,P_{\mathit{i}|j}\right\}\times {\left\{H_j,P_{\mathit{j}|i}\right\}}^{\mathit{T}}=H_i{H}_j^T+H_i{P}_{j|i}^T+P_{\mathit{i}|j}{H}_j^T+P_{\mathit{i}|j}{P}_{j|i}^T$(6)  ~ Eq (8):

(6) $H^0=\left\{H_i,P_{\mathit{i}|j}\right\}\times {\left\{H_j,P_{\mathit{j}|i}\right\}}^{\mathit{T}}=H_i{H}_j^T+H_i{P}_{j|i}^T+P_{\mathit{i}|j}{H}_j^T+P_{\mathit{i}|j}{P}_{j|i}^T$(6)

where $H_i$ represents the $\mathit{i}$-th word embedding of the sequence $X_i$, and $P_{\mathit{j}|i}$ represents the relative position embedding of the sequence $X_i$. $\mathit{T}$represents the transpose operation.

A decoupling matrix of content embedding and relative position embedding are used to calculate the weight of attention between words. The projection matrix is calculated as follows:

(7) $Q_c=H^0{W}_{\mathit{q},\mathit{c}},K_c=H^0{W}_{\mathit{k},\mathit{c}},V_c=H^0{W}_{\mathit{v},\mathit{c}},Q_r=\mathit{P}{W}_{\mathit{q},\mathit{r}},K_r=\mathit{P}{W}_{\mathit{k},\mathit{r}}$(7)

where $Q_c$, $K_c$ and $V_c$represents the content vectors generated by projection matrices, $W_{\mathit{q},\mathit{c}}$，$W_{\mathit{k},\mathit{c}}$，$W_{\mathit{v},\mathit{c}}$ $\in R^{\mathit{d}\times \mathit{d}}$ respectively, and $\mathit{P}\in R^{2\mathit{K}\times \mathit{d}}$ represents the vector of the relative position embedding in a shared layer. $Q_r$ and $K_r$ represent the relative position embedding generated by projection matrices $W_{\mathit{q},\mathit{r}}$，$W_{\mathit{k},\mathit{r}}\in R^{\mathit{d}\times \mathit{d}}$, respectively, as shown in Eq (8)(8) $z_i=Q_i^c{K}_j^{cT}+Q_i^c{{\mathit{K}}_{\mathit{r},\mathit{\delta }\left(\mathit{i},\mathit{j}\right)}}^{\mathit{T}}+K_j^c{{\mathit{Q}}_{\mathit{r},\mathit{\delta }\left(\mathit{i},\mathit{j}\right)}}^{\mathit{T}}$(8) :

(8) $z_i=Q_i^c{K}_j^{cT}+Q_i^c{{\mathit{K}}_{\mathit{r},\mathit{\delta }\left(\mathit{i},\mathit{j}\right)}}^{\mathit{T}}+K_j^c{{\mathit{Q}}_{\mathit{r},\mathit{\delta }\left(\mathit{i},\mathit{j}\right)}}^{\mathit{T}}$(8)

where $z_i$ is the$\mathit{i}$-th element in the attention matrix $\mathrm{z}$, $Q_i^c$ represents the $\mathit{i}$-th row in $Q_c$, $K_j^c$ represents the $\mathit{j}$-th row in $K_c$, $K_{\mathit{r},\mathit{\delta }\left(\mathit{i},\mathit{j}\right)}$ represents the relative distance $\mathit{\delta }\left(\mathit{i},\mathit{j}\right)$ of the $\mathit{r}$-th row in $K_r$, and $Q_{\mathit{r},\mathit{\delta }\left(\mathit{i},\mathit{j}\right)}$ represents the relative distance $\mathit{\delta }\left(\mathit{i},\mathit{j}\right)$ of the $\mathit{r}$-th row in $Q_r$. We calculate the disentangled attention $z_i$, as shown in Eq. (9)(9) $z_i=\mathit{Softmax}\left(\frac{z_i}{\sqrt{3\mathit{d}}}\right)\times V_c$(9) :

(9) $z_i=\mathit{Softmax}\left(\frac{z_i}{\sqrt{3\mathit{d}}}\right)\times V_c$(9)

where $\mathit{Softmax}\left(\cdot \right)$ represents an activation function, and we feed $z_i$ to the FFN and normalize $z_i$ to get the normalized output$r_i$, as shown in Eq (10)(10) $r_i=\mathit{LN}\left(\mathit{FFN}\left(z_i\right)+z_i\right)$(10) :

(10) $r_i=\mathit{LN}\left(\mathit{FFN}\left(z_i\right)+z_i\right)$(10)

where $\mathit{LN}\left(\cdot \right)$ represents a $\mathit{LayerNorm}$ function (Lei Jimmy et al. 2016), and $\mathit{FFN}\left(\cdot \right)$ represents a feed forward neural network.

Finally, we use Transformer Decoders (Ashish et al. 2017) to obtain intermediate embedding, and we input the intermediate embedding into the pooling layer. We take the vector $\mathit{S}{ }^{\mathrm{\prime }}$ of the pooling layer at the $\mathit{CLS}$ as the word vector of $X_i$, defined as $\mathit{S}{ }^{\mathrm{\prime }}\in R^{d\times d_q}$. We regard $\mathit{S}{ }^{\mathrm{\prime }}$ as a node feature of the semantic graph. Algorithm procedure of the sentence coding layer is shown in Algorithm 2.

Table

Algorithm 2: Sentence Coding Layer

Input: $\mathit{S}=\left[\mathit{CLS}\right]S_1\left[\mathit{SEP}\right]S_2\left[\mathit{SEP}\right]$/*A sentence pair */ Output: $\mathit{S}{ }^{\mathrm{\prime }}$/* Sentence coding */ Progress: 1: $\mathit{InitializeS}{ }^{\mathrm{\prime }}\in R^{d\times d_q}$ 2: for $\mathit{i}=1\mathit{to}\mathit{transforme}{r}_n\left(H^{\mathit{n}-1}\right).\mathit{size}$ do 3: $H^0\leftarrow H_i{H}_j^T+H_i{P}_{j|i}^T+P_{\mathit{i}|j}{H}_j^T+P_{\mathit{i}|j}{P}_{j|i}^T$ 4: $\mathit{Matrices}{Q}_c,K_c,V_c,Q_r,K_r\in R^{\mathit{d}\times \mathit{d}}$ 5: $z_i\leftarrow \mathit{DisentangledAttention}(Q_c,K_c,V_c,Q_r,K_r,\mathit{H}$) 6: $r_i\leftarrow \mathit{LayerNorm}\left(z_i\right)$ 7: $r_i\leftarrow \mathit{FFN}\left(r_i\right)$ 8: $\mathit{S}{ }^{\mathrm{\prime }}\leftarrow \mathit{S}{ }^{\mathrm{\prime }}+r_i$ 9: end for 10: $\mathit{S}{ }^{\mathrm{\prime }}\leftarrow \mathit{Decoders}\left(\mathit{S}{ }^{\mathrm{\prime }}\right)$

Algorithm 2

outlines the process for calculating the sentence coding layer. In Step 1, the output variable $\mathit{S}{ }^{\mathrm{\prime }}$ is declared. Then, in Step 2, the encoder layers are traversed. The content embedding and relative position embedding of the text are obtained in Step 3, and five projection matrices are generated in Step 4. The disentangled attention is then calculated in Step 5, which is normalized in Step 6. The attention is inputted to the FFN in Step 7, and the output vector of each encoder layer is accumulated in Steps 8–9. Finally, the output vector of the encoder layer is fed into the decoder to obtain the word vector $\mathit{S}{ }^{\mathrm{\prime }}$ in Step 10.

Semantics Graph Layer

We construct a semantic graph layer based on the graph attention network (Velikovi et al. 2017), which can effectively fuse the word vector of the text with the dependency representation. Specially, we build a heterogeneous text graph that contains word nodes and dependency relations. The word vector representation of the text is the node feature of the graph, and the dependency representation of the text is the edge feature of the graph. The semantic graph layer can learn the relation weights between word nodes using the graph attention mechanism, as shown in Figure 4.

Figure 4. Semantics graph layer.

When constructing an edge, we do not take into account the dependency type and dependency direction. If two words in a sentence have dependency relations, we construct an edge between the corresponding nodes. And we represent the F dimensional semantic feature of each node in the graph attention network as $\mathit{h}=\left\{{\vec{h}}_1,{\vec{h}}_2,\dots ,{\vec{h}}_{\mathit{N}}\right\},{\vec{h}}_{\mathit{i}}\in R^F$, all node semantic features of the output are expressed as$\mathit{h}{ }^{\mathrm{\prime }}=\left\{{\vec{h}}_1^{\prime },{\vec{h}}_2^{\prime },\dots ,{\vec{h}}_{\mathit{N}}^{\prime }\right\},{\vec{h}}_{\mathit{i}}^{\prime }\in R^{\mathit{F}{\mathit{ }}^{\mathrm{\prime }}}$. Let the vector features of word nodes $u_i,u_j$ be $h_i$ and $h_j$, respectively. For a word node $u_i$ we calculate the attention coefficient $e_{\mathit{ij}}$ of all neighboring word nodes $u_j$ and $u_i$, as shown in Eq. (11):

(11) $e_{\mathit{ij}}=\mathit{a}\left(\mathit{W}{\vec{h}}_{\mathit{i}},\mathit{W}{\vec{h}}_{\mathit{j}}\right),\mathit{j}\in N_i$(11)

where $\mathit{a}$ represents a shared attention weight, $\mathit{W}\in R^{\mathit{F}{\mathit{ }}^{\mathrm{\prime }}\times \mathit{F}}$ represents a learnable weighting matrix, which convert these features of word nodes into high-dimensional features.

In order to assign the attention weights between different word nodes, we normalize the attention coefficient $e_{\mathit{ij}}$ between word node $u_i$ and all neighboring word nodes $u_j$, and obtain an attention score ${\alpha }_{\mathit{ij}}$, as shown in Eq. (12):

(12) ${\alpha }_{\mathit{ij}}=\mathit{softma}{x}_j\left(e_{\mathit{ij}}\right)=\frac{\mathit{exp}\left({\mathit{e}}_{\mathit{ij}}\right)}{{\sum }_{k\in N_i}\mathit{exp}\left(e_{\mathit{ik}}\right)}$(12)

where we use a $\mathit{LeakyReLU}\left(\cdot \right)$ activation function to optimize Eq. (12), as shown in Eq. (13).

(13) ${\alpha }_{\mathit{ij}}=\frac{exp\left(L\mathrm{eakyReLU}\left({\vec{a}}^{\mathit{T}}\left[\mathit{W}{\vec{h}}_{\mathit{i}}\mathit{W}{\vec{h}}_{\mathit{j}}\right]\right)\right)}{{\sum }_{k\in N_i}\mathit{exp}\left(\mathit{LeakyReLU}\left({\vec{a}}^{\mathit{T}}\left[\mathit{W}{\vec{h}}_{\mathit{i}}\mathit{W}{\vec{h}}_{\mathit{k}}\right]\right)\right)}$(13)

And we perform a weighted summation of the attention features of all word nodes, and obtain the semantic feature representation ${\vec{h}}_{\mathit{i}}^{\prime }$ of word nodes, as shown in Eq. (14):

(14) ${\vec{h}}_{\mathit{i}}^{\prime }=\mathit{\sigma }\left(\sum _{j\in N_i}{\alpha }_{\mathit{ij}}\mathit{W}{\vec{h}}_{\mathit{j}}\right)$(14)

where $\mathit{\sigma }\left(\cdot \right)$ represents an activation function, and $h_i$ represents the feature representation of the $\mathit{i}$-th word node in the attention layer.

In order to make the attention more accurate, we use multi-head attention mechanism to assign $\mathit{K}$ groups of independent attention weight matrices to Eq. (14), as shown in Eq. (15):

(15) ${\vec{h}}_{\mathit{i}}^{\prime }=\mathit{\sigma }\left(\frac{1}{K}\sum _{\mathit{k}=1}^K\sum _{j\in N_i}{\alpha }_{ij}^k{W}^k{\vec{h}}_{\mathit{j}}\right)$(15)

Finally, to improve the semantic representation of text, we concatenate the obtained text sentence representation$S^{\prime }$ with the text semantic graph representation ${\vec{h}}_{\mathit{i}}^{\prime }$, and obtain the final text semantic representation, as shown in Eq. (16):

(16) $\mathit{S}{{\mathit{ }}^{\mathrm{\prime }}}_{\mathit{O}}=\mathit{Concat}\left({\vec{h}}_{\mathit{i}}^{\prime },\mathit{S}{ }^{\mathrm{\prime }}\right)$(16)

where $\mathit{Concat}\left(\cdot \right)$ represents the concatenated operation. Algorithm procedure of the semantics graph layer is shown in Algorithm 3.

Table

Algorithm 3: Semantics Graph Layer

Input: $\mathit{S}{ }^{\mathrm{\prime }}\in R^{d\times d_q}$, adjacency matrix $\mathit{A}\leftarrow {\left({\mathit{a}}_{\mathit{ij}}\right)}_{\mathit{n}\times \mathit{n}}$ Output: Semantics graph representation $\mathit{S}{{\mathit{ }}^{\mathrm{\prime }}}_{\mathit{O}}$ Progress: 1: $\mathit{Initialize}\mathit{S}{{\mathit{ }}^{\mathrm{\prime }}}_{\mathit{O}}\in R^{d\times d_q}$ 2: $\mathit{node}\mathit{\hspace{0.17em}}\leftarrow \mathit{S}{ }^{\mathrm{\prime }}$ 3: $\mathit{edge}\mathit{\hspace{0.17em}}\leftarrow \mathit{A}$ 4: ${\vec{h}}_{\mathit{i}}^{\prime }\leftarrow \mathrm{GAT}\left(\mathit{\hspace{0.17em}}\mathit{node},\mathit{\hspace{0.17em}}\mathit{edge}\right)\mathit{\hspace{0.17em}}$ 5: $\mathit{S}{{\mathit{ }}^{\mathrm{\prime }}}_{\mathit{O}}\leftarrow \mathit{Concat}\left({\vec{h}}_{\mathit{i}}^{\prime },S^{\prime }\right)$

Algorithm 3

Algorithm 3 describes the process for calculating the semantics graph layer. Firstly, the output variable $\mathit{S}{{\mathit{ }}^{\mathrm{\prime }}}_{\mathit{O}}$ (Step 1), and we regard the word vector $\mathit{S}$ is considered as the node feature of the graph (Step 2). Next, we take the adjacency matrix $\mathit{A}$ is then taken as the edge feature of the graph (Step 3), and we feed node and edge features are fed into GAT to obtain the graph attention vector (Step 4). Finally, the word vectors and graph attention vectors are concatenated to obtain the semantic graph vector $\mathit{S}{{\mathit{ }}^{\mathrm{\prime }}}_{\mathit{O}}$ (Step 5).

Similarity Computing Layer

The similarity computing layer is created to build a text classifier that calculates the semantic similarity between two sentences. A fully connected neural network is used to classify the text vector and obtain the semantic similarity. In Section 3.4, the final text semantic representation $\mathit{S}{{\mathit{ }}^{\mathrm{\prime }}}_{\mathit{O}}$ is obtained. A fully connected neural network is created for binary classification using $\mathit{S}{{\mathit{ }}^{\mathrm{\prime }}}_{\mathit{O}}$ as input. The network’s classification output (0–1) is obtained using the cross-entropy loss function. The network is then trained with a pre-trained language model DeBERTv3 (as described in Section 3.3) using a fine-tuned training pattern to obtain the final output, as illustrated in Figure 1. The formula for the similarity calculation is shown in Eq. (17):

(17) $\mathit{y}=\mathit{f}\left(\mathit{W}\right)\times \mathit{S}{{\mathit{ }}^{\mathrm{\prime }}}_{\mathit{O}}+\mathit{b}$(17)

where $\mathit{f}\left(\cdot \right)$ is the activation function, $\mathit{W}$ represents the weight matrix, $\mathit{S}{{\mathit{ }}^{\mathrm{\prime }}}_{\mathit{O}}$ represents the semantic vector, and $\mathit{b}$ is the learnable parameters. And we use the cross-entropy loss function $\mathit{L}$ to measure the prediction quality, as shown in Eq. (18):

(18) $\mathit{L}=\frac{1}{N}{\sum }_{i=1}^N{L}_i=-\left[y_i\cdot log\left(p_i\right)+\left(1-y_i\right)\cdot log\left(1-p_i\right)\right]$(18)

where $y_i$ represents the label of sample $\mathit{i}$, and $p_i$represents the probability that sample $\mathit{i}$ is predicted to be positive value.

Experiments and Results

In this section, we first describe the details of the experiments and datasets in Sections 4.1 and 4.2, then introduce the baselines in Section 4.3 and introduce the metrics in Section 4.4, and show the results of the main experiment in Section 4.5. In Section 4.6 we show results on the low-resource setting. Finally, we perform an ablation testing to prove the effectiveness of each component in Section 4.7.

Experimental Details

The proposed STSG was implemented using the PyTorch 1.7 framework on a Windows server with an Intel(R) Core-(TM) i7-10700F CPU @ 2.90 GHz, 32 G memory, and an NVIDIA GeForce RTX 3060 GPU. DeBERTV3 (Pengcheng, Jianfeng, and Weizhu 2023) suggests using these hyper-parameters to fine-tune DeBERTaV3 in downstream tasks. To ensure a fair comparison with our baseline model DeBERTV3, we have chosen the hyper-parameters of STSG to be consistent with it. The original DeBERTV3 article has demonstrated the validity of these parameters, which are described in Table 2.

Table 2. Parameters of STSG.

Parameters	Value
Learning rate	3e-5
Batch size	32
Vocab size	30522
Hidden size	768
Hidden layers	12
Attention heads	12
Text lengths	510
Optimizer	AdamW (Ilya and Frank 2019)
Loss function	Cross-Entropy (See Equation 27(27) $\mathit{MSE}=\frac{1}{n}\sum _{\mathit{i}-1}^n{\left(x_i-y_i\right)}^2$(27) )

Datasets

To assess the performance of STSG in short text similarity, we utilized the MRPC dataset (Alex et al. 2019), which is the most commonly used authoritative evaluation benchmark for such tasks (Devlin et al. 2018; Pengcheng, Jianfeng, and Weizhu 2023; Raffel et al. 2020; Yung-Sung et al. 2022). Additionally, we employed the challenging low-resource BIOSSES dataset (Gizem, Hakime, and Arzucan 2017) to validate the feasibility of STSG in low-resource text semantic similarity tasks, as shown in Table 3:

1. MRPC (The Microsoft Research Paraphrase Corpus) is a dataset consisting of 5800 sentence pairs extracted from online news. The category distribution is unbalanced, with 68% of the pairs being positive samples, which poses a significant challenge to the model’s computation.

2. BIOSSES is a dataset comprising 100 sentence pairs that were selected from medical abstracts in the biomedical field. The dataset uses score intervals of 0-5 to judge the similarity between the pairs. Due to its low-resource nature and extremely sparse samples, the dataset presents a challenge for models to learn from small samples.

Table 3. Summary of datasets.

Dataset	Train	Dev	Test	Domain
MRPC	3668	408	/	News
BIOSSES	64	16	20	Biomedical

Baselines

We used the following 10 models as the experimental baseline.

1. BERT (Devlin et al. 2018). This model uses an existing unlabeled corpus to pre-train the Transformer model to obtain word embedding vectors, and then fine-tunes the model to complete the sentence similarity computing.

2. T5 (Raffel et al. 2020). This model uses the unified pre-trained language models to learn multiple different NLP tasks.

3. ELECTRA (Kevin et al. 2020). This model uses a discriminative pre-trained text encoder to calculate the similarity of two sentences.

4. ERNIE2.0 (Yu et al. 2020). This model is a continuous learning framework for deeply integrated knowledge to continuously learn vocabulary, syntactic, and semantic knowledge by introducing more tasks.

5. SpanBERT (Joshi et al. 2020). This model is a pre-trained model based on word segmentation, which adds a mask to the random adjacent word to calculate text similarity.

6. DistilBERT (Sanh et al. 2019). This model is a pre-trained universal language representation model to calculate text similarity.

7. ESM-2 (Zemin et al. 2022). ESM-2 calculates the sequence text data using large-scale pre-trained models.

8. XLNet (Yang et al. 2019). XLNet is a generalized autoregressive pre-trained model to calculate text similarity.

9. DeBERTa (He et al. 2021). DeBERTa uses the disentangled attention to improve BERT and further improve the performance of the model.

10. DeBERTav3 (Pengcheng, Jianfeng, and Weizhu 2023). This model is an efficient and simple pre-trained model to calculate text similarity.

Metrics

(1) Precision. Precision describes how many of the number of similar sentence pairs (classification result of 1) predicted by the model are true:

(19) $\mathit{Precision}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FP}}$(19)

(2) Recall. Recall describes the percentage of all similar sentence pairs (classification result 1) correctly predicted by the STSG model:

(20) $\mathit{Recall}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}}$(20)

(3) Accuracy. Accuracy describes the percentage of all true outcome values of the STSG model:

(21) $\mathit{Accuracy}=\frac{\mathit{TP}+\mathit{TN}}{\mathit{TP}+\mathit{FP}+\mathit{TN}+\mathit{FN}}$(21)

(4) F1-score. The F1-score is the mean value of the Precision and Recall. The value range of F1-score is [0, 1]:

(22) $\mathit{F}1\mathit{Score}=\frac{2\times \mathit{Precision}\times \mathit{Recall}}{\mathit{Precision}+\mathit{Recall}}$(22)

where $\mathit{TP}$ is the number of similar sentence pairs correctly predicted by the model, $\mathit{TN}$ is the number of non-similar sentence pairs predicted by the model, $\mathit{FP}$ is the number of similar sentence pairs incorrectly predicted by the model, and $\mathit{FN}$ is the number of non-similar sentence pairs not predicted by the model.

(5) Pearson. Pearson metric reflects the difference between the predicted result of the model and the correct result, where 0 represents no correlation and 1 represents a complete correlation. The corresponding formula is defined as Eq. (23):

(23) $\mathit{Pearson}=\frac{n\left({\sum }_{i=1}^n{x}_i{y}_i\right)-{\sum }_{i=1}^n{x}_i{\sum }_{i=1}^n{y}_i}{\sqrt{\mathit{n}\left({\sum }_{i=1}^n{x}_i^2\right)-{\left({\sum }_{i=1}^n{x}_i\right)}^2}\sqrt{\mathit{n}\left({\sum }_{i=1}^n{y}_i^2\right)-{\left({\sum }_{i=1}^n{y}_i\right)}^2}}$(23)

(6) Spearman. The Spearman metric measures the rank correlation coefficient between the predicted outcome of the model and the correct outcome. The value ranges from −1 to 1, where −1 indicates no correlation and 1 indicates perfect correlation. The corresponding formula is defined as Eq. (24):

(24) $\mathit{Spearman}=1-\frac{6{\sum }_{i=1}^n{d}_i^2}{\mathit{n}\left(n^2-1\right)}$(24)

(7) Kendall. Kendall’s correlation is a rank correlation coefficient used to assess the correlation between two random variables based on the rank of the data objects. The corresponding formula is defined as Eq. (25):

(25) $\mathit{Kendall}=\frac{\mathit{c}-\mathit{d}}{\sqrt{\left(c+d+t_x\right)\left(\mathit{c}+\mathit{d}+t_y\right)}}$(25)

where $\mathrm{c}$ and $\mathrm{d}$ represent the number of concordant and divergent pairs, and ${\mathrm{t}}_{\mathrm{x}}$ and ${\mathrm{t}}_{\mathrm{y}}$ denote the number of tied rankings in ${\mathrm{x}}_{\mathrm{i}}$ and ${\mathrm{y}}_{\mathrm{i}}$, respectively.

(8) MAE. Mean Absolute Error (MAE) can be calculated as the mean absolute error between the true value and the predicted value, as shown in Eq. (26):

(26) $\mathit{MAE}=\frac{1}{n}\sum _{\mathit{i}-1}^n\left|x_i-y_i\right|$(26)

(9) MSE. Mean Squared Error (MSE) can calculate the mean-squared error between the true and predicted values, as shown in Eq. (27):

(27) $\mathit{MSE}=\frac{1}{n}\sum _{\mathit{i}-1}^n{\left(x_i-y_i\right)}^2$(27)

where $x_i$ represents the true result, $y_i$ represents the predicted result, $d_i$ is the level difference between $x_i$ and $y_i$, and $\mathit{n}$ is the number of samples.

Results of the Main Experiment

(1) Results on MRPC dataset

Table 4 shows the accuracy of STSG compared to advanced methods based on the MRPC dev dataset. STSG achieves the highest performance among all the methods, with an F1-score of .946. The proposed STSG has a recall of .913, outperforming other methods. The pre-trained language model based achieves similarity computation with F1-scores ranging from .900 to .926. Examples of such models include BERT, T5, ELECTRA, DeBERTa, DeBERTav3 and SpanBERT. Comparative learning methods, such as Sentence-BERT, DiffCSE, and SimCSE, have achieved F1-scores ranging from .760 to .777. The MRPC dataset has an unbalanced category distribution, with 68% of sentence pairs being positive samples. As a result, contrastive learning-based methods typically yield low F1-scores. However, it has been shown that using pre-trained language models such as DeBERTav3 and DeBERTa alone can improve results. It is worth noting that pre-trained language models may be difficult for learning semantic of sentences, resulting in a large discrepancy between F1-scores and Recall scores. In contrast, STSG based on a combination of syntactic and pre-trained models is better at capturing the semantic of sentences. As a result, the similarity calculation is better, and the difference between the F1-score and the Recall and Accuracy metrics is smaller.

Table 4. Evaluation on MRPC dataset.

Model	Params(M)	Vocabulary (K)	Precision	Recall	Accuracy	F1-score
BERT	110	30	0.848	0.820	0.860	0.900
T5	220	32	0.863	0.858	0.879	0.913
ELECTRA	335	30	0.888	0.825	0.877	0.915
ERNIE2.0	750	30	0.887	0.853	0.890	0.922
SpanBERT	335	28	0.867	0.838	0.875	0.911
DistilBERT	66	30	0.847	0.791	0.848	0.895
ESM-2	150	33	0.642	0.583	0.701	0.805
XLNet	110	32	0.873	0.826	0.873	0.911
Sentence-BERT (Nils and Iryna 2019)	110	30	-	-	-	0.760
DiffCSE (Yung-Sung et al. 2022)	355	50	-	-	-	0.769
SimCSE (Tianyu, Xingcheng, and Danqi 2021)	355	50	-	-	-	0.777
DeBERTa	100	50	.866	.832	.873	0.910
DeBERTav3	86	128	0.886	0.873	0.897	0.926
STSG	100	128	0.916	0.913	0.926	0.946

(2) Evaluation on BIOSSES dataset

Table 5 shows the accuracy of STSG in comparison to other methods, using the BIOSSES test dataset. STSG achieves the best values for MAE and MSE errors, with a Pearson value of .915 and a Spearman’s value of .835. This primary cause of these results is the limited number of training samples in the BIOSSES dataset, which comprises only 64 pairs of sentences and more noisy data. Methods that rely on pre-trained language models can be challenging to apply to low-resource tasks. For example, models like DeBERTav3 and BioBERT may not yield high Pearson values. The STSG enhances the global semantic information of sentences by utilizing syntactic dependencies. Compared to DeBERTav3, Pearson values have improved by 8.67%. These results demonstrate that combining syntactic dependency and pre-trained language models can be effective in textual semantic similarity tasks with small datasets.

Table 5. Evaluation on BIOSSES dataset.

Model	Params (M)	Vocabulary (K)	MAE	MSE	P-value	Pearson	Spearman	Kendall
ELMo (Yifan, Shankai, and Zhiyong 2019)	-	-	-	-	-	0.602	-	-
BioBERT (Yifan, Shankai, and Zhiyong 2019)	-	-	-	-	-	0.827	-	-
BlueBERT (Yifan, Shankai, and Zhiyong 2019)	-	-	-	-	-	0.916	-	-
BERT	110	30	0.916	1.215	0.000	0.800	0.750	0.530
T5	220	32	0.872	1.170	7.994	0.877	0.786	0.616
ELECTRA	335	30	1.003	1.636	0.000	0.808	0.820	0.667
ERNIE2.0	750	30	0.896	1.170	1.129	0.871	0.754	0.564
SpanBERT	335	28	0.990	1.432	0.000	0.810	0.702	0.530
DistilBERT	66	30	0.992	1.552	0.009	0.630	0.670	0.496
ESM-2	150	33	1.048	1.580	0.037	0.524	0.461	0.325
XLNet	110	32	0.930	1.306	1.165	0.870	0.822	0.633
DeBERTa	100	50	0.788	0.987	0.000	0.779	0.819	0.667
DeBERTav3	86	128	0.992	1.555	4.248	0.842	0.714	0.513
STSG	100	128	0.518	0.376	6.715	0.915	0.835	0.650

The MAE and MSE metrics are used to assess the deviation between the predicted values of the model and the true values of the samples. The smaller the MAE and MSE values, the better the predictive performance of the model. Table 5 shows that some methods have high Pearson values but large MAE and MSE values due to poor processing of sentence semantics, such as ERNIE2.0 and XLNet. The proposed STSG has a MAE of .518 and a MSE of .376, which is 40.60% lower than T5’s MAE.

Results on the Low-Resource Setting

Table 6 shows the experiments conducted on various training ratios of the MRPC dev dataset. We divided the training set of MRPC into new datasets with different ratios to train the model (called training ratios), and used the validation set of 100% size to verify the performance of the model to test the generalization ability of STSG on the classification problem.

Table 6. Low-resource settings of MRPC datasets.

Training Ratio	Train	Dev	Test
1%	36	408	/
5%	183	408	/
10%	366	408	/
20%	733	408	/
50%	1834	408	/
100%	3668	408	/

And we use different training ratios on the MRPC dataset to show the Receiver Operating Characteristic Curve (ROC) curves. Figure 5 describes the ROC curves of the proposed STSG and several baselines on the MRPC dataset. Specifically, Figure 5(a) shows that the Area Under Curve (AUC) of STSG on a training ratio of 100% is .913. Similarly, Figure 5(b) shows that the AUC value of STSG on a training ratio of 50% is .869. Moreover, Figure 5(c) shows that the AUC value of STSG on a training ratio of 20% is .772. Finally, Figure 5(d) shows that the AUC of STSG on a training ratio of 10% is .693. The pre-trained language model approach performs poorly on the low-resource dataset in terms of AUC values due to the imbalance of positive and negative samples in MRPC. It struggles to accurately identify negative examples in sentences. The results demonstrate that STSG achieves a high True Positive Ratio for positive samples at different training ratios, while maintaining a low the False Positive Ratio for negative samples.

Figure 5. ROC curve on MRPC dataset.

Table 7 show that the F1-score of the proposed STSG on a training ratio of 1% is .812 and the accuracy is .684. This represents an increase in F1-score of 14.04% and accuracy of 12.50% compared to BERT. On a training ratio of 50%, the F1-score of STSG is .909 and the accuracy is .873.

Table 7. Evaluation on MRPC dataset by different training ratios.

Training Ratio	Model	Precision	Recall	F1-score	Accuracy
1%	BERT	0.548	0.549	0.712	0.608
	DistilBERT	0.516	0.511	0.750	0.625
	SpanBERT	0.553	0.535	0.768	0.650
	T5	0.438	0.484	0.782	0.647
	STSG	0.341	0.500	0.812	0.684
5%	BERT	0.631	0.549	0.808	0.694
	DistilBERT	0.845	0.516	0.816	0.694
	SpanBERT	0.800	0.533	0.821	0.703
	T5	0.661	0.528	0.814	0.694
	STSG	0.760	0.559	0.826	0.716
10%	BERT	0.694	0.552	0.819	0.706
	DistilBERT	0.774	0.539	0.822	0.706
	SpanBERT	0.774	0.539	0.822	0.706
	T5	0.734	0.533	0.819	0.700
	STSG	0.781	0.676	0.853	0.777
20%	BERT	0.631	0.549	0.808	0.694
	DistilBERT	0.662	0.593	0.812	0.711
	SpanBERT	0.672	0.627	0.812	0.721
	T5	0.701	0.561	0.821	0.711
	STSG	0.807	0.757	0.875	0.819
50%	BERT	0.783	0.762	0.864	0.809
	DistilBERT	0.820	0.713	0.870	0.804
	SpanBERT	0.812	0.771	0.879	0.826
	T5	0.804	0.808	0.876	0.831
	STSG	0.863	0.836	0.909	0.873

The experimental results in Table 7 show that the proposed of STSG outperforms other baselines with different ratios on the MRPC dataset. The existing methods ignore the semantic correlation between words, resulting in unstable results during evaluation under different training ratios. In contrast, the proposed STSG learns dependencies between words and incorporates the feature learning of domain nodes, leading to stable prediction performance.

Figure 6 shows that the Pearson of STSG is .895 for a training ratio of 50%, .509 for a training ratio of 20%, and .578 for a training ratio of 10%. These results show that STSG can achieve accurate predictions on small datasets with different proportions of BIOSSES, and its generalization ability exceeds that of similar baselines. STSG takes into account the feature learning of domain nodes, which enables the model to better recognize the context in short texts.

Figure 6. Pearson on BIOSSES dataset.

Table 8 shows the experiments conducted on different training ratios of the BIOSSES test dataset. We split the training set of BIOSSES into new datasets with different ratios to train the model, and use the validation set of 100% to verify the performance of the model to test the generalization ability of the proposed STSG.

Table 8. Low-resource settings of BIOSSES datasets.

Training Ratio	Train	Dev	Test
10%	6	16	20
20%	12	16	20
50%	32	16	20
100%	64	16	20

Figure 7 presents the experimental results for different training scales of the BIOSSES test dataset. The BIOSSES training set was split into new datasets with varying scaling sizes to train the model. The model’s MAE and MSE were validated using 100% and 50% sized validation sets. With a 50% training scaling, STSG achieved an MAE of .605 and an MSE of .555. This represents an 11.03% decrease in MAE and a 36.21% decrease in MSE compared to T5. Compared to other methods, STSG exhibits smaller MAE and MSE values at different training ratios. Methods based on pre-trained language models tend to yield large error values under low-resource experiments because these models are not fine-tuned for downstream tasks and are difficult to adapt to small sample datasets. STSG utilizes syntactic information and textual semantic graphs to improve the model’s generalization ability, resulting in strong predictive performance even on small datasets.

Figure 7. Errors on BIOSSES dataset.

Ablation Testing

Tables 9 and 10 show the ablation testing results of STSG on MRPC and BIOSSES, respectively. The STSG model that does not use a dependency parsing layer is represented as -w/o DP. Table 9 shows that the F1-score of -w/o DP on the MRPC dataset is .926, which is a decrease of 2.11% compared to STSG. Similarly, Table 10 shows that the Pearson of -w/o DP is .842, which is a decrease of 7.98% compared to STSG. These results show that the proposed STSG takes into account the syntactic structure features between sentence pairs, which makes the model understand the true intention of the sentence expression, thus improving the accuracy of the similarity computation task.

Table 9. Ablation testing on MRPC dataset.

Model	Precision	Recall	Accuracy	F1-score
STSG	0.916	0.913	0.926	0.946
-w/o DP	0.886	0.873	0.897	0.926

Table 10. Ablation testing on BIOSSES dataset.

Model	MAE	MSE	P-value	Pearson
STSG	0.518	0.376	6.715	0.915
-w/o DP	0.992	1.555	4.248	0.842

Conclusion and Future Works

Short text semantic similarity computation is a fundamental problem in natural language processing, which aims to predict the similarity between two sentences. To tackle the shortcomings of current short text semantic similarity methods, we introduce STSG, which is based on dependency parsing and pre-trained language models. The model utilizes syntactic information and incorporates it into pre-trained language models, thereby augmenting the overall semantic information of sentences to address the issue of semantic sparsity. We propose a textual semantic graph layer that utilizes word vectors and dependency parsing relations as features of Graph Attention Networks (Velikovi et al. 2017) to enhance the word relevance modeling problem. However, the STSG model has limitations, such as its inability to process two sentences with more than 512 tokens. Additionally, the tool can only calculate semantic similarity for English text. Moreover, due to limited computational resources, we are unable to experiment with the latest large models such as GPT4 (Achiam et al. 2023). Our future plans involve addressing the current limitations of STSG and proposing a new method for measuring semantic similarity of text. This will be accomplished by combining textual semantic dependency analysis with large-scale language models.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Open Project of Sichuan Provincial Key Laboratory of Philosophy and Social Science for Language Intelligence in Special Education under Grant No. YYZN-2023-4 and the Ph.D. Fund of Chengdu Technological University under Grant No. 2020RC002.