Hierarchical multi-instance multi-label learning for Chinese patent text classification

Abstract

To further enhance the accuracy of the Chinese patent classification, this paper proposes a model, based on the patent structure and takes the patent claim as subjects, with multi-instance multi-label learning as the main method. Firstly, the patent claims are divided into multiple independent texts using the sequence number as the splitting token. For each patent, multiple claims are regarded as multiple instances, and the corresponding IPCs serve as its multiple labels. Next, the concept of secondary_label is introduced following the composition rules of IPC, and the relationships between instances and multiple secondary_labels are mined through the construction of fully-connected layers. To capture more comprehensive semantic information of instances, BIGRU and self-attention are employed to enhance semantics and reduce information loss during the training process. Finally, the max-pooling operations are utilised to obtain the predicted categories of patents based on capturing the relationships between instances and different hierarchical labels. Experimental results on the '2017 Chinese patent dataset' demonstrate that the multi-instance multi-label approach can effectively mine deeper relationships between patents and labels in classification tasks. As a result, our model significantly improves the accuracy of patent text classification.

KEYWORDS:

• Patent text classification
• IPC
• secondary_label
• multi-instance multi-label learning
• patent claim

1. Introduction

As a basic technology in the patent management process, patent text classification plays an important role in the organisation, analysis, retrieval, and other tasks of patent documents. With the development of technology, there has been a continuous increase in the number of patent applications, posing a significant challenge to the patent management process.

Current research on patent classification aims to reduce reliance on manual methods and enhance the efficiency of patent documents management. Some progress has been made using abstracts and titles as the basis. Previous researchers proposed an approach that combined transformers with hierarchical algorithms (Huang et al., 2019) and a model that constructed the semantic network (Sarica et al., 2020) to solve the patent text classification task. However, the specific writing structure of patents and the abundance of technical information contained in patent claim provide the potential for improving classification accuracy. For example, in the patent “关系抽取模型的建立方法以及关系抽取方法 (method for establishing relation extraction model and method for relation extraction)”, a key sentence included in the second claim, “基于知识图谱的实体概念结构构建 …  (the construction of entity concept structure based on knowledge graph …)”, has been ignored in the classification process, which may cause incomplete prediction results. Besides that, the structural characteristic of IPC is often overlooked in related studies, which may lead to low classification accuracy.

Based on the considerations above, a model that can not only consider the technical information in patent claim but also leverage the specific structure of IPC needs to be proposed. For the sake of achieving the aforementioned motivation, the following two issues should be taken into account. (1) How to make the most of technical information hidden in patent claim and improve the representation ability of input claims for better training effectiveness. (2) In view of the structural characteristic of IPC, how to construct deeper connections between patent claim and IPCs, and further improve the classification accuracy.

Given the two considerations above, a hierarchical multi-instance multi-label learning for Chinese patent text classification, abbreviated as HMM-CPT, is proposed. Different from the previous work, the patent claim and IPC structure are given lots of attention as the focus of our study. Our research group has carried out various research on text analysis tasks and provided strong support for the work of this paper (Zhang et al., 2023; Zhang et al., 2022).

The overall framework of the HMM -CPT method is shown in Figure 1. Our contributions are summarised as follows.

• Construct a relationship mining model with the aim of establishing associations between claims and IPCs, thereby obtaining a more comprehensive connection between instances and labels.

• Proposed a hierarchical multi-instance multi-label learning framework, utilising the hierarchical structure of IPC. By leveraging the information association at the current level, more accurate predictions can be obtained for the prior level categories.

Figure 1. The framework of the HMM-CPT method.

In this article, the framework of hierarchical multi-instance multi-label learning for Chinese patent classification is composed of three parts: data preprocessing, instance-label connection construction, and bag-label relationship obtaining. The novelties and advantages of our model are demonstrated as follows:

1. The training data is prepared in the following ways: Each patent is treated as an independent 'bag', and patent claims in each 'bag'are divided into multiple texts, which enables the information in patent claims to be comprehensively mined.

2. Multiple fully-connected layers are constructed to establish the connections between each claim and secondary_labels. This kind of mapping connection makes relationship mining more comprehensive.

3. We add a max-pooling layer composed of two max-pooling operations in different directions, which results in a more accurate prediction at the prior level.

Overall, the method we proposed not only enriches the research content of patent text classification but also establishes a deeper connection between patent texts and labels, which helps enhance the effectiveness of patent text classification.

The remainder of this paper starts with a brief review of related literature on patent text classification tasks. Then we present the process of experiment data preparation in section 3, where the model construction process, training process, and other details of the HMM-CPT are demonstrated completely. In the fourth section, several groups of experiments are conducted to verify the effectiveness of our proposed method. A general review of this paper and the prospect of sustainable research work in the future are summarised in the last section.

2. Related work

The existing patent classification methods can be roughly classified into two categories, which are methods based on machine learning and deep learning, respectively. And current researchers put their sights on improvement and optimisation based on these two kinds of methods.

2.1. The methods based on machine learning

The methods based on machine learning refer to utilising prior knowledge to obtain a regular model through abundant studies and then using the obtained model to predict the data with unknown results (Choi et al., 2019). Many classical models and algorithms have been applied to text classification tasks, such as k-NN (Wang et al., 2019), naïve Bayes(NB) (Chen et al., 2019), support vector machine (SVM) (Goudjil et al., 2018), bag-of-words model(cBOW) (Soumya George & Joseph, 2014), etc. Based on the naïve Bayes method, Xiao et al. (2018) established a text classification model, which realised the effective classification of patent texts in the security field. For the structural characteristics of patent text, Bao et al. (2018) put forward a framework based on multi-instance learning, in which the patent text is taken as a bag, and the abstract and title are taken as instances. In another work, the semantic information of labels is used to define the specific representation between text and labels, and the self-attention mechanism is introduced to enhance the ability of text presentation and further verify the advance of the proposed multi-label classifier. (Xiao et al., 2019).

With the unsolved problem that some samples, which belong to a single category, were ignored in the text feature acquisition process, Hu et al. (2020) improved the information gain function and introduced the weight coefficient to adjust the information gain value of important features, thus promoting the accuracy of patent classification. The unsolved problem still challenges the researchers in that the automatic classification method could not identify unknown words. For this issue, Xiao et al. (2021) fused sentence vectors with proper noun vectors and constructed a patent classification model.

2.2. The methods based on deep learning

Deep learning is an overall framework that combines mathematical knowledge with computer algorithms and acquires the preset results through large-scale data training and calculation. Deep learning is characterised by strong learning ability, adaptability, portability, and good performance in large-scale data processing, etc. thus, it is commonly introduced into massive text-processing tasks. Typical deep learning algorithms include Convolutional Neural Networks(CNN) (Jiang et al., 2022; Wang et al., 2019), Recurrent Neural Networks(RNN) (Mansueli et al., 2022), Generative Adversarial Networks(GANs) (Croce et al., 2020), Deep Reinforcement Learning(DRL) (Keneshloo et al., 2019), LSTM (Huan et al., 2022; Su, 2018; Zhang et al., 2023), etc. Lee et al. (2022) proposed a multi-model deep learning framework by using quantitative information and texts of patents to improve the patent classification effectiveness. The employment of LSTM in patent classification tasks is also very common. It can be used to construct a corpus training model to predict the patent categories (Ma et al., 2018). And some other scholars verified that CNN performs well on patent automatic classification in multiple fields (Wang et al., 2022; Hu et al., 2018). Due to the demands for massive patent classification, Lyu et al. (2020) designed seven kinds of automatic methods based on deep learning combining the word sequence features, context features, and key classification features of patent texts, and further validated the effectiveness of the deep learning method in CPT classification tasks.

In addition to the scholars mentioned above, other researchers also worked hard to improve the effectiveness of patent classification. Yu et al. (2020) put forward a model that fused two-channel features that were generated by processed text mapping, to improve the accuracy of the automatic classification for CPT. In the case of patent text on a large scale, difficulties arise due to similar content in CPT. For that problem, Wu et al. (2020) introduced an attention mechanism into the bidirectional long-term and short-term memory network and further distinguished similar texts by weighing them differently. Several BERT optimal models were used to classify the multi-label patents automatically, on the basis that the first four digits of IPC numbers were selected as classification labels (Xinyu et al., 2022; Zhang et al., 2022). In order to achieve better classification results, Wen et al. (2021) proposed a multi-level model that combined ALBERT with bidirectional gating units, which can enhance the semantic representation of long-distance keywords to some extent. Similar work was carried out by Zhang who attached great importance to the semantic meaning of the technical text in the patent (Zhang et al., 2021).

2.3. Research on multi-instance multi-label learning for text classification

The concept of MIML(Multi-Instance Multi-Label learning) is proposed to address the multi-classification problem in texts or images (Li et al., 2017). Its basic idea is described as follows. The data in the dataset are taken as numbers of bags, and each bag contains several instances in it; a bag can possess several labels, and the numbers of instances and labels are not necessarily equal (Xiang et al., 2018). MIML can construct a learning model according to the labelled bags, which have been classified correctly, to predict the unclassified bags. Through the analysis of the above concepts and the structure of patent texts, Feng and Zhou (2017) put forward the deep MIML network, which can conduct the generation and follow-up learning of instances. Further, by means of establishing the relation between instances and the corresponding labels, the deep MIML network has shown good performance on text classification tasks. In another idea, the multi-label problem in patent texts is transformed into the label sorting problem, thus the Fast MIML model is constructed to mine the labels in the unpredicted texts (Huang et al., 2018). In order to conduct a more systematic analysis of the application of the MIML method in patent text classification, Bao applied seven methods based on MIML structure to the classification task for CPT, respectively. Then, the effectiveness of MIML was validated through many experiments (Bao et al., 2021).

The above research conducted by many scholars indicates the adaptiveness of the MIML method in dealing with the CPT classification task. However, most of those methods took abstracts and titles as research objects, and lacked mining and analysis for quantitative technical information in patent claim. Besides, most of these studies have not considered the hierarchical structure characteristics of IPC, which leads to inaccurate predictions of patent labels, thus causing low classification accuracy. In such a case, a hierarchical multi-instance multi-label learning for Chinese patent texts, abbreviated to HMM-CPT, is put forward in this paper, in which the Bi-GRU (Lu et al., 2020)network and self-attention mechanism (Sivakumar & Rajalakshmi, 2021) were employed to improve the effectiveness of the classification. The data composition and methodology are elaborated on in the following sections.

3. Preparing works

According to the aforementioned related works, a message reveals that it is feasible to apply MIML methods to solve the patent text classification problem, and it does make good effects. Further, the composition of the patent text and the formal structure of claims give the feasibility for using the MIML network to mine the deeper contents.

3.1. Dataset

In this part, the format and processing of the patent claim, as well as the IPC structure, will be described in detail.

C1. The patent claim

The patent claim covers all innovative features and contents described in the specification, including independent and subordinate claims, numbered with Arabic figures. An example is given in Table 1. Claim 2 is a subordinate claim of Claim 1, and these two claims are divided by sequence numbers. All IPCs belonging to this patent are listed in the third column. This kind of structure provides the potential for the application of the multi-instance multi-label approach.

Table 1. The claims and IPCs of a patent.

Patent title	Claims	IPCs
一种便于外设进出的集装箱自动开关装置 (An automatic container opening and closing device convenient for peripheral equipment to enter and exit)	一种便于折叠的可移动衣柜, 包括：可折叠柜体、 …  … 采用倒挂式连接 (A movable wardrobe that is convenient for folding; is composed of a foldable cabinet body, … which is connected upside down.) 根据权利要求1所述可折叠柜体和可折叠内壳, 特征在于采用阻燃、抗静电的聚丙烯作为材料. (According to the first claim, the foldable cabinet and foldable inner shell are characterised in that flame retardant and antistatic polypropylene is used as the material)	G06K17/00：在包括G06K1/00至G06K15/00两个或多个大组中的设备之间实现协同作业的方法或装置 (A method or device for achieving cooperative operation between two or more devices in a large group including G06K1/00 to G06K15/00) G07C9/00：独个输入口或输出口登记器 (Single input or output register)

C2 The preprocessing of the CPT

The patent claim text is usually larger in length compared with the abstract, and both of them have a similar writing format, such as '根据权利要求1所述的 …  … (according to the first claim that …)，其特征在于 …  … (the characteristic is …)， 一种 …  … 装置 (a device that …) ， 特征在于 …  … (and the characteristic is …)'. These words or phrases, which are used to organise the text structure, do not provide much useful information in the training process. Therefore the dataset needs to be preprocessed, and the steps are as follows:

1. Stop words removing. Summarise the writing formats of the claims to construct specific stop word lists for the dataset, and remove meaningless words and phrases that have structural functions but no useful information.

2. Text division. Take the single patent as a unit, and divide the claims in the dataset with the sequence number as the splitting token.

3. Word segmentation. Jieba tool is utilised to process the patent claims in the CPT

C3. The IPC

The IPC structure is a hierarchical structure, composed of five levels: section, class, subclass, main group, and subgroup. According to statistical data from the website 'incoPAT', the IPC system possesses 8 sections, 131 classes, and over 630 subclasses. An example is given in Table 2. In the practical case study, the first three layers are our main focus.

Table 2. An example of IPC hierarchical structure.

IPC hierarchical structure	IPC	Description
Section	F	Mechanical engineering lighting;Heating; Weapons; Blasting
Class	F02	Combustion engines; Hot-gas or combustion-product engine plants
Subclass	F02D	Controlling combustion engines
Main group	F02D 41	Electrical control of the supply of combustible mixture or its constituents
Subgroup	F02D 41/02	Circuit arrangements for generating control signals

Definition 1. (secondary_label)

In our proposed method, the concept of the secondary_label is defined as follows: (1) $\begin{array}{rl}& forIPCQ\\ & Q_{\mathit{\text{section-level-category}}}:label\\ & Q_{\mathit{\text{class-level-category}}}:\mathit{\text{secondary\_label of Q}}\end{array}$(1) Formula(1) is explained by taking the data in Table 2 as an example. If the section level is the final classification result we want, the class level will be treated as the research object. That is, F is used as the label, F02 is used as a secondary_label of F.

3.2. The multi-instance multi-label structure of patent texts

The idea of the multi-instance multi-label deep learning network can be described as follows: regarding the dataset as a set of bags, each bag is composed of multiple instances and possesses multiple labels. In this paper, as shown in Figure 2, one patent is regarded as a bag, each claim of this patent is regarded as an instance, and all IPCs belonging to this patent are used as multi-labels. The section-level and class-level of the IPC numbers are taken as labels in this paper shown in Table 3.

Figure 2. The MIML structure for patent text.

Table 3. Example of text division and labels of Chinese patent.

Title	The text division of patent claim	Labels
营养泵的换液阀及营养泵 (Liquid changing valve of nutrition pump and nutrition pump)	1.一种营养泵的换液阀, 包括阀体, 所述阀体上 …  … 所述挤压块由缺口进入到沟槽内与沟槽另一侧槽壁挤压液管. (A liquid changing valve of a nutrition pump, composed of a valve body, and on the valve body … , an extrusion block enters a groove through a gap and extrudes a liquid pipe with the groove wall on the other side of the groove.)	Section	Class
		F：机械工程 (machinery engineering) A：人类生活必需(requirements for human-lives)	F04:液体变容式机械; 液体泵或弹性流体泵 (Liquid variable volume machinery; liquid or elastic fluid pump) A61: 医学或兽医学; 卫生学 (Medicine or veterinary medicine; hygiene)
	2. 营养泵的换液阀, 在挤压块行程的起始端分别设置有光电传感器. (The liquid changing valve of the nutrition pump; photoelectric sensors are provided at the starting end of the extrusion block stroke respectively.)
	3. 一种营养泵, 包括泵体, 所述泵体上安装有权利要求1或2所述的换液阀. (A nutrition pump comprises a pump body, and the pump body is provided with a liquid changing valve according to claim 1 or 2.)

4. Methodology

In order to mine the technical information in patent claims and thereby improve the classification accuracy of CPT, MIML learning was introduced into the patent text classification task to predict the categories for unclassified patents by training a model based on the information of classified patents.

According to the idea of MIML learning, the mathematical expression in this paper is as follows: the CPT dataset $D=\{(X_1,L_1),(X_2,L_2),\dots \dots ,(X_p,L_p)\}$, $X$and $L$represents the instance set and label set of each bag, respectively.$W_i=\{x_{\text{1}},x_{\text{2}}\dots \dots x_n\},L_i=\{l_1,l_2,\dots \dots l_m\},n,m>=1$, $W_i$and $L_i$represent the instance set and label set of X_i. $sl=\{s{l}_1,s{l}_2,\dots ,s{l}_t\}$ represents the secondary_label set of X_i.

4.1. Hierarchical multi-instance multi-label learning method for CPT classification

Based on the idea of the MIML learning, the HMM-CPT(Hierarchical Multi-instance Multi-label learning for Chinese Patent Text classification) is proposed. The overall process structure is shown in Figure 3, which consists of the input layer, modelling layer, and pooling layer. The function and mechanism of each layer are elaborated on in detail successively.

Figure 3. Hierarchical multi-instance multi-label learning method for CPT classification.

The process of this model is described as follows: Firstly, the BERT model is employed to obtain the text feature for input patent claims. Then, label features and text features are fed into the multiple fully-connected layers to establish connections between claims and secondary_labels. Further, the self-attention mechanism and Bi-GRU are introduced to enhance the text feature representation and the reservation of context semantics. Finally, two max-pooling operations with different aims are conducted to obtain the relation between bags and labels. The whole algorithm process can be shown by Algorithm 1.

The algorithm of the HMM-CPT structure is shown as follows:

Table

Algorithm 1: Hierarchical MIML learning for CPT classification
Input: Chinese patent texts
Output: patent-label mapping relationship.
1.  #data_preprocessing: stop word, text division, word segmentation
2.  data_preprocessing
3.  word = jieba (S)
4.  text = bert(S)
5.  # instance generation
6.  claims_text = get_claims_text(dataset)
7.  labels = get_labels(dataset)
8.  model.init(config)
9.  tokenizer.init(config)
10.  s∈S, l∈L
11.  for i in n
12.   for j in t
13.    convolutional (si,slj,.)
14.    output = martrix(t*t)
15.    max-pooling ()
16.   model.backward()
17.  max-pooling ()
18.  end for
19.  save_model()
20.  deep_model = load_model()
21.  results = test(test_data, deep_model)
22.  end

Algorithm 1 describes the overall process of the HMM-CPT, in which two loop statements are included. Steps 1–4 are used to preprocess the patent text dataset, and the preprocessed data are fed into the next step; steps 5–9 are employed to prepare the text data for subsequent input; steps 10–17 are introduced to mine the instance-label relation; the prediction results work out in steps 17–21. For popular understanding, in this algorithm, n represents the number of instances, and t represents the number of secondary_labels. Two loop statements are included in this algorithm, and thus the time complexity of this algorithm is O(nt).

4.2. Input layer

For a given Chinese patent text in our dataset,$X=\{x_1,x_2,\dots ,x_i,\dots x_n\},1\le i\le n$, $x_i$represents the i-th claim.$L=\{l_1,l_2,\dots ,l_j,\dots l_m\},1\le j\le m$, $l_j$represents the j-th label. The input layer aims to preprocess the input text to generate the instances for the following training. Before obtaining the text representation, the patent claims of each patent are divided into multiple independent texts, as shown in Algorithm 2.

The algorithm of the text division is shown as follows:

Table

Algorithm 2: the process of text division
Input: Claims in Chinese patent texts
Output: single claims.
1.  data_preprocessing
2.  full_claims_text = get_claims_text(dataset)
3.  for full_claims_text in dataset:
4.    if sequence_number < = 1;
5.     output = claim
6.    else:
7.     read sequence_number in full_claims_text
8.     se_num = get_sequence_number
9.     splitting_token('//') =  full_claims_text(se_num)
10.     text_splitting(splitting_token)
11.    end for
12.  output = text_splitting(full_claims_text)
13.  end

Algorithm 2 describes the text division process in the front part of the HMM -CPT. The patent claims texts are read through steps 1–2; steps 3–11 describe the division operation on claim texts; concretely, the specific symbol '//', which is derived from the replacement with the sequence numbers, is used as the splitting token; with steps 12–13, the divided claim texts are put out for the subsequent process. One loop statement and one judgment statement are included in this algorithm, and thus the time complexity of the algorithm is O(n), where n represents the number of claims.

The text feature of input claims is obtained by the BERT model. The process is formulated as follows: (2) $text\mathrm{\_}feature(x_i)=BERT(input\mathrm{\_}claim{x}_i)$(2) The mechanism of the BERT model is shown in Figure 4.

Figure 4. The structure of the BERT model.

4.3. Modelling layer

The purpose of the modelling layer is to model the connection between instances(that is, the claim text) and secondary_labels. The fully-connected layers are employed to establish the mapping between instances and secondary_labels through multiple convolutional operations, and what we get is the instance-label layer mentioned in Figure 5.

Figure 5. The construction process of the instance-label layer.

A two-dimensional fully-connected layer, which can be used to deal with the single-instance multi-label classification, is proposed at first, aiming to build the connections between one claim and all secondary_labels. That is, once the representation of claim x is obtained, a two-dimensional matrix of size t*t is constructed. t represents the number of secondary_labels. The purpose of the fully-connected layer is to construct the mapping from a single claim to all secondary_labels of the corresponding patent.

In such a fully-connected layer, the (i, j)-th node represents the matching score between the claim x and the i-th secondary_label of the j-th label. Here, we choose the softmax function as the activation function. The form of activation is as follows: (3) $\begin{array}{rl}{\mathbf{c}}_{i,j}& =f({\mathit{\omega }}_{i,j}+{\mathit{b}}_{i,j})\end{array}$(3) (4) $\begin{array}{rl}softmax(\mathit{z}{)}_i& \mathbf{=}\frac{e^{z_i}}{{\sum }_{j=1}^N{e}^{z_j}},i=1,\dots ,N\end{array}$(4) In the above formula, f(•) is the activation function, weight vector ω_i,j is the matching template for the i-th secondary_label of the j-th label. $\mathit{z}$represents the vector mapping of instance i, and N represents the dimension of the vector.

Further, the Bi-GRU and self-attention mechanism are introduced to enhance the text representation of instances. Finally, two max-pooling layers in different directions are conducted. The process can be expressed as: (5) $\text{mapping}\sum _{i=1}^n(X_i,L_i)=convo(\sum _{j=1}^t(x_i,\text{ s}{\text{l}}_j))$(5) The self-attention mechanism computing process is shown in Figure 6,

Figure 6. Self-attention mechanism computing process.

According to Figure 6, this computing process can be formulated as follows: (6) ${\mathit{s}}_{1,i}=exp(h_{1,i})/{\sum }_{j}exp(h_{1,i})$(6)

4.3. Max-pooling layer

In this layer, the relation between instances and labels is finally obtained by conducting two max-pooling operations on the instance-label layer constructed in the previous step. Concretely, the first pooling operation is a max-pooling on every instance-secondary_label layer, which aims to pick out the highest-scored secondary_label for every label; the second one is a max-pooling on the instance level, by which the sorted matching scores between labels and the corresponding patent are obtained. The whole process is described in Figure 7 and formula (7); (7) $\begin{array}{rl}& hier\mathrm{\_}miml=max-poolin{g}_{bag}\sum _{i=1}(X_i,L_j)\{{\mathit{M}}_{\mathbf{1}}\}\\ & {\mathit{M}}_{\mathbf{1}}=max-poolin{g}_{label}\text{mapping}(\sum _{j=1}(x_i,\text{s}{l}_j)\end{array}$(7)

Figure 7. The max-pooling process.

5. Experiments

In this section, the experimental details are elaborated on, including the experimental data, setting up, results, and analysis.

5.1. Experimental data

2461 patents in the mechanical engineering field were selected from the first 15000 patents of the '2017 Chinese patent dataset' as our experimental data.¹ The selection principles of patent texts are as follows:

1. The number of claims is greater than or equal to 6 (according to the average number of claims calculated from the first 15,000 patents), which is intended to ensure the sufficiency of information and the presence of multi-instance characteristics.

2. The number of IPCs is greater than or equal to 2, which aims to make sure the patents have multi-label attributes.

Some examples of IPC deconstruction are listed in Table 4.

Table 4. Examples of IPC deconstruction.

Section(label)	Class(label; secondary_label of section)	Sub-class(secondary_label for class)
A	A01; A21; A22	A01C; A01D; A21B; A21C; A22B; A22C
B	B01; B02; B04	B01B; B01D; B02B; B02C; B04B; B04C
C	C01; C02; C06	C01B; C01F; C02F; C06B; C06C
D	D01; D07; D21	D01B; D01C;D07B; D21B; D21F
E	E05; E06; E21	E05;F E05G; E06B; E06C; E21C; E21F
F	F15; F16; F17	F15B; F15D; F16C; F16D; F17B; F17C
G	G06; G07; G08	G06E; G06J; G07B; G07F; G08C; G08G
H	H03; H04; H05	H03B; H03C; H04 K; H04L; H05C; H05H

Dataset 1:title-abstract-IPC. The patent titles and abstracts are taken as the main experimental objects. The dataset is divided into the training set and testing set at the ratio of 4:1.

Dataset 2: title-abstract-claims-IPC. The patent titles, abstracts, and claims are taken as the main experimental objects. The dataset is divided into the training set and testing set at the ratio of 4:1 as well.

Some samples in our datasets are listed in Table 5.

Table 5. Samples in the patent dataset.

Patent titles	Patent abstracts	Patent claim	IPC
一种建筑施工用的防护栏 (Protective fence for building construction)	本实用新型公开了一种建筑施工用的防护栏， …  … ，可折叠，利于推广 (The utility model discloses a protective fence for building construction, … , which can be folded and is conducive to promotion.)	一种建筑施工用的防护栏，包括 …  … 插销 (A protective fence for building construction includes … .. bolt) 一种 …  … 材料层 (A kind of … material layer) …  …	Section/Class	Class/Subclass
			E/04 F/21 A/62	E04/H F21/S F21/V A62/C
一种输电线路巡检鸟巢清除车 (The utility model relates to a bird nest removal vehicle for transmission line inspection)	本实用新型提供一种输电线路巡检鸟巢清除车， …  … ，功能更加强大 (The utility model provides a bird’s nest removal vehicle for transmission line inspection … , more powerful)	一种输电线路巡检鸟巢清除车，包括 …  … 连接(a bird nest removal vehicle for transmission line inspection, includes … connection) 输电线路巡检鸟巢清除车，所述雷达 …  … 模块连接(bird nest removal vehicle for transmission line inspection, the radar …  … module connection) …  …	H/02 A/01	H02/G A01/M

5.2. Evaluation index

Three indexes are used to evaluate the experimental results, that is, Hamming Loss(HL), Average Precision(AP), and F1-score. The explanations and formulas of the three indexes are as follows: (8) $\begin{array}{rl}HL& =\frac{1}{M}\sum _{\text{i}=1}^M\frac{|Y_i\oplus Z_i|}{L}\end{array}$(8) (9) $\begin{array}{rl}Acc& =\frac{TP}{TP+FP}\end{array}$(9) (10) $\begin{array}{rl}F1-score& =\frac{2^{\ast }Ac{c}^{\ast }Recall}{Acc+Recall}\\ Acc& =\frac{TP}{TP+FP}\\ Recall& =\frac{TP}{TP+FN}\end{array}$(10) HL indicates the misclassification degree of the claim on a certain label in the test set. The smaller the value of HL, the higher the prediction accuracy of a certain label.

AP indicates the accuracy of the patent text classification prediction, that is, the proportion of the correctly predicted samples in the total patents. The larger the AP value, the higher the prediction accuracy.

F1-score indicates the accuracy of the model prediction. The higher the value, the better the performance of the model.

In the above formulas (6), (7), and (8), M and L represent the total number of patents and labels respectively; Y_i and Z_i represent the actual number and predicted number for the i-th patent. The specific symbol ⊕ represents the exclusive-or operation (XOR), and f(x_i, y_j) indicates the matching score between instance x_i and label y_j.

5.3. Experimental methods

To verify the effectiveness of our proposed method based on the hierarchical MIML learning oriented to Chinese patent texts, the experiment is carried out in the following ways.

1. Data processing: taking a single patent as a unit, the patent claims are divided with the sequence number as the splitting token; the stop words are removed; the jieba tool is used for word segmentation; finally, the input text is obtained.

2. Model training: For each patent text, its title, abstract, and multiple claims serve as multiple instances. the text features are obtained by the BERT model, and then fed into the convolutional neural network for training together with the labels. Concretely, the Bi-GRU and self-attention mechanism are used to enhance the representation of instances.

3. Model testing: taking the patent texts in the test set as the object, several groups of experiments are set to test the classification model constructed after the above two steps.

5.4. Experimental settings

Several models with good performance were selected as the comparison objects, based on the reference of multiple literature sources. The models mentioned in our experiments are listed in Table 6.

Table 6. Models and the corresponding descriptions.

Models	Model Descriptions
HMM-CPT	Our proposed model aims to mine the technical information hidden in patent claims, based on MIML learning. the bi-gru and self-attention mechanism are employed to enhance the semantic representation
BiLSTM + ATT	Paying attention to the contextual information of patent abstracts, adopting bidirectional recurrent neural networks and self-attention mechanisms to make the learned contextual information more comprehensive.
BERT-wmm-ext	One variant of the BERT model, which is suitable for pre-training in the Chinese language.
WPOS-GRU	A patent classification method based on two-channel feature fusion; enhances the textual representation of patent abstracts through separate training of word meanings and parts of speech.
ALBERT + GRU	A patent classification model combines ALBERT and GRU, the former is used to improve the representation ability for word vectors, and the latter aims to preserve the semantic association for long-distance words
WBAT	Word2Vec + Bi-GRU + ATT + TextCNN, which considers contextual features and bi-directional word order features simultaneously, and strengthens the key features in patent texts

It should be noted that, in our experiments, HMM-CPT is a multi-instance multi-label method. The patent title, abstract, and multiple claims are taken as the multi-instance input of this model. As for the five models from BiLSTM + ATT to WBAT in Table 6, the patent title, abstract, and patent claim are input into these models as a whole.

The models listed in Table 6 are experimentally evaluated on two datasets, respectively. The two datasets are Dataset 1:title-abstract-IPC, and Dataset 2: title-abstract-claims-IPC.

5.5. Experimental results and analysis

The experiments are carried out at two levels of the IPC: the section level and the class level. at the section level, the class level digits of IPC are taken as the secondary_label. While the subclass level digits of IPC are taken as the secondary_label at the class level.

The experimental results are listed in Table 7 and Table 8.

Table 7. The experimental results at the section level.

	Dataset 1			Dataset 2
Models	HL	Acc(%)	F1	HL	Acc(%)	F1
BiLSTM + ATT	0.047	83.7	0.7677	0.045	84.6	0.7773
HMM-CPT	0.067	86.9	0.7824	0.049	88.4	0.7967
BERT-wmm-ext	0.064	79.5	0.7563	0.054	82.8	0.7742
WPOS-GRU	0.057	78.7	0.7466	0.048	83.2	0.7670
ALBERT + GRU	0.049	82.6	0.7607	0.051	85.1	0.7792
WBAT	0.070	83.4	0.7631	0.067	85.3	0.7827

Table 8. The experimental results at the class level.

	Dataset 1			Dataset 2
Models	HL	Acc(%)	F1	HL	Acc(%)	F1
BiLSTM + ATT	0.065	71.6	0.6494	0.045	75.6	0.6733
HMM-CPT	0.070	73.1	0.6690	0.052	77.4	0.6937
BERT-wmm-ext	0.073	71.8	0.6327	0.054	75.4	0.6642
WPOS-GRU	0.062	70.6	0.6445	0.047	74.6	0.6659
ALBERT + GRU	0.067	72.3	0.6547	0.049	76.2	0.6825
WBAT	0.073	72.5	0.6632	0.062	77.0	0.6914

Based on the experimental results in Tables 7 and 8, we have made the following analysis.

1. From a vertical perspective, as shown in Tables 7 and 8, our proposed method HMM-CPT outperforms other methods on Acc and F1-score. We have analyzed the reasons and summarised them as follows:

   1. In the experiment, except for HMM-CPT, all other models treats the title, abstract, and patent claim as a single instance. Although the amount of information is the same, the connection established between a single instance and multiple labels is relatively limited. Therefore, the accuracy of label prediction is lower.

   2. The HMM-CPT method treats the title, abstract, and each claim as separate instances, and then establishes the relationship between each instance and each label. Consequently, the instance-label relationships obtained through training are more comprehensive, leading to better performance in the label prediction step.

   3. The structure of both the text and labels can also be utilised to obtain better classification results. By establishing connections between instances and secondary_labels, the knowledge scope associated with the labels is expanded. This allows for more accurate predictions when classifying unclassified patents.

2. From a horizontal perspective, The experimental accuracy of each model on Dataset 2(title-abstract-claims-IPC) is higher than that on Dataset 1(title-abstract-IPC).

   1. Based on our analysis, we believe that a patent abstract generally encompasses the overall content of a patent. However, due to its concise language, some technical details may not be fully reflected. The patent claims contain more information compared to the abstract, and this information helps classify and predict patent texts.

   2. Compared to the patent specification, the advantage of the patent claim lies in its textual structure. The patent claim is composed of multiple independent claims, and the semantics of each claim are relatively complete. This structure provides us with an opportunity to use a multi-instance multi-label learning approach.

In summary, our proposed model takes the patent claim, which contains more useful information, as the main research object. In addition, our proposed method combines multi-instance multi-label learning with the IPC hierarchical structure, which enables a more comprehensive connection established between labels and instances. That is, the HMM-CPT constructs the relationships between claims and multiple secondary_labels, thereby obtaining more comprehensive and accurate results in label-level prediction.

As can be seen from the experimental data, current deep learning models can indeed achieve good classification performance, which provides a significant reference for this research. It is worth noting that, the model WBAT, Word2Vec + Bi-GRU + ATT + TextCNN, did not perform well at the HL value, and the reason may be that this method did not concentrate on IPC structure but text feature mining. In our experiment, the whole IPCs are treated as labels for WBAT. As shown in Table 4, the Acc-value of WBAT is the highest except for our proposed method. Figures 8–10 represent the results more intuitively.

Figure 8. HL results on dataset1&2 at section level(left) &class level(right).

Figure 9. Acc(%) results on dataset1&2 at section level(left) &class level(right).

Figure 10. F1-score results on dataset1&2 at section level(left) &class level(right).

5.6. Ablation study

An ablation experiment was designed to verify the rationality of each module that constitutes our proposed model. The results of the ablation experiment are shown in Tables 9 and 10.

Table 9. ablation results at the section level.

	Dataset 1			Dataset 2
Models	HL	Acc(%)	F1	HL	Acc(%)	F1
MIML	0.064	82.7	0.7993	0.063	83.1	0.7482
MIML + ATT	0.060	84.9	0.7823	0.060	87.6	0.7794
MIML + Bi-GRU	0.060	85.4	0.7605	0.056	87.4	0.7869
MIML + Bi-GRU + ATT	0.068	87.0	0.7993	0.052	89.2	0.7970

Table 10. Ablation results at the class level.

	Dataset 1			Dataset 2
Models	HL	Acc(%)	F1	HL	Acc(%)	F1
MIML	0.078	69.8	0.6256	0.051	72.1	0.5283
MIML + ATT	0.071	72.7	0.6705	0.059	75.4	0.6748
MIML + Bi-GRU	0.074	72.5	0.6453	0.057	76.3	0.6844
MIML + Bi-GRU + ATT	0.072	73.6	0.7042	0.060	77.9	0.7062

From the results of the ablation experiments in Tables 9 and 10, it can be observed that the addition of ATT and Bi-GRU indeed contributes to improving the accuracy of the Chinese patent text classification proposed in this paper. Concretely, the results of the ablation experiments in Figures 9 and 10 demonstrate that the inclusion of ATT and Bi-GRU contributes to the improvement of the accuracy of the proposed Chinese patent text classification method. In terms of the Acc and F1 values, the difference between MIML + ATT and MIML + Bi-GRU is not significant, but there is an obvious enhancement observed when both ATT and Bi-GRU are combined in the MIML + ATT + Bi-GRU model. It is worth noting that ATT focuses on emphasising the semantic relevance of keywords in sentences, whereas Bi-GRU excels in enhancing the overall semantic coherence of sentences. Therefore, the integration of the multi-instance multi-label framework with ATT and Bi-GRU for text feature representation is a rational approach for improving the accuracy of Chinese patent text classification.

5.7. Other cases

In the previous set of experiments, when extracting experimental data, the principle 'the number of claims is greater than or equal to 6' was based on the average quantity observed in the first 15,000 patents of the 2017 Chinese patent dataset. Another question that needs to be considered is whether the more claims a patent contains, the higher its classification accuracy is.

For this issue, another set of experiments was designed. The patents were selected from the first 15,000 in our dataset with varying numbers of claims, ranging from 2 to 15, and each group consisted of approximately 100 patents. The experimental results are presented in the form of a line graph, as shown in Figure 11.

Figure 11. The impact of the number of claims on the classification accuracy.

From the two lines in Figure 11, it can be observed that as the number of claims increases from 2 to 6, the classification accuracy gradually improves. However, beyond 6 claims, the accuracy levels off without significant improvement. After analyzing the experimental results and patent claim text, it can be concluded that there is redundancy in the information contained in patent claims. When more than 8 or even more claims are introduced, although the supplementary content increases, the usable information does not significantly increase. This indicates that the inclusion of patent claims does provide additional information for patent classification, but a higher quantity does not necessarily result in better classification performance. Based on the accuracy at the section level and class level, it can be concluded that the optimal number of claims for patent classification is between 6 and 8.

6. Conclusions

To fully mine the information in the Chinese patent texts, and improve the accuracy of classification, the HMM-CPT, a method based on multi-instance multi-label learning, is proposed. Concretely, the proposal of this method is grounded on a full consideration of patent text structure, and the patent claim is taken as the main research object. Besides, the hierarchical structure of the label is also taken into consideration. Specifically, the IPCs are deconstructed into several parts for different requirements. By deeply mining technical information hidden in patent claims and constructing the relationship between claims and IPCs, the classification accuracy for Chinese patent texts is improved.

Although our proposed method can improve the classification accuracy for Chinese patent texts to some extent, there are still limitations to consider. The technical information in patent claims can provide supplementary information for the construction of patent text classification models. However, due to its writing style, there is also some repetitive text in it, which unavoidably leads to information redundancy in the classification process and wastes computational resources. In the future, increased consideration can be given to the structure and writing style of the patent claim to explore ways to reduce information redundancy in patent claim texts, which aims to provide underlying theoretical support for Chinese patent text classification.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the National Natural Science Foundation of China (Grant number 62076006), the University Synergy Innovation Program of Anhui Province (Grant number GXXT-2021-008), the Opening Foundation of State Key Laboratory of Cognitive Intelligence (Grant number COGOS-2023HE02), and the University Natural Science Research Project of Anhui Province(Grant number 2023AH050846).

Notes

1 Data source address: https://opendata.pku.edu.cn/file.xhtml?fileId=6605; https://patents.qizhidao.com/; https://github.com/SL-wamg/CPT