Diagnostic validation and development of an algorithm for identification of intussusception in children using electronic health records of Ningbo city in China

ABSTRACT

Background

Monitoring the risk of intussusception after the introduction of rotavirus vaccines is recommended by the World Health Organization (WHO). Although the validity of intussusception monitoring using electronic health records (EHRs) has been confirmed previously, no similar studies have been conducted in China. We aimed to verify the diagnosis and determine an algorithm with the best performance for identification of intussusception using Chinese EHR databases.

Research design and methods

Using the Regional Health Information Platform in Ningbo, patients aged 0–72 months from 2015 to 2021 with any related visits for intussusception were included. The algorithms were based on diagnostic codes or keywords in different clinical scenarios, and their performance was evaluated with positive predictive value (PPV) and sensitivity in line with the Brighton guidelines.

Results

Brighton level 1 intussusception was confirmed in 2958 patients with 3246 episodes. Fine-tuned algorithms combining the appearance of the relevant ICD-10 codes or the Chinese keyword ‘Chang Tao’ in any diagnostic reports with the results of enema treatments or related surgeries showed the highest sensitivity, while the highest PPV was obtained by further criteria based on typical radiographic appearances.

Conclusion

Intussusception could be identified and validated internally using EHRs in Ningbo.

KEYWORDS:

• Algorithms
• China
• electronic health records
• intussusception
• post-market monitoring
• verification study

1. Introduction

Intussusception is the second-most common cause of acute abdomen in children and the most common cause of small bowel obstruction in young infants [1]. Most cases of intussusception occur in children aged between 3 months and 3 years, and the peak incidence is observed from 5 to 9 months of age [1]. The etiology of intussusception is poorly understood [2], and several factors, including viral illness, abdominal trauma, tumors, parasites, and vaccination, have been reported to be potential causes of intussusception [1]. In some studies, rotavirus vaccination has been reported to be associated with an increased risk of intussusception [3,4]. However, according to the latest WHO position paper on rotavirus vaccines, the pathogenic mechanisms underlying intussusception following rotavirus vaccination remain poorly defined, and post-licensure evaluations of rotavirus vaccines have revealed that the risk of intussusception varies according to vaccine and study location [5]. Of course, it also emphasized that the benefit of rotavirus vaccines is greater than the small risk of intussusception [5]. Therefore, the WHO recommends timely monitoring of the risk of intussusception after the introduction of these vaccines [5]. In China, the Lanzhou lamb rotavirus vaccine and Rotateq vaccine became available on the market in 2000 and 2018 respectively [6]. Recently, in a cross-sectional survey of 10 provinces using a multistage sampling scheme, as a non-National Immunization Program (NIP) vaccine, the coverage rate of the third dose of rotavirus vaccine was 1.8% in China in 2019, and the coverage rate varied from provinces [7]. However, no safety surveillance reports for the risk of intussusception associated with rotavirus vaccines has been reported or published in China.

The use of electronic medical records for post-market monitoring of vaccines to facilitate timely and dynamic reporting of the efficacy and safety of vaccines based on real-world data (RWD) has become an important trend worldwide [8]. In this regard, the Food and Drug Administration (FDA) Framework for Real-World Evidence Program emphasizes the importance of data reliability and relevance when utilizing RWD to generate real-world evidence [9,10]. The accuracy of the key variables is an important factor influencing the reliability of RWD data. Indeed, in order to maxi mize the internal validity of studies and minimize the misclassification bias in background rate estimation, association analysis, and other safety monitoring studies, investigators should ensure that the selected outcome variables are validated, whether they are defined by a code-based electronic algorithm or an algorithm that combines structured and unstructured data elements [11].

A few previous studies have validated the outcomes of intussusception using data from electronic health records or claims databases, but no such study has been conducted in China [12–14]. Since the Regional Health Information Platform (RHIP) in the city of Ningbo, China, has been recently proven to be a feasible and credible database for vaccine safety monitoring [15,16], this study aimed to use the RHIP in Ningbo to verify the diagnosis and (or) determine the algorithm with the best performance for identification of intussusception in line with the Brighton Collaboration criteria, thereby paving the way for safety monitoring of the rotavirus vaccine in China.

2. Methods

2.1. Data sources and study population

Ningbo is a city in Zhejiang Province, located on the east coast of China, with more than 9.54 million permanent residents at the end of 2021. According to a survey conducted in Yinzhou, a developed district of Ningbo city, the coverage of 3 doses of rotavirus vaccine was 0.41% in 2017 [17]. The RHIP in Ningbo was established in 2011 and launched by the Health Commission of Ningbo with the aim of developing an integrated and standardized medical information network. The RHIP in Ningbo has two primary data sources: the digital platform of the Centers for Disease Control and Prevention (CDC), which includes primary healthcare, chronic or infectious disease surveillance, vaccine registration, and death registration data, and the digital hospital platform, which collects data from electronic medical records (EMR). These data are transformed into a structured format and linked with a unique national identifier or healthcare identifier. By 2015, the RHIP in Ningbo covered more than 87% of the residents and had collected nearly all of their healthcare information from birth to death. The vaccine registration system of the RHIP in Ningbo covers almost all permanent resident children aged <6 years. Several recent studies have used this database to monitor the post-marketing safety of various vaccines [15,18]. Detailed information on the RHIP in Ningbo can be found in previous studies [15,16].

Although most of intussusception occur in children <3 years of age, in our study, we noticed there were 748 (23.09%) cases of intussusception occurred in children between 3 years and 6 years. Hoping that the algorithm of intussusception could be applied to a wider population and serve a broader purpose, in this study, we included subjects aged 0–72 months. The eligibility criteria for the present study were as follows: (1) registered in the RHIP in Ningbo for no less than 180 days between 1 January 2015 and 30 June 2021; (2) aged 0–72 months during the observation period; and (3) diagnosed as showing intussusception during the observation period.

2.2. Identification of visits and episodes involving intussusception

Potential intussusception-related visits were identified on the basis of the International Classification of Diseases (ICD) code K56.1 in the Tenth Revision or the Chinese Term ‘Chang Tao’ (肠套). This approach was used for identification of all potential intussusception visits by evaluation of outpatient and emergency records, admission and discharge hospitalization records, records of imaging examinations such as X-ray, ultrasound, computed tomography (CT), and air or hydrostatic enema examinations, and surgical notes. A single episode was defined as a set of consecutive visits within 48 hours for each patient. Any clinical visits that occurred ≥48 hours after individual visit last time was considered to indicate the start of a new repeat episode of intussusception. We also defined visits with nonspecific image features (including abdominal ultrasound or CT scan showing the concentric ring sign, the target sign, the doughnut sign, the pseudokidney or sandwich sign, and abdominal X-ray showing fluid levels and dilated bowel loops) in ultrasound, X-ray, or CT scans as a suspected visit for intussusception.

2.3. Case adjudication/validation

The validation process for intussusception cases was in line with the Brighton guidelines [19]. First, a standard and anonymous electronic case report form (eCRF) was constructed for collection of information regarding the onset month, sex, radiographic examination results, enema treatment, and surgical operation. Senior clinical experts in China reviewed the eCRFs. Second, two local clinical experts were invited to separately judge the eCRFs of potential cases in accordance with the Brighton criteria for intussusception. In the Brighton guideline, cases meeting the definition standard of level 1 are named as ‘definite’ cases, while cases that meet the definition standard of the lower two levels are named as ‘probable’ and ‘possible’ cases, which are defined on the basis of criteria representing less direct evidence of intussusception. Since level 1 is the gold standard for clinical diagnosis of intussusception according to the views of Chinese pediatric profession and clinical experts, and the valid cases in our study had sufficient information to determine as level 1. Therefore, in this study, we only focused on the definite cases. The vaccination history of all potential cases was blinded. The cases that met the definition for level 1 intussusception in the Brighton criteria were judged as Brighton level 1 cases, while the cases that did not meet this definition were judged as cases without intussusception. Cases that could not be judged due to lack of information such as imaging, surgical, or autopsy reports were classified as uncertain. When the opinions of the two clinical experts were inconsistent, another nationally renowned expert was invited to participate in the discussion and reach a final decision on the basis of the voting results after the discussion.

2.4. Identification algorithms

We used the following algorithms to identify intussusception episodes in this study: (1) intussusception-related ICD-10 code alone (K56.1); (2) intussusception-related Chinese Term (‘Chang Tao’ [肠套]) alone; and (3) intussusception-related ICD-10 code (K56.1) or intussusception-related Chinese Term (‘Chang Tao’ [肠套]). The performance of these algorithms was evaluated in different clinical scenarios, including evaluations based on outpatient and emergency records, evaluations using admission and discharge hospitalization records, and evaluations based on imaging examinations (including X-ray, ultrasound, and CT). Then, these algorithms were fine-tuned using criteria based on typical radiographic appearances (concentric circle sign is a typical radiographic appearance described as a target sign or doughnut sign on transverse sections and a pseudokidney or sandwich sign on longitudinal sections. It was recommended by pediatric experts and surgeon that any value of the longitudinal cutting length and transverse cutting diameter of concentric circles exceeding 20 mm may suggest a high potentiality of intussusception in clinical practice. Therefore, these features were applied to improve the performance of algorithms for those with diagnose coding or terms only.), records of enema treatment, and records of related surgery in order to improve their accuracy.

2.5. Identification rule based on expert agreement

Using the eCRF for assessment of potential intussusception, this study verified and evaluated the performance of each algorithm in different clinical scenarios through internal verification. Then, for algorithms showing high accuracy, an identification rule based on expert agreement was further summarized in accordance with the Brighton guidelines, the experience of experts in the diagnosis of intussusception, and related clinical knowledge.

2.6. Statistical analysis

The performance of each identification algorithm was assessed by determining its sensitivity and positive predictive value (PPV), and 95% confidence intervals were calculated using the binomial distribution [20]. We also evaluated the differences in demographic characteristics between intussusception cases judged by experts and the intussusception cases identified by the common algorithms used to identify intussusception in previous vaccine safety studies, which were based on identification of diagnostic codes or keywords in hospitalization records or outpatient and emergency records [21–24], as well as the algorithms with high accuracy in our study. The demographic characteristics evaluated in these comparisons included age, sex, birth weight, birth city, location, and mobility. Categorical characteristics were presented as frequency (percentage) and continuous characteristics were presented as mean ± standard deviation. SAS 9.4 was used for statistical analyses.

3. Results

3.1. Case extraction and adjudication

During the study period, a total of 34,363 patients aged 0–72 months with potential intussusception over 50,250 visits were identified in the RHIP in Ningbo. On the basis of the inclusion and exclusion criteria, after excluding patients who had been registered in the RHIP in Ningbo for less than 180 days, 33,666 permanent residents (97.97%) with a total of 46,110 episodes of potential intussusception were included in the study. After reviewing the information from radiology and surgical reports, 32,578 patients (94.81%) with 42,748 episodes had at least one instance of intussusception-related diagnostic information in the records of imaging examinations, enema treatment, or related surgeries. Among these patients, 8235 patients (23.96%) with a total of 9229 episodes had information regarding nonspecific image features in their medical records. After evaluations by the clinical experts on the basis of the Brighton criteria, 2958 patients (8.61%) with 3246 episodes were classified as showing Brighton level 1 intussusception, while 800 patients (2.33%) with 818 episodes were classified as showing uncertain findings due to the lack of information. The detailed process for identifying suspected intussusception cases is shown in Figure 1.

Figure 1. Flow chart for judging suspected cases of intussusception.

Abbreviation: RHIP, Regional health information platform

3.2. Performance of the identification algorithms

Algorithms including K56.1 or ‘Chang Tao’ [肠套] showed higher sensitivity without significantly reducing the PPV in comparison with algorithms including only one of K56.1 or ‘Chang Tao’ [肠套]. Using the algorithm including K56.1 or ‘Chang Tao’ [肠套], 377, 8788 and 12,418 potential intussusception episodes were identified by evaluation of hospitalization records, outpatient and emergency records, and imaging examination records, respectively. In the evaluation of hospitalization records, the algorithm identified 278 Brighton level 1 episodes with high PPV and low sensitivity. Similarly, in evaluations based on outpatient and emergency records and imaging examination records, 2676 and 2723 Brighton level 1 episodes, respectively, were identified by this algorithm with the high sensitivity and low PPV.

In combination with the records of enema treatments or related surgeries, algorithms that included K56.1 or ‘Chang Tao’ [肠套] in any diagnosis terms of intussusception had a PPV of 80.14% (95% CI: 78.81%, 81.42%) and the highest sensitivity of 90.63% (95% CI: 89.58%, 91.62%). On adding the criteria for typical radiographic appearances, the PPV of the algorithms increased while their sensitivity decreased. The highest PPV (87.35%, 95% CI 86.08, 88.56%) was observed when searching for K56.1 or ‘Chang Tao’ [肠套] in the diagnosis terms, restricting cases to those with records of enema treatment or records of related surgeries, and identifying cases with any value of the longitudinal cutting length and transverse cutting diameter of concentric circles >30 mm (Table 1).

Table 1. Positive predictive value and sensitivity of algorithm for episodes with intussusception.

Algorithms	K56.1 or ‘Chang Tao’ (肠套)		ICD10: K56.1		Chinese Term: ‘Chang Tao’ (肠套)
	SEN^*d	PPV^*e	SEN	PPV	SEN	PPV
Diagnosis of hospitalization records	8.56 (7.62, 9.58)	73.74 (68.99, 78.11)	5.98 (5.19, 6.85)	79.84 (74.23, 84.69)	7.58 (6.69, 8.54)	77.36 (72.36, 81.84)
Diagnosis of outpatient and emergency records	82.44 (81.09, 83.73)	30.45 (29.49, 31.42)	71.47 (69.88, 73.02)	31.80 (30.73, 32.88)	78.19 (76.73, 79.60)	30.22 (29.24, 31.22)
Diagnosis of imaging examinations^*a	83.89 (82.58, 85.14)	21.93 (21.20, 22.67)	0.77 (0.50, 1.13)	5.43 (3.55, 7.92)	83.86 (82.55, 85.11)	21.93 (21.21, 22.67)
Any diagnose terms and records of enema treatment	89.90 (88.81, 90.91)	80.19 (78.85, 81.47)	71.23 (69.63, 72.78)	80.36 (78.86, 81.80)	89.28 (88.16, 90.32)	80.14 (78.81, 81.43)
Any diagnose terms and records of enema treatment or related surgeries	90.63 (89.58, 91.62)	80.14 (78.81, 81.42)	71.53 (69.95, 73.08)	80.37 (78.88, 81.81)	90.05 (88.97, 91.06)	80.17 (78.84, 81.45)
Any diagnose terms and records of enema treatment or related surgeries and records with type A^*b typical radiographic appearance	76.40 (74.90, 77.85)	87.35 (86.08, 88.56)	64.39 (62.71, 66.04)	87.96 (86.59, 89.24)	76.03 (74.53, 77.49)	87.33 (86.05, 88.54)
Any diagnose terms and records of enema treatment or related surgeries and records with type B^*c typical radiographic appearance	77.79 (76.32, 79.21)	85.02 (83.68, 86.28)	64.39 (62.71, 66.04)	85.69 (84.24, 87.06)	77.42 (75.94, 78.85)	84.96 (83.62, 86.23)

^*aImaging tests included X-ray, ultrasound and CT.

^*bthe type A of typical radiographic appearance was that any value of the longitudinal cutting length and transverse cutting diameter of concentric circles exceeds 30 mm.

^*cthe type B of typical radiographic appearance was that any value of the longitudinal cutting length and transverse cutting diameter of concentric circles exceeds 20 mm.

^*dPPV = (Number confirmed episodes with medical records available/Number potential episodes with medical records available) *100%.

^*eSEN means Sensitivity, Sensitivity = (Number confirmed episodes with medical records available/Number potential episodes overall) *100%

3.3. The rule based on expert agreement

In the diagnostic rule for intussusception, the potential cases were first identified on the basis of the presence of the diagnostic codes or keywords for intussusception in any of the diagnosis terms, records of enema treatment, or records of surgeries. Among potential cases, those showing nonspecific image features were classified as suspected cases for further judgment; cases that did not show these features were diagnosed as not showing intussusception. Finally, for cases with records of air or hydrostatic enema examination, if the examination records described disappearance of nonspecific image features and entry of air into the small intestine, intussusception was diagnosed. If the examination records described opening of the ileocecal region during the examination, no intussusception was diagnosed. In addition, cases with records of related surgeries were diagnosed as showing intussusception. Detailed information on the diagnostic rules for intussusception based on experts is provided in Figure 2.

Figure 2. Diagnostic rule of intussusception based on experts.

3.4. Demographics by different algorithms

With reference to the Brighton level, episodes of intussusception judged by experts and those identified by a common algorithm using hospitalization records showed differences in demographic characteristics such as age and sex. Episodes identified by the common algorithm using hospitalization records involved a younger average patient age, a higher proportion of patients aged 0–11 months, and a greater proportion of males. In comparison, no differences in demographic characteristics were observed between Brighton level 1 episodes of intussusception judged by experts and episodes identified by the common algorithm using outpatient and emergency records, episodes identified by the algorithm with highest sensitivity, and episodes identified by the algorithm with the highest PPV (Table 2).

Table 2. Demographic characteristics by different identification algorithms.

Variables	Brighton level one episodes (N = 3246)	Common algorithm used in hospitalization records^*a (N = 278)	Common algorithm used in outpatient and emergency records^*b (N = 2676)	Algorithm with highest SEN^*c (N = 2942)	Algorithm with highest PPV^*d (N = 2480)
Age
Mean ± SD	24.91 ± 15.49	22.13 ± 16.70	24.34 ± 15.25	24.61 ± 15.48	24.40 ± 15.20
0–5	82(2.53)	24(8.66)	68(2.55)	79(2.69)	59(2.38)
6–11	659(20.35)	84(30.32)	573(21.46)	620(21.12)	521(21.02)
12–17	526(16.24)	35(12.64)	456(17.08)	487(16.59)	419(16.9)
18–23	477(14.73)	30(10.83)	380(14.23)	417(14.21)	366(14.76)
24–29	436(13.46)	20(7.22)	352(13.18)	388(13.22)	339(13.67)
30–35	311(9.6)	17(6.14)	246(9.21)	277(9.44)	233(9.4)
36–47	425(13.12)	37(13.36)	344(12.88)	382(13.02)	317(12.79)
48–71	323(9.97)	30(10.83)	251(9.4)	285(9.71)	225(9.08)
Gender
Female	1066(32.84)	78(28.67)	873(32.62)	962(32.70)	804(32.42)
Male	2180(67.16)	200(71.94)	1803(67.38)	1980(67.30)	1676(67.58)
Birth Weight, Kg	3.37 ± 0.45	3.38 ± 0.51	3.38 ± 0.45	3.38 ± 0.46	3.38 ± 0.45
Born city
Ningbo	2662(82.01)	224(80.58)	2183(81.58)	2406(81.78)	2047(82.54)
Others	584(17.99)	54(19.42)	493(18.43)	536(18.22)	433(17.46)
Location
Urban	1945(59.92)	165(59.35)	1608(60.09)	1769(60.13)	1509(60.85)
Fringe	759(23.38)	67(24.1)	624(23.32)	687(23.35)	582(23.47)
Rural	542(16.7)	46(16.55)	444(16.59)	486(16.52)	389(15.69)
Mobility
No	2696(83.06)	233(83.81)	2210(82.59)	2437(82.83)	2064(83.23)
Yes	550(16.94)	45(16.19)	466(17.41)	505(17.17)	416(16.77)

^*aAlgorithm searching K56.1 or ‘Chang Tao’ [肠套] in the diagnosis of hospitalization records.

^*bAlgorithm searching K56.1 or ‘Chang Tao’ [肠套] in the diagnosis of outpatient and emergency records.

^*cAlgorithm including K56.1 or ‘Chang Tao’ [肠套] in any diagnose terms and combined with records of enema examinations or records of related surgeries.

^*dAlgorithm including K56.1 or ‘Chang Tao’ [肠套] in any diagnose terms, and restricting to episodes having records of enema treatment or records of related surgeries, with any value of the longitudinal cutting length and transverse cutting diameter of concentric circles exceeds 30 mm.

4. Discussion

Accurate identification of the outcome of interest is important to reduce bias and improve the internal validity of studies. In accordance with the Brighton guidelines, this study attempted to identify all potential intussusception cases using information from outpatient and inpatient visits, imaging examinations, enema treatments, and surgical records. After evaluations by experts, a total of 2958 patients with 3246 episodes were finally determined to meet the criteria for Brighton level 1 intussusception. In terms of the performance of the algorithms, the algorithm including K56.1 or ‘Chang Tao’ [肠套] in any of the diagnostic terms of intussusception in combination with the records of enema treatments or related surgeries had the highest sensitivity (90.63%, 95% CI 89.58, 91.62%) and a high PPV, while the highest PPV (87.35%, 95% CI 86.08, 88.56%) was observed when searching K56.1 or ‘Chang Tao’ (肠套) in any diagnosis terms and restricting to episodes having records of enema treatments or records of related surgeries, with any value of the longitudinal cutting length and transverse cutting diameter of concentric circles > 30 mm. The algorithm searching for diagnostic codes or keywords in hospitalization records had a low sensitivity (less than 10%), while the PPV was less than 40% when searching for diagnostic codes or keywords in outpatient and emergency records.

Our results are in agreement with those obtained in other countries. First, as for hospitalized intussusception cases, the PPV of ICD-9 or ICD-10 code from a study in Ontario of Canada was 72.4% (95% CI: 65.4–78.7%) [13]. Similarly, in the Valencia of Spanish, the PPV of the ICD-9 code in any discharged diagnosis was 85.4% (95% CI: 78.8, 90.6%) [25]. Obviously, no distinct difference was observed between these previous studies and our findings that the PPV of ICD-10 code was 79.84% (95% CI: 74.23, 84.6%).Moreover, a study in United States infants found that the PPV of combination of ICD-9 code or Current Procedural Terminology (CPT) codes for inpatient and emergency intussusception cases together was 46.0% [24]. The value was slightly higher than the PPV of ICD-10 code in outpatient or emergency department setting (31.80%), and lower than the PPV of ICD-10 code in inpatient department setting (79.84%) in our study. Regarding the majority of valid cases in our study were from clinic visits, the results of the two studies could be similar. Second, some clinical features, such as the typical radiological findings, surgery notes and specific clinical symptoms, were attempted to fine-tuned and improve performance of the above algorithms. For example, in a national random sample of Swedish children younger than 3 years, the PPV was 84% (95% CI: 82, 86%) when typical radiological findings or surgery were required for a diagnosis of intussusception and up to 89% when clinically likely cases were also included [14]. Likewise, the PPV of inpatient diagnose coding or Chinese terms was 73.74% (95% CI:68.99, 78.11%), and when adding that with typical radiographic appearances, enema treatment, or surgery records in our study, the performance raised to 85.02% (95% CI:83.68, 86.28%) or much higher level.In addition to PPV, a few studies reported the sensitivity of algorithms; one study from the United States found that the sensitivity of abdominal radiography for identifying children at risk of intussusception was 77% (95% CI: 60, 89%), which was a little lower than the sensitivity of the algorithm in the present study that the diagnosis in imaging examinations can identify 83.89% (95% CI: 82.58, 85.14%) of all cases [26]. Plus, a Canadian study reported that the sensitivity of an algorithm searching for ICD code for intussusception in emergency department visit records was 65.1% (95% CI: 55.0, 74.2%) [13], which was close to the sensitivity of 71.47% (95% CI: 69.88, 73.02%)for ICD-10 code outpatient and emergency records observed in this study.

Our study found that algorithms combining records of enema examinations or related surgeries with searches for diagnostic codes or keywords in any of the diagnosis terms showed the highest sensitivity. The PPV of the algorithm increased further when longitudinal cutting length and transverse cutting diameters of concentric circles > 30 mm was added as a criterion, but the sensitivity decreased simultaneously. In active surveillance for vaccine safety, algorithms with higher sensitivity are important to identify vaccine safety signals in a timely and fast manner. In contrast, in causal association studies with a sufficient sample size, algorithms with higher PPV are a better choice because they can minimize the misclassification bias of outcomes. In our study, since case verification and algorithm formulation were both blinded to vaccination history, the misclassification bias was attributable to an undifferentiated misclassification of vaccination history. Therefore, although the misclassification bias would have resulted in the estimates tending to the null hypothesis, its impact was very limited when using the algorithm with the highest PPV to conduct a causal association study.

Our study is the first to use data from electronic health records in China to verify the outcomes of intussusception. Since the RHIP covered almost all permanent resident children aged below 6 years, we could identify all potential cases of intussusception in permanent residents aged 0–72 months in Ningbo city and extract all related records, including those related to diagnosis, imaging examinations, enema examinations, and surgeries. Therefore, after validating the algorithms, we could identify algorithms with the highest sensitivity or PPV to provide a scientific basis for algorithm selection in different scenarios of active surveillance. We also compared the demographic characteristics of patients with intussusception identified using different algorithms. The results suggested that identification based solely on searches of diagnostic codes or keywords in hospitalization records may introduce selection bias. Therefore, hospital-based active surveillance should include hospitals at different levels and from different regions to reduce potential bias. Moreover, in addition to the different algorithms, based on the Brighton guidelines, we also summarized a rule for rapid identification of intussusception cases based on experts’ experience in the diagnosis of intussusception and related clinical knowledge.

This study had some limitations. First, this was an internal validation study based on a data platform. Since the data for clinical symptoms such as bloody stool and abdominal pain were not transformed into a structured format, we failed to evaluate the performance of these variables in the optimization algorithm. Moreover, the records for some patients (2.33%) did not include related information for case judication. Therefore, we were unable to determine whether these cases did not involve intussusception because of incomplete information. These missing data may have influenced our findings, but since they represented only a small proportion of cases, their impact should be very limited. Finally, patients in the emergency and outpatient departments could not be distinguished in this study, so we were unable to separately evaluate the performance of the algorithms while searching for diagnostic codes or keywords in records from the emergency department.

5. Conclusion

Intussusception can be identified and validated internally using the electronic health records in Ningbo. An electronic algorithm combining all diagnostic terms for intussusception with those for air enema or related surgery could be an alternative method for capturing the occurrence of intussusception. In the present study, an algorithm including K56.1 or ‘Chang Tao’ (肠套) in any of the diagnostic terms combined with records of enema treatments or related surgeries showed the highest sensitivity (90.63%; 95% CI: 89.58%, 91.62%) and high PPV (80.14%; 95% CI: 78.81%, 81.42%), while the highest PPV (87.35%; 95% CI: 86.08%, 88.56%) and a high sensitivity (76.40%; 95% CI: 74.90, 77.85%) were observed with the addition of any value of the longitudinal cutting length and transverse cutting diameter of concentric circles > 30 mm.

Declaration of interest

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

Reviewer disclosures

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Author contributions

S. Zhan and G. Xu had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. S. Deng, Z. Liu are co–first authors. Concept and design: S. Deng, Z. Liu, S. Zhan. Acquisition, analysis, or interpretation of data: S. Deng, Z. Liu, J. Yang, L. Zhang, R. Ma, N. Li, G. Xu.

Drafting of the manuscript: S. Deng, Z. Liu. Critical revision of the manuscript for important intellectual content: All authors. Statistical analysis: S. Deng, Z. Liu, J. Yang.

Obtained funding: S. Zhan. Administrative, technical, or material support: T. Shou, J. Zhu, Y. He. Supervision: Z. Liu, G. Xu, S. Zhan. Final approval of manuscript: all authors.

Ethics approval and patient consent

The study protocol was approved by the ethical review committee of the Peking University Health Science Center (IRB. No: IRB00001052-15, 045). The requirement for informed consent was waived.

Data availability statement

The data that support the findings of this study are from the Ningbo Center for Disease Control and Prevention, but are not publicly available due to privacy or ethical restrictions. Data are available on request from the corresponding author and with permission of Ningbo Center for Disease Control and Prevention.

Additional information

Funding

This manuscript was funded by the National Natural Science Foundation of China (grant number 81973146), the National Natural Science Foundation of China for young scholars (grant number 82204175), and the National Key Research and Development Project of China (grant number 2016YFC0901105). This work was also supported in part by the Bill & Melinda Gates Foundation (grant number INV-035024). Under the grant conditions of the Foundation, a Creative Commons Attribution 4.0 Generic License has already been assigned to the Author Accepted Manuscript version that might arise from this submission. The funders had no role in study design, data collection, analysis, and preparation of the manuscript.