ABSTRACT
To evaluate the performance of “Vaccination Rates Monitoring Report System” implemented by Shenzhen CDC, we conducted an analysis of the data quality and identify key areas for system improvement. Following evaluation guidelines provided by WHO and United States CDC, we established six evaluation attributes: representativeness, simplicity, acceptability, data reliability, stability and timeliness. In eastern, central and western regions of Shenzhen, we selected one district from each region, of which the local CDC and ten CHSCs under jurisdiction were chosen for evaluation. On-site inspections, questionnaires survey and interviews were utilized for data collection, while the Likert scale method was used for attributes rating evaluation. A total of 70 participants were surveyed, consisting of 60 CHSCs and 10 CDCs staff. The gender ratio was 1:2.5 (males to females), with the majority falling within the 25–34 age range (46%). Most participants held full-time positions (80%) and had more than 5 years of work experience (62%). The system achieved 100% coverage of all CHSCs and CDCs (100%). The cumulative percentage scores for the overall favorable options of simplicity, acceptability, data reliability, stability, and timeliness were 79%, 85%, 73%, 50%, and 71% respectively. The system operates normally with strong representativeness. Acceptability was rated as “good.” Simplicity, data reliability, and system timeliness were rated as “average,” while system stability was rated as “poor.” Based on these survey results, developers should urgently investigate reasons for poor stability, particularly addressing concerns from CHSCs users. Additionally, the issues and shortcomings identified in other attributes should also be gradually improved.
Introduction
Vaccination plays a crucial role in preventing infectious diseases and is considered the most cost-effective measure. Maintaining high vaccination rates within the population is a key factor in establishing an immunologic barrier. Evaluating vaccination rates provides valuable insights into the effectiveness of immunization programs works and helps assess the potential spread of infectious diseases. Since 1999, China has been monitoring the vaccination rates for children’s National Immunization Program Vaccines (NIPVs) and using broadly consistent reporting format to collect vaccination data, so to assess the authenticity, accuracy, and reliability of vaccination rates through scientific evaluation and analysis.Citation1 Vaccination rates monitoring is based on a professional system platform. In December 2014, the “China Immunization Program Information Management System (National Network) Vaccination Monitoring Module” was officially launched.Citation2 and immunization program doctors could report vaccination data step by step through statistical operations, forming a complete monitoring network at the Community Health Service Center (CHSC, referring to institution with vaccination qualifications)-county/district Center for Disease Control and Prevention (CDC)-city CDC-province CDC-national CDC levels. This system enables decision-makers to timely identify weak vaccination areas or for specific vaccine, providing a visual comparisons, allowing for targeted policy support and measures for checking and filling vaccination gaps.
In 2016, the State Council’s “13th Five-Year Plan for Health and Wellness” proposed a mandatory target of >90% NIPVs coverage for eligible children at the township (town, street) level. As a result, local governments in each area have established management and evaluation systems for routine childhood immunization targets, incorporating NIPVs coverage into government performance assessments. In Shenzhen, the vaccination rates have traditionally been reported through the National Network, which is shared with other regions nationwide. However, since 2023, Shenzhen has developed an independent monitoring platform for vaccination rates, which is integrated with other immunization program modules in a comprehensive Immunization Information Systems (IIS). This system enables the city to achieve comprehensive, scientific, and systematic management of immunization program data. Despite efforts to debug the system during its initial launch, it still does not fully meet the expectations for some users in certain aspects. Therefore, a comprehensive evaluation of the “Shenzhen CDC IIS – Vaccination Rates Monitoring Report System” (hereinafter as system) was conducted to better understand, improve and enhance its operation so that to achieve a perfect using experience for staff, and to make the data recording and retrieval process more scientific.
System and methods
System characteristics
The system was designed for the “National Routine Immunization Vaccination rates Monitoring Program”, as a subsystem of the IIS in Shenzhen, it has various functionalities such as data entry, tabulation report, quick statistics, query and export. All of CHSCs in Shenzhen are required to input statistical data into the system and upload it before the 5th of each month. The input data comprises two types of reports: “7-1 of NIPVs vaccination rates” and “7-2 of Non-National Immunization Program Vaccines (NNIPVs) does”. In the “7-1” report, the denominator of the vaccination rates refers to the number of eligible individuals for receiving the vaccine. It specifically includes the counts of children within the CHSCs’ jurisdiction who have reached the prescribed age for that particular vaccine dose during any day of the given month, as defined by the national immunization program. It also incorporates the number of children who have missed that vaccine dose. CHSCs staff is responsible for entering the denominator into the system when individuals receive the vaccine for the first time. On the other hand, the numerator represents the actual counts of individuals who receive a specific vaccine dose in a given month. This information is automatically recorded by the computer after vaccination. When the CHSCs need to report the vaccination rates data every month, data aggregation becomes necessary. The process of aggregation has been simplified through system integration, as staff now only need to execute a specific a computer command to aggregate and upload the data. During the upload process, manual modification of the data is possible. After proofreading, CDCs staff are responsible for conducting a comprehensive review of the data reported by CHSCs and uploading it to the superior department before the 10th of each month. The data undergoes a step by step check before finally being uploaded to the national CDC. If an error is identified, the data must be withdrawn from the target institution and undergo the necessary steps again ().
Figure 1. System structure and data flow.
Methods
According to the evaluation guidelines of disease surveillance system of World Health Organization (WHO) and US CDC,Citation3 six evaluation attributes of system representativeness, simplicity, acceptability, data reliability, stability and timeliness were determined. We created a semi-structured questionnaire that encompassed general personal information, evaluation questions related to each attribute, and overall opinions and suggestions regarding the usage experience of the system. In addition to the Shenzhen CDC, we selected three administrative districts from eastern, central, and western region of Shenzhen, namely Luohu, Futian, and Baoan to gather responses. Each administrative district had 10 streets under its jurisdiction, and we selected the largest CHSC from each street. From each CHSC, we selected two different staff to complete the questionnaire, ensuring no missing items (open-ended questions were not mandatory). Furthermore, we surveyed 9 CDC staff from the three administrative districts (3 staff per district) and 1 staff from Shenzhen, resulting in a total collection of 70 questionnaires. The online questionnaire was used to investigate the multiple choice questions, while the face-to-face or telephone survey was conducted to answer open-ended interview questions. Prior to the formal investigation, a district CDC and a CHSC were selected to conduct a preliminary investigation, and the findings were used to enhance and refine our questionnaire. Each attribute score of the questionnaire referred to the evaluation method (Likert scale method) conducted by Sahil Sharma,Citation4 where attributes with an overall favorable option cumulative score percentage above 90% were classified as “excellent,” those between 80% and 89% were classified as “good,” those between 60% and 79% were labeled as “average,” those between 40% and 59% were denoted as “poor,” and those below 40% were considered “very poor.”
Representativeness
Representativeness of the surveillance system is the occurrence of a health-related event over time and its distribution in the population by place and person.Citation5 In this evaluation, we conducted the assessment by reviewing the regular reports provided by all monitoring institutions in Shenzhen, as well as considering the coverage rates of these institutions in terms of their number and type.
Simplicity
It was obtained from questionnaire survey, including seven questions as follows: “quickly logging into the system,” “convenient to query historical data,” “system increases daily workload,” “content should be disposed in system,” “system straightforward reflects the vaccination rates of all vaccines,” “average time spent on the system each time,” and “overall, system is easy to use.”
Acceptability
It was obtained from questionnaire survey, including four questions as follows: “system language is simple and easy to understand,” “system interface is beautiful and comfortable,” “do not feel repellent when using system,” and “overall, feel satisfaction with the system.”
Data reliability
It was obtained from two approaches. One is to review the CHSCs’ logs to look up the real data and compare it to the reported vaccination rates. Another is questionnaire survey, including four questions as follows: “automatic statistical results of the system meet the reporting requirements”, “data generated from system is consistent with real situation”, “willing to use system data for auxiliary description of true working”, and “modify the system data before reporting”.
Stability
It was obtained from questionnaire survey, including three questions as follows: “system blinks or crashes when using,” “system will re-login if not operates for a long time,” “system automatically modifies the data because of its own defects.”
Timeliness
It was conducted base on the submission deadline. CHSCs are required to submit before the 5th of each month, while district CDCs are before the 10th. Data was obtained from questionnaire survey, which included one question asking “comply with quality requirements and submit reports on time.”
Statistical analysis
Excel 2013 was used to establish the database and perform logical verification. SPSS 19.0 software was used for general description analysis, chi-square test was used to compare the rates between different groups, and the test level was taken as ɑ = 0.05. All questionnaires were required to achieve a 100% response rates (excluding open-ended questions) to be included in the research analysis, otherwise, it will be considered invalid and excluded from the analysis.
Ethics approval
Our study was approved by the Ethics Committee of Luohu District Center for Disease Control and Prevention, Shenzhen, China. The study complied with the guidelines of the Declaration of Helsinki.
Results
Basic information of the respondents
A total of 60 staff from CHSCs and 10 staff from CDCs were surveyed using questionnaires. The respondents consisted of 29% males and 71% females. The majority of the participants fell within the age range of 25–34 years (46%). The job attribute for most staff was full-time (80%), including all CDCs staff. Regarding the staff involved in the vaccination rates report, the majority had been working for more than 5 years (53%) ().
Table 1. Basic information of survey respondents.
System representativeness
The system encompasses all CDCs and CHSCs in Shenzhen. All institutions monitor and report vaccination rates through this system once a month. The latest report was conducted in June, with a coverage, utilization and reporting rate of 100%.
System simplicity
Among CHSCs and CDCs staff, 80% considered “logging into the system is quick.” 73% and 90%, respectively, thought that it is “convenient to query historical data.” 28% and 0%, respectively, thought that “the system increases the daily workload.” 83% and 60%, respectively, thought that “content should be disposed in system” was relatively easy. 63% and 100%, respectively, considered that “the system can straightforward reflect vaccination rates of all vaccines.” When reporting vaccination rates, the majority of CHSCs staff (48%) and all of the CDCs staff (100%) spent over 15 minutes on average each time. In general, the system was considered easy to use, with 72% and 90% agreeing (). The overall cumulative percentage for the favorable option of simplicity was 79%.
Table 2. Investigation on system simplicity.
System acceptability
70% of CHSCs staff and 100% of CDCs staff thought that “the system language is simple and easy to understand.” 78% and 90%, respectively, thought that “the system interface is beautiful and comfortable.” 77% and 100%, respectively, said that “they do not feel repellent when using the system.” 72% and 90%, respectively, were “generally satisfied with the system” (). The overall cumulative percentage for the favorable option of acceptability was 85%.
Table 3. Investigation on system acceptability.
Data reliability
The assessment of data reliability involved utilizing a questionnaire survey and checking the consistency of outpatient logs with the system. 82% and 90% of staff from CHSCs and CDCs, respectively, thought that “automatic statistical results of the system could basically meet the reporting requirements.” However, none of CDCs staff considered “fully capable meet.” 83% and 40% of staff believed that “the data generated from system is basically consistent with the real situation.” Similarly, no CDCs staff considered “fully consistent.” 75% and 60% of staff were “willing to use the data as an auxiliary description of true working,” but none of CDCs staff were “very willing.” Regarding the statement “staff should modify the system data before reporting,” 95% and 100% selected “occasionally” or higher (). Based on the year-end assessment on CHSCs in 2022, one of them with a low comprehensive evaluation was selected from each of the three districts, and the second dose of “Group A and C Meningococcal Polysaccharide Vaccine” (abbreviate A+C vaccine), with a generally low vaccination rate, were checked by examining the actual vaccination data recorded in the outpatient logs from January to June 2023. The results showed that the vaccination rates in Luohu, Futian, and Baoan districts were 43%, 55%, and 49%, respectively. These rates were compared and statistically verified against the reported vaccination rates in the system and were found to be significantly lower than the reported rates, showing a significant discrepancy (). The overall cumulative percentage for the favorable option of data reliability was 73%.
Table 4. Investigation on data reliability.
Table 5. Investigation on agreement between actual and reporting vaccination rates of “second dose A+C vaccine”.
System stability
85% of CHSCs staff thought that “the system occasionally or often blinks or crashes when using,” whereas CDCs staff thought that it will never happen. 77% and 80% of staff, respectively, thought that “system will re-login if not operates for a long time.” 42% of CHSCs staff thought that “the system occasionally or often automatically modifies the data after reporting due to its own defects,” similarly, CDCs staff thought that it will never happen (). The overall cumulative percentage for the favorable option of system stability was 50%.
Table 6. Investigation on system stability.
Timeliness of the report
In the findings regarding the question “When reporting data, you will upload base on quality requirements and within the specified time,” 62% of CHSCs staff thought that “it can be done all the time,” while 37% were “frequently” and 1% were “occasionally.” However, for CDCs staff, 80% were “all the time,” while 20% were “frequently.” The overall cumulative percentage for the favorable option of reporting timeliness was 71%.
Discussion
The utilization of electronic health information to support clinical and public health services has witnessed an upward trend in recent years, such as IIS, which is confidential, population-based, computerized databases that record all immunization doses administered by participating providers to persons residing within a given geopolitical area.Citation6 The application of the IIS is also very extensive, in USA，according to the research by Lynn,Citation7 IIS provides consolidated immunization histories and clinical decision support to providers. At the population level, IIS provides aggregate data on vaccinations, enabling public health interventions to enhance vaccination rates, identify vulnerable populations at risk for vaccine-preventable diseases, and optimize program resources. In Italy, VincenzaCitation8 explored the advantages of using IIS as a tool to combat vaccine hesitancy. Similarly, SarahCitation9 described the processes employed in Canada to record childhood and adolescent vaccinations using both IIS and non-IIS methods, outlining the respective strengths and limitations of these systems and processes, emphasizing their significant differences.
The vaccination rates monitoring report system in Shenzhen was independently developed by the city, its functions have been integrated into the CHSCs’IIS, significantly improving data preservation capabilities. Within the specified reporting time, staff can calculate relevant data for any specified time range and directly upload it once generated. This method is considerably simpler and more efficient than the previously utilized National Network. Previously, when data were reported through the National Network, staff were required to export the report from the vaccination system and subsequently upload it to the National Network, undergoing a step-by-step approval process. If any unreasonable or clear logical errors were identified within the report, staff had the option to modify the data before uploading. However, this practice could significantly compromise the information accuracy.
In terms of system simplicity, some staff may find the system login inconvenient. This is because the system developer, likely motivated by network security management requirements, requires users to log into a VPN client before accessing the system. Only after successfully establishing a connection are users able to log in and utilize the system normally. During interviews, some individuals mentioned that they found it cumbersome to enter a verification code at each login, considering it as an obstacle in daily operations. When accessing historical data, some individuals found it inconvenient. As the system was recently implemented, unfamiliarity with accessing historical data might be a concern. Furthermore, integrating the vast amount data from the old system (National Network) has resulted in instances of data loss, this necessitates constant improvements and patches over an extended period of time. Some CHSCs staff believed that the system has increased their workload due to the time spent investigating data errors. Furthermore, delays in response from the developer’s technical support have further escalated this workload. Although the system was designed to enable one-click data submission, a minority of CHSCs and CDCs staff found it to be not user-friendly. When rushing to submit data, considering data quality and conducting logical checks become demanding tasks. For CDCs staff, this step is particularly taxing. Obvious errors in data aggregation require extensive revisions, resulting in unnecessary consumption of time and energy, as well as guilt feelings for work mistakes. The system does not allow modifications to individual vaccination rate after generating a draft, instead, all vaccines must be reentered, which is considered as a cumbersome process by users. The system clearly presents vaccination rates for all program vaccines, but three CHSCs respondents had different opinions. Background checks revealed that these individuals had less than a year in their positions, with one being part-time, possibly affecting their understanding of vaccination rates. To ensure strict quality control, CHSCs staff must verify the system-generated data, usually taking over 15 minutes. While for CDCs staff, who should review all CHSCs data, need even more time. Despite these challenges, most staff found the system relatively easy to use and noted improvements compared to the old system. Although it is not yet perfect, being developed by a local developer in Shenzhen, it has the adaptability to continuously incorporate suggestions to address minor issues and facilitate ongoing improvements.
Monitoring vaccination rates is a vital component of routine immunization program works. For professionals in this field, particularly those working in the CDCs with strong educational backgrounds, generally possess a comprehensive understanding of relevant terminology. However, a survey revealed that some CHSCs staff found the systematic language complex. in Shenzhen, where stringent quality requirements exist for vaccination rates monitoring, the calculation method for monthly rates involves concepts of cumulative vaccination rates and complexities like excluding and selecting target populations. This complexity highlights the necessity of enhancing staff training in their daily work. The old system operated independently, while the new system has been integrated with various other systems. This integration illustrates an extensive architecture that facilitates the sharing and exchanged of data, even though each system still maintains its independent links. The integration may give the impression of limited independence, leading some individuals to perceive a lack of “aesthetic and comfort” in the interface. To address this concern, future system updates should focus on enhancing visibility and improving color differentiation within the interface, this will result in improved visual effects and minimize any aversion experienced by staff. Based on overall satisfaction, the majority of staff were satisfied with the system. However, a minority voiced dissatisfaction due to various reasons, including delayed technical support, particularly during weekends. Since CHSCs continue operations on weekends while developers do not, this misalignment negatively impacts satisfaction levels. It is recommended that the developers establish a small team to be available during weekends for prompt resolution of potential issues, thus enhancing responsiveness and overall satisfaction.
When reporting vaccination rates on the old system, staff could freely modify the data, which could be attributed to the separation of the system from the vaccination process, a deficient supervision mechanism, and inadequate management methods. Moreover, such modifications were tacitly accepted and sometimes even encouraged by higher management. Because CHSCs were required to achieve a yearly cumulative vaccination rates of 95% in all NIPVs to be considered qualified, but it was often difficult to achieve completely, leading to data manipulation to cope with administrative pressure and evade accountability, which seriously undermined the authenticity of the work. For higher management, this situation rendered them unable to discern variations in vaccination practices across different regions, impeding the development of effective improvement strategies. A key objective of the system is to ensure data authenticity. The system emphasizes accuracy but is predicated on achieving a “passing” grade in the yearly assessment (cumulative vaccination rates with 95% in all NIPVs). Consequently, meeting monthly reporting requirements is essential, failure to do so may result in being unqualified at the year’s end due to “accumulated difficulties.” For CDCs staff, the review and approval process are more challenging, as frequent modification to the data are common. Notably, no CDCs staff selected “fully capable to meet the reporting requirements.” Furthermore, satisfying reporting requirements doesn’t guarantee complete data accuracy, when data seriously deviates from standards, CHSCs staff would be required to make appropriate modifications. Fortunately, the majority believed the system’s data is “basically consistent” with reality, but no CDCs staff considered it “completely consistent,” which reflects the system’s success to some extent. Therefore, not all staff were willing to use this data for “auxiliary description of true working,” especially for CDCs staff. In addition to the questionnaire, the system’s data reliability was also evaluated concerning the vaccination rate of “second dose of A+C vaccine.” The assessment revealed significant deviations between actual (clinic logs) and reported vaccination rates. In China, the majority of NIPVs are administered before 18 months of age with a generally high coverage, often exceeding 95%. Such as the hepatitis B vaccine (administered at 0, 1, and 6 months) and the measles-containing vaccine (administered at 8 and 18 months). However, the second dose of A+C vaccine is administered at 6 years of age, while parental awareness of vaccination significantly declines at this stage, and no matter how much doctors advise parents, the effect is minimal. Therefore, the vaccination rates of vaccines for older children (3 years and older) are generally low. For instance, apart from the second dose of A+C vaccine, there are also similar vaccines such as the first dose of A+C vaccine (administered at 3 years old) and the fourth dose of polio vaccine (administered at 4 years old), while with the second dose of A+C vaccine being often the most severe. Due to the frequent inconsistency between the generated data and the reporting requirements of these vaccines, as previously mentioned, under management pressure, the staff may manually modify the data, which the system cannot detect and describe. Clinic logs are the authentic records left by CHSCs when vaccinating children. These records cannot be modified and are essentially official records. The number of children to be vaccinated in the clinic log of a certain CHSC is fixed when a child is born or newly moved into the jurisdiction of the CHSC, the CHSC then has the obligation to complete the vaccination for these children according to the immunization schedule. If vaccination is not administered within the designated period, it will be recorded as missing. If a child has a temporary contraindication, according to the regulations for calculating vaccination rates in China, the child will still be counted as one of denominators for calculation. Additionally, if a child is not under the jurisdiction for more than 3 months, they need to be migrated out of the vaccination system, otherwise, the child will also be counted as one of target population, even though they cannot actually receive the vaccination. All these situations contribute to a decrease in vaccination rates. Therefore, whether due to objective reasons or inadequate performance, in order to avoid penalties, data may be modified when reporting to the system. Commonly, the modification type refers to the vaccines for older children just mentioned (inconsistent rates between logs and reports often only involve these vaccines), and the method is to appropriately reduce the respective denominators to increase the vaccination rates. Fortunately, such practices are less common in the new system compared to the old. Given these reasons, it is recommended that the top-level managers treat real data reasonably and friendly, in order to avoid placing excessive administrative pressure on grassroots staff. Simultaneously, it is suggested that the developer enhances the functionality to track data modification traces, enabling the ability to investigate the authenticity of vaccination data for a CHSC or region at any time, and evaluate the objective vaccination rates.
In terms of system stability, most CHSCs staff reported intermittent issues such as system crashes, application closures, and occasional lagging that necessitates a system refresh to display data. However, CDCs staff did not encounter these problems. This discrepancy may be attributed to the integration of multiple systems and the continuous real-time changes in data at the CHSCs. Additionally, some CHSCs staff reported data confusion after merging multiple duplicate records, resulting in inaccurate calculations of vaccination doses. Even more concerning, a few staff reported that the system occasionally modifies data automatically due to inherent defects after submission. The underlying reasons for such occurrences remain unclear, but this issue has been submitted to the developer for resolution. Regarding timeliness, the majority of CHSCs staff were able to submit reports punctually, but a small portion experience delays due to data being returned for multiple modifications. Strengthening training and cultivating a sense of responsibility among these staff are essential to rectifying this issue. As for CDCs staff, they generally manage to submit reports on time, any few delays that do occur were typically the result of excessive modifications by CHSCs.
It is worth mentioning that, in our analysis, we did not observe any influence on genders and age against system evaluation, perhaps the job does not have a gender preference in its selection process, and the work pressure and energy required are within the normal range for both genders. Shenzhen is a city with a younger population, and the majority of our survey respondents were young individuals with similar educational and lifestyle backgrounds, it was difficult to observe any differences in the usage of modern equipment. In addition, in the evaluation process, some aspects of certain attributes were analyzed based on the subjective opinions of the respondents. This is determined by the methodology of the system evaluation, many scholars have conducted similar researches such as on rubella, pneumonia, antimicrobial resistance, hypertension monitoring and malaria surveillance system.Citation10–14 They also to some extent used the subjective opinions in analyzing the simplicity, flexibility, stability, acceptability, and other aspects of the systems. However, these did not hinder their contribution to the development of professional field and providing valuable references for other scholars’ research. Moreover, in this study, we quantitatively rated the subjective opinions, which enhances the scientific interpretability. Hence, we believe that it still plays a certain role in promoting the industry’s development.
Conclusion
The vaccination rates monitoring report system in Shenzhen is operating normally. All institutions in Shenzhen use this system to monitor and report vaccination rates, achieving a coverage rate and utilization rate of 100%, indicating a strong level of representativeness. The system’s acceptability was rated as “good,” while attributes such as system simplicity, data reliability, and system timeliness received an “average” rating. However, system stability was rated as “poor.” It is crucial for the developer to thoroughly investigate the factors for the “poor stability” rating, particularly for CHSCs users, as the need for improvement is more urgent in that case. The absence of an “excellent” rating for any attributes highlights the necessity for substantial enhancements and updates to the system, especially concerning data reliability. Efforts must be made to fortify the authenticity and operability of data storage and retrieval. Providing a trace query for data modification is essential to ensure the system has the ability to restore data as much as possible. Nevertheless, the system, built upon the foundation of the original National Network system, has made significant strides in various areas, positively contributing to daily works. It still warrants further promotion once the issues outlined in this article have been addressed.
Limitations
This evaluation work still has some limitations. Firstly, our comprehensive evaluation of the system are unique nationwide, with no peer-reviewed research or English-language studies for reference and comparison, this could result in cognitive biases due to regional characteristics. Secondly, there are over 500 CHSCs and more than 1500 staff in Shenzhen, our questionnaire survey did not achieve full coverage, which may have resulted in the omission of certain information to some extent. Thirdly, when using the Likert scale, we did not follow the standard algorithm but refer to an article practice instead. We assigned scores to each evaluation attribute option based on our own situation and calculated the percentage of cumulative scores for favorable options, whether this approach is more scientifically valid needs to be verified by further peer research. Lastly, we did not conduct a scientific comparison between the new and old systems, which resulted in a weak presentation in superiority of the new system.
Acknowledgments
Linxiang Chen, Ziqi Wang, and Lina Duan contributed to the questionnaire and study design, data collection and analysis, data interpretation, and manuscript writing. Xiaojun Zheng helped in revising and improving writing. FangFang Lu and Huawei Xiong helped develop the study. Data were collected by Jin Liao, Chunmiao Peng, Kangming Chen, and Wenli Zhang. The original draft was written by Linxiang Chen and Ziqi Wang. The article’s final version has been approved by all writers. We expressed our gratitude to immunization clinics for their unwavering assistance with the questionnaire survey. We thanked Mr Yucheng Xu for his assistance with data analysis.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Additional information
Funding
References
- Ye J, Coi L, Yu W. Surveillance for routine vaccination coverage with national immunization program vaccines among age-eligible children in China, 2016–2019. Chin J Vaccines Immun. 2022;28(1):94–9.
- Cui J, Coi L, Zheng J, Cao L, Yuan P, Wang M, Xiao Q , Wang H. Analysis of reported coverage rates of vaccines in national immunization program in China 2014. Chin J Vaccines Immun. 2016;22(1):34–40.
- German RR, Lee LM, Horan JM, Milstein B, Pertowski CA. Updated guidelines for evaluating public health surveillance systems: recommendations from the guidelines working group. MMWR Recom Rep. 2001;50(RR–13).
- Sharma S, Goel K, Kaushal K, Grover GS, Dikid T, Singh GB. An evaluation of malaria surveillance system in Punjab, India, 2020. Indian J Comm Med. 2023;48(3):407–12. doi:10.4103/ijcm.ijcm_305_22.
- Shedura VJ, Hussein AK, Nyanga SK, Kamori D, Mchau GJ. Evaluation of the influenza-like illness sentinel surveillance system: a national perspective in Tanzania from January to December 2019. PLoS One. 2023;18(3):e283043. doi:10.1371/journal.pone.0283043.
- Pabst LJ, Williams W. Immunization information systems. J Public Health Manag Pract. 2015;21(3):225–226. doi:10.1097/PHH.0000000000000202.
- Scharf LG, Coyle R, Adeniyi K, Fath J, Harris L, Myerburg S, Kurilo MB, Abbott E. Current challenges and future possibilities for immunization information systems. Acad Pediatr. 2021;21(4):57–64. doi:10.1016/j.acap.2020.11.008.
- Gianfredi V, Moretti M, Lopalco PL. Countering vaccine hesitancy through immunization information systems, a narrative review. Hum Vaccin Immunother. 2019;15(11):2508–26. doi:10.1080/21645515.2019.1599675.
- Wilson SE, Quach S, MacDonald SE, Naus M, Deeks SL, Crowcroft NS, Mahmud SM, Tran D, Kwong JC, Tu K, et al. Immunization information systems in Canada: attributes, functionality, strengths and challenges. A Canadian immunization research network study. Can J Public Health. 2017;107(6):575–82. doi:10.17269/CJPH.107.5679.
- Gavhi F, De Voux A, Kuonza L, Motaze NV. Evaluation of the rubella surveillance system in South Africa, 2016–2018: a cross-sectional study. PLoS One. 2023;18(6):e287170. doi:10.1371/journal.pone.0287170.
- Dadzie D, Addo-Lartey AA, Peprah NY, Kenu E. Evaluation of surveillance system for pneumonia in children below five years, tema metropolis, Ghana, 2012–2016. Ghana Med J. 2020;54(2):11–17. doi:10.4314/gmj.v54i2s.3.
- Collineau L, Godebert E, Thibaut S, Lemenand O, Birgand G, Caillon J, Bourely C. Evaluation of the French surveillance system for epidemiological surveillance of antimicrobial resistance in the community and nursing homes. JAC Antimicrob Resist. 2022;4(4):c78. doi:10.1093/jacamr/dlac078.
- Sesay U, Gebru GN. Evaluation of a hypertension surveillance system, Kenema Government Hospital, Sierra Leone, 2021. Prev Chronic Dis. 2023;20:E15. doi:10.5888/pcd20.220230.
- Amede PO, Umeokonkwo CD, Abege S, Akawe J, Derek J, Adedire E, Balogun MS. Evaluation of malaria surveillance system in Benue State, Nigeria. Malar J. 2022;21(1):348. doi:10.1186/s12936-022-04367-4.