Evaluating the reactogenicity of COVID-19 vaccines from network-meta analyses

ABSTRACT

Background

Evidence-based reassurances addressing vaccine-related concerns are crucial to promoting primary vaccination, completion of the primary series, and booster vaccination. By summarizing and comparing the reactogenicity of COVID-19 vaccines authorized by the European Medicines Agency, this analysis aims to support in-formed decision-making by the lay public and help overcome vaccine hesitancy.

Research design and methods

A systematic literature review identified 24 records reporting solicited adverse events for AZD1222, BNT162b2, mRNA-1273, NVX-Cov2373, and VLA2001 in individuals aged 16 or older. Network meta-analyses were conducted for each solicited adverse events reported for at least two vaccines that were not compared head-to-head but could be connected through a common comparator.

Results

A total of 56 adverse events were investigated through network meta-analyses within a Bayesian framework with random-effects models. Overall, the two mRNA vaccines were found to be the most reactogenic vaccines. VLA2001 had the highest likelihood of being the least reactogenic vaccine after the first and second vaccine dose, especially for systemic adverse events after the first dose.

Conclusions

The reduced chance of experiencing an adverse event with some COVID-19 vaccines may help to overcome vaccine hesitancy in population groups with concerns about the side effects of vaccines.

KEYWORDS:

• COVID-19
• Network meta-analysis
• Reactogenicity
• Safety
• Systematic literature review
• Vaccine

1. Introduction

COVID-19 is an airborne disease caused by the SARS-CoV-2 virus, with over 630 million confirmed cases and over 6.6 million deaths reported worldwide as of 8 November 2022 [1]. The first case of COVID-19 was reported in Wuhan, in December 2019, which prompted institutions and pharmaceutical companies to develop vaccines that could stop the spread of the virus and reduce the number of hospitalizations and deaths [2], The first COVID-19 vaccine was approved in December 2020, and since then, 40 vaccines have been authorized, each in at least one country, six of which approved for use in the European Union (EU) by the European Medicines Agency (EMA) [3]. Moreover, a total of 12.93 billion doses have been administered globally, with nearly one billion in the European Union [4]. Despite a decrease in the number of severe symptomatic infections and deaths, COVID-19 is still a global pandemic, and SARS-CoV-2 reinfections can occur. Several studies have revealed that immunity, induced by vaccines or natural infection, wanes after 5–8 months [5,6]. In response to this, additional booster doses have been administrated and continue to be recommended by public health authorities, especially for high-risk populations [7]. Nevertheless, 24.4% of adults in the EU have never received a COVID-19 vaccine as of 8 November 2022 [4].

Hesitancy toward COVID-19 vaccines is one of the main barriers preventing full vaccine uptake. Multiple studies have shown that safety concerns constitute one of the main elements of hesitancy [8–11]. The speed of development of the vaccines and potential side effects are common reasons for unwillingness to receive COVID-19 vaccines [8–11]. This was also identified in a technical report by the European Centre for Disease Prevention and Control (ECDC) which addressed public health considerations in support of the COVID-19 vaccination strategies [12]. One of the identified obstacles to reaching full vaccine uptake is ‘concerns about side effects.’ In addition, according to surveys, individuals who experienced any degree of side effects – or whose friends or family members have – may be less likely to receive a booster vaccine. Clinical trials have revealed that safety profiles differ between vaccines. The reactogenicity of a vaccine depends on several variables, including the platform used. The currently approved vaccines have been developed using several platforms. These technologies range from traditional platforms to emerging approaches such as messenger ribonucleic acid (mRNA) vaccines. As of 24 June 2022, VLA2001, the only inactivated whole-virus vaccine accessible in the EU, was granted full marketing authorization by the EMA [13]. Therefore, the vaccines currently available in the European geographic area are AZD1222 (AstraZeneca/University of Oxford) BNT162b2 (Pfizer/BioNTech), Ad26.COV2-S (Janssen), mRNA-1273 (Moderna), NVX-CoV2373 (Novavax), and VLA2001 (Valneva) [3]. Evidence for the reactogenicity of these vaccines is provided by the outcomes of the clinical trials and surveillance data. However, since there has not been a clinical trial directly comparing these vaccines, no conclusions about their relative reactogenicity can be drawn.

Evidence-based reassurances addressing vaccine-related concerns – including those regarding side effects – are crucial to promoting primary vaccination, completion of the primary series, and booster vaccination [12]. By summarizing and comparing the reactogenicity of COVID-19 vaccines, this analysis aims to support informed decision-making by the lay public and help overcome vaccine hesitancy. Moreover, it generates additional evidence useful to policymakers to produce vaccination promotion strategies addressing misinformation, distrust, or a lack of clear information in population groups with low vaccination uptake. This study consists of a systematic literature review (SLR) and several network meta-analyses (NMAs) to summarize and compare the reactogenicity following primary series vaccination with COVID-19 vaccines currently approved by the EMA. The NMA was chosen as the methodology for this study because it allows comparing any number of treatments that have not been compared in head-to-head clinical trials, by combining evidence from a network of multiple randomized controlled trials (RCTs) in a single analysis [14].

2. Materials and methods

The Preferred Reporting Items for Systematic Reviews and Meta-analysis Involving a Network Meta-analysis (PRISMA-NMA) guidelines were followed during the execution and reporting of the SLR and NMA [15,16].

2.1. Search strategy and selection criteria

The search was conducted in PubMed, Embase, the Cochrane Database of Systematic Reviews, and the Cochrane Central Register of Controlled Trials. A gray literature search was conducted to identify the most recent data not yet indexed in the medical literature databases. The complete search details can be found in [Appendix A; Section 1]. Requirements for eligible studies included reporting solicited safety outcomes for a clinical trial after the first or second dose for a two-arm trial evaluating EMA-approved vaccines or placebo. Only studies reporting outcomes on two- arm clinical trials were included to ensure the comparability of populations and connectivity of the interventions within a network. This choice reflects the methodology used to gather data on AEs – in these studies, participants were required to record specifically solicited AEs uniformly into a registry. This ensures the comparability of the AE data and is, therefore, a more appropriate data source to inform an NMA than unsolicited AE data. No limitations were set on publication time or language. All identified RCTs were assessed for risk of bias using the Cochrane Collaboration’s risk of bias assessment tool. A summary of the complete eligibility criteria is presented in Table 1.

Table 1. Inclusion and exclusion criteria for the SLR based on the Population, Intervention, Comparator, Outcome, and Study design (PICOS) framework.

Category	Inclusion criteria	Exclusion criteria
Population (P)	Healthy individuals aged ≥ 16 years	Individuals age < 16 years
Intervention (I)	EMA approved COVID-19 vaccines with two primary series doses	EMA-approved COVID-19 vaccines with a dose number for the primary series different than 2
Comparators (C)	EMA approved COVID-19 vaccines with two primary series doses, a placebo	EMA-approved COVID-19 vaccines with dose numbers for the primary series different than 2, not COVID-19 vaccine, COVID-19 vaccines not approved by EMA
Outcomes (O)	Stratified and aggregated data for solicited AEs	Studies that do not report at least one of the outcomes of interest
Study design (S)	RCTs—both parallel-group and crossover (double-blind, single-blind, open-label) for clinical trials phase 2/3 or phase 3	RCTs for clinical trials phase 1, 2, or 1/2 RCTs on booster doses All other study designs that are not RCTs
Language	English language	None
Time limit	No restriction	None
Country	No restriction	None
Abbreviations: AE, adverse event; EMA, European Medicines Agency; RCT, randomized clinical trial.
A complete description of the PICOS is provided in Appendix A; Section 1.1

2.2. Data synthesis

A full overview of the identified data was used to perform a feasibility assessment regarding their final application in the NMAs [Appendix B; Section 1.1]. This assessment comprised three steps: specifying possible treatment effect modifiers (TEMs) regarding the reporting of AEs after a vaccine, ensuring that the transitivity assumption is not violated for the data included in the NMAs by comparing the heterogeneity in baseline characteristics for the included trials, and identifying the connected network for each AE endpoint of interest reported. Trials that did not have any reported AEs (zero-cell events) as the trial result for any of the outcomes of interest were excluded from the networks because models without events in at least one treatment arm and small sample sizes are numerically unstable [17]. A vaccine was only included in the network when data was reported for that endpoint. Analyses were performed on all endpoints for which at least two vaccines that were not compared in a head-to-head trial reported data and could be connected through a common comparator. Based on the grading of the AE, a network was created for each reported AE of interest.

A study by Beatty et al. found factors that were potentially associated with participant-reported AEs after COVID-19 vaccination and used them to identify TEMs for the feasibility assessment [18]. Beatty et al. concluded that factors associated with patient-reported adverse effects are vaccine dose, vaccine brand, age, female sex, and COVID-19 infection before vaccination. The vaccine dose of these TEMS was excluded because the NMA will estimate the relative effect for AE incidences using the same dose. Additionally, infection with COVID-19 before vaccination was not included because this is not a commonly reported patient characteristic, and the vaccine brand was ruled out, given that identification of differences between vaccines is the goal of this study. Hence, only age and female sex were determined as potential TEMs and used in the feasibility assessment [Appendix B; Section 1.1.1]. Older-aged individuals were shown to have lower odds of reporting adverse effects (any grade and severe), while higher odds were associated with the female sex [18].

2.3. Data analysis

The NMAs were conducted within a Bayesian framework on a log odds scale. Based on a meta-analysis by Turner et al., an informative prior distribution was used for between-study heterogeneity variance [19]. The posterior distributions resulting from the analysis were assessed using Markov Chain Monte Carlo (MCMC) simulations. The convergence of the posterior estimates was assessed using trace plots and Gelman-Rubin-Brooks plots. Both random-effects (RE) models and fixed-effects (FE) models were applied to each network, with the RE model as the base case to better account for the possible presence of heterogeneity [Appendix B; Section 1.2]. The Deviance Information Criterion was checked to compare the RE and FE models and to estimate the goodness of fit of each model [20]. To visualize the results, forest plots, rankograms, probability of being best-tolerated vaccine, and the surface under the cumulative ranking curve (SUCRA) scores without placebo were generated. The forest plots show the relative safety estimates compared to the placebo for each vaccine. The rankograms and SUCRA scores provide an overview of the treatment hierarchy, where the SUCRA score measures the probability of a vaccine being better than another vaccine in the network, on average [21]. The vaccine with the greatest SUCRA score is most likely to outperform the comparators and, as a result, it can be interpreted as the vaccine most likely to be the least reactogenic.

3. Results

The search for the SLR was performed on 27 July 2022 and identified 1,168 different publications. After 238 duplicates had been removed, 930 studies were screened by title and abstract, of which 42 articles were eligible for full-text screening. Following the exclusion of 30 records at full-text screening, 12 records reporting data for eight RCTs were included. There were 12 additional records found through the gray literature search, and data was therefore extracted from a total of 24 records, reporting outcomes for nine RCTs. A PRISMA flow diagram of the complete search is presented in Figure 1, and an overview of the identified trials is given in Appendix A; Section 2.1. The articles reported data on all five EMA-approved two-dose COVID-19 vaccines. A placebo was the most used comparator in the trials.

Figure 1. PRISMA flow diagram of the SLR for the NMAs.

3.1. Transitivity

We performed a feasibility assessment to ensure that all studies included are similar and to increase the likelihood that the transitivity assumption holds. Therefore, differences in the study designs and patient characteristics between the trials were evaluated to identify factors that might affect the summary measures of treatment effects. All trials were homogeneous in their study design except for the 18–30-year-old cohort in the trial comparing VLA2001 with AZD1222 (NCT04864561), which was open-label and non-comparative. The study design for the 30–65-year-old cohort was a comparative double-blinded randomized trial. The difference in study design and the separate data collection of this cohort – as well as the absence of the 18–30-year-old comparator cohort – can be explained by age restrictions applied to the AZD1222 vaccine by the British government. From April 2021, AZD1222 was no longer recommended for individuals aged 30 and under due to a possible correlation between the vaccine and very rare but serious cases of thrombosis in combination with thrombocytopenia [22]. Hence, the outcomes for this cohort were excluded from the analysis as a consequence of the absence of a comparable cohort and the higher risk of bias of an open-label study design.

By checking baseline characteristics such as age, sex, race, and comorbidities, differences between trials were identified and evaluated considering the defined TEMs. Two trials (NCT04368728 and NCT04649151) were excluded from the NMA due to the significantly lower mean age of participants (19.4 and 14.3 years old, respectively). Despite the exclusion of the 18–30-year-old cohort in the trial comparing VLA2001 with AZD1222, the population of the 30–65-year-old cohort in this trial still has a lower mean age than the general trend represented in the other studies. Nevertheless, given the lack of data for this new disease area, the novelty of the COVID-19 vaccine as a preventive treatment, and the importance of overcoming vaccine hesitancy, this study was judged similarly enough not to be excluded. In the end, six clinical trials were used as data sources to perform the NMAs [Appendix B; Section 2.1 & 2.4].

3.2. Connected network

For each of the reported AEs, networks were created separately for any grade and Grade 3+ AEs. Networks could not be created for AEs of Grade 1 or 2 because these outcomes were not commonly reported in the identified studies. The definitions of the endpoints were homogeneous, and the time point at which the outputs were collected was seven days after each injection in all RCTs. A total of 56 connected networks were constructed, comprising 36 different solicited systemic AEs and 20 solicited local AEs. A complete network connecting all the included vaccines is presented in Figure 2. An overview of all networks and the inclusion of the vaccines in each network is provided in Appendix B; Section 2.2.

Figure 2. Network plot of the complete identified network.

No long-term fluctuations were identified in the trace plots for any NMA, and the density plots followed a normal distribution for all effect size estimates, indicating good convergence. Furthermore, none of the simulations appeared to have autocorrelation, showing that the MCMC chains created were sufficiently thinned. An RE and an FE model were both run for each NMA. In around 40% of the NMAs, the deviance information criterion (DIC) values were in favor of the RE model [Appendix B; Section 2.10]. Combined with the slight differences in baseline characteristics found in the feasibility assessment [Appendix B; Section 2.4], the choice was made to exclusively present the outcomes for the RE models. Forest plots relative to placebo and rankograms were created for each NMA [Appendix B; Section 2.6 & 2.7].

3.3. Local adverse events

Of the nine NMAs on local AEs of any grade, seven had a connected network including all five EMA-approved vaccines, and the remaining two networks included four EMA-approved vaccines. The results on AEs after the first dose show that VLA2001 has the highest SUCRA scores and is therefore most likely to be the least-reactogenic vaccine when compared with the other vaccines in the network for all local AEs (Table 2) except for injection site swelling. In particular, VLA2001 was estimated to have a 98.8% probability of ranking the best-tolerated vaccine for any grade of injection site redness [Appendix B; Section 2.8]. This is in line with the outcomes of the overall reported local AEs after the first dose (Figure 3) for which VLA2001 is estimated to have a 99.8% probability of being the highest-ranking and thus best-tolerated vaccine [Appendix B; Section 2.8]. Since the 95%-credible intervals do not overlap, we can conclude that VLA2001 has a high certainty of being the best-tolerated vaccine for the overall reported AEs. NVX-CoV2373 is most likely the second-best-tolerated vaccine according to the SUCRA score and the median OR estimates for each endpoint. The results for the overall reported local AEs of any grade after the second dose show that AZD1222 has the highest likelihood of being less reactogenic than the other vaccines (SUCRA score: 0.895), closely followed by VLA2001 (SUCRA score: 0.839). This is also reflected in the SUCRA scores of the individual endpoints for AEs of any grade after the second dose. The vaccine with the highest likelihood of being the most reactogenic is mRNA-1273. This vaccine has the highest estimated odds ratio for the overall local AEs (Figure 3) and the lowest SUCRA score (Table 2) in four of the five endpoints for local AEs after the second dose.

Figure 3. Forest plots for any grade local solicited AE after first dose (A) and second dose (B).

Table 2. SUCRA ranking scores for local AEs of any grade after the first and the second dose.

Dose	Adverse event	AZD1222	BNT162B2	mRNA1273	NVXCoV2373	VLA2001
Dose 1	Injection Site Pain	0.5	0.056	0.195	0.753	0.996
	Injection Site Redness	0.359	0.421	0.098	0.625	0.996
	Injection Site Swelling	0.088	0.616	0.256	0.963	0.577
	Injection Site Tenderness	0.01	NA	0.707	0.324	0.959
	Reported Local AE	0.501	0.046	0.206	0.748	1
Dose 2	Injection Site Pain	0.933	0.24	0.011	0.509	0.807
	Injection Site Redness	0.464	0.804	0.287	0.319	0.624
	Injection Site Swelling	NA	0.864	0.022	0.614	NA
	Injection Site Tenderness	0.53	NA	0.983	0.011	0.477
	Reported Local AE	0.895	0.25	0.027	0.489	0.839

3.4. Systemic adverse events

Of the 20 NMAs on systemic AEs of any grade, 10 had a connected network including all five EMA-approved vaccines. For the systemic AEs of any grade after the first and the second dose, VLA2001 has the highest likelihood of being the best-tolerated vaccine, with probabilities of this ranking ranging between 86.1% and 99.8% for the first dose, and 74.4% and 97.2% for the second dose [Appendix B; Section 2.8]. The estimated odds ratio of reporting any overall systemic AEs after the first dose is significantly lower for VLA2001 than the other EMA-approved vaccines (Figure 4). Therefore, there is a high certainty that for this endpoint, VLA2001 is better tolerated than the other vaccines. The other vaccines have point estimates close to each other with overlapping 95% credible intervals, which indicates a higher uncertainty around the ranking of these vaccines. AZD122 has the highest odds ratio compared to placebo to report overall systemic AEs after the first dose, with SUCRA scores lower than the other vaccines in almost all the endpoints (Table 3). Therefore, there is a high likelihood that AZD1222 is the most reactogenic vaccine for systemic AEs after the first dose. However, it has a high likelihood of being better tolerated after the second dose compared to NVX-CoV2372, mRNA-1273, and BNT126b2. The vaccine mRNA-1273 has the lowest SUCRA scores in all the endpoints after the second dose. Finally, the protein subunit vaccine NVX-CoV2373 was estimated to have the highest probability of being the best-tolerated vaccine for arthralgia and nausea/vomiting after the first and second doses.

Figure 4. Forest plots for any grade systemic solicited AE after first dose (A) and second dose (B).

Table 3. SUCRA ranking scores for systemic AEs of any grade after the first and the second dose.

Dose	Adverse event	AZD1222	BNT162b2	mRNA1273	NVXCoV2373	VLA2001
Dose 1	Arthralgia	NA	0.135	0.449	0.916	NA
	Chills	0.504	0.995	0.002	NA	NA
	Fatigue	0.033	0.286	0.447	0.74	0.994
	Fever	0.006	0.247	0.517	0.773	0.957
	Headache	0.081	0.327	0.451	0.644	0.997
	Malaise	0.002	NA	NA	0.998	NA
	Myalgia	0.011	0.376	0.557	0.562	0.995
	Nausea/vomiting	NA	NA	0.354	0.646	NA
	Reported systemic AE	0.033	0.412	0.379	0.677	0.999
	Vomiting	0.459	0.541	NA	NA	NA
Dose 2	Arthralgia	NA	0.402	0.126	0.971	NA
	Chills	1	0.333	0.167	NA	NA
	Fatigue	0.777	0.23	0.028	0.494	0.971
	Fever	0.607	0.258	0.089	0.624	0.922
	Headache	0.754	0.299	0.023	0.432	0.992
	Malaise	0.999	NA	NA	0.001	NA
	Myalgia	0.814	0.349	0.016	0.386	0.935
	Nausea/vomiting	NA	NA	0.065	0.935	NA
	Reported systemic AE	0.777	0.296	0.009	0.447	0.971
	Vomiting	0.814	0.186	NA	NA	NA

3.5. Severe (Grade 3+) adverse events

Due to scarce event data, fewer networks including all five vaccines could be created for AEs of Grade 3 or higher severity. Of the 28 endpoints, 16 had one or more trials in which patients did not report any Grade 3 or higher AEs. An absence of any severe AE was particularly reported for VLA2001 and a placebo [Appendix B; Section 2.5]. On the other hand, mRNA-1273 reported AEs of Grade 3 or higher for all endpoints. In the end, most of the networks for local AEs compared only two vaccines, whereas larger networks could be created for systemic AEs. Overall, AZD1222 has the lowest SUCRA scores for all systemic AEs after the first dose but has the highest scores after the second dose. The vaccine with the lowest probability of being the best-tolerated vaccine for the largest number of systemic AEs after the second dose is mRNA-1273.

4. Discussion

Despite the availability of COVID-19 vaccines and the ongoing global pandemic, 24.2% of the adult population in the EU has never received a COVID-19 vaccine [4]. The novelty of this disease area means that direct comparative evidence on COVID-19 vaccine reactogenicity from extensive clinical trials is not yet available. However, more data on the relative safety of the currently approved vaccines can help to overcome vaccine hesitancy and improve vaccine uptake [12]. With this review and NMAs, we provide a novel comparison of the safety profiles of EMA-approved COVID-19 vaccines. For each analysis, four metrics to infer the relative reactogenicity between interventions were estimated for each vaccine and endpoint. The outcomes of this study will support policymakers in designing evidence-based vaccination promotion strategies as well as facilitate informed decision-making to help overcome the vaccine hesitancy that remains a barrier to vaccine uptake for COVID-19.

4.1. Key findings

We evaluated the safety profiles of five EMA-approved vaccines by comparing them for 56 endpoints. These endpoints correspond to 15 solicited AEs of any grade and Grade 3 or higher after the first and the second dose. Most of the endpoints were systemic AEs (65.5% of the endpoints). Overall, our results had higher certainty in the outcomes for AEs of any grade than for Grade 3 or higher, and systemic AEs outcomes were more certain than the local AEs outcomes. VLA2001 has the highest probability of being the least-reactogenic vaccine for 17 endpoints, especially for systemic AEs. Moreover, it was estimated to be significantly better tolerated than the other vaccines in four endpoints after the first dose: overall local AEs, overall systemic AEs, myalgia, and headache. For nine severe AEs, VLA2001 reported the absence of AEs and therefore could not be included in the network. AZD1222 had a higher certainty of being better tolerated than NVX-CoV2373, mRNA-1273, and BNT162b2 for several systemic AEs after the second dose, but the estimated odds ratios were lower than those of VLA2001. On the other hand, AZD1222 has a likelihood of being the least tolerated vaccine for systemic AEs after the second dose. The two-mRNA vaccines were significantly less likely to be the least reactogenic of the vaccines, with no significant differences between the two vaccines for most of the endpoints. The vaccine mRNA-1273 is most likely the least tolerated vaccine for systemic AEs after the second dose. In several endpoints, there was less certainty in the true estimates of the odds ratios of the vaccines when compared with a placebo. However, we can conclude that the inactivated whole virus vaccine VLA2001 has the highest likelihood of being the best-tolerated vaccine after both the first and second doses, with the lowest estimated odds ratio in 17 endpoints. Moreover, both mRNA vaccines are likely to be the most reactogenic vaccines.

4.2. Reactogenicity of vaccine platforms

We compared our results with the outcomes of two meta-analyses evaluating the safety profiles of COVID-19 vaccines [23,24]. The outcomes generated by these studies are in line with the results of our NMAs. In both our NMAs and the meta-analyses, inactivated virus vaccines were most likely to be the best-tolerated vaccines. Furthermore, the meta-analyses identified mRNA and viral vector vaccines as the most reactogenic, and mRNA vaccines were found to be less tolerated. The different outcomes of the meta-analyses and NMAs can be explained by the use of aggregated clinical trial data after both doses from all three phases of the clinical trial, and a smaller number of studies included in the NMAs than in the meta-analyses. Our findings confirm the role of vaccine platforms in determining the reactogenicity of COVID-19 vaccines. However, more research indirectly or directly comparing a larger number of trials and COVID-19 vaccines is needed to make a more concrete statement on the reactogenicity of platforms for COVID-19 vaccines. Additionally, the possible reasons behind the differences in the safety profiles of these vaccine technologies are left unaddressed by all these studies. A recent study by Zhang et al. aimed to assess the correlation between the reactogenicity and immunogenicity of two COVID-19 vaccines [25]. However, no correlation between the neutralizing antibody response and any systemic or local AEs was identified. Thus, the relationship between reactogenicity and immunogenicity remains unclear, highlighting a research gap in this area. Additionally, further research is necessary to elucidate the relationship between adverse events incidence and vaccine technology.

4.3. Adverse events of vaccines compared to placebo

There were 11 endpoints for which vaccines were estimated to be less reactogenic than placebo, with no significant differences estimated between the vaccines and placebo for most of these endpoints. This could be explained by the possible presence of bias due to the limited availability of RCT data and the estimation of relative estimates through a common comparator arm. Another plausible explanation could be a reported nocebo effect by patients taking the placebo. This finding agrees with a review by Howick et al. which showed that 49.1% of the patients in the placebo arms of RCTs experience AEs due to the drug [26]. Moreover, our findings are in line with a study by Haas et al. on COVID-19 vaccines in which placebo recipients experienced more systemic AEs than local AEs after both doses [27].

4.4. The approach of the analysis

The current analyses of evidence regarding COVID-19 vaccines were performed using the NMA methodology, which was deemed most appropriate as it allows the indirect comparison of all identified vaccines in a single analysis. A Bayesian framework was used to describe the model estimates and the uncertainty around their distributions in detail. This enabled us to make assertions as to the expected ranking of the vaccines without statistical significance, with the additional benefit that ranking outcomes are the most tangible and interpretable in terms of vaccine safety [28]. RE models were used as the base case to account for heterogeneity in baseline characteristics between the trial populations; moreover, the DIC values for RE and FE models were similar for almost all analyses [Appendix B, section 2.10], indicating that both analyses were comparable without a preferred model [29]. This study has generated new evidence on the reactogenicity of EMA-approved vaccines by performing analyses and generating comparative data of networks that include at least two treatments yet to be compared in a head-to-head clinical trial.

4.5. Limitations of the study

The extensiveness of the network for some of the analyses was limited by the presence of reported zero events, especially for AEs of Grade 3 or higher. For methodological reasons, trials in which the vaccine or placebo reported zero events could not be included in the networks. This implies that the high tolerability of some vaccines is not reflected in the analyses. Future research is needed on the impact of rare event data on estimates and on methods to incorporate extremely rare event data in NMAs. The presence of zero events can also be explained by the limited availability of phase 3 trial evidence due to the novelty of the disease area and the limited sample size of some of the included trials. In addition, the scarceness of the evidence affected our ability to check the consistency of the direct and indirect evidence in the networks, which could potentially impact the reliability and accuracy of our findings. To strengthen the robustness and generalizability of outcomes, future research could explore the utilization of a larger number of trials and individual patient data. Another limitation of this study concerns the type of safety data used to conduct the analyses. The search aimed to identify solicited adverse events after the first and second doses. In clinical trials, this kind of data is collected in a structured manner by requiring participants to record a prespecified set of adverse events, generally seven days after vaccination. However, solicited adverse events alone do not represent the full safety profile. Due to the heterogeneity in data collection of unsolicited adverse events and adverse events of special interest, a comparison for these endpoints was considered too prone to bias to be included in the analysis. Even though differences in the reporting of solicited adverse events between studies may arise, this data remains the best available source for comparing the reactogenicity of COVID-19 vaccines.Furthermore, the fact that the trials used in this study only included healthy individuals may impact the generalizability of the results toward an unhealthier part of the population.

5. Conclusions

The current study is the first evidence-synthesis study of COVID-19 vaccine reactogenicity. Our results emphasize the differences in the safety profiles of two-dose EMA-approved COVID-19 vaccines. Additional evidence on vaccines can be a useful tool to reduce misinformation, distrust and lack of clear information, as well as to promote primary vaccination, completion of the primary series, and booster vaccination. The reduced chance of experiencing an AE with some vaccines, such as the inactivated whole virus vaccine VLA2001, might help to overcome vaccine hesitancy in population groups with concerns about the side effects of vaccines. Future studies could explore this finding further through the comparison of additional COVID-19 vaccines as more clinical trial evidence is produced.

Declaration of Interest

G Tiozzo, T Louwsma, S Konings are employees of Asc Academics. Asc Academics has received consultancy fees for this project from Valneva SE. G Vondeling and JP Gomez are paid employees at Valneva SE. M Postma had received funding from Janssen-Cilag B.V for projects unrelated to the current study. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or material discussed in the manuscript apart from those disclosed.

Reviewer disclosures

Peer reviewers on this manuscript have received an honorarium for their review work. Peer reviewers on this manuscript have no other relevant financial or other relationships to disclose.

Author Contributions

Conceptualization, G.T., T.L., S.K., G.V. and J.P.G.; methodology, G.T., T.L. and S.K.; software, S.K.; formal analysis, G.T. and S.K.; data curation, G.T. and S.K.; writing – original draft preparation, G.T. and T.L.; writing – review and editing, S.K., G.V., J.P.G., M.P and R.F.; visualization, S.K.; supervision, G.V., J.P.G., M.P. and R.F.; project administration, G.T. and T.L. All authors have read and agreed to the published version of the manuscript.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/14760584.2023.2208216.

Additional information

Funding

This paper was funded by Valneva SE. Valneva SE had no role in the set-up of the study or interpretation of the results.