Foundations for improved vaccine correlate of risk analysis using positive-unlabeled learning

ABSTRACT

Insights into mechanisms of protection afforded by vaccine efficacy field trials can be complicated by both low rates of exposure and protection. However, these barriers do not preclude the discovery of correlates of reduced risk (CoR) of infection, which are a critical first step in defining correlates of protection (CoP). Given the significant investment in large-scale human vaccine efficacy trials and immunogenicity data collected to support CoR discovery, novel approaches for analyzing efficacy trials to optimally support discovery of CoP are critically needed. By simulating immunological data and evaluating several machine learning approaches, this study lays the groundwork for deploying Positive/Unlabeled (P/U) learning methods, which are designed to differentiate between two groups in cases where only one group has a definitive label and the other remains ambiguous. This description applies to case–control analysis designs for field trials of vaccine efficacy: infected subjects, or cases, are by definition unprotected, whereas uninfected subjects, or controls, may have been either protected or unprotected but simply never exposed. Here, we investigate the value of applying P/U learning to classify study subjects using model immunogenicity data based on predicted protection status in order to support new insights into mechanisms of vaccine-mediated protection from infection. We demonstrate that P/U learning methods can reliably infer protection status, supporting the discovery of simulated CoP that are not observed in conventional comparisons of infection status cases and controls, and we propose next steps necessary for the practical deployment of this novel approach to correlate discovery.

KEYWORDS:

• Machine learning
• vaccine
• efficacy trials
• correlates of protection
• immunity

Introduction

The fundamental goal of late-stage vaccine clinical field trials is to determine efficacy. Such studies expand safety data to a broader population in support of widespread use. Beyond this primary motivation, for effective vaccines, they also offer the potential to derive biological insights into potential mechanisms of protection based on the identification of correlates of infection risk (CoR). Whether directly responsible for protection or reliable surrogates for mechanisms that were not directly assessed, immune markers statistically associated with infection status endpoints provide key signposts in further research and development, and can also suggest novel direct interventions.^1,2 However, the ability to confidently and completely identify CoR depends upon many factors, including the level of protection observed, the magnitude of differences in immunogenicity data between infected and uninfected subjects, and study power. Study power further depends on factors such as study group sample size and the dynamic range of immune response markers.

HIV-1 vaccine efficacy trials in particular have been complicated by low rates of exposure and low levels of protection.³ In spite of these barriers, several CoR have been defined using case–control study designs, in which the profiles of infected (case) and uninfected (control) subjects are compared.^4,5 However, as opposed to controlled human challenge studies, exposure and infection are not strictly associated in field trials. Subjects who are infected have definitively been exposed to the virus, while subjects who are uninfected may have been either exposed and protected or simply never exposed. Thus, the “control” class in case–control analysis is split between individuals who lack protective immunity and were not exposed to the virus and individuals who have protective immunity and either were or were not exposed to the virus. The resulting dilution of protected subjects with unprotected but unexposed subjects reduces the likelihood of identifying CoR. The ability to robustly infer the protection status of uninfected individuals would thus improve the ability to identify CoR and fill critical gaps in knowledge regarding immune responses that contribute to protection. Such inferences could provide substantial improvements in CoR discovery for trials performed with low efficacy and/or low rates of pathogen exposure.

Machine learning (ML) methods are statistical methods that seek to identify meaningful patterns in data.^6,7 These approaches offer the potential to identify groups of subjects across which relevant immunological differences are detected and have been employed previously in CoR analysis studies. Examples of these applications span various infectious diseases, input data types, and experimental designs.^8–19 Case–control experimental designs seek to compare the response profiles for individual variables between case (infected) and control (uninfected) vaccine recipients. Such comparisons, however, are not well-suited to “wide” data, which is comprised of many immunogenicity parameters measured across relatively few subjects,^20–23 and instead typically rely on pre-selection of a small number of candidate measures selected to maintain power.²⁴ ML methods, with strong analytical negative controls such as cross-validation and permutation, can provide a sound approach to identify CoR in wide data: these approaches have identified consistent HIV vaccine CoR in animal model studies and human vaccine efficacy trials evaluating diverse vaccine regimens.^9,10,25

ML methods can be broadly classified into two categories: unsupervised and supervised. Unsupervised ML methods do not use class labels (e.g. “infected” and “uninfected”) to separate data. Instead, these methods rely on the internal structure of the data to identify subjects with shared response attributes. Supervised ML methods, however, make use of known class labels to identify characteristics that discriminate between classes.²⁶

In typical analysis of case–control studies, infection status is used as a proxy for protection status when performing CoR analysis. As outlined previously, however, being uninfected does not map one-to-one with being protected. Supervised methods work best when they are given data that is correctly and completely labeled, but this imperfect mapping of infection status to protection status leads to sub-optimal subject classification and poses a limit to CoR identification. In contrast, unsupervised methods are free to classify the data without being influenced by these imperfect infection status labels, but in the process they ignore what we do know about infected subjects: that they are unprotected.²⁷ Neither class of ML methods takes full advantage of the available information.

These limitations warrant a different approach, one that takes more complete advantage of the available data. To offer an analytical alternative to the discovery of CoR, our goal becomes to define a way to reclassify subjects as “was/would have been protected” and “was/would have been unprotected,” if all subjects had been exposed.

A potential solution to this problem is semi-supervised machine learning (SSL), which uses a limited number of labeled samples as a starting point for building a classifier, and then attempts to fit the remaining data.²⁸ SSL approaches thus resemble supervised ones initially, with a few examples of labeled data at the outset, before being turned loose, unsupervised, to “figure out” the rest of the samples. Most SSL methods require examples from both classes, which in the context of vaccine efficacy field trial data would mean examples of both protected and unprotected subjects. Unfortunately, these protection status labels, which would best enable correlate discovery, are precisely what vaccine efficacy field trial data lack.

A sub-class of SSL methods that is well-suited for this task is Positive/Unlabeled (P/U) Learning. These methods are trained using data for a single class (the “positive” data) and then used to classify the remaining data (the “unlabeled” data).^29,30 In contrast to traditional SSL methods, P/U methods are centered on learning what one class “looks like.” These approaches directly map to the infection status label problem encountered in efficacy trials: one set of subjects are known to have been unprotected (positive/infected), and the uninfected group is presumed to be comprised of a mixture of protected and unprotected.

Here, using a simplified model data set, we explore the ability of positive unlabeled ML methods to contribute to CoR discovery.

Methodology

Data simulation

Immunogenicity data for two classes, unprotected and protected subjects, were simulated. Unprotected class subject data was generated by simulating an initial multivariate normal dataset of three independent variables and 5n (n = 1000) data points with standard deviation sd. Excess (5×) data points were simulated to ensure a large available pool of data points from which data points could be sampled with reduced likelihood of overlap. These variables serve as modeled immunogenicity “features.” Assuming that vaccine-induced responses would be greater for each of the response features among protected subjects, protected class subject data was generated by simulating a multivariate normal dataset of three independent variables and 5n data points such that the centroid of the protected data set lay at a distance d from the centroid of the unprotected dataset in each feature dimension. The value of distance d was varied from 0 to 5 sd in order to consider different effect sizes in terms of immunogenicity profile differences. Thus, each of the three simulated immunogenicity “feature” values correlated with protection status.

By definition, the infected subject class is unprotected. Hence, infected subject data was generated by sampling without replacement from the unprotected dataset for a total of n data points. Uninfected subject class data were generated by sampling without replacement from the unprotected and protected datasets for a total of n data points. Multiple such datasets were generated by sampling different proportions from the unprotected and protected datasets. This proportion represents the simulated efficacy (e) of the vaccine and was varied from 0.05 to 0.5. The uninfected data set thus contained n(e) number of points from the protected dataset and n(1-e) points from the unprotected dataset. For example, an efficacy of 0.3 resulted in 30% of the uninfected subject data being sampled from the protected subject group, with the remaining 70% of the n data points sampled from the unprotected subject group. Conditions resembling the efficacy of a vaccine above 50% were not considered because at such higher efficacy values, most of the data points from the uninfected population would belong to the uninfected protected dataset, and the value of excluding the uninfected unprotected subjects to support identification of correlates of risk is expected to be limited. This data generation process was repeated across 10 different random seeds.

To investigate the effect of sample size on correlate discovery, the ability of machine learning methods to define potential correlates of protection for smaller datasets was tested by simulating datasets of smaller sample size (n = 500, 250, 50). These lower bounds represent numbers of infection cases similar to those reported in HIV vaccine efficacy trials

Consistent with our goal to explore initial proof of concept, we have not considered analysis of a placebo group, longitudinal immunogenicity data, information regarding time of infection, impacts of censoring, or adjustments of individual subject risk based on clinical or other co-variates. The modeled data sets were complete and did not incorporate imputation or missing values or exclusion of subjects. Additionally, protection status was treated as a binary classification, as is risk, rather than the continuous variables they are expected to be in reality.

Training and testing using different classification methods

Training and testing for inference accuracy was performed using feature 1 and feature 2 data. The third feature was held out to support evaluation of the ability of protection status inferences to support CoP discovery. This choice was made to avoid bias associated with evaluating the model based on data that was used for classification method training.

K-means

The K-means clustering method was used to identify two clusters using feature 1 and feature 2 data of uninfected and infected subjects without any group labels (Supplemental Figure S1A). The R package ‘stats’ was used to carry out k-means clustering.³¹ The third feature was held out for model evaluation. Because K-means clustering is an unsupervised ML method, it cannot independently identify which of the two clusters is the protected cluster and unprotected cluster, leading to two possibilities when labeling the clusters. The final class labels were based on which classification led to higher model accuracy, as defined by the fraction of correct predictions ($\frac{\mathit{TP}+\mathit{TN}}{\left(\mathit{TP}+\mathit{TN}+\mathit{FP}+\mathit{FN}\right)}$). After clustering was completed, inter-cluster variance (SSB) and intra-cluster variance (SSW) were calculated. The goodness of fit of the model was evaluated ($\frac{\mathit{SSB}}{\mathit{SSB}+\mathit{SSW}}$). Based on how the K-means algorithm works, in this case, the algorithm would classify the input data points into two clusters. Hence, as a control, goodness of fit of the model was evaluated when distributions of the same class were used as an input. If the calculated goodness of fit for the tested model did not exceed the goodness-of-fit value of this control condition, the model’s cluster assignments were not accepted. In these cases, it was determined that the model did not characterize the data sufficiently better than a single cluster.

Support Vector Machine (SVM)

An SVM classifier was trained using features 1 and 2 with the protection status labels so as to serve as a positive control. Given the nature of the data and its progression along the XYZ diagonal, the SVM was trained using a linear kernel (Supplemental Figure S1A). The R package ‘e1071’ was used to construct the SVM,³² which was optimized for class separation by maximizing the margin between the classes’ closest points using C-type classification to set the hyperplane. To reduce the potential of overfitting while balancing the need to ensure a sufficient number of protected subjects were included among the uninfected, repeated 3-fold cross validation was performed, and the mean values across test set folds were considered as the values for the particular evaluation parameter at given seed, distance and efficacy value across replicates.

Reliable Negative (RN) SVM

Infected subjects were considered to be the positive class and uninfected subjects were considered unlabeled. It was assumed that the unlabeled (uninfected) subjects most distinct from the positive (infected) class belonged to the protected class and thus represented “reliable negatives.” The number of reliable negatives identified was half the number of the protected subjects in the unlabeled dataset. These reliable negatives were identified based on the Euclidean distance observed in feature 1 and feature 2 values. This number was set to balance the goal of maintaining a high likelihood of identifying reliable negatives belonging to the protected data set while providing sufficient data points to train the SVM. The positive (unprotected) class and the identified reliable negative (protected) class together formed the training set. The SVM was trained using feature 1 and feature 2 data of the training set, again using a linear kernel (Supplemental Figure S1B).

Model evaluation

The accuracy of each model described above was evaluated in terms of Matthews’ Correlation Coefficient (MCC). Predicted protection status labels were compared with actual protection status labels to identify true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN) (Supplemental Table S1). MCC was calculated using the formula $\frac{\left[\left(\mathit{TP}\ast \mathit{TN}\right)-\left(\mathit{FP}\ast \mathit{FN}\right)\right]}{\mathit{sqrt}\left[\left(\mathit{TP}+\mathit{FP}\right)\ast \left(\mathit{TP}+\mathit{FN}\right)\ast \left(\mathit{TN}+\mathit{FP}\right)\ast \left(\mathit{TN}+\mathit{FN}\right)\right]}$, as defined by using the package ‘caret.’³³ Negative predictive value i.e. protection predictive values in this case ($\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FN}}$), specificity ($\frac{\mathit{TN}}{\mathit{TN}+\mathit{FP}})$, sensitivity ($\frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}})$ were also calculated.³⁴ Reported values represent averages across replicates.

Correlate discovery

The ability of infection status labels, actual protection status labels, and the inferred protection status predictions to define correlates of protection was evaluated for feature 3 using a two-sided Wilcoxon-Mann-Whitney U-test with the null hypothesis that the two groups being compared were identical. The R package ‘stats’ was used to perform the Wilcoxon-Mann-Whitney U-test.³¹ Cliff’s delta values for feature 3 were likewise calculated to determine the effect size of group differences. The reported values represent averages across replicates. The R package ‘effsize’ was used to calculate the Cliff’s delta.³⁵ To define the robustness of the RN-SVM model in terms of risk of false discovery, Cliff’s delta was also calculated after the values of feature 3 were scrambled.

Results

Modeling immunogenicity data for variably effective and immunogenic vaccines

Given the nature of human vaccine efficacy field trials, exposure and infection are not strictly associated (Figure 1a,b). Case–control study experimental designs aim to compare features of subjects with the disease or condition under study (cases) and a similar group of people who do not (controls). In vaccine efficacy field trials, case–control designs typically compare immune response features of infected vaccine recipients with those observed among uninfected vaccine recipients (Figure 1c). However, unless all vaccine recipients are exposed or the vaccine is 100% effective, the uninfected class is composed of subjects that exhibit a protective response (UP-uninfected protected) and those that do not (UU-uninfected unprotected) but remained uninfected because they were not exposed (Figure 1c). The ability to resolve CoR depends upon both the level of protection observed (efficacy) and the magnitude of differences in immunogenicity data between protected and unprotected subjects (effect size/distance) (Figure 2a).

Figure 1. The labeling problem in correlates analysis. (a) In field trials of vaccine efficacy with pathogens that are not universally prevalent, the infected study group is comprised of vaccine recipients who were definitively exposed to the virus, while the uninfected study group consists of both vaccine recipients who were and others who were not exposed to the virus. (b) The ideal information needed to support discovery of Correlates of Protection (CoP) is the true protection status of exposed individuals. (c) Case-control analysis based on infection status defines Correlates of Risk (CoR) based on comparison of immune response features of infected vaccine recipients (gray) with those observed among uninfected vaccine recipients (purple). However, unless all vaccine recipients were exposed and/or the vaccine is completely effective, the uninfected class is comprised of subjects that exhibit a protective response (red, UP-uninfected protected) and those that do not (blue, UU-uninfected unprotected), but remained uninfected because they were not exposed. While CoR discovery power is improved by increasing the sample size of the uninfected class, it remains reduced by the inclusion of UU subjects.

Figure 2. Graphical illustration of simulated data. (a) The ability to resolve CoR depend critically on the overall efficacy (left) of the vaccine, which is the fraction of vaccine recipients that mounted a protective response, and the effect size (right), which is the magnitude and spread of response distributions between protected (red) and unprotected (blue) participants, visualized for two exemplary variables. (b) To model immunogenicity data for differently distinct and effective vaccines, two populations (U and P) were assigned three normally distributed, uncorrelated immune response parameters (μ) that varied in distance (d) from each other defined in terms of standard deviations from the mean (SD). Parameter space ranged from distances of 0.0–5.0 SD and efficacy (e) of 5–50%.

To address the impact of each of these parameters, immunogenicity data consisting of three independent, normally distributed features were simulated for protected and unprotected populations that differed by varying response magnitude distances (d) (Figure 2b). To reflect the infection status labels available in real-world vaccine efficacy field trials, these protected and unprotected subject profiles were then sampled at varying degrees of efficacy (e) up to 50% to generate a dataset representing the uninfected class. By definition, the infected class was generated by sampling only from the unprotected subject class profiles. Collectively, uninfected and infected subject data profiles were generated for a set of 80 different data compositions, each simulated across ten different random seeds. This set included eight different efficacy and ten different distance measures, which were designed to cover the condition space in which case–control correlate analysis based on infection status alone has reduced power and P/U learning methods might add value to CoR discovery.

Inferences of protection status can identify CoR not detected using infection status

Simulated data was used to evaluate the effect of efficacy and distance on case–control CoR analysis based on infection status. Over a wide range of efficacy and distance measures, comparisons between variable distributions for infected versus uninfected subjects and between infected and uninfected protected (UP) subjects were made. Conditions reflecting 15% and 30% efficacy rates separated by 0.5 and 2d were modeled and graphed for a single seed (Figure 3, biplots). Splitting the uninfected class into uninfected protected and uninfected unprotected improved in power and confidence in the correlates (Figure 3, violin plots). This analysis represents the gain one would expect from perfect protection label inferences.

Figure 3. Disambiguation of UP and UU improves power and confidence in correlates. Variable (V) biplots of feature 1 and feature 2 for infected (gray) and uninfected (purple) vaccine recipients (top) and for uninfected unprotected (blue) and uninfected protected (red) vaccine recipients (bottom) for distributions with variable efficacy (15% and 30% efficacy) and effect size (0.5 SD and 2 SD). Violin plots show group distributions for V2 (p values by ANOVA with Tukey’s MHC; *p < .05; **p < .01; ****p < .0001).

Setting baseline for ability of ML methods to classify data based on protection status

Before assessing the performance of P/U ML techniques to supplement correlates discovery, it was essential to establish baseline expectations for case–control (infected-uninfected) analysis based on infection status and the accuracy of fully supervised and unsupervised machine learning classification. To accomplish this goal, K-means (unsupervised ML) clustering was performed and SVM (supervised ML) models were trained using two of the three immunogenicity features in the dataset and evaluated for classification performance. The third feature was left out to be used for model evaluation.

The performance of K-means sets a lower baseline for expectation from positive unlabeled ML models as it represents the ability of ML to classify data points according to protection status without considering any outcome information. The K-means algorithm was employed on both infected and uninfected subjects for each dataset. As the distance between the underlying protected and unprotected subject immunogenicity profile data increased, so did the ability of this unsupervised approach to discriminate between protected and unprotected subjects, as defined by Matthew’s Correlation Coefficient (Figure 4a). Similarly, performance improved as vaccine efficacy was increased. Classification performance was also defined using Protection Predictive Value, the proportion of people predicted by the algorithm as protected which are truly protected (Figure 4b), Specificity, the probability of being predicted as protected, conditioned on truly being protected (Figure 4c) and Sensitivity, the probability of being predicted as unprotected, conditioned on truly being unprotected (Figure 4d). Performance was also compared to the benchmark set by a case–control analysis based on infection status.

Figure 4. Accuracy of protection status label inferences. (a–d) Evaluation of ML algorithms to accurately predict protection status. Mathew’s correlation coefficient (MCC) (a), Protection Predictive Value (b), Specificity (c) and Sensitivity (d) calculated using data with protection status labels inferred from unsupervised analysis (K-means), supervised analysis (SVM) or P/U ML (RN-SVM) as compared to case-control analysis (infection status labels) or ground truth (actual protection status). Performance metrics are presented for input data sets with variable immunogenicity response variable distance (x-axis) and efficacy (columns). Error bars represent standard deviation across replicates. MCC could not be calculated when all samples were predicted to a single class.

Given the ability to infer protection class, the ability of K-means classifications to support discovery of the held out third feature as a CoR was assessed. The effect sizes assessed by Cliff’s delta (Figure 5a) and statistical significance of differences between predicted groups of measurement evaluated by Mann-Whitney U-test (Figure 5b) increased as well. Thus, it became easier for the model to produce clusters that closely resembled protection status as distance and efficacy increased. Again, confidence in CoR discovery using these approaches was compared to ground truth protection status labels and infection status labels for each simulated immunogenicity data condition.

Figure 5. Strength and confidence of correlate discovery from inferred protection status. (a,b) Evaluation of effect size and confidence in correlate identification. Cliff’s delta (a) and statistical significance (b) of the held-out immunogenicity feature calculated from inferred protection status labels inferred from unsupervised analysis (K-means), supervised analysis (SVM) or P/U ML (RN-SVM), as compared to case-control analysis labels (infection status labels) or ground truth (actual protection status). Correlate metrics are presented for input data sets with variable immunogenicity response variable distance (x-axis) and efficacy (columns). Statistical significance was calculated using a two-sided Wilcoxon-Mann-Whitney U test with the null hypothesis that the two groups being compared were identical. Error bars represent standard deviation across replicates. Values could not be calculated when all samples were predicted to a single class.

The performance of SVM sets the upper bound expected from positive unlabeled ML models as it represents the ability of ML to classify data points according to protection status. An SVM model was trained using protection status labels with three-fold cross-validation. As with K-means, SVM afforded increases in both classification accuracy and CoR discovery with increasing distance and efficacy (Figure 4, Supplemental Figure S2). Under most distance and efficacy conditions, the performance of SVM exceeded that of K-means. The gap between performance of K-means and SVM as well as their improvements relative to infection status labels commonly used in the analysis of case–control studies supported the potential for P/U ML methods to offer a useful additional analytical approach.

Semi-supervised learning improves the ability to discover CoR

To directly assess the value of P/U ML to contribute to CoR discovery, a reliable negative (RN) method was employed using two of the three simulated response features. RN SVM takes a two-step approach by first identifying the unlabeled (uninfected) samples most distinct from the positive (infected) class under the hypothesis that these samples represent the protected class.^29,36 Then, these “reliable negative” samples are used to classify the protection status of the remaining unlabeled (uninfected) samples.

Protection status labels generated by RN-SVM were more accurate than labels based on infection status for most conditions tested, as demonstrated by Matthews Correlation Coefficient, Protection Predictive Value, Specificity, and Sensitivity evaluations (Figure 4a–d). These results suggest that this approach can successfully identify uninfected subjects that were likely to be infected had they been exposed. To test the resulting improvement in power to identify CoP, the statistical significance (Wilcoxon Mann Whitney U-test) and effect size (Cliff’s Delta) of differences for newly labeled samples were evaluated for the held-out feature and demonstrated a robust ability to improve identification of CoP (Figure 5).

In general, RN-SVM outperformed K-means at lower distance and efficacy values, and was equivalent to that of SVM at higher values (Figures 4 and 5, Supplemental Figure S2). In almost all conditions, RN-SVM provided better identification of uninfected unprotected and uninfected protected data points as compared to infection-status-based analysis of case–control data. Thus, usage of RN-SVM would help to identify correlates not identified through standard analysis.

Efficacy plays an important role in inference accuracy in datasets of smaller size

Beyond efficacy and distance, the number of samples available is also expected to impact the likelihood and confidence of CoR identification. While a useful upper end, n = 1000 represents a greater number of infection cases than are often observed in clinical trials. Because considerably smaller numbers of infection cases have supported identification of CoR for vaccines, with low efficacy using case–control designs based on infection status,^37,38 we wished to begin to explore anticipated limits to method utility posed by study size. Hence, data simulated for studies in which 500, 250, and 50 infection cases were observed was analyzed. This lower bound represents numbers of infection cases similar to those reported in HIV vaccine efficacy trials. In general, the model performance of RN-SVM when classifying data samples for smaller datasets was similar to that observed at n = 1000 (Supplemental Figures S3 and S4). While efficacy played a more dominant role in determining model performance as the number of available true protected points in the infected dataset became limiting, this analysis showed that even small studies may benefit from the application of P/U ML approaches for correlate discovery (Figure 6).

Figure 6. Effect of sample size on protection status inferences and correlates discovery. (a) Classification accuracy (MCC) of protection status labels predicted by RN-SVM for different sample sizes (n = 50, 250, 500, 1000). (b) Correlate effect size (Cliff’s delta) for the held-out variable. Metrics are presented for input data sets with variable immunogenicity response distance (x-axis) and efficacy (columns) and sample sizes. Error bars represent standard deviation across replicates. Values could not be calculated when all samples were predicted to a single class.

Robustness of reliable negative SVM

Feature 3 was scrambled and Cliff delta re-assessed as a means to test the robustness of the modeling approach with respect to the risk of false discovery. This analysis demonstrated that scrambling of data points resulted in low Cliff delta values (Figure 7), suggesting a low risk of false discovery when infection status inferences are applied to discovery of correlates among features not considered in model training.

Figure 7. Robustness of Reliable Negative SVM algorithm evaluated using correlate effect size. Correlate effect size (Cliff’s delta) for the held-out and scrambled held out variable calculated based on inferred protection status label for RN-SVM algorithm as compared to ground truth protection status. Metrics are presented for input data sets with variable immunogenicity response distance (x-axis) and efficacy (columns). Each row represents the different sample sizes. Error bars represent standard deviation across replicates.

Discussion

Vaccine trials are absolutely necessary to establish the ability of a vaccine to protect at-risk populations from infectious diseases. While more direct study designs, such as challenge, ring trials, household, or partner studies, among other designs, can condition analysis on exposure, field trials, the most rigorous means to assess vaccine efficacy, rely on natural exposure to the pathogen of interest, which may be a relatively infrequent event.³⁹ Given a low rate of exposure and sample size, limits to the infection status information available to support CoR analysis reduce power to both define vaccine efficacy and identify CoR in comparative analyses of infection cases and controls, despite the potential to be refined by incorporation of baseline prognostic or exposure risk variables. Imperfect efficacy reduces statistical power further.^40,41 The combined effect of these constraints has sometimes resulted in trial designs in which relatively few immunological endpoints are pre-selected prospectively for correlates analysis.^4,24–45 This strategic prioritization maximizes power by design but can leave a knowledge gap of untapped insights into mechanisms of protection.

Beyond these practical constraints, the real-world information from trials defines subjects by infection status. However, because the uninfected class is expected to be composed of protected as well as unexposed individuals, its ability to define correlates is reduced as compared to the ideal labels of protection status. We hypothesized that this “labeling problem” warrants consideration of a different approach. So, our goal was to comparatively evaluate the means to reclassify subjects as “was/would have been protected” and “was/would have been unprotected” if all subjects had been exposed, in order to improve CoR detection. To this end, we evaluated the potential of a Positive Unlabeled Machine Learning using simulated data across a wide range of effect size (0–5 SD) and efficacy (0.05–0.5) values. Efficacy rates higher than 50% were not considered, as under these circumstances most of the data points from the uninfected population belong to an uninfected protected dataset.

To this end, K-means clustering appears to offer some useful discriminative capability. As a fully unsupervised method, this approach can be considered to set a baseline expectation as to how well a model can perform when it does not consider protection/infection outcomes at all. The fact that it can outperform the infection status-based comparisons of case–control study designs under a variety of input data conditions suggests its potential value to correlate discovery. In contrast, fully supervised classification based on fully labeled data represents the upper bound of performance that might be expected from P/U learning approaches. Here, under conditions with relatively higher efficacy and distance, one such method (SVM) approached ground truth.

RN SVM P/U learning, one of the many reported semi-supervised approaches, is centered on learning what the definitively labeled (infected) class “looks like” and then classifying the remaining data as “infected-like” and not “infected-like,” representing the putatively protected class. This approach typically exhibited performance falling between fully unsupervised and fully supervised approaches in each of the performance metrics evaluated. Similarly, its ability to identify and be confident in CoP fell between these comparators. Importantly, however, both K-means and RN SVM outperformed the standard CoR identification approach of comparing responses between infected and uninfected subjects, and in some cases approached ground truth. Collectively, these results suggest the potential utility of both unsupervised and P/U learning approaches to generate hypotheses regarding candidate CoP. For small values of distance and efficacy, SVM often classified all samples to be members of the same class. However, K-means and RN-SVM still yielded class predictions under these circumstances, raising concerns as to the possibility of overfitting and false discovery of CoR. Reassuringly, statistically significant differences in the held-out immunogenicity feature between groups that were too similar to be differentiated by supervised methods were not observed, though they may be expected to result from analysis of features considered in K-means and RN-SVM classification. To safeguard against such false discoveries in the absence of held out data, comparisons to results observed in the context of infection label permutation may be useful.

Limitations to the present analysis include the simplified data streams employed: variables followed normal distributions, and a set of only two features was used to support model inferences. Though not correlated with each other, both were distinct from the distribution among the unprotected subjects. In real-world immunogenicity data, hundreds of response variables may be measured, many of which may not relate to protection at all, and those that do would be expected to exhibit different “distances” from their distributions in infected subjects. Selecting features to train the model might be important in such cases. Additionally, some fraction of subjects may present as non-responsive to the vaccine. Here, vaccine “take” was modeled as being complete. Under what conditions these attributes might generally reduce or enhance modeling performance remains to be determined. To this end, further synthetic data sets that explore this complexity can be simulated. Beyond further efforts with synthetic data, inferences of protection status after selective blinding of vaccine efficacy studies with other designs, such as challenge experiments for which ground truth protection status is known, could provide further support for the potential utility of protection inferences to support candidate CoP identification. Similarly, for disease contexts in which biomarkers of exposure exist, these markers could be explored as a means to either validate the accuracy of or better inform protection status inferences. After validation on more complex datasets, this approach could be deployed in the context of case–control experimental designs as a complement to other analytical approaches to identify candidate correlates in vaccine trials with limited efficacy. Application of these approaches would support the potential to extract additional insights from trials that fail to meet endpoint efficacy criteria or to prioritize subjects among the controls for further analysis. While we have framed our interest in the context of prospective, interventional vaccine efficacy studies, should further work support utility, we envision that extension to other trial designs is feasible.

Overall, based on the simplified analysis presented here, P/U ML methods show potential to solve the protection status labeling problem inherent in vaccine efficacy field trials, motivating the follow-up proposed above. Further, alternative and more sophisticated P/U ML methods such as iterative SVM, SPY-SVM, and bagging SVM can be explored for their ability to solve the protection status labeling problem. Given our results, we hypothesize that they could lead to increased confidence in correlates identified from comparisons based on infection status alone as well as identification of new putative correlates of protection, particularly in the context of smaller sample sizes and low efficacy, and even in the context of trials that have failed to meet endpoint efficacy criteria. Additionally, should further work support generalized utility, application of the proposed approach has the potential to allow studies with shorter duration or fewer participants to yield the same candidate correlates as more resource-intensive designs. Correlates identified by protection status inferences rightly represent hypotheses regarding potential CoP; that this same limitation applies to traditional CoR. While only pathogen challenge can identify definitive CoP, the prospect of deriving greater benefit from experimental studies in humans motivates pushing the boundaries of trial analysis, particularly as immunological data sets become more and more sophisticated, and inference methods improve in accuracy, widespread utility, and scientific acceptance.

Author contribution

Investigation: N.K. and K.M. Coding: N.K. and K.M.; Data analysis and data visualization: N.K.; Writing-original draft: N.K. and K.M.; Writing-reviewing and editing: all authors. Supervision: M.E.A.; Conceptualization and funding: M.E.A.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplemental data

Supplemental data for this article can be accessed on the publisher’s website at https://doi.org/10.1080/21645515.2023.2204020.

Additional information

Funding

The study was supported by the National Institutes of Health NIAID under grants [R56AI165448 and P01AI162242].