F*, an interpretable transformation of the F measure, equates to the critical success index

ABSTRACT

When analysing large datasets, suitable outcome measures are needed. For example, in clinical medicine, we often want to know how well a test identifies people with or without a particular diagnosis. Traditionally, this evaluation has been calculated in terms of Sensitivity and Specificity, and Positive and Negative Predictive Values (PPV, NPV), but many other test outcome measures are available. Recently, a measure termed F* was described, as an interpretable transformation of the F measure. The F measure is a unitary metric calculated as the harmonic mean of Sensitivity and PPV. We show that F* is in fact identical to a previously described measure which is monotonically related to the F measure, and which has been variously named in previous publications as the ratio of verification, the Jaccard similarity measure or index, the threat score, the Tanimoto index, and the critical success index (CSI). The origins of these different terms, which date from the late 19th–20th century, in different scientific disciplines (weather forecasting, ecology, machine learning) may explain the repeated independent redescription of this measure. More recently, the advantages of applying CSI, a metric routinely used to verify weather forecast accuracy, have been demonstrated in medical diagnostic accuracy studies.

PLAIN LANGUAGE SUMMARY

When medical studies have collected large amounts of health data on people classified in the data as having a particular disease, it may often be difficult to “see the wood from the trees” in terms of being sure that that disease classification is actually correct for each person in the dataset. Therefore, it is important that researchers first verify the disease classification accuracy of the data against an external “gold standard” source of information holding the correct diagnosis. This is usually a patient’s medical records. A comparison of the health data disease classification against the medical records diagnosis generates various measures of disease classification accuracy, all of which have advantages and disadvantages as verification measures. Here, we examine a verification measure which has recently been characterised as “F*”. It turns out, in fact, as shown by elementary mathematical techniques, that F* is not novel; having been previously described under different names for more than a century in various scientific disciplines (weather forecasting, ecology, machine learning). Our preference is to call this measure the “critical success index” (CSI). It is a single measure which, in clinical medicine, combines information about the classification performance with respect to both test and patient. It is easy to calculate, simple to use, and to interpret (higher values are better).

KEYWORDS:

• Binary classification
• critical success index
• F measure
• F*
• Jaccard index

1. Introduction

We live in the age of “Big Data”. However, the question of how best to “mine” these data to extract what “really matters”, in terms of correct identification of items being searched for, or guiding future actions or interventions based on the aggregate data, remains to be resolved. Is there a single measure, or a minimum set of measures, which can meaningfully evaluate large amounts of data? What methods should be used?

For example, in medical practice, diagnostic and screening accuracy studies of clinical investigations or, increasingly, of diagnostic algorithms are often reported using some of the many measures which may be derived from a standard 2×2 contingency table. This cross-classifies actual and predicted classes of the diagnosis [1]. Typically, these outcomes include paired measures of diagnostic or screening classification by the test, Sensitivity (Sens) and Specificity (Spec), and paired measures of classification of the patient, Positive and Negative Predictive Values (PPV, NPV). All of these are recognised to have potential shortcomings as outcome measures. A single, unitary or global measure which combines information relevant to both test and patient classification may thus be desirable.

One such measure is the F measure, defined as the harmonic mean of PPV (also known as precision) and Sens (also known as recall) [2]. This measure corresponds to the coefficient previously described by Dice [3], and independently by Sørensen [4], hence sometimes known as the Dice coefficient or the Sørensen-Dice coefficient, and to the approach advocated by van Rijsbergen [5].

In terms of the base data from a 2×2 contingency table containing N elements with four degrees of freedom, the following four classifications may be made: TP = true positive (or a “hit” or “true hit”), FP = false positive (or “false hit” or “false alarm”), FN = false negative (or a “miss”), and TN = true negative (or a “non-event” or “correct rejection”). From these: $\begin{array}{r}\text{F}=2\text{.TP/}(2\text{.TP}+\text{FP}+\text{FN})\end{array}$This may also be expressed in terms of PPV and Sens, as: $\begin{array}{rl}\text{F}& =2\text{.PPV.Sens/}(\text{PPV}+\text{Sens})\\ & =2/[1/\text{Sens}+1/\text{PPV}]\end{array}$that is, F is the harmonic mean of PPV and Sens.

More recently, Hand et al. [6] have described “F*” as “an interpretable transformation of the F measure” [6], where: $\begin{array}{r}{\mathrm{F}}^{\ast }=\text{F}/(2-\text{F})\end{array}$They acknowledge that “researchers may recognise this as the Jaccard coefficient widely used in areas where TN may not be relevant”. As will be shown, these authors have indeed redescribed an already existing binary classification measure, first reported in the late nineteenth century as the ratio of verification in the context of forecasting tornadoes [7], and subsequently as the Jaccard index or similarity coefficient (J) [8], the threat score [9], the Tanimoto index [10], and later still as the critical success index (CSI) [11,12]. Here we use the latter terminology.

2. Mathematical proofs of identity of F* and CSI

The identity of F* and CSI may be shown in several ways using elementary mathematical methods.

2.1. From the base data of a 2 × 2 contingency table

Hand et al. [6] showed that: $\begin{array}{r}{\mathrm{F}}^{\ast }=\text{TP}/(\text{N}-\text{TN})\end{array}$This also holds for CSI, since in terms of the base data: $\begin{array}{rl}\text{CSI}& =\text{TP}/(\text{TP}+\text{FP}+\text{FN})\\ & =\text{TP}/(\text{N}-\text{TN})\end{array}$Hence F* = CSI, QED.

2.2. From the monotonic relationship of F to CSI

The monotonic relationship between F and CSI, as shown for example by Jolliffe [13] (modified), is given by: $\begin{array}{r}\text{F}=2\text{.CSI}/(1+\text{CSI})\end{array}$The equivalence of F* and CSI may thus be shown. Since Hand et al. [6] showed that: $\begin{array}{r}{\mathrm{F}}^{\ast }=\text{F}/(2-\text{F})\end{array}$Then rearranging: $\begin{array}{rl}\text{F}& ={\text{F}}^{\ast }\cdot (2-\text{F})\\ & =2{.\mathrm{F}}^{\ast }-{\mathrm{F}}^{\ast }\text{.F}\end{array}$Dividing through by F and rearranging: $\begin{array}{rl}\left(2.{\mathrm{F}}^{\ast }/\mathrm{F}\right)-{\mathrm{F}}^{\ast }& =1\\ {\mathrm{F}}^{\ast }+1& =2.{\mathrm{F}}^{\ast }/\mathrm{F}\end{array}$Hence: $\begin{array}{r}\text{F}=2{.\mathrm{F}}^{\ast }/({\mathrm{F}}^{\ast }+1)\end{array}$Hence F* = CSI, QED.

2.3. From the combination of PPV (or precision) and Sens (or recall)

Like F, CSI may be characterised in terms of PPV and Sens: $\begin{array}{r}\text{CSI}=1/\left[\right(1/\text{PPV})+(1/\text{Sens})-1]\end{array}$Again, the equivalence of F* and CSI may be shown. Hand et al. [6] found that: $\begin{array}{r}{\mathrm{F}}^{\ast }=(\text{PPV}\times \text{Sens})/\text{PPV}+\text{Sens}-(\text{PPV}\times \text{Sens})\end{array}$Dividing through by (PPV × Sens) gives: $\begin{array}{r}{\mathrm{F}}^{\ast }=1/\left[\right(1/\text{Sens})+(1/\text{PPV})-1]\end{array}$Hence F* = CSI, QED.

2.4. From the combination of Sens, PPV, P, and Q

In the 2×2 contingency table, prevalence or base rate P = (TP + FN)/N, and bias or threshold Q = (TP + FP)/N. Thus, from Powers [2]: $\begin{array}{rl}\text{F}& =2\text{.Sens.P}/(\text{Q}+\text{P})\\ & =2\text{.PPV.Q}/(\text{Q}+\text{P})\end{array}$For CSI the equations are [1]: $\begin{array}{rl}\text{CSI}& =1/\left[\right(\text{Q}+\text{P})/\text{Sens.P}-1]\\ & =1/\left[\right(\text{Q}+\text{P})/\text{PPV.Q}-1]\end{array}$Since from Hand et al. [6]: $\begin{array}{r}{\mathrm{F}}^{\ast }=\text{F}/(2-\text{F})\end{array}$Then substituting and rearranging: $\begin{array}{rl}{\mathrm{F}}^{\ast }& =\left[2\text{.Sens.P}/\right(\text{Q}+\text{P}\left)\right]/(2-[2\text{.Sens.P}/(\text{Q}+\text{P})\left]\right)\\ & =1/\left[\right(\text{Q}+\text{P})/\text{Sens.P}-1]\end{array}$ $\begin{array}{rl}{\mathrm{F}}^{\ast }& =\left[2\text{.PPV.Q}/\right(\text{Q}+\text{P}\left)\right]/(2-[2\text{.PPV.Q/}(\text{Q}+\text{P})\left]\right)\\ & =1/\left[\right(\text{Q}+\text{P})\text{/PPV.Q}-1]\end{array}$Hence F* = CSI, QED.

3. Conclusion

As mentioned above, Hand et al. noted in their formulation of F* that “researchers may recognise this as the Jaccard coefficient widely used in areas where TN may not be relevant” [6] and they cite Jaccard’s 1908 paper [14], although others [2] cite his 1901 paper [15] as the forerunner of the 1912 English translation [8].

We suggest that this is a parameter which, like F [3–5], has undergone periodic redescriptions. The first report of which we are aware is Gilbert’s “ratio of verification” of 1884 [7], predating the Jaccard similarity coefficient [8]. This latter measure is equivalent in set theory to union over intersection, which was also proposed by Tanimoto in 1958 when working for IBM [10], without reference to either Gilbert or Jaccard. The same measure has also been described by Palmer & Allen in 1949 as the threat score [9], and as the CSI by Donaldson et al. [11] in 1975 and by Schaefer [12] in 1990, and now as F* by Hand et al. [6] These multiple redescriptions may reflect use of this measure by researchers in different disciplines (weather forecasting, ecology, machine learning) unaware of prior authors and unbeknownst to later authors. This case illustrates the potential for convergent evolution of verification metrics over time. A system for greater cohesion between disciplines over the years may be required to prevent duplication in literature. We see little merit in using the F* nomenclature rather than CSI, as the latter makes clear its difference from, and hence avoids any confusion with, F.

The CSI has recently been exported to the domain of clinical medicine. Examples of its use include the evaluation of the accuracy of instruments used in day-to-day clinical practice for screening cognitive function in patients with possible dementia or mild cognitive impairment [16], as well as in diagnostic accuracy studies of administrative epilepsy data [17]. In these studies the identity of F* and CSI has been confirmed using the respective datasets. We have also suggested possible application of CSI in assessing both NICE criteria for 2-week-wait suspected brain and CNS cancer referrals [18] and polygenic hazard scores [19]. These are all situations in which large numbers of TN may complicate the interpretation of more traditional measures such as PPV and Sens. The large number of TN instances make it difficult to discriminate diagnostic algorithm accuracy on the basis of negative predictive value (NPV) and specificity as these become inflated (usually approaching 100%). It is not uncommon for diagnostic accuracy studies to rank algorithms in order of highest to lowest PPV and Sens, with priority given to those with higher values in both estimates [17,20]. This approach is challenged by lack of a single threshold, and the trade-off relationship between PPV and Sens. A convenient way to combine PPV and Sens into a single metric to make it easier and more objective to rank diagnostic algorithms by their accuracy is to use CSI [17,20].

Many outcome measures may be quantitatively, semi-quantitatively, or qualitatively classified, for example, as poor, fair, good, or excellent. Is it possible to classify CSI values in one or other of these ways? To date, diagnostic accuracy thresholds based on combined PPV and Sens have yet to be determined. However, as for Sens, Spec, PPV, and NPV, all of which are conditional probabilities ranging from 0 to 1, a higher value is evidently better. Empirical recommendations may thus be made, such as the proposal that CSI ≥0.80 would be acceptable as diagnostically accurate from prior epilepsy literature, based on observation that both the underlying PPV and Sens were ≥80% at this CSI threshold [17]. Future work should be undertaken to explore and cross-validate this threshold in other datasets within and outside epilepsy. Clearly, absolute values may be used when comparing different testing strategies or algorithms [17,20]. CSI may also be treated as a proportion and a non-exact binomial method used to calculate its confidence intervals, thereby aiding the ability to make valid comparisons [1]. As CSI values are derived from both Sens and PPV, they are dependent on both test threshold (Q) and class prevalence (P) [21].

Authors’ contributions

Conceptualization, A.J.L; Methodology, A.J.L.; Formal Analysis, A.J.L; Writing – Original Draft, A.J.L.; Writing – Review & Editing, G.K.M., A.J.L., Supervision, A.J.L.

Availability of data and materials

No datasets used and/or analysed during the current study.

Declarations

Ethics approval and consent to participate: not applicable as this is a commentary article on a previously publish study.

Consent for publication: not applicable.

Acknowledgements

We are grateful to Professor Iain Buchan and Dr Glen Martin for strategic advice on this work.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

GKM is supported by a National Institute for Health and Care Research (NIHR) Clinical Lectureship (CL-2022-07-002).