On Coping in a Non-Binary World: Rejoinder to Biedermann and Kotsoglou

This article responds to:

Commentary on “Three-Way ROCs for Forensic Decision Making” by Nicholas Scurich and Richard S. John (in: Statistics and Public Policy)

In a sworn declaration written for the prosecution, Biedermann, Budowle, and Champod (2022) stated:

In our view, the key to dealing with “inclonclusives” [sic] is transparency. “Inconclusives” fall into a response category in its own right that can and should be summarized in a separate statistic that is useful for monitoring various aspects of a study (e.g., examiners’ level of responsiveness). “Inconclusives” should not be dismissed; they should be used within the scope in which they can be meaningfully informative. (p. 28)

We agree that inconclusives should not be dismissed, which is the central point of our article: “The performance of forensic firearm examiners cannot be determined without considering their accuracy in judging all three categories of evidence: matching [same source], nonmatching [different source], and inconclusive” (Scurich and John 2023, p. 9).

In their letter to the editor, however, Biedermann and Kotsoglou (2023) advocate for an entirely different position regarding inconclusives:

[R]ather than trying to handle a consequence of this reporting scheme, i.e. the conclusion category “inconclusive”, it would make more sense to abandon the reporting scheme altogether in favour of a more methodologically defensible reporting format that focuses on assessing and reporting the value of findings only. …[Scurich & John] provide us with yet another argument in support of the call to abandon the traditional reporting format of “identification–inconclusive–exclusion”. (p. 5)

Biedermann and Kotsoglou (2023) may well be right that reporting categorical decisions is not ideal. However, our article provided a framework to evaluate how firearm examiners do make decisions rather than how Biedermann and Kotsoglou believe they ought to. Consistent with Arkes and Koehler (2022), whom Biedermann and Kotsoglou cited, “in our article we accepted the world as it currently exists, one in which examiners use categorical conclusions” (p. 176). Biedermann and Kotsoglou’s opinion about how examiners ought to conduct their examinations does not address the substance of our article.

Most of the letter to the editor is devoted to repeating (“reiterating”) Biedermann and Kotsoglou’s (2021) contention that there is no category of inconclusive evidence. Figure 1 in our article clearly demonstrates the reality of inconclusive evidence in the forensic firearm and toolmark domain. In that example, one of the consecutively fired cartridge cases contained no so-called individual characteristics (or striations), and therefore it is not possible—in principle—to make an identification or elimination. The evidence is simply inconclusive. Our article also discussed comparison Set 17 from Law and Morris (2021), who noted:

While the ground truth was an identification in this set, the firearm used, a Ruger[textregistered] LC9, had a smooth breech face and firing pin. The smooth firearm surfaces led to few, if any, reproducible individual characteristics on the fired cartridge cases (refer to the images of Set 17 in the Appendix S1) (p. 8)

Rather than engage our article and explain why these examples are misguided or could be addressed by different statistical techniques, Biedermann and Kotsoglou (2023) merely state:

In forensic science comparison contexts, there are simply no ground truth states other than ‘same source’ and ‘different sources’. To claim or suggest otherwise would violate the principle of the excluded middle (citing themselves).

The letter to the editor does not define “the principle of excluded middle.” Their previous paper gives this definition: “Whatever our degree of belief is, a forensic item or trace either has or does not have a given person of interest as its source: Tertium not [sic] datur (“no third possibility being given”) Aristotle’s principle of the excluded middle” (Biedermann and Kotsoglou 2021, p. 10). However, Biedermann and Kotsoglou do not explain how Aristotle’s metaphysical concept applies to the instant matter, nor do they address the possibility that objective facts may include not only the source but also the condition of the items being compared. So far as we can tell, the unelaborated citation to Aristotle is merely a rhetorical device with no substantive contribution. It most certainly does not justify their extreme conclusion that three-way ROCs are “inappropriate across all domains where the term “inconclusive” is being used” (Biedermann and Kotsoglou 2023, p. 4).

The fact remains that inconclusive decisions are frequent in firearm examiner studies (Scurich 2022). Our paper described one approach that can be used to evaluate those decisions as part of overall examiner performance, and we demonstrated why other treatments of inconclusives lead to implausible results. Biedermann and Kotsoglou’s letter fails to engage our analysis or provide any practical solution to the current situation. It thus “constitute[s] an obstacle to long-awaited fundamental progress in the field” (Biedermann and Kotsoglou 2023, p. 1).

Disclosure Statement

No potential conflict of interest was reported by the author(s).