Probability or confidence, a distinction without a difference?

ABSTRACT

How analytic uncertainty is presented can affect critical policymaker decisions. Analysts are told to offer separate: (1) assessments of the likelihood an analytic statement is true and (2) confidence statements that express how strongly analysts believe their assessments. The distinction between likelihood and confidence sounds clear but is not. The IC defines confidence poorly. Most intelligence products fail to distinguish likelihood from confidence and use the terms interchangeably to mean probability. The IC should abandon its confusing shorthand confidence descriptions. Instead, analysts should provide explanations of the nature and strength of evidence, and the reasoning behind analytic judgments.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. Fish, “Legal Arguments.” Schauer, The Proof. Schauer, Thinking Like a Lawyer. Farnsworth, The Legal Analyst. Harris, “Justice Jackson.” This article was inspired by all of the cited works. Supreme Court Justice Ketanji Brown Jackson’s 31 October 2022 hypothetical was offered in a case on race considerations for college admissions.

2. Director of National Intelligence, ICD 203, Analytic Standards. The initial version of ICD 203 was issued in June 2007.

3. Friedman and Zeckhauser, “Analytic Confidence,” 1069.

4. The specific charge might be under 18 U.S.C. § 2332a.

5. Farnsworth, The Legal Analyst, 257–272. This is referred to as a ‘standard of proof’. The literature on the standards of proof – its forms, meanings, interpretations, and consequences – is vast. An excellent place to start is the cited work.

6. Bush, Decision Points. Hayden, Playing to the Edge. Gates, Duty, 171–177. The cited works are the primary sources. A CIA-produced video on Al Kibar is dated 25 April 2008, and can be accessed at https://www.youtube.com/watch?v=yj62GRd0Te8. Hayden refers to the creation of the video on page 267.

7. Hayden, Playing to the Edge, 261.

8. Ibid., 262.

9. Bush, Decision Points, 421.

10. Hayden, Playing to the Edge, 268.

11. Ibid., 256.

12. Hayden offered more than one-word summaries for his confidence. The DNI guidance does not require that.

13. Weinstein and Dewsbury, “Comment on the meaning”, I wish to thank sincerely the anonymous reviewer for her/his insightful comments on an earlier draft of this article. The reviewer pointed out, for example, fascinating and relevant materials in the journal Law, Probability and Risk, such as the cited work.

14. Gates, Duty, 171–172. At least one curious point remains. The low confidence assessment offered by Hayden rested on the failure of the U.S. Intelligence Community to locate a plutonium reprocessing facility or identify work on weaponization. Given the failure to identify the reactor for some two years, and the security practices of the Syrians, how much weight should be accorded to negative evidence? Did we have confidence that we would have detected these, if they existed? After all, the IC had not identified positive evidence to refute the argument that Syria had a nuclear weapons program. As Gates reports his question to President Bush when the reactor was first brought to the president’s attention, ‘How can we have any confidence at all in the estimates of the scope of the North Korean, Iranian, or other possible programs [given this failure]?’ Perhaps he later should have asked how we could have confidence in the negative evidence that played such a large role in the decision to forego an attack.

15. Ibid., 177.

16. Irwin and Mandel, “How Intelligence Organizations Communicate,” See cited work for a thorough discussion. Those scholars conclude, “Ultimately, we argue that current confidence standards are poorly conceived, ambiguous, vague, and unclear, and may effectively augment the potential for miscommunication, which the IC seeks to mitigate”. The cited work provides many examples of how confidence is expressed in different organizations (or, directives about how it should be expressed).

17. ”Retired Admiral Bill McRaven and Michael Morell”.

18. Ibid. Morell’s description of the process the CIA took, including the formation of two Red Teams, appears to detail additional studies of likelihood – not determinations of confidence in those likelihoods. The teams reviewed all the evidence and examined alternative possibilities explicitly. Morell uses the word ‘confidence’ as a substitute for ‘likelihood’ in the interview. He notes that one Red Team stated, ‘We think he’s there, but we’re not as confident as the analysts who have been working this directly’. Is this indeed an expression of confidence in the judgment or, rather, probability?

Bowden, The Finish, 256. CNN, “The Axe Files”. Other reports show the Red Team presented likelihood estimates of 30 or 40 per cent compared to likelihood estimates from other CIA advisers of 60 and 95 per cent. None of the available sources of the famous 28 April 2011 meeting with President Obama cite clear confidence assessments. There is a curious, short discussion of high, medium, and low confidence levels in Bowden, but this is not explained and Bowden reports its occurrence at a meeting well before the 28 April session. DNI Clapper’s jumbled use of terms in the cited CNN work is telling. In discussing the meeting with Obama on Bin Laden, Clapper stated, “We put a lot of discussion [into] percentages of confidence, which to me is not particularly meaningful. In the end it’s all subjective judgment anyway”..

19. Memorandum for National Intelligence Board Principals, 8 November 1984. “Confidence” is used in the cited work as a synonym for “likelihood”. A Confidence Level Table specifies terms that fit into five levels of likelihood.

20. Intelligence Analysis of the Long Range Missile Threat, 45.

21. Director of National Intelligence, ICD 203, Analytic Standards. Director of Central Intelligence, Assessed Manpower, 8–9. The initial version of ICD 203 was issued in June 2007. The 1986 Memorandum uses the terms high, medium, and low confidence along similar lines, but defines none of the terms. The report describes why confidence in assessments (ranges of estimates) have changed in terms of the available evidence. The provision of a “ranged estimate” and a separate ‘minimum estimate’ outside the range is curious.

22. National Intelligence Council, Iran: Nuclear

23. National Intelligence Council, Updated Assessment on COVID-19.

24. Shaw, Brown, and Bromley, “Strategic Stories”.

25. Levine, “Principles of Intelligence Analysis”.

26. McHenry, “Three IARPA forecasting efforts.” Horowitz et al., “Keeping Score”, 6. The prediction market run by the Intelligence Community and other efforts to improve the accuracy of predictions (e.g., the Good Judgment Project) require participants to offer de facto probability judgments. They do not, however, ask participants to express their short-hand confidence in their judgments. One might argue that the size of participants’ bets in the market expresses their confidence, but there are many factors influencing the size of bets, not least, how risk averse they are. The cited work by Horowitz et al summarizes prediction markets and other forms of crowdsourced predictions with the following: ‘Avoiding false confidence regarding probability assessments is crucial, which requires effectively communicating forecasts in ways that make clear the odds, the reasoning behind those odds, the track record of the approach, and the limits of the approach’. Note there is no call for a separate confidence judgment.

27. Friedman, War and Chance, 144–146. Mui, “Gut Check,” Friedman’s book is a rich source of material on probability assessments and related issues.

28. Farnsworth, The Legal Analyst, 258. A discussion of civil cases, in which the standard of proof generally is the preponderance of the evidence (judged as greater than 50 per cent), illustrates this. “It might be hard for the parties to provide any decider – whether jury or judge – with the knowledge they would need to make more precise estimates. The trier of the facts would end up making guesses based on trivial foundations … The judge or jury is more likely to be equal of the task if called on just to sort the case onto one side or another of the “more likely than not” threshold”.

29. Schauer, The Proof, 169–170. “Juries are not required to explain the grounds for their verdict, and it is impermissible for a judge to require a jury to subdivide its verdict into particular answers to particular questions”.

30. Friedman and Zeckhauser, “Analytic Confidence”.

31. Ibid., 1070.

32. Ibid., 1071.

33. Ibid., 1072.

34. Ibid., 1074.

35. Ibid., 1075.

36. Friedman, War and Chance. Friedman has made good progress with his delightful book.

37. This short inset cannot do justice to Friedman’s and Richard Zeckhauser’s work. They point out, for example, the potential roles of past behaviors, precedents, and base rates in logical argumentation, and their shortfalls. Inevitably, their base rate example reminded me of the apocryphal story of the senior British foreign official who retired in the 1950s after a long career. The official noted that throughout his career he regularly opined that a major war in Europe would not occur, and he was only wrong twice.

38. Memorandum from Lawrence K. Gershwin.

39. National Intelligence Council, Updated Assessment of Anomalous Health Incidents, 3.

Additional information

Notes on contributors

Robert Levine

Robert Levine retired from the Central Intelligence Agency after 33 years in the United States Intelligence Community. He served as a senior military analyst and ran the internal analytic quality evaluation program for the CIA’s Directorate of Analysis. Dr. Levine taught intelligence and national security policy at the National War College, the CIA, within the Intelligence Community, and at Mercyhurst College. He received his PhD from the Rand Graduate Institute. He is currently a lecturer at John Hopkins University. His co-edited book with Roger Z. George, The CIA Intelligence Analyst: Views from the Inside, will be published by Georgetown University Press in 2024. Nothing in this article should be construed as asserting or implying US government endorsement of its factual statements and interpretations.