Agreement between cause of death assignment by computer-coded verbal autopsy methods and physician coding of verbal autopsy interviews in South Africa

ABSTRACT

Background

The South African national cause of death validation (NCODV 2017/18) project collected a national sample of verbal autopsies (VA) with cause of death (COD) assignment by physician-coded VA (PCVA) and computer-coded VA (CCVA).

Objective

The performance of three CCVA algorithms (InterVA-5, InSilicoVA and Tariff 2.0) in assigning a COD was compared with PCVA (reference standard).

Methods

Seven performance metrics assessed individual and population level agreement of COD assignment by age, sex and place of death subgroups. Positive predictive value (PPV), sensitivity, overall agreement, kappa, and chance corrected concordance (CCC) assessed individual level agreement. Cause-specific mortality fraction (CSMF) accuracy and Spearman’s rank correlation assessed population level agreement.

Results

A total of 5386 VA records were analysed. PCVA and CCVAs all identified HIV/AIDS as the leading COD. CCVA PPV and sensitivity, based on confidence intervals, were comparable except for HIV/AIDS, TB, maternal, diabetes mellitus, other cancers, and some injuries. CCVAs performed well for identifying perinatal deaths, road traffic accidents, suicide and homicide but poorly for pneumonia, other infectious diseases and renal failure. Overall agreement between CCVAs and PCVA for the top single cause (48.2–51.6) indicated comparable weak agreement between methods. Overall agreement, for the top three causes showed moderate agreement for InterVA (70.9) and InSilicoVA (73.8). Agreement based on kappa (−0.05–0.49)and CCC (0.06–0.43) was weak to none for all algorithms and groups. CCVAs had moderate to strong agreement for CSMF accuracy, with InterVA-5 highest for neonates (0.90), Tariff 2.0 highest for adults (0.89) and males (0.84), and InSilicoVA highest for females (0.88), elders (0.83) and out-of-facility deaths (0.85). Rank correlation indicated moderate agreement for adults (0.75–0.79).

Conclusions

Whilst CCVAs identified HIV/AIDS as the leading COD, consistent with PCVA, there is scope for improving the algorithms for use in South Africa.

KEYWORDS:

• Physician-coded
• computer-coded
• verbal autopsy
• agreement

Responsible Editor Stig Wall

Responsible Editor Stig Wall

Acknowledgments

We would like to express our thanks to the National and Provincial Departments of Health and facility managers for allowing us access to medical and forensic records, Geospace for conducting the fieldwork and Monique Maqungo and Tracy Glass, SAMRC, for support with cleaning, checking and preparation of the data.

Author contributions

P Groenewald, D Bradshaw, J Joubert and S J Clark conceptualised and designed the research. P Groenewald was responsible for data collection and curation, and project management. J Thomas, Z Li and C Kabudula conducted the data analysis. D Bradshaw, S J Clark, P Groenewald, J Thomas, Z Li were responsible for data interpretation. P Groenewald wrote the original draft. SJ Clark reviewed the literature and wrote the paper context section. D Morof, a representative of the United States of America Government and funded through Centers for Disease Control, provided technical input and support. All authors reviewed and edited the manuscript.

Disclosure statement

Apart from the funding support from PEPFAR and CDC Foundation declared below none of the authors except Samuel J Clark declared any potential conflicts of interest. Samuel Clark declared grants received from NIH, Bill and Melinda Gates Foundation, Vital Strategies, Emory University and the University of Toronto, consulting fees from Vital strategies, CDC Foundation, Emory University and the University of Toronto, and support for meetings and travel from WHO, Vital strategies and CDC Foundations for related work on verbal autopsy cause-coding algorithms.

Ethics and consent

The project protocol was reviewed by the SAMRC Ethics committee and approved on 27 June 2017 (EC004–2/2017). This project was also reviewed in accordance with CDC human research protection procedures and was determined to be research, but CDC investigators did not interact with human subjects or have access to identifiable data or specimens for research purposes (8 April 2017). Written consent was obtained from all participants. Since access to medical and forensic records was only required for retrospective record review after death a waiver of the need for individual consent from family members for access was requested and approved by the Ethics Committee.

Paper context

Increasingly national-scale verbal autopsy implementations rely on cause-coding algorithms to automate coding of verbal autopsies to inform national burden of disease metrics. This comparison of the performance of InterVA-5, inSilicoVA and Tariff 2.0 against a reference PCVA dataset, which included community deaths, has corroborated the urgent need to improve the performance of these algorithms. A standard method for comparison of cause-coding algorithm performance and standard, freely available datasets for testing and validation are warranted.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/16549716.2023.2285105

Notes

¹ The term InSilicoVA-NT was used by Jha et al [33] to differentiate the standard InSilicoVA which uses InterVA conditional probabilities and does not require a training dataset, from a version of InSilicoVA where a training dataset replaced the inbuilt conditional probabilities.

Additional information

Funding

This project was supported by the President’s Emergency Plan for AIDS Relief (PEPFAR) through the Centers for Disease Control and Prevention (CDC) under the terms of cooperative agreement [GH001150]. Additional funding was provided by CDC Foundation.Disclaimer: The findings and conclusions in this manuscript are those of the authors and do not necessarily represent the official position of the funding agencies.