Using routine data for long term impact evaluation: Methodological reflections from a complex health system intervention in a low-income context

References

• Ahanhanzo, Y. G., Ouedraogo, L. T., Kpozèhouen, A., Coppieters, Y., Makoutodé, M., & Wilmet-Dramaix, M. (2014). Factors associated with data quality in the routine health information system of Benin. Archives of Public Health, 72(1), 1–14. https://doi.org/10.1186/2049-3258-72-25
   PubMedGoogle Scholar
• Arel-Bundock, V., & Pelc, K. J. (2018). When can multiple imputation improve regression estimates? Political Analysis, 26(2), 240–245. https://doi.org/10.1017/pan.2017.43
   Web of Science ®Google Scholar
• Bernal, J. L., Cummins, S., & Gasparrini, A. (2017). Interrupted time series regression for the evaluation of public health interventions: A tutorial. International Journal of Epidemiology, 46(1), 348–355. https://doi.org/10.1093/ije/dyw098
   PubMed Web of Science ®Google Scholar
• Boslaugh, S. (2007). An introduction to secondary data analysis. Secondary data sources for public health: A practical guide (pp. 2–10). Cambridge University Press.
   Google Scholar
• Brocklehurst, P., Price, J., Glenny, A. M., Tickle, M., Birch, S., Mertz, E., & Grytten, J. (2013). The effect of different methods of remuneration on the behaviour of primary care dentists. Cochrane Database Syst Rev(11), Cd009853. https://doi.org/10.1002/14651858.CD009853.pub2
   PubMed Web of Science ®Google Scholar
• Cattaneo, M. D., & Titiunik, R. (2021). Regression discontinuity designs. arXiv preprint arXiv, 2108, 09400. https://doi.org/10.48550/arXiv.2108.09400
   Google Scholar
• Chansa, C., Mukanu, M. M., Chama-Chiliba, C. M., Kamanga, M., Chikwenya, N., Bellows, B., & Kuunibe, N. (2019). Looking at the bigger picture: Effect of performance-based contracting of district health services on equity of access to maternal health services in Zambia. Health Policy and Planning, 35(1),36–46. %J Health Policy and Planning. https://doi.org/10.1093/heapol/czz130
   Web of Science ®Google Scholar
• Clarke, G. M., Conti, S., Wolters, A. T., & Steventon, A. (2019). Evaluating the impact of healthcare interventions using routine data. BMJ, 365. https://doi.org/10.1136/bmj.l2239
   Google Scholar
• Curley, C., Krause, R. M., Feiock, R., & Hawkins, C. V. J. U. A. R. (2019). Dealing with missing data: A comparative exploration of approaches using the integrated city sustainability database. 55(2), 591–615. https://doi.org/10.1177/1078087417726394
   Google Scholar
• De Allegri, M., Lohmann, J., & Schleicher, M. (2018). Results-based financing for health: Impact evaluation in Burkina Faso. https://www.rbfhealth.org/sites/rbf/files/documents/Burkina-Faso-Impact-Evaluation-Results-Report.pdf
   Google Scholar
• Dettori, J. R., Norvell, D. C., & Chapman, J. R. (2018). The sin of missing data: Is all forgiven by way of imputation? 8(8), 892–894. https://doi.org/10.1177/2192568218811922
   Google Scholar
• Devkaran, S., & O’Farrell, P. N. (2015). The impact of hospital accreditation on quality measures: An interrupted time series analysis. BMC Health Services Research, 15(1), 137. https://doi.org/10.1186/s12913-015-0784-5
   PubMedGoogle Scholar
• Dong, Y., & Peng, C. Y. (2013). Principled missing data methods for researchers. Springerplus, 2(1), 222. https://doi.org/10.1186/2193-1801-2-222
   PubMed Web of Science ®Google Scholar
• Evans, R. S. (2016). Electronic health records: Then, now, and in the future. Yearbook of Medical Informatics, 25(S 01), S48–S61. https://doi.org/10.15265/IYS-2016-s006
   Google Scholar
• Fretheim, A., Zhang, F., Ross-Degnan, D., Oxman, A. D., Cheyne, H., Foy, R., Goodacre, S., Herrin, J., Kerse, N., McKinlay, R. J., Wright, A., & Soumerai, S. B. (2015). A reanalysis of cluster randomized trials showed interrupted time-series studies were valuable in health system evaluation. Journal of Clinical Epidemiology, 68(3), 324–333. https://doi.org/10.1016/j.jclinepi.2014.10.003
   PubMed Web of Science ®Google Scholar
• Gimbel, S., Micek, M., Lambdin, B., Lara, J., Karagianis, M., Cuembelo, F., Gloyd, S. S., Pfeiffer, J., & Sherr, K. (2011). An assessment of routine primary care health information system data quality in Sofala Province, Mozambique. Population Health Metrics, 9(1), 1–9. https://doi.org/10.1186/1478-7954-9-12
   PubMedGoogle Scholar
• Greene, H. W. (2003). Econometric Analysis (Fifth) ed.). Pearson Education Inc.
   Google Scholar
• Horton, N. J., & Kleinman, K. P. (2007). Much ado about nothing: A comparison of missing data methods and software to fit incomplete data regression models. The American Statistician, 61(1), 79–90. https://doi.org/10.1198/000313007x172556
   PubMed Web of Science ®Google Scholar
• Hox, J. J., & Boeije, H. R. (2005). Data collection, primary versus secondary. Amsterdam: Elsevier. https://doi.org/10.1016/B0-12-369398-5/00041-4
   Google Scholar
• Hughes, R. A., Heron, J., Sterne, J. A., & Tilling, K. J. I. J. O. E. (2019). Accounting for missing data in statistical analyses: Multiple imputation is not always the answer. 48(4), 1294–1304. https://doi.org/10.1093/ije/dyz032
   Google Scholar
• Jacob, R., Somers, M. A., Zhu, P., & Bloom, H. (2016). The validity of the comparative interrupted time series design for evaluating the effect of school-level interventions. Evaluation Review, 40(3), 167–198. https://doi.org/10.1177/0193841x16663414
   PubMed Web of Science ®Google Scholar
• Khandker, S. R., Koolwal, G. B., & Samad, H. A. (2010). Handbook on impact evaluation: Quantitative methods and practices. World Bank Publications.
   Google Scholar
• Kuunibe, N. (2020). Using routine panel and time series data to assess program impact in low–and middle-income countries: The case of performance-based financing in rural Burkina Faso. Oxford Academic Journals.
   Google Scholar
• Kuunibe, N., Lohmann, J., Hillebrecht, M., Nguyen, H. T., Tougri, G., & De Allegri, M. (2020). What happens when performance‐based financing meets free healthcare? Evidence from an interrupted time‐series analysis. Health Policy and Planning, 35(8),906–917. %J Health Policy and Planning. https://doi.org/10.1093/heapol/czaa062
   PubMed Web of Science ®Google Scholar
• Lagarde, M. (2012). How to do (or not to do) … Assessing the impact of a policy change with routine longitudinal data. Health Policy and Planning, 27(1), 76–83. https://doi.org/10.1093/heapol/czr004
   PubMed Web of Science ®Google Scholar
• Leys, C., Delacre, M., Mora, Y. L., Lakens, D., & Ley, C. (2019). How to classify, detect, and manage univariate and multivariate outliers, with emphasis on pre-registration. International Review of Social Psychology, 32(1). https://doi.org/10.5334/irsp.289
   Web of Science ®Google Scholar
• Linden, A. (2015). Conducting interrupted time-series analysis for single- and multiple-group comparisons. The Stata Journal, 15(2), 480–500. https://doi.org/10.1177/1536867X1501500208
   Web of Science ®Google Scholar
• Linden, A. (2018). Using forecast modelling to evaluate treatment effects in single-group interrupted time series analysis. J Eval Clin Pract. https://doi.org/10.1111/jep.12946
   Web of Science ®Google Scholar
• McLintock, K., Russell, A. M., Alderson, S. L., West, R., House, A., Westerman, K., & Foy, R. (2014). The effects of financial incentives for case finding for depression in patients with diabetes and coronary heart disease: Interrupted time series analysis. BMJ Open, 4(8), e005178. https://doi.org/10.1136/bmjopen-2014-005178
   PubMed Web of Science ®Google Scholar
• Michielutte, R., Shelton, B., Paskett, E. D., Tatum, C. M., & Velez, R. 2000. Use of an interrupted time-series design to evaluate a cancer screening program. Health Education Research 15: 615–623. http://her.oxfordjournals.org/content/15/5/615.full.pdf 5
   PubMed Web of Science ®Google Scholar
• Ministère de la Santé Burkina Faso. (2015). Metadonnees des Indicateurs du systeme National d’information Sanitaire (SNIS). http://www.cns.bf/IMG/Metadonnees/Meta_donnees_SNIS.pdf
   Google Scholar
• Muthee, V., Bochner, A. F., Osterman, A., Liku, N., Akhwale, W., Kwach, J., Onyango, F., Odhiambo, J., Onyango, F., Puttkammer, N., & Prachi, M. (2018). The impact of routine data quality assessments on electronic medical record data quality in Kenya. PLoS One, 13(4), e0195362. https://doi.org/10.1371/journal.pone.0195362
   PubMed Web of Science ®Google Scholar
• Powell, A., Davies, H., & Thomson, R. (2003). Using routine comparative data to assess the quality of health care: Understanding and avoiding common pitfalls. Quality and Safety in Health Care, 12(2), 122–128. https://doi.org/10.1136/qhc.12.2.122
   PubMedGoogle Scholar
• Pratama, I., Permanasari, A. E., Ardiyanto, I., & Indrayani, R. (2016, 24-27 October.). A review of missing values handling methods on time-series data. Paper presented at the 2016 international conference on information technology systems and innovation (ICITSI) (pp. 1-6). Institute of Electricals and Electronics Engineers (IEEE). https://doi.org/10.1109/ICITSI.2016.7858189
   Google Scholar
• Schneider, A., Donnachie, E., Tauscher, M., Gerlach, R., Maier, W., Mielck, A., Linde, K., & Mehring, M. (2016). Costs of coordinated versus uncoordinated care in Germany: Results of a routine data analysis in Bavaria. BMJ Open, 6(6), e011621. https://doi.org/10.1136/bmjopen-2016-011621
   PubMed Web of Science ®Google Scholar
• Schulte, P. J., & Mascha, E. J. (2018). Propensity score methods: Theory and practice for anesthesia research. Anesthesia & Analgesia, 127(4), 1074–1084. https://doi.org/10.1213/ANE.0000000000002920
   PubMed Web of Science ®Google Scholar
• Serumaga, B., Ross-Degnan, D., Avery, A. J., Elliott, R. A., Majumdar, S. R., Zhang, F., & Soumerai, S. B. (2011). Effect of pay for performance on the management and outcomes of hypertension in the United Kingdom: Interrupted time series study. BMJ, 342(jan25 3), d108. https://doi.org/10.1136/bmj.d108
   PubMedGoogle Scholar
• Shin, Y. (2017). Time Series Analysis in the social sciences the fundamentals (1) ed.). University of California Press.
   Google Scholar
• Smith, M., Lix, L. M., Azimaee, M., Enns, J. E., Orr, J., Hong, S., & Roos, L. L. (2018). Assessing the quality of administrative data for research: A framework from the Manitoba centre for health policy. Journal of the American Medical Informatics Association, 25(3), 224–229. https://doi.org/10.1093/jamia/ocx078
   PubMed Web of Science ®Google Scholar
• StataCorp, L. P. (2013), Stata multiple imputation reference manual. Release. 13. https://www.stata.com/manuals13/mi.pdf
   Google Scholar
• Stausberg, J. (2014). International prevalence of adverse drug events in hospitals: An analysis of routine data from England, Germany, and the USA. BMC Health Services Research, 14(1), 1–9. https://doi.org/10.1186/1472-6963-14-125
   PubMedGoogle Scholar
• Steptoe, A., Breeze, E., Banks, J., & Nazroo, J. (2013). Cohort profile: The English longitudinal study of ageing. International Journal of Epidemiology, 42(6), 1640–1648. https://doi.org/10.1093/ije/dys168
   PubMed Web of Science ®Google Scholar
• Todd, O. M., Burton, J. K., Dodds, R. M., Hollinghurst, J., Lyons, R. A., Quinn, T. J., … Conroy, S. (2020). New horizons in the use of routine data for ageing research. Age and Ageing, 49(5), 716–722. https://doi.org/10.1093/ageing/afaa018
   PubMed Web of Science ®Google Scholar
• Tsvetanova, A., Sperrin, M., Peek, N., Buchan, I., Hyland, S., & Martin, G. P. (2021). Missing data was handled inconsistently in UK prediction models: A review of method used. Journal of Clinical Epidemiology, 140, 149–158. https://doi.org/10.1016/j.jclinepi.2021.09.008
   PubMed Web of Science ®Google Scholar
• Tung, Y. C., Chang, G. M., & Cheng, S. H. (2015). Long-term effect of fee-for-service-based reimbursement cuts on processes and outcomes of care for stroke: Interrupted time-series study from Taiwan. Circulation: Cardiovascular Quality and Outcomes, 8(1), 30–37. https://doi.org/10.1161/circoutcomes.114.001086
   PubMed Web of Science ®Google Scholar
• WHO. (2007). Health System Strengthening Interventions: Making the Case for Impact Evaluation. https://www.who.int/alliance-hpsr/resources/alliancehpsr_briefingnote2.pdf
   Google Scholar
• WHO. (2021). Toolkit for Analysis and Use of Routine Health Facility Data: General Principles. https://www.who.int/data/data-collection-tools/health-service-data/toolkit-for-routine-health-information-system-data/modules
   Google Scholar
• Wing, C., Simon, K., & Bello-Gomez, R. A. (2018). Designing difference in difference studies: Best practices for public health policy research. Annual Review of Public Health, 39(1), 453–469. https://doi.org/10.1146/annurev-publhealth-040617-013507
   PubMed Web of Science ®Google Scholar
• Wooldridge, J. M. (2010). Econometric analysis of cross section and panel data. MIT press.
   Google Scholar
• Zombre, D., De Allegri, M., & Ridde, V. (2017). Immediate and sustained effects of user fee exemption on healthcare utilization among children under five in Burkina Faso: A controlled interrupted time-series analysis. Social Science & Medicine, 179, 27–35. https://doi.org/10.1016/j.socscimed.2017.02.027
   PubMed Web of Science ®Google Scholar