Selecting Performance Indicators and Targets in Health Care: An International Scoping Review and Standardized Process Framework

References

• Cassel CK, Conway PH, Delbanco SF, Jha AK, Saunders RS, Lee TH. Getting more performance from performance measurement. N Engl J Med. 2014;371(23):2145–2147. doi:10.1056/NEJMp1408345
   PubMed Web of Science ®Google Scholar
• Panzer RJ, Gitomer RS, Greene WH, Webster PR, Landry KR, Riccobono CA. Increasing demands for quality measurement. JAMA. 2013;310(18):1971–1980. doi:10.1001/jama.2013.282047
   PubMed Web of Science ®Google Scholar
• Lawrence M, Olesen F. Indicators of quality in health care. Eur J Gen Pract. 1997;3(3):103–108. doi:10.3109/13814789709160336
   Google Scholar
• Meyer GS, Nelson EC, Pryor DB, et al. More quality measures versus measuring what matters: a call for balance and parsimony. BMJ Qual Saf. 2012;21(11):964–968. doi:10.1136/bmjqs-2012-001081
   PubMed Web of Science ®Google Scholar
• Greenburg A, Dale A. Measuring what matters in hospitals. Health Quality Ontario (HQO); 2021. Available from: https://www.hqontario.ca/Blog/hospital-care/measuring-what-matters-in-hospitals. Accessed April 5, 2022.
   Google Scholar
• Mannion R, Braithwaite J. Unintended consequences of performance measurement in healthcare: 20 salutary lessons from the English National Health Service. Intern Med J. 2012;42(5):569–574. doi:10.1111/j.1445-5994.2012.02766.x
   PubMed Web of Science ®Google Scholar
• Smith PC, Mossialos E, Papanicolas I. Performance measurement for health system improvement: experiences, challenges and prospects: background document 2. Regional World Health Organization. Office for Europe; 2008. Available from: https://apps.who.int/iris/handle/10665/350328. Accessed April 5, 2022.
   Google Scholar
• Jones P, Shepherd M, Wells S, Le Fevre J, Ameratunga S. What makes a good healthcare quality indicator? A systematic review and validation study. Emerg Med Austral. 2014;26(2):113–124. doi:10.1111/1742-6723.12195
   PubMed Web of Science ®Google Scholar
• Campbell SM, Braspenning JA, Hutchinson A, Marshall M. Research methods used in developing and applying quality indicators in primary care. BMJ Qual Saf Health Care. 2002;11(4):358–364. doi:10.1136/qhc.11.4.358
   PubMed Web of Science ®Google Scholar
• Teare GF. Measurement of quality and safety in healthcare: the past decade and the next. Healthc Quart. 2014;17:45–50. doi:10.12927/hcq.2014.23950
   PubMedGoogle Scholar
• Perla R. Commentary: health systems must strive for data maturity. Am J Med Qual. 2013;28(3):263–264. doi:10.1177/1062860612465000
   PubMed Web of Science ®Google Scholar
• Blumenthal D, McGinnis JM. Measuring vital signs: an IOM report on core metrics for health and health care progress. JAMA. 2015;313(19):1901–1902. doi:10.1001/jama.2015.4862
   PubMed Web of Science ®Google Scholar
• Canadian Institute for Health Information and Statistics Canada. Rethink, renew, retire: report from the fourth consensus conference on evaluating priorities for Canada’s health indicators; 2015. Available from: https://secure.cihi.ca/free_products/Rethink_Renew_Retire.pdf. Accessed April 5, 2022.
   Google Scholar
• National Quality Forum. Committee guidebook for the NQF measure endorsement process; 2019. Available from: https://www.qualityforum.org/Measuring_Performance/Measuring_Performance.aspx. Accessed April 5, 2022.
   Google Scholar
• Madsen MM, Eiset AH, Mackenhauer J, et al. Selection of quality indicators for hospital-based emergency care in Denmark, informed by a modified-Delphi process. Scand J Trauma Resusc Emerg Med. 2016;24(1):11. doi:10.1186/s13049-016-0203-x
   PubMed Web of Science ®Google Scholar
• Schull MJ, Guttmann A, Leaver CA, et al. Prioritizing performance measurement for emergency department care: consensus on evidence-based quality of care indicators. CJEM. 2011;13(5):300–309, E28-E43. doi:10.2310/8000.2011.110334
   Google Scholar
• Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. PLOS Medicine. 2021;18(3):e1003583. doi: 10.1371/journal.pmed.1003583
   Google Scholar
• Mays N, Pope C, Popay J. Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Policy. 2005;10(1_suppl):6–20. doi:10.1258/1355819054308576
   PubMedGoogle Scholar
• Aktaa S, Batra G, Wallentin L, et al. European Society of Cardiology methodology for the development of quality indicators for the quantification of cardiovascular care and outcomes. Eur Heart J Qual Care Clin Outcomes. 2020. doi:10.1093/ehjqcco/qcaa069
   Web of Science ®Google Scholar
• Bianchi V, Spitale A, Ortelli L, Mazzucchelli L, Bordoni A. Quality indicators of clinical cancer care (QC 3) in colorectal cancer. BMJ open. 2013;3(7):e002818. doi:10.1136/bmjopen-2013-002818
   PubMed Web of Science ®Google Scholar
• Bramesfeld A, Wrede S, Richter K, et al. Development of quality indicators and data assessment strategies for the prevention of central venous catheter-related bloodstream infections (CRBSI). BMC Infect Dis. 2015;15:435. doi:10.1186/s12879-015-1200-9
   PubMed Web of Science ®Google Scholar
• Casey MM, Moscovice I, Klingner J, Prasad S. Rural relevant quality measures for critical access hospitals. J Rural Health. 2013;29(2):159–171. doi:10.1111/j.1748-0361.2012.00420.x
   PubMed Web of Science ®Google Scholar
• Chrusch CA, Martin CM; Project TQIICC. Quality improvement in critical care: selection and development of quality indicators. Can Respir J. 2016;2016:2516765. doi:10.1155/2016/2516765
   PubMed Web of Science ®Google Scholar
• Elliot C, Mcullagh C, Brydon M, Zwi K. Developing key performance indicators for a tertiary children’s hospital network. Austra Health Rev. 2018;42(5):491–500. doi:10.1071/AH17263
   PubMed Web of Science ®Google Scholar
• Emond YE, Stienen JJ, Wollersheim HC, et al. Development and measurement of perioperative patient safety indicators. Br J Anaesth. 2015;114(6):963–972. doi:10.1093/bja/aeu561
   PubMed Web of Science ®Google Scholar
• Fekri O, Leeb K, Gurevich Y. Systematic approach to evaluating and confirming the utility of a suite of national health system performance (HSP) indicators in Canada: a modified Delphi study. BMJ open. 2017;7(4):e014772. doi:10.1136/bmjopen-2016-014772
   PubMed Web of Science ®Google Scholar
• Goldfarb M, Bibas L, Newby LK, et al. Systematic review and directors survey of quality indicators for the cardiovascular intensive care unit. Int J Cardiol. 2018;260:219–225. doi:10.1016/j.ijcard.2018.02.113
   PubMed Web of Science ®Google Scholar
• Grace SL, Poirier P, Norris CM, Oakes GH, Somanader DS, Suskin N. Pan-Canadian development of cardiac rehabilitation and secondary prevention quality indicators. Can J Cardiol. 2014;30(8):945–948. doi:10.1016/j.cjca.2014.04.003
   PubMed Web of Science ®Google Scholar
• Gurvitz M, Marelli A, Mangione-Smith R, Jenkins K. Building quality indicators to improve care for adults with congenital heart disease. J Am Coll Cardiol. 2013;62(23):2244–2253. doi:10.1016/j.jacc.2013.07.099
   PubMed Web of Science ®Google Scholar
• Guth RM, Storey PE, Vitale M, et al. Decision analysis for metric selection on a clinical quality scorecard. Am J Med Qual. 2016;31(5):400–407. doi:10.1177/1062860615589117
   PubMed Web of Science ®Google Scholar
• Mangione-Smith R, Schiff J, Dougherty D. Identifying children’s health care quality measures for Medicaid and CHIP: an evidence-informed, publicly transparent expert process. Acad Pediatr. 2011;11(3):S11–21. doi:10.1016/j.acap.2010.11.003
   PubMed Web of Science ®Google Scholar
• Martinez DA, Kane EM, Jalalpour M, et al. An electronic dashboard to monitor patient flow at the Johns Hopkins hospital: communication of key performance indicators using the donabedian model. J Med Syst. 2018;42(8):133. doi:10.1007/s10916-018-0988-4
   PubMed Web of Science ®Google Scholar
• Mazzone PJ, Vachani A, Chang A, et al. Quality indicators for the evaluation of patients with lung cancer. Chest. 2014;146(3):659–669. doi:10.1378/chest.13-2900
   PubMed Web of Science ®Google Scholar
• Moehring RW, Anderson DJ, Cochran RL, et al. Expert consensus on metrics to assess the impact of patient-level antimicrobial stewardship interventions in acute-care settings. Clin Infect Dis. 2017;64(3):377–383. doi:10.1093/cid/ciw787
   PubMed Web of Science ®Google Scholar
• Morris AM, Brener S, Dresser L, et al. Use of a structured panel process to define quality metrics for antimicrobial stewardship programs. Infect Control Hosp Epidemiol. 2012;33(5):500–506. doi:10.1086/665324
   PubMed Web of Science ®Google Scholar
• Perera R, Dowell A, Crampton P. Painting by numbers: a guide for systematically developing indicators of performance at any level of health care. Health Policy. 2012;108(1):49–59. doi:10.1016/j.healthpol.2012.07.008
   PubMed Web of Science ®Google Scholar
• Profit J, Gould JB, Zupancic JAF, et al. Formal selection of measures for a composite index of NICU quality of care: baby-MONITOR. J Perinatol. 2011;31(11):702–710. doi:10.1038/jp.2011.12
   PubMed Web of Science ®Google Scholar
• Reiter A, Geraedts M, Jäckel W, Fischer B, Veit C, Döbler K. Selection of hospital quality indicators for public disclosure in Germany. Z Evid Fortbild Qual Gesundhwes. 2011;105(1):44–48. doi:10.1016/j.zefq.2010.12.024
   PubMedGoogle Scholar
• Sauvegrain P, Chantry AA, Chiesa-Dubruille C, Keita H, Goffinet F, Deneux-Tharaux C. Monitoring quality of obstetric care from hospital discharge databases: a Delphi survey to propose a new set of indicators based on maternal health outcomes. PLoS One. 2019;14(2):e0211955. doi:10.1371/journal.pone.0211955
   PubMed Web of Science ®Google Scholar
• Schnitker LM, Martin-Khan M, Burkett E, Beattie ERA, Jones RN, Gray LC. Process quality indicators targeting cognitive impairment to support quality of care for older people with cognitive impairment in emergency departments. Acad Emerg Med. 2015;22(3):285–298. doi:10.1111/acem.12616
   PubMed Web of Science ®Google Scholar
• Science M, Timberlake K, Morris A, Read S, Le Saux N. Quality metrics for antimicrobial stewardship programs. Pediatrics. 2019;143(4). doi:10.1542/peds.2018-2372
   PubMed Web of Science ®Google Scholar
• SooHoo NF, Lieberman JR, Farng E, Park S, Jain S, Ko CY. Development of quality of care indicators for patients undergoing total hip or total knee replacement. BMJ Qual Saf. 2011;20(2):153–157. doi:10.1136/bmjqs.2009.032524
   PubMed Web of Science ®Google Scholar
• Stang AS, Straus SE, Crotts J, Johnson DW, Guttmann A. Quality indicators for high acuity pediatric conditions. Pediatrics. 2013;132(4):752–762. doi:10.1542/peds.2013-0854
   PubMed Web of Science ®Google Scholar
• Stegbauer C, Willms G, Kleine-Budde K, Bramesfeld A, Stammann C, Szecsenyi J. Development of indicators for a nationwide cross-sectoral quality assurance procedure for mental health care of patients with schizophrenia, schizotypal and delusional disorders in Germany. Z Evid Fortbild Qual Gesundhwes. 2017;126:13–22. doi:10.1016/j.zefq.2017.07.006
   PubMed Web of Science ®Google Scholar
• Thern J, de With K, Strauss R, Steib-Bauert M, Weber N, Kern WV. Selection of hospital antimicrobial prescribing quality indicators: a consensus among German antibiotic stewardship (ABS) networkers. Infection. 2014;42(2):351–362. doi:10.1007/s15010-013-0559-z
   PubMed Web of Science ®Google Scholar
• Tsiamis E, Millar J, Baxi S, et al. Development of quality indicators to monitor radiotherapy care for men with prostate cancer: a modified Delphi method. Radiother Oncol. 2018;128(2):308–314. doi:10.1016/j.radonc.2018.04.017
   PubMed Web of Science ®Google Scholar
• van der Wees PJ, Verkerk EW, Verbiest MEA, et al. Development of a framework with tools to support the selection and implementation of patient-reported outcome measures. J Patient Rep Outcomes. 2019;3(1):75. doi:10.1186/s41687-019-0171-9
   PubMedGoogle Scholar
• Van Grootven B, McNicoll L, Mendelson DA, et al. Quality indicators for in-hospital geriatric co-management programmes: a systematic literature review and international Delphi study. BMJ Open. 2018;8(3):e020617. doi:10.1136/bmjopen-2017-020617
   PubMed Web of Science ®Google Scholar
• van Heurn E, de Blaauw I, Heij H, et al. Quality measurement in neonatal surgical disorders: development of clinical indicators. Eur J Pediatr Surg. 2015;25(6):526–531. doi:10.1055/s-0034-1396416
   PubMed Web of Science ®Google Scholar
• Wood L, Bjarnason GA, Black PC, et al. Using the Delphi technique to improve clinical outcomes through the development of quality indicators in renal cell carcinoma. J Oncol Pract. 2013;9(5):e262–267. doi:10.1200/JOP.2012000870
   PubMedGoogle Scholar
• Health Quality Ontario (HQO). Hospital sector indicator reduction and management strategy; 2016. Available from: https://www.hqontario.ca/System-Performance/Measuring-System-Performance/Hospital-Sector-Indicator-Reduction-and-Management-Strategy. Accessed April 5, 2022.
   Google Scholar
• Ontario Hospital Association and Health Quality Ontario. A sustainable indicator reduction and management strategy for the Ontario Hospital Sector; 2019. Available from: https://www.oha.com/health-system-transformation/high-performing-health-system/a-sustainable-indicator-reduction-and-management-strategy. Accessed April 5, 2022.
   Google Scholar
• Health Quality & Safety Commission. Health quality & safety indicators: summary feedback document; 2012. Available from: https://www.hqsc.govt.nz/assets/Health-Quality-Evaluation/PR/HQSI-Feedback-and-Engagement-document-July12-web.pdf. Accessed April 5, 2022.
   Google Scholar
• The King’s Fund. Getting the measure of quality: opportunities and challenges; 2010. Available from: https://www.kingsfund.org.uk/publications/getting-measure-quality. Accessed April 5, 2022.
   Google Scholar
• National Institute for Health and Care Excellence. NICE indicator process guide; 2019. Available from: https://www.Nice.Org.Uk/Media/Default/Get-Involved/Meetings-In-Public/Indicator-Advisory-Committee/Ioc-Process-Guide.Pdf. Accessed April 5, 2022.
   Google Scholar
• The Health Foundation. The measurement maze; 2019. Available from: https://www.health.org.uk/publications/reports/the-measurement-maze. Accessed April 5, 2022.
   Google Scholar
• HANYS. Quality measurement: focus on the measures that matter; 2016. Available from: https://www.hanys.org/communications/pr/2016/2016-04-20_quality_measurements_focus_on_measures_that_matter.cfm. Accessed April 5, 2022.
   Google Scholar
• National Quality Forum. Measure sets and measurement systems: multistakeholder guidance for design and evaluation; 2020. Available from: https://www.qualityforum.org/Publications/2020/07/Measure_Sets_and_Measurement_Systems__Multistakeholder_Guidance_for_Design_and_Evaluation.aspx. Accessed April 5, 2022.
   Google Scholar
• Institute of Medicine. Vital signs: core metrics for health and health care progress; 2015. Washington, DC: The National Academies Press. Available from: https://www.nap.edu/catalog/19402/vital-signs-core-metrics-for-health-and-health-care-progress. Accessed April 5, 2022.
   Google Scholar
• De koning J, Smulders A, Klazinga N. The Appraisal of Indicators Through Research and Evaluation (AIRE) Instrument. Amsterdam: Academic Medical Center; 2006.
   Google Scholar
• Tricco AC, Lillie E, Zarin W, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169(7):467–473. doi:10.7326/M18-0850
   PubMed Web of Science ®Google Scholar