Selecting Performance Indicators and Targets in Health Care: An International Scoping Review and Standardized Process Framework

Figures & data

Figure 1 The flow of study identification and selection according to PRISMA-ScR guidelines.

Note: Adapted from Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. PLOS Medicine. 2021;18(3):e1003583. Creative commons license and disclaimer available from: http://creativecommons.org/licenses/by/4.0/legalcode.¹⁷

Abbreviation: PRISMA-ScR, preferred reporting items for systematic reviews and meta-analyses extension for scoping reviews.

Table 1 Peer Reviewed and Grey Literature by Country

Country	Peer Reviewed Literature	Grey Literature
Australia	3	0
Canada	8	3
France	2	0
Germany	3	0
Netherlands	3	0
New Zealand	1	1
Norway	0	0
Sweden	0	0
Switzerland	1	0
United Kingdom	1	3
United States	11	4
Total	33	11

Table 2 Peer Reviewed and Grey Literature by Field of Study

Acute Care Clinical Area	Peer Reviewed Literature	Grey Literature
Cancer	4	0
Cardiology	4	0
Critical Care	1	0
Emergency Care	2	0
Geriatrics	1	0
Hospital or Health Systems	6	11
Infection Control	4	0
Maternity	2	0
Mental Health	1	0
Patient Safety	2	0
Pediatrics	3	0
Surgery	3	0
Total	33	11

Table 3 Scoping Review Peer-Reviewed Literature Summary

Article Info				Indicators Addressed			Consensus Method Used	Article Summary
First Author	Year	Jurisdiction	Field of Study	Clinical Quality	Business Based	Target Setting
Aktaa^Citation19	2020	UK	Cardiology	Yes	No	No	Not Applicable	Paper proposes a 4-step process for KPI selection in cardiology, including identification of domains of care by constructing a conceptual framework; construction of candidate QIs via a systematic review of the literature; selection of a final set of QIs by obtaining expert opinions using the modified-Delphi method; and validation. Paper noted that expert panels have inherent bias. Therefore expansion of participants is important mitigation.
Bianchi^Citation20	2013	Switzerland	Cancer	Yes	No	No	Modified-Delphi	Colorectal Cancer Quality Indicator (QI) selection process governed by an expert panel identified 27 QIs from an original list of 149. QIs were rated using a Likert Scale and within clinical categories that followed the care continuum. Validation of the final QI set of was led by an academic researcher. Noted limitation of physician only panel. Offers a template for indicator definition sheets.
Bramesfeld^Citation21	2015	Germany	Infection Prevention and Control	Yes	No	No	Modified-Delphi	Study identified 32 indicators for measuring the prevention and management of Catheter Related Blood Stream Infections. Process considered relevance and feasibility criteria. Panelists participated in a pre-survey workshop. QIs were classified as process, outcome or structural. Likert scale was used to rate QIs.
Casey^Citation22	2013	USA	Hospital System	Yes	No	No	Modified-Delphi	Paper summarizes a panel process that examined the relevance of nationally reportable indicators to rural hospitals. Process included an expert panel that voted on the indicators to give Rural hospitals direction on which indicators are best to be used and how they align to national indicator reporting. Categorized the indicators into clinical categories; voting was noted but scale not described.
Chrusch^Citation23	2016	Canada	Critical Care	Yes	No	Yes	Nominal Group Technique	Paper describes a multiple case study in which conferences were held to have experts select indicators for comparing ICU performance. Organizations test indicators and report back on how they were used and the data results. Results identified 22 ICU indicators. Validation of indicators conducted.
Elliot^Citation24	2018	Australia	Hospital System	Yes	Yes	No	Modified-Delphi	Paper describes a 5-step process used to systematically select 20 indicators to monitor hospital strategic plan. 725 indicators were narrowed down to 110 by staff. Executives selected 20 clinical and business indicators. Five phases: (1) identification of potential indicators; (2) consolidation into a pragmatic set; (3) analysis of potential indicators against criteria; (4) mapping indicators to strategic plan; (5) key stakeholder presentation
Emond^Citation25	2015	Netherlands	Surgery	Yes	No	No	Modified-Delphi	Article describes a process that selected patient safety indicators in surgery. Process was governed by steering committee and expert panel of hospital leaders. 11 indicators were selected and validated in 8 hospitals. Patients and managers were on the panel.
Fekri^Citation26	2017	Canada	Hospital System	Yes	No	No	Modified-Delphi	Paper describes process used to select a national set of indicators. Technical group narrowed first set of metrics via quantitative survey followed by a consensus conference of end-users. 37 of 56 indicators were selected. Process included clear guiding principles.
Goldfarb^Citation27	2018	USA	Cardiology	Yes	No	No	Modified-Delphi	Systematic review of cardiology quality indicators was completed ahead of an international expert panel survey. Fifteen QIs were selected from an original list of 108, using a Likert scale. QIs were categorized as process, outcome or structural. Expert panel consisted of only physicians.
Grace^Citation28	2014	Canada	Cardiology	Yes	No	No	Modified-Delphi	Study identified quality indicators in cardiac rehabilitation. Process has three stages including ratings by working groups and validation of final QIs by stakeholders. Process resulted in a final list of 5 QIs from a list of 37. Qualitative and quantitative validation of QIs was completed.
Gurvitz^Citation29	2013	USA	Cardiology	Yes	No	No	Modified-Delphi	Paper describes indicators selection process aimed at monitoring quality improvement for adults with congenital heart disease (ACHD) conditions. Expert panel only included Physicians. 55 of 61 indicators were selected based on literature review and clinical guidelines. indicators were not independently validated.
Guth^Citation30	2016	USA	Patient Safety	Yes	No	No	Kepner-Tregoe Decision Analysis	Case study report on process used to select indicators for a hospital quality scorecard. Governing committee and working groups, narrowed 750 indicators to 25. Process included metric collection; harm evaluation; metric viability; ability to implement; categorizing metrics; assess impact; and risk assessment.
Mangione-Smith^Citation31	2011	USA	Pediatrics	Yes	No	Yes	Modified-Delphi	Paper summarizes a process that selected quality indicators for a health insurance program. Voting on a Likert scale resulted in 25 of 199 indicators being chosen. Noted field testing was needed to set targets.
Martinez^Citation32	2018	USA	Hospital System	Yes	No	No	Participatory Design Approach	Article describes how a hospital prioritized metrics for an electronic dashboard. Resulted in 10 indicators mapped to the Donabedian framework of process, outcome, and structure. Process asked end-users about barriers to using indicators. Noted that different audiences need different indicators.
Mazzone^Citation33	2014	USA	Cancer	Yes	No	No	Modified-Delphi	Panel of physicians selected Quality Indicators (QIs) to evaluate lung cancer processes of care. Narrowed original list of 18 QIs to 7. Assessed indicators using clearly defined criteria. Assessed indicators using defined criteria. Validity included testing QIs in 3 organizations. Paper noted bias of physician only panel.
Moehring^Citation34	2017	USA	Infection Prevention and Control	Yes	No	No	Modified-Delphi	Study selected indicators to aid decision making in antimicrobial stewardship Programs. Process governed by a panel of physicians and pharmacists. Panel rated QIs against 4 questions versus defined criteria. 14 metrics were selected from an original list of 90 using a Likert scale.
Morris^Citation35	2012	Canada	Infection Prevention and Control	Yes	No	No	Modified-Delphi	Paper describes process where expert panel rated potential indicators using a set of criteria. Panelists rated indicators on a Likert scale and could add anonymous comments. 4 indicators from an original list of 14 were selected. No patient or family member participated in process.
Perera^Citation36	2012	New Zealand	Hospital System	Yes	No	Yes	Not Applicable	Paper describes indicator framework. Framework includes prioritization of indicators; delineation of intent; implementation requirements; development of indicator specifications; assessment of indicator purpose, and target development. Paper notes indicators for one purpose may be inappropriate for another. indicator credibility relies on having defined purpose. Targets need to be developed based on current performance and understanding of barriers to attaining targets.
Profit^Citation37	2011	USA	Pediatrics	Yes	No	No	Modified-Delphi	Study selected indicators for neonatal intensive care units. Process resulted in 9 of 28 indicators aligned with IOM dimensions of quality using clear assessment criteria and indicator definitions. Expert panel did not include an administrator.
Reiter^Citation38	2011	Germany	Hospital System	Yes	No	No	QUALIFY Instrument	Paper describes selecting hospital quality indicators deemed suitable for hospital disclosure. Working groups of clinicians and representatives selected 31 of 55 indicators for disclosure.
Sauvegrain^Citation39	2019	France	Maternity	Yes	No	No	Delphi Survey	Paper describes process to select indicators for obstetrical care. Scientific committee and expert panel selected 13 indicators from a list of 28 that were derived from current database and literature review. Noted training ahead of process was not done but should be in future. Stated indicator targets should be discussed as an accompany process. Noted panel participants will have biases.
Schnitker^Citation40	2015	Australia	Emergency	Yes	No	No	Modified-Delphi	Study selected process quality indicators (PQIs) to monitor Emergency Department patients with cognitive impairment. Approach included building a list of PQIs based on a literature review. Process resulted in in 11 PQIs being selected from original list of 22. Process field tested indicators for data quality ahead of final selection. Noted a panel of local experts have biases and recommend involving outside experts.
Schull^Citation16	2011	Canada	Emergency	Yes	No	No	Modified-Delphi	Study selected national measures for Emergency Departments. Process resulted in selection of 48 of 170 candidate indicators. Categorized indicators by clinical domain. Noted when a panel is system-based it can underrepresent smaller and rural hospitals.
Science^Citation41	2019	Canada	Infection Prevention and Control	Yes	No	No	Modified-Delphi	Study identified metrics for antimicrobial stewardship programs. Process was governed by a steering committee and expert panel. Process resulted in the selection of 4 metrics. Noted that bias in panels can be mitigated by neutral facilitator.
Soohoo^Citation42	2010	USA	Surgery	Yes	No	Yes	Modified-Delphi	Study selected indicators for total joint replacement patients. Panel of orthopedic surgeons selected 68 indicators from an original list of 101. Field tested indicators for data quality and to inform the setting of targets.
Stang^Citation43	2013	Canada	Pediatrics	Yes	No	Yes	Modified-Delphi	Study identified indicators for high acuity pediatric conditions. An interdisciplinary advisory group selected 62 indicators from a list of 97. Noted that field testing of final indicators can inform potential benchmarks and targets.
Stegbauer^Citation44	2017	Germany	Mental Health	Yes	No	No	Modified-Delphi	Study selected indicators for schizophrenia. Expert panel narrowed 847 indicators to a list of 27 using 2 main criteria: relevance and schizophrenia. Indicator had to be defined in terms of matching an outcome (goal) and be tied to a treatment (process). Patients were on panel.
Thern^Citation45	2014	Germany	Infection Prevention and Control	Yes	No	No	Modified-Delphi	Study selected 42 indicators from a list of 99. Process included surveying experts ahead of the development of an indicator list, a literature search, ranking of indicators using a Likert scale and an in-person conference. Stated that final list of indicators should be validated for data quality.
Tsiamis^Citation46	2018	Australia	Cancer	Yes	No	No	Modified-Delphi	Physician panel selected indicators to monitor radiotherapy for men with prostate cancer. Process included literature review and categorizing QIs along the continuum of care. 17 out of an original list of 114 QIs were selected. Noted physician only panel could have bias. Noted most QIIs selected were process metrics.
van der Wees^Citation47	2019	Netherlands	Patient Safety	Yes	No	No	User Based Design	Paper proposed a framework to select Patient Reported Outcomes Measures. Framework developed using a design approach based on user needs and was guided by a project team of experts and end-user representatives.
Van Grootven^Citation48	2018	USA	Geriatrics	Yes	No	No	Delphi	Study selected indicators to evaluate in-hospital geriatric programs. 31 of 44 indicators were chosen using Likert scale against 2 criteria: appropriateness and feasibility. Panelists had at least 2 years of experience in geriatric medicine. Panel demographics balanced age and gender to ensure equity.
van Heurn^Citation49	2015	Netherlands	Surgery	Yes	No	Yes	Modified-Delphi	Panel of surgeons selected 24 neonatal surgical indicators an original list of 220. Paper emphasized importance of validation data and having external experts review final list for link to best practice. Study stated indicators need validation to inform targets.
Wood^Citation50	2013	Canada	Cancer	Yes	No	Yes	Modified-Delphi	Study selected indicators in renal cell carcinoma. Panel selected 23 indicators from an original list of 34 that were generated from a literature search and panel input. Categorization of indicators followed continuum of care. Noted physician only panel should include other professions. Noted indicator data should be tested to inform targets.

Table 4 Scoping Review Grey-Literature Summary

Article Info				Indicator Type Addressed			Consensus Method Used	Article Summary
First Author	Year	Jurisdiction	Field of Study	Clinical Quality	Business Based	Target Setting
Health Quality Ontario^Citation51	2016	Canada	Hospital	Yes	No	No	Modified-Delphi	Agency aimed to reduce number of patient safety indicators. 11 indicators selected from original inventory of 180. Structured process included clear aim, guiding principles, literature search, voting using a Likert scale, and involved representation from clinical experts, sector representatives and patients.
CIHI^Citation13	2015	Canada	System	Yes	No	No	Conference followed by Working Groups	Agency prioritized a national set of indicators. Document explains process of conference, criteria and post conference work that led to a manageable list. Broad representation but no patient or front-line manager. Had clear indicator assessment criteria. Conclusion noted requirement to validate indicators for data quality.
Ontario Hospital Association^Citation52	2019	Canada	Hospital	Yes	No	No	Modified-Delphi	Process aimed to reduce amount of measurement. Criteria used included public accountability, system monitoring, local monitoring and indicator retirement. Over 500 indicators reduced to 156 with 144 indicators retired. Expert panel did not include patients or frontline staff but noted they were required in future. Noted targets needed but did not address directly.
Health Quality and Safety Commission New Zealand^Citation53	2012	New Zealand	System	Yes	No	No	Modified-Delphi	Paper summarizes process used to select 17 indicators for public reporting and quality improvement. Process included a steering committee, advisory group, and a use of defined criteria. Panel included managers and patients.
The King’s Fund^Citation54	2010	UK	System	Yes	No	Yes	Not Applicable	Paper provides guidance on measuring acute care quality. Key topics include defining measurement; identifying audiences and purposes of indicators; impact indicators and benchmarks have on staff; and steps to select indicators. Paper emphasizes indicators and targets will motivate or unintendedly harm users. As such, processes need to ensure data is tailored to right audience.
National Institute for Health and Care Excellence^Citation55	2019	UK	System	Yes	No	No	Modified-Delphi	Document describes how national system indicators were selected and how indicators are to be used. Document shares the principles and aims of indicator selection, committee structures, testing of indicators, and consultation with stakeholders. Validation included qualitative feedback from end-users. Process involved managers and public. Emphasizes regular review required for acceptability.
The Health Foundation^Citation56	2019	UK	System	Yes	No	No	Qualitative Interviews	Multiple-case study interviewed unit-level staff on how best to reduce indicators to manageable number to enable improvement. Categorized indicators into Donabedian framework and patient reported outcome and experience measures. Assessment criteria included indicators being easily understood, relevant to area, and actionable.
Hospital Association of New York State^Citation57	2016	USA	Hospital	Yes	No	Yes	Not Applicable	Discussion paper proposes indicator selection process. Processes should aim to have indicators match clinical reality and allow improvement; include assessment criteria; use ranking methodologies; and validate indicators for data quality. Report suggests indicator assessment criteria should include fit with priorities; performance history; relevance; actionability; and financial impact.
National Quality Forum^Citation14	2019	USA	System	Yes	No	No	Modified-Delphi Process	Guide explains governance model, process and criteria used to select national indicators. Process included interdisciplinary membership, feedback from stakeholders ahead of and during process and clear assessment criteria. Indicators categorized using Donabedian framework of structure, process, and outcomes.
National Quality Forum^Citation58	2020	USA	System	Yes	Yes	No	Not Applicable	Paper discusses work of committee that examined definitions, best practices, data issues and impact of measurement. Paper offers a four-step process to assess and select indicators and noted costs and efficiency indicators should be considered. Paper stated processes should include education on how to use indicators.
Institute of Medicine^Citation59	2015	USA	System	Yes	No	Yes	Modified-Delphi Process	Paper proposes 15 indicators that measure health outcomes while reducing burden of measurement on clinicians and enhancing transparency and comparability. Report provides an overview of process followed, including criteria set used. Calls on system to test indicators for both statistical and face validity.

Table 5 The 5-P Indicator Selection Process Framework

Domain	Elements	Element Description
Purpose	Clarify Aim	Articulate the rationale for conducting an indicator and target selection exercise. By stating the process aim, whether it is to align indicators to an operational process, a strategic plan, a regulatory requirement, or public reporting, the work can be scoped properly.
	Develop Guiding Principles	Establish principles to ensure participants understand the values by which the process is being conducted. Principles may include openness, transparency, scientific soundness, relevance, accountability, scope, and span of control.
	Identify Level of Use	Identify the organizational unit that will use the indicators to ensure relevancy to end-users. As an example, indicators used by a board to monitor quality outcomes may be different than indicators selected by a clinical unit focused on process improvement.
Polity	Build Governance Structures	Identify a structure that will manage indicator and target selection to ensure it is completed. These structures may include a steering committee, a project management team, a data quality advisory group, and an expert panel that will assess potential indicators and targets.
	Recruit Participants	Select and recruit expert panel members. Panels should be diverse and multi-disciplinary to ensure equity and a broad view of how indicators and targets will be used. Composition of panels should consider the process aim and level of use when selecting participants.
Prepare	Seek End-User Input	Seek input from end-users to understand their experiences with the potential indicators under consideration and solicit ideas on the draft criteria they may recommend in evaluating indicators.
	Research Evidence-Based Literature	Identify the range of indicators used in their area or that are required by regulation. A search of literature and evidence-based guidelines, and government mandated indicators will help organizations identify a comprehensive set of indicators to assess.
	Build an Inventory of Potential Indicators	Compile a comprehensive list of indicators with definitions and data sources, so participants understand each indicator to be evaluated. If the process addresses target selection, the nature of the target (eg, past performance, benchmark, best practice) should be explained.
	Categorize Potential Indicators into Strategic Themes	Categorize indicators into themes aligned with the organization’s strategy, quadrants of the balanced scorecard, or the Donabedian framework of outcomes, process, and structure. By creating categories, process participants and end-users will better understand the linkage an indicator has with the identified purpose.
	Orient and Train Participants	Provide participants with orientation materials on the process aim, definition and purpose of each indicator, potential targets, and methods they will use to recommend indicators and targets.
Procedure	Utilize a Consensus Building Method	Identify and use a recognized consensus building method such as the Delphi, modified-Delphi, or Normative Group Technique. This is particularly important when indicators are being identified to measure a new strategy compared to a quality improvement project.
	Identify a Facilitator	Select an independent facilitator so as to not bias the process. The facilitator should be a third-party, or a neutral party from an organization’s performance measurement department.
	Establish Indicator Selection Criteria	Set criteria by which the assessment of indicators will be based. Common criteria include those prescribed by the Appraisal of Indicators through Research and Evaluation (AIRE) tool such as relevance, scientific soundness, feasibility, and validity. Criteria may change based on the aim statement and level of use described in the “Purpose” Domain.
	Analytically Assess Indicators	Identify a Likert assessment scale participants will use to evaluate indicators against criteria, and how assessments will be completed, either via survey, in person, or both.
	Set Indicator Targets	Assign a target for each indicator. Considerations may include maintaining performance if the current indicators result is ahead of a benchmark, attempting to reach a benchmark if performance is behind ideal performance, or making progress towards the benchmark should it be deemed unattainable within the period in which the indicator is being measured.
Prove	Assess Data Quality	Validate the final list of indicators by testing data quality. Processes may wish to defer the setting of specific indicator targets until after this phase to ensure targets are based on valid data trends.
	Validate with End-Users	Seek feedback from end-users on the relevance the final set of indicators and targets have to their environment and performance requirements, and whether the identified target motivates the end-user to implement improvement actions.

Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. PLOS Medicine. 2021;18(3):e1003583. doi: 10.1371/journal.pmed.1003583

 Google Scholar

Aktaa S, Batra G, Wallentin L, et al. European Society of Cardiology methodology for the development of quality indicators for the quantification of cardiovascular care and outcomes. Eur Heart J Qual Care Clin Outcomes. 2020. doi:10.1093/ehjqcco/qcaa069

 Web of Science ®Google Scholar

Bianchi V, Spitale A, Ortelli L, Mazzucchelli L, Bordoni A. Quality indicators of clinical cancer care (QC 3) in colorectal cancer. BMJ open. 2013;3(7):e002818. doi:10.1136/bmjopen-2013-002818

 PubMed Web of Science ®Google Scholar

Bramesfeld A, Wrede S, Richter K, et al. Development of quality indicators and data assessment strategies for the prevention of central venous catheter-related bloodstream infections (CRBSI). BMC Infect Dis. 2015;15:435. doi:10.1186/s12879-015-1200-9

 PubMed Web of Science ®Google Scholar

Casey MM, Moscovice I, Klingner J, Prasad S. Rural relevant quality measures for critical access hospitals. J Rural Health. 2013;29(2):159–171. doi:10.1111/j.1748-0361.2012.00420.x

 PubMed Web of Science ®Google Scholar

Chrusch CA, Martin CM; Project TQIICC. Quality improvement in critical care: selection and development of quality indicators. Can Respir J. 2016;2016:2516765. doi:10.1155/2016/2516765

 PubMed Web of Science ®Google Scholar

Elliot C, Mcullagh C, Brydon M, Zwi K. Developing key performance indicators for a tertiary children’s hospital network. Austra Health Rev. 2018;42(5):491–500. doi:10.1071/AH17263

 PubMed Web of Science ®Google Scholar

Emond YE, Stienen JJ, Wollersheim HC, et al. Development and measurement of perioperative patient safety indicators. Br J Anaesth. 2015;114(6):963–972. doi:10.1093/bja/aeu561

 PubMed Web of Science ®Google Scholar

Fekri O, Leeb K, Gurevich Y. Systematic approach to evaluating and confirming the utility of a suite of national health system performance (HSP) indicators in Canada: a modified Delphi study. BMJ open. 2017;7(4):e014772. doi:10.1136/bmjopen-2016-014772

 PubMed Web of Science ®Google Scholar

Goldfarb M, Bibas L, Newby LK, et al. Systematic review and directors survey of quality indicators for the cardiovascular intensive care unit. Int J Cardiol. 2018;260:219–225. doi:10.1016/j.ijcard.2018.02.113

 PubMed Web of Science ®Google Scholar

Grace SL, Poirier P, Norris CM, Oakes GH, Somanader DS, Suskin N. Pan-Canadian development of cardiac rehabilitation and secondary prevention quality indicators. Can J Cardiol. 2014;30(8):945–948. doi:10.1016/j.cjca.2014.04.003

 PubMed Web of Science ®Google Scholar

Gurvitz M, Marelli A, Mangione-Smith R, Jenkins K. Building quality indicators to improve care for adults with congenital heart disease. J Am Coll Cardiol. 2013;62(23):2244–2253. doi:10.1016/j.jacc.2013.07.099

 PubMed Web of Science ®Google Scholar

Guth RM, Storey PE, Vitale M, et al. Decision analysis for metric selection on a clinical quality scorecard. Am J Med Qual. 2016;31(5):400–407. doi:10.1177/1062860615589117

 PubMed Web of Science ®Google Scholar

Mangione-Smith R, Schiff J, Dougherty D. Identifying children’s health care quality measures for Medicaid and CHIP: an evidence-informed, publicly transparent expert process. Acad Pediatr. 2011;11(3):S11–21. doi:10.1016/j.acap.2010.11.003

 PubMed Web of Science ®Google Scholar

Martinez DA, Kane EM, Jalalpour M, et al. An electronic dashboard to monitor patient flow at the Johns Hopkins hospital: communication of key performance indicators using the donabedian model. J Med Syst. 2018;42(8):133. doi:10.1007/s10916-018-0988-4

 PubMed Web of Science ®Google Scholar

Mazzone PJ, Vachani A, Chang A, et al. Quality indicators for the evaluation of patients with lung cancer. Chest. 2014;146(3):659–669. doi:10.1378/chest.13-2900

 PubMed Web of Science ®Google Scholar

Moehring RW, Anderson DJ, Cochran RL, et al. Expert consensus on metrics to assess the impact of patient-level antimicrobial stewardship interventions in acute-care settings. Clin Infect Dis. 2017;64(3):377–383. doi:10.1093/cid/ciw787

 PubMed Web of Science ®Google Scholar

Morris AM, Brener S, Dresser L, et al. Use of a structured panel process to define quality metrics for antimicrobial stewardship programs. Infect Control Hosp Epidemiol. 2012;33(5):500–506. doi:10.1086/665324

 PubMed Web of Science ®Google Scholar

Perera R, Dowell A, Crampton P. Painting by numbers: a guide for systematically developing indicators of performance at any level of health care. Health Policy. 2012;108(1):49–59. doi:10.1016/j.healthpol.2012.07.008

 PubMed Web of Science ®Google Scholar

Profit J, Gould JB, Zupancic JAF, et al. Formal selection of measures for a composite index of NICU quality of care: baby-MONITOR. J Perinatol. 2011;31(11):702–710. doi:10.1038/jp.2011.12

 PubMed Web of Science ®Google Scholar

Reiter A, Geraedts M, Jäckel W, Fischer B, Veit C, Döbler K. Selection of hospital quality indicators for public disclosure in Germany. Z Evid Fortbild Qual Gesundhwes. 2011;105(1):44–48. doi:10.1016/j.zefq.2010.12.024

 PubMedGoogle Scholar

Sauvegrain P, Chantry AA, Chiesa-Dubruille C, Keita H, Goffinet F, Deneux-Tharaux C. Monitoring quality of obstetric care from hospital discharge databases: a Delphi survey to propose a new set of indicators based on maternal health outcomes. PLoS One. 2019;14(2):e0211955. doi:10.1371/journal.pone.0211955

 PubMed Web of Science ®Google Scholar

Schnitker LM, Martin-Khan M, Burkett E, Beattie ERA, Jones RN, Gray LC. Process quality indicators targeting cognitive impairment to support quality of care for older people with cognitive impairment in emergency departments. Acad Emerg Med. 2015;22(3):285–298. doi:10.1111/acem.12616

 PubMed Web of Science ®Google Scholar

Schull MJ, Guttmann A, Leaver CA, et al. Prioritizing performance measurement for emergency department care: consensus on evidence-based quality of care indicators. CJEM. 2011;13(5):300–309, E28-E43. doi:10.2310/8000.2011.110334

 Google Scholar

Science M, Timberlake K, Morris A, Read S, Le Saux N. Quality metrics for antimicrobial stewardship programs. Pediatrics. 2019;143(4). doi:10.1542/peds.2018-2372

 PubMed Web of Science ®Google Scholar

SooHoo NF, Lieberman JR, Farng E, Park S, Jain S, Ko CY. Development of quality of care indicators for patients undergoing total hip or total knee replacement. BMJ Qual Saf. 2011;20(2):153–157. doi:10.1136/bmjqs.2009.032524

 PubMed Web of Science ®Google Scholar

Stang AS, Straus SE, Crotts J, Johnson DW, Guttmann A. Quality indicators for high acuity pediatric conditions. Pediatrics. 2013;132(4):752–762. doi:10.1542/peds.2013-0854

 PubMed Web of Science ®Google Scholar

Stegbauer C, Willms G, Kleine-Budde K, Bramesfeld A, Stammann C, Szecsenyi J. Development of indicators for a nationwide cross-sectoral quality assurance procedure for mental health care of patients with schizophrenia, schizotypal and delusional disorders in Germany. Z Evid Fortbild Qual Gesundhwes. 2017;126:13–22. doi:10.1016/j.zefq.2017.07.006

 PubMed Web of Science ®Google Scholar

Thern J, de With K, Strauss R, Steib-Bauert M, Weber N, Kern WV. Selection of hospital antimicrobial prescribing quality indicators: a consensus among German antibiotic stewardship (ABS) networkers. Infection. 2014;42(2):351–362. doi:10.1007/s15010-013-0559-z

 PubMed Web of Science ®Google Scholar

Tsiamis E, Millar J, Baxi S, et al. Development of quality indicators to monitor radiotherapy care for men with prostate cancer: a modified Delphi method. Radiother Oncol. 2018;128(2):308–314. doi:10.1016/j.radonc.2018.04.017

 PubMed Web of Science ®Google Scholar

van der Wees PJ, Verkerk EW, Verbiest MEA, et al. Development of a framework with tools to support the selection and implementation of patient-reported outcome measures. J Patient Rep Outcomes. 2019;3(1):75. doi:10.1186/s41687-019-0171-9

 PubMedGoogle Scholar

Van Grootven B, McNicoll L, Mendelson DA, et al. Quality indicators for in-hospital geriatric co-management programmes: a systematic literature review and international Delphi study. BMJ Open. 2018;8(3):e020617. doi:10.1136/bmjopen-2017-020617

 PubMed Web of Science ®Google Scholar

van Heurn E, de Blaauw I, Heij H, et al. Quality measurement in neonatal surgical disorders: development of clinical indicators. Eur J Pediatr Surg. 2015;25(6):526–531. doi:10.1055/s-0034-1396416

 PubMed Web of Science ®Google Scholar

Wood L, Bjarnason GA, Black PC, et al. Using the Delphi technique to improve clinical outcomes through the development of quality indicators in renal cell carcinoma. J Oncol Pract. 2013;9(5):e262–267. doi:10.1200/JOP.2012000870

 PubMedGoogle Scholar

Health Quality Ontario (HQO). Hospital sector indicator reduction and management strategy; 2016. Available from: https://www.hqontario.ca/System-Performance/Measuring-System-Performance/Hospital-Sector-Indicator-Reduction-and-Management-Strategy. Accessed April 5, 2022.

 Google Scholar

Canadian Institute for Health Information and Statistics Canada. Rethink, renew, retire: report from the fourth consensus conference on evaluating priorities for Canada’s health indicators; 2015. Available from: https://secure.cihi.ca/free_products/Rethink_Renew_Retire.pdf. Accessed April 5, 2022.

 Google Scholar

Ontario Hospital Association and Health Quality Ontario. A sustainable indicator reduction and management strategy for the Ontario Hospital Sector; 2019. Available from: https://www.oha.com/health-system-transformation/high-performing-health-system/a-sustainable-indicator-reduction-and-management-strategy. Accessed April 5, 2022.

 Google Scholar

Health Quality & Safety Commission. Health quality & safety indicators: summary feedback document; 2012. Available from: https://www.hqsc.govt.nz/assets/Health-Quality-Evaluation/PR/HQSI-Feedback-and-Engagement-document-July12-web.pdf. Accessed April 5, 2022.

 Google Scholar

The King’s Fund. Getting the measure of quality: opportunities and challenges; 2010. Available from: https://www.kingsfund.org.uk/publications/getting-measure-quality. Accessed April 5, 2022.

 Google Scholar

National Institute for Health and Care Excellence. NICE indicator process guide; 2019. Available from: https://www.Nice.Org.Uk/Media/Default/Get-Involved/Meetings-In-Public/Indicator-Advisory-Committee/Ioc-Process-Guide.Pdf. Accessed April 5, 2022.

 Google Scholar

The Health Foundation. The measurement maze; 2019. Available from: https://www.health.org.uk/publications/reports/the-measurement-maze. Accessed April 5, 2022.

 Google Scholar

HANYS. Quality measurement: focus on the measures that matter; 2016. Available from: https://www.hanys.org/communications/pr/2016/2016-04-20_quality_measurements_focus_on_measures_that_matter.cfm. Accessed April 5, 2022.

 Google Scholar

National Quality Forum. Committee guidebook for the NQF measure endorsement process; 2019. Available from: https://www.qualityforum.org/Measuring_Performance/Measuring_Performance.aspx. Accessed April 5, 2022.

 Google Scholar

National Quality Forum. Measure sets and measurement systems: multistakeholder guidance for design and evaluation; 2020. Available from: https://www.qualityforum.org/Publications/2020/07/Measure_Sets_and_Measurement_Systems__Multistakeholder_Guidance_for_Design_and_Evaluation.aspx. Accessed April 5, 2022.

 Google Scholar

Institute of Medicine. Vital signs: core metrics for health and health care progress; 2015. Washington, DC: The National Academies Press. Available from: https://www.nap.edu/catalog/19402/vital-signs-core-metrics-for-health-and-health-care-progress. Accessed April 5, 2022.

 Google Scholar