Qualitative evaluations of women’s leadership programs: a global, multi-sector systematic review

ABSTRACT

Objective

The contribution of women’s leadership programs to gender change in organisations is controversial, and evidence of programs’ effectiveness is siloed across countries, sectors and industries. This systematic review aimed to provide a summary of current global efforts to evaluate women’s leadership programs.

Method

A systematic review protocol was registered with Open Science Framework prior to data extraction. Eight databases from multidisciplinary fields including (but not limited to) behavioural, social, physical, health and life sciences, management and business, and gender and women’s studies were searched for academic papers examining the outcomes of women’s leadership programs. Twenty-four studies were appraised for methodological quality using Joanna Briggs Institute guidelines and 16 studies (11 peer reviewed articles and five theses) were included in the review.

Results

Data were synthesized using an updated Kirkpatrick typology with seven categories used to classify evaluation outcomes. Subjective outcome levels were addressed more frequently than objective levels. Promotion to a leadership position was the sole objective outcome addressed, but methodological limitations of the included studies mitigate a link between programs and women’s career advancement.

Conclusions

Currently, the global evidence of women’s leadership programs’ impact on individuals and organisations is inconclusive. This systematic review emphasises the need for enhanced methodological and theoretical rigour to guide the development of future women’s leadership programs and their evaluation.

Key Points

What is already known about this topic:

1. Reasons for the persistent underrepresentation of women in leadership are multiple and complex, and women leaders face both structural and individual barriers in their efforts to ascend to the top levels of organisations.

2. Research has largely moved away from a micro-level focus on how best to assimilate women into existing, male-dominated workplaces, to a focus on dismantling structural barriers to women’s leadership such as meso-level organisational policies and practices and macro-level national and societal factors such as culture and legislation. In practice, however, micro-level strategies to address gender imbalance in leadership are frequently employed, and there is much debate regarding their capacity to contribute to gender change in organisations.

3. Our knowledge about the impact of women’s leadership programs is limited and fragmented, as is our understanding of how this impact is assessed.

What this paper adds:

1. This systematic review is the first to map the quality and nature of women’s leadership program evaluations globally, demonstrating the applicability of a systematic review methodology to leadership, management, and organisational psychology research.

2. Our paper challenges the premise that individual level strategies can affect organisational and cultural change by examining the evidence of the effectiveness of micro-level approaches to women’s leadership development.

3. The paper includes practical recommendations to advance women’s leadership program evaluation practice and research, emphasising the need for enhanced methodological rigour and realistic expectations regarding both the program and the evaluation.

KEYWORDS:

• Evaluation
• gender
• leadership
• organisational psychology
• systematic review
• women

Despite compelling ethical (Kalev et al., 2006; Skaggs, 2008) and economic (Alcázar et al., 2013; Desvaux et al., 2008; Noland et al., 2016; Woetzel, 2015) arguments for more women in leadership, and decades of equal employment opportunity legislation and affirmative action programs, women hold less than a quarter (24%) of senior roles globally (Grant Thornton, 2018). Reasons for the persistent underrepresentation of women in leadership are multiple and complex, and women leaders face both structural and internal barriers in their efforts to ascend to the top levels of organisations. Structural factors such as normative perceptions of men as leaders (Ibarra et al., 2013), gendered career paths and gendered work (Hoobler et al., 2011), and lack of access to networks and sponsors (Davey, 2008), and individual factors such as a relative lack of confidence, communication skills, self-efficacy and willingness to self-promote (Kay & Shipman, 2014; O’Neil & Hopkins, 2015; Pelfrey et al., 2022) are commonly cited barriers to women’s career progression.

Metz and Kumra (2019) identify a clear evolution in the academic literature on women’s leadership from a focus on fixing the women (1967–1976), to fixing the organisation (1977–1996), to fixing the environment (1998-present). These periods describe approaches to gender equality in leadership that can also be categorised as micro-, meso-, and macro-level, respectively (van Esch et al., 2017). Research has largely moved away from a micro-level focus on how best to assimilate women into existing, male-dominated workplaces, to a focus on dismantling structural barriers to women’s leadership such as meso-level organisational policies and practices and macro-level national and societal factors such as culture and legislation (Metz & Kumra, 2019).

In practice, however, micro-level strategies to address gender imbalance in leadership are frequently employed (J. A. de Vries & van den Brink, 2016; European Commission, 2010; Leimon et al., 2010; Nentwich, 2006; Zahidi & Ibarra, 2010). Women’s leadership programs are an enduring example of a micro-level approach, encouraging personal agency, self‐awareness, self‐confidence, and the development of leadership skills and networks within an all-female environment (O’Neil & Bilimoria, 2005; Weyer, 2007). Rather than being a forgotten relic of the “fix the women” era, demand for women’s leadership programs is high (J. de Vries, 2010; Debebe et al., 2016; Kassotakis, 2017; Kolhatkar, 2016), and many major organisations have a women’s leadership program (Anderson et al., 2008; Sandler, 2014). Taking a global perspective, over the last two decades 36% of Australian universities had implemented staff development programs exclusively for women (Tessens, 2008). In the UK, this figure was 31% (Bagilhole, 2002), and in the US, over one third of the top-ranking business schools offer women’s-only leadership programs (Sugiyama et al., 2016).

Leadership programs cost organisations billions of dollars annually (Gurdjian et al., 2014; Kassotakis & Rizk, 2015) and it is reasonable to argue that investment of resources to develop women leaders should be focused on evidence-based results (Kassotakis, 2017). However, there is a dearth of leadership development frameworks specifically for women (Gipson et al., 2017) and management journals rarely publish leadership development studies with woman-only participants (Madsen & Scribner, 2017; Ngunjiri & Gardiner, 2017). As such, there is little empirical evidence to suggest if and when women’s leadership programs are most beneficial (Harris & Leberman, 2012; Kulik & Roberson, 2008b) and there is much debate regarding their capacity to contribute to gender change in organisations (J. A. de Vries & van den Brink, 2016; R. J. Ely & Meyerson, 2000; Nentwich, 2006; Zanoni et al., 2010).

Much of the scholarly literature that informs our current understanding of women’s leadership development is descriptive (J. de Vries, 2010), or focuses on practical recommendations for organisations to implement (R. J. Ely et al., 2011; Hopkins et al., 2008) rather than implementing and evaluating these recommendations within organisational settings (Gipson et al., 2017). There are few studies that both describe and evaluate women’s leadership programs, let alone compare their outcomes (Bierema, 2017; Debebe, 2009; Debebe et al., 2016; Hayward & Voller, 2010) and any evidence that does exist for the effectiveness of women’s leadership programs is siloed within individual programs, industries, sectors, and countries (Kulik & Roberson, 2008a; Ngunjiri & Gardiner, 2017). As such, our knowledge about the impact of women’s leadership programs is limited and fragmented, as is our understanding of how this impact is assessed. Responding to calls in the literature for research to inform the improved evaluations of women’s leadership programs (Debebe et al., 2016; Gipson et al., 2017; Kassotakis, 2017), we identify a need to universally appraise, synthesise and report on the current nature of women’s leadership programs and the quality of women’s leadership program evaluations.

For the purpose of the current work, evaluation is defined as a systematic determination of the merit, worth or value of a program and includes “the collection of descriptive and judgemental information that is necessary to make decisions about the utility of training efforts and identify areas for modification and improvement” (K. Ely et al., 2010, p. 588). Critically, evaluations need to be of sufficient quality to inform learning and accountability regarding a program’s effectiveness (Hirsh et al., 2011; Raifman et al., 2017; Stufflebeam, 2001).

Evaluation may serve a number of purposes and may be conducted through a number of methods (Easterby-Smith, 1994; Voller, 2010). Women’s leadership programs vary in their design, target audience, organisational and cultural contexts, and intended outcomes (Clarke, 2011; J. de Vries, 2010; Debebe et al., 2016; Vinnicombe & Singh, 2002), and their evaluation may consequently vary in response to the nature and emphasis of the program itself (Debebe et al., 2016; Grove et al., 2005). Evaluations may be conducted internally (i.e., the evaluators are affiliated with the organisation) or externally (Hirsh et al., 2011; Torres & Preskill, 2001), and for summative purposes (i.e., to assess the results of a women’s leadership program after completion) and/or formative purposes (i.e., to improve program development and implementation during the intervention; Easterby-Smith, 1994; Ford & Sinha, 2008; Voller, 2010). Data may be collected once, or at multiple times pre-, mid-, post-program, and/or longitudinally, and from one or multiple stakeholders (Grove et al., 2005; Hirsh et al., 2011).

Although there is no standard evaluation model for women’s leadership programs, Kirkpatrick’s (1979, 1996) four-step taxonomy (measuring reactions, learning, behaviour and results) dominates the general training evaluation literature (K. Ely et al., 2010; Ford & Sinha, 2008; Hirsh et al., 2011), and provides a useful framework for mapping program outcomes. Reactions (level 1) refers to individuals’ subjective evaluations of their training experiences. Learning (level 2) refers to the retention of training material. Behaviour (level 3) refers to the influence of a program on an individual’s work-related behaviours. Lastly, Results (level 4) refers to the influence of training on organisational objectives. Examining the frequency with which these outcome levels are measured, as well as the purposes and methods of women’s leadership program evaluations, will aid our understanding of the function of evaluation in different contexts, which outcomes levels are considered pertinent and possible to measure, the validity of reported outcomes, and the sustainability of outcomes over time.

Evaluations can be based on qualitative or quantitative data, or both (mixed methods). Qualitative data from participant observations, case studies, in depth interviews, narrative and document analysis, and focus groups can be particularly useful for evaluators to explore and explain the effectiveness or otherwise of interventions (Farmer et al., 2006; Foster-Fishman et al., 2005; Lockwood et al., 2017; Marecek, 2003). Questions relating to the utility of women’s leadership programs and their impact can be addressed through open-ended responses from participants, facilitators, managers and other relevant stakeholders (Farquhar et al., 2006; Lockwood et al., 2017). Not only can these responses provide evaluators with participant reactions to the program, but they can also provide information regarding knowledge gained and transfer of learning once participants return to work (Basl, 2000). To our knowledge, there are no existing qualitative reviews in this area.

This systematic review focuses on women’s leadership program evaluations based on qualitative data. We have included descriptive and demographic information relating to the included publications and women’s leadership programs to provide a complete and exhaustive summary of the current global status of academic efforts to evaluate women’s leadership programs within organisations. Though commonly used in the health sciences to inform best practice, systematic reviews in the field of Industrial and Organisational psychology are rare (Briner & Rousseau, 2011; Rojon et al., 2011) and those which exist are often poorly executed (Schalken & Rietbergen, 2017). For these reasons, the methodology for this qualitative systematic review is informed by guidelines from the Joanna Briggs Institute (Lockwood et al., 2017), for a rigorous and defensible methodology.

Review questions

The main questions for this systematic review are:

1. What is the methodological quality of women’s leadership program evaluations?

2. What are the descriptive and demographic characteristics of women’s leadership program evaluations and of the programs themselves?

3. How are women’s leadership programs evaluated? Secondary questions include: What data collection methods and time points are used?; For what purposes are program evaluations conducted?; What evaluation methodologies are used?; and What levels of outcomes are measured?

Method

Review registration

The systematic review protocol was registered with the Open Science Framework a priori to the commencement of data extraction.

Inclusion and exclusion criteria

This systematic review considered peer reviewed and non-peer reviewed academic papers written in English from multidisciplinary fields including (but not limited to) behavioural, social, physical, health and life sciences, management and business, and gender and women’s studies. The focus was on qualitative data including (but not limited to) research designs and methods such as grounded theory, ethnography, phenomenology, action research, and feminist research. Included studies were those that evaluated women’s leadership programs conducted within organisational settings. There were no limits applied in terms of country, industry, or sector. Included participants were women who had completed a women’s leadership program, as well as program providers and other relevant stakeholders from organisations that had hosted a program. Excluded studies were non-academic publications, papers using quantitative or mixed methods, and papers that did not evaluate women’s leadership programs or identify women’s leadership program outcomes.

Search strategy

A specialist librarian assisted in developing the search strategy. Initially developed in PsycINFO, the search strategy was then adapted for all other databases to suit local syntax, thesaurus terms and subject headings. A fully mapped search strategy for PsycINFO appears in Appendix A. The following electronic databases were systematically searched from the date of database inception to December 2021: PsycINFO, Web of Science, Emerald Insight, Business Source Complete, Health Business Elite, Academic Search Complete, Sociological Abstracts, and Scopus. All relevant journal articles were manually reviewed for forward and backward citations to ensure no relevant papers were missed. That is, the reference lists of all relevant papers were checked for additional relevant papers (forward citations), and a Google Scholar search looked for additional relevant papers that had cited the original journal articles (backward citations). Academic grey literature was sought from Google Scholar and ProQuest. Each database was searched, and citations were imported into reference management software (EndNote). Duplicate articles were identified by EndNote and removed by hand. The titles and abstracts of all remaining articles were screened by the first author and irrelevant articles were excluded.

Assessment of methodological quality

Full text qualitative papers selected for retrieval were assessed by two independent reviewers for methodological rigour prior to inclusion in the review using the JBI Critical Appraisal Checklist for Qualitative Research (Lockwood et al., 2015), a standardised critical appraisal instrument from the Joanna Briggs Institute System for the Unified Management, Assessment and Review of Information (JBI SUMARI). All papers were independently reviewed by the first author and a second reviewer; the second and third authors independently reviewed half the papers each. Any disagreement that arose between authors was to be resolved by the third reviewer, however, the reviewers were able to resolve all disagreements within their original pairings.

Data extraction

Qualitative data extracted from papers included information regarding the publications, the women’s leadership programs, and the program evaluations. Descriptive and demographic features of the included studies included paper type (peer reviewed journal article or thesis), authors, country of origin, and the industries in which these studies were conducted. Program-related data included program aims, participant and cohort descriptions, recruitment procedures, program length, and whether the program was discrete or embedded in wider gender change efforts. Information pertaining specifically to women’s leadership program evaluations included whether the evaluation was conducted internally or externally, for summative or formative purposes, the methods, sources and time points of data collection, and the levels of outcomes measured.

Data synthesis

Many researchers have supplemented Kirkpatrick’s model with objective outcome categories designed to better inform the modification and improvement of programs and gauge program impact at the structural level. Previous systematic reviews examining the effectiveness of general leadership development programs (Frich et al., 2015; Voller, 2010) have adopted the typology of evaluation outcomes outlined in Table 1. In keeping with this typology, seven categories were used to classify evaluation outcomes: Reaction (1), Knowledge (subjective) (2A), Behaviour/Expertise (3A), System results/Performance (subjective) (4A) and System results/Performance (objective) (4B).

Table 1. Adapted Kirkpatrick typology of evaluation outcomes for leadership programs.

Level	Label	Description
Level 1	Reaction	How participants feel about the program and their satisfaction with different components
Level 2A	Knowledge (subjective)	Principles, facts, attitudes, and skills learned during or by the end of the program, as communicated in statements, opinion, belief, or judgement by the participant or trainer
Level 2B	Knowledge (objective)	Principles, facts, attitudes, and skills learned during or by the end of the program, measured by objective means
Level 3A	Behaviour/ expertise (subjective)	Changes in on-the-job behaviour perceived by participants, or global perceptions by peers or a supervisor
Level 3B	Behaviour/ expertise (objective)	Tangible results that evaluate changes in on-the-job behaviour or supervisor rating of observable behaviour
Level 4A	System results/ performance (subjective)	Organisational results perceived by respondents and group effectiveness perceived by subordinates
Level 4B	System results/ performance (objective)	Tangible organisational results such as reduced costs, improved quality, and promotions

From Frich et al. (2015)

Results

Study inclusion

A Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) flow diagram was created to demonstrate the number of studies remaining at each stage of the review process, and to document the reasons for study exclusion (see Figure 1). There were 42,994 studies identified from the search strategy and a total of 34,678 were eligible for inclusion after duplicates were removed. All papers were screened for inclusion based on title and abstract and 34,589 were excluded. A total of 89 publications were assessed for inclusion through full text review using the inclusion/exclusion criteria. Sixty-six studies were then excluded after full text review, leaving 24 studies that were appraised for inclusion. Of these, eight were excluded due to extremely poor methodological quality (i.e., few items on the JBI tool were endorsed or program outcomes were unclear). Appendix B contains information on excluded studies and the reasons for exclusion.

Figure 1. PRISMA diagram.

Alt Text: A flow diagram depicts the flow of information through the different phases of this systematic review. It maps out the number of records identified, included and excluded, and the reasons for exclusions.

Methodological quality

Table 2 contains the quality appraisals of the 16 included studies. Eleven of these were peer reviewed articles and five were theses. The most common weaknesses in the study designs were failure to address the influence of the researcher on the research, and vice-versa (nine papers), and a lack of evidence of ethical approval (nine papers). Six papers lacked a statement locating the researcher culturally or theoretically, and five lacked congruity between the stated philosophical perspective and the research methodology. Participants and their voices were adequately represented, and conclusions drawn in the research report flowed from the analysis, or interpretation, of the data in 14 papers. There was congruence between the research methodology and the methods used to collect data, the research methodology and the interpretation of data, and the research methodology and the interpretation of results in 15 papers. The research methodology and research questions or objectives were congruent in all 16 papers.

Table 2. JBI-Qualitative critical appraisal checklist.

	Studies appraised
	Broyles (2019)*	Brue and Brue (2016)	Calizo (2011)*	Clarke (2011)	Debebe (2017)	Debebe (2011)	Debebe (2009)	J. A. de Vries and van den Brink (2016)	J. de Vries (2010)*	Gibbs (2016)*	Harris and Leberman (2012)	Kvach et al. (2017)	Louw and Zuber-Skeritt (2009)	Monks and Barker (1999)	Selzer et al. (2017)	Vinas (2018)*
Q1: Is there congruity between the stated philosophical perspective and the research methodology?	Y	Y	Y	N	Y	Y	N	Y	Y	Y	Y	N	N	N	Y	Y
Q2: Is there congruity between the research methodology and the research question or objectives?	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y
Q3: Is there congruity between the research methodology and the methods used to collect data?	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	?	Y	Y
Q4: Is there congruity between the research methodology and the representation and analysis of data?	Y	Y	Y	Y	Y	Y	?	Y	Y	Y	Y	Y	Y	Y	Y	Y
Q5: Is there congruity between the research methodology and the interpretation of results?	Y	Y	Y	Y	Y	Y	?	Y	Y	Y	Y	Y	Y	Y	Y	Y
Q6: Is there a statement locating the researcher culturally or theoretically?	Y	Y	Y	N	Y	N	N	Y	Y	Y	N	N	Y	Y	Y	N
Q7: Is the influence of the researcher on the research, and vice-versa, addressed?	Y	N	Y	N	Y	N	N	Y	Y	Y	N	N	N	N	Y	N
Q8: Are participants, and their voices, adequately represented?	Y	Y	Y	Y	Y	Y	Y	N	Y	Y	Y	Y	Y	N	Y	Y
Q9: Is the research ethical according to current criteria or, for recent studies, and is there evidence of ethical approval by an appropriate body?	Y	Y	Y	N	N	N	N	?	Y	Y	N	Y	N	N	Y	N
Q10: Do the conclusions drawn in the research report flow from the analysis, or interpretation, of the data?	Y	Y	Y	Y	Y	Y	N	Y	Y	Y	?	Y	Y	Y	Y	Y

Y = Yes; N = No;? = Unclear.

*Thesis.

Characteristics of included studies

Table 3 contains descriptive and demographic features of the evaluation studies and details of the women’s leadership programs.

Table 3. Descriptive and demographic characteristics of the studies and details of the women’s leadership programs.

	Publication Details			Program Description
Authors	Publication type	Country	Industry	Program Name	Program Aim	Program Length	Discrete or Embedded	Participant Description
Broyles (2019)	Thesis	USA	Higher Education	Leading Edge Institute	… inspiring, preparing and empowering “the women of Alabama to lead our State to a more equitable and prosperous future”.	1 week	Not stated	N = 10 Ps from multiple program cohorts Various education, experience and career levels Recruitment procedures not stated
Brue and Brue (2016)	Paper	USA	STEM	Oklahoma Career Tech Women in Leadership Program (OCTWL)	The goal was to empower women to develop leadership strategies, provide a forum to share leadership insights, and address organizational challenges faced by women.	10 months (4 meetings over a 10 month period)	Not stated	N = 7 Ps from multiple program cohorts Various education, experience and career levels Recruitment procedures not stated
Calizo (2011)	Thesis	USA	Higher Education	Women’s Institute for Leadership Development Program (WILD) & Higher Education Collaborative for Women’s Leadership Development (HEC)	The goal of the (WILD) program was to ―advance women leaders in central academic as well as top level administrative positions [and to] position women to attain a variety of higher level positions at most public and private colleges and universities. The goal of HEC is to ―establish a strong network of women in the Greater Cincinnati higher education community‖.	Unclear (Workshops over Fall semester plus internship, lengths varied)	Embedded	N = 6 Ps from multiple cohorts and multiple programs Open to both faculty and staff Associate Professor or higher/Director level or higher
Clarke (2011)	Paper	Australia	Various	The High Impact Women in Leadership Program (HIWL)	The program was developed in response to the South Australian Government Strategic Plan (2007) which set targets for women in leadership positions comprising 50 percent of general membership of State Government Boards and Committees, Chairpersons of State Government Boards and Committees, members of Parliament and public sector employees at executive level by 2014.	6 months (2 × 2 day modules, 1 day module each month x 6 months)	Not stated	N = 17 Ps from multiple program cohorts Senior level managers and women at lower levels who have been identified as having high potential for leadership
Debebe (2017)	Paper	(as above)	(as above)	(as above)	(as above)	(as above)	(as above)	(as above)
Debebe (2011)	Paper	(as above)	(as above)	(as above)	(as above)	(as above)	(as above)	(as above)
Debebe (2009)	Paper	Global	Government	Women’s Leadership Series (WLS) comprising “Women’s Leadership & Management” and “Negotiation Skills for Women”	The WLS is administered by the CGIAR Gender and Diversity (G&D) Program as part of its comprehensive program of increasing the percentage of highly qualified developing-country and female staff in the CGIAR Centers.	1 week	Embedded	N = 24 Ps from multiple program cohorts Various education, experience and career levels Recruitment procedures not stated
J. A. de Vries and van den Brink (2016)	Paper	Australia	Higher Education & Policing	Not stated	… sought to turn the inevitable focus on the development of individual women inherent in women’s programs from a weakness (fixing women) to a strength, through the additional concurrent focus on gendered organizational change.	6 months + (Time commitment varies from year to year. More than 5 workshop days, other activities and mentoring over a period of 6+ months) Policing: Participants’ time commitment was 9 days in total, plus Peer Learning Group and mentoring commitments	Embedded	N = 23 (program participants) N = 6 (champions) N = 14 (mentors) Ps from multiple cohorts, from multiple cohorts in multiple industries Recruitment of Ps counter to the prevailing culture – placed emphasis on selecting a diverse and broadly representative cohort group
J. de Vries (2010)	Thesis	(as above)	(as above)	(as above)	…to enable women at the university to develop leadership skills and knowledge in order to increase their participation in positions of leadership and in all the university’s decision-making processes; and to contribute to culture change in the University to encourage and welcome women’s involvement in leadership and decision-making positions.	(as above)	(as above)	(as above)
Gibbs (2016)	Thesis	USA	Academic Medicine	Hedwig van Ameringen Executive Leadership in Academic Medicine Program (ELAM)	“ELAM is designed to develop the personal and professional skills required to lead and manage successfully in today’s multifaceted health care environment, with the unique challenges facing women in leadership positions”.	1 year (no detail regarding meeting frequency)	Not stated	N = 19 Ps from multiple cohorts Associate or full professor ranks Rigorous selection process – must have leadership potential and commit to ongoing professional development
Harris and Leberman (2012)	Paper	New Zealand	Higher Education	New Zealand Women in Leadership Program (NZWIL)	The program was designed to build on the existing programs in two universities and break the circle of underrepresentation in senior positions as well as address the need for more female leaders in New Zealand universities. The program aims to recognize and enhance women’s leadership capacities and influence within universities.	5 days	Not stated	N = 96 (survey), 76 (phone interview) Ps from multiple program cohorts Women at upper-middle levels (academic and general staff positions) who are in, or aspire to be in, leadership positions
Kvach et al. (2017)	Paper	Ethiopia/USA	Academic Medicine	Not stated	By the end of the fellowship, participants were expected to achieve the following objectives: 1) Describe concepts of gender equity and health equity research; 2) plan a project to promote gender equity in their home institution; 3) prepare a professional development plan, including leadership, research and teaching goals, potential challenges, and strategies to achieve their goals; 4) present a summary of their proposals to UW faculty leaders; 5) network and collaborate with external partners; and 5) serve as academic leaders and champions for gender equity in their home institutions and in Ethiopia.	2 weeks + (2 week fellowship plus encouraged to maintain contact with mentor)	Embedded	N = 9 Ps from multiple cohorts Early or mid-level academic positions
Louw and Zuber-Skeritt (2009)	Paper	South Africa/Australia	Higher Education	The Australia-South Africa Links Program (Leadership Development Program, LDP)	The main purpose of the Links Programme was to achieve greater quality and quantity of research output and publications, and improve academic leadership and higher-degree supervision among women.	2 years (3 × 1 week residential workshops over 2 years)	Not stated	N = 14 Ps from single cohort Academics (mostly in staff development positions)
Monks and Barker (1999)	Paper	Ireland	Higher Education	Management Development – A Female Perspective	… the purpose of the course was … to allow individual women academics and administrators to explore issues that impacted on their lives and career development and to begin to facilitate the setting of individual agendas for future action.	2 days	Not stated	N = 32 Ps from multiple cohorts Various education, experience and career levels
Selzer et al. (2017)	Paper	USA	Higher Education	Not stated	The goals of the program included increased self-awareness and competencies in the following broad leadership development components: visioning and strategic alignment, finance and operations, understanding and building culture, and negotiation, teams, and conflict.	7 months (3.5 hours per month over 7 months)	Embedded	N = 3 (the authors) Ps from single cohort Exclusive program with competitive selection process (Must have Master’s Degree, certain rank and paygrade, minimum length of employment with the university) Open to both faculty and staff
Vinas (2018)	Thesis	USA	STEM	Not stated	The intended goals of the program … are to further develop the leadership talent of their women leaders by offering them the space and time to focus on themselves and their development, and through these means to encourage their retention.	3 days	Embedded	N = 18 Ps from multiple cohorts Director-level minimum, most with an MBA, recognised as “high potential”

Descriptive and demographic features

A total of 16 papers, evaluating 12 programs, were reviewed. The same women’s leadership program is discussed in Debebe (2009, 2011, 2017), J. de Vries (2010), J. A. de Vries and van den Brink (2016), Calizo (2011) and Selzer et al. (2017).

Of the 12 programs evaluated, five were delivered in the USA, two in Australia, and one program each in New Zealand and Ireland. A further three programs were delivered across multiple countries; one paper evaluated a collaborative program between Ethiopia and the USA, another between South Africa and Australia, and one program included global participants. Three papers evaluated women’s leadership programs in more than one industry. Six programs were set in the context of higher education, two each in academic medicine and STEM (Science, Technology, Engineering and Maths), one program each in policing and government, and one program was open to participants from multiple industries.

Women’s leadership program descriptions

Program aims varied considerably. Two programs sought to identify and explore issues that impacted on women’s personal and professional development (Brue & Brue, 2016; Monks & Barker, 1999) whereas four stated the explicit purpose of increasing the percentage of women leaders (Calizo, 2011 [WILD program]; Clarke, 2011; Debebe, 2009, 2011, 2017; Harris & Leberman, 2012). The development of personal and professional skills required for leadership featured in seven programs (Broyles, 2019; Brue & Brue, 2016; Gibbs, 2016; Harris & Leberman, 2012; Selzer et al., 2017; Vinas, 2018). The onus was on participants to act as champions for gender equity within their organisation in Kvach et al. (2017), but the program examined in J. de Vries (2010) and J. A. de Vries and van den Brink (2016) had a concurrent focus on gendered organisational change. Other program aims included establishing a strong network of women (Calizo, 2011), encouraging the retention of women (Vinas, 2018), and an increase in the quality and quantity of research publications (Louw and Zuber-Skeritt, 2009).

Samples ranged from three to 76 participants. Participants were recruited from a single program cohort (i.e., one in which the participants all start and finish at the same time) in two papers (Louw and Zuber-Skeritt, 2009; Selzer et al., 2017). The remaining 14 papers recruited participants from multiple program cohorts. There were single papers recruiting participants from two (Clarke, 2011), three (Brue & Brue, 2016) and four (Vinas, 2018) program cohorts, and 11 papers recruited participants from unspecified multiples of program cohorts (Broyles, 2019; Calizo, 2011; J. de Vries, 2010; J. A. de Vries & van den Brink, 2016; Debebe, 2009, 2011, 2017; Gibbs, 2016; Harris & Leberman, 2012; Kvach et al., 2017; Monks & Barker, 1999). Participants varied between studies by way of their position, department, industry, sector, and nationality. Eight papers did not clearly refer to how women were selected for the leadership program (Broyles, 2019; Brue & Brue, 2016; Debebe, 2009, 2011, 2017; Kvach et al., 2017; Louw & Zuber-Skeritt, 2009; Monks & Barker, 1999). Where recruitment details were stated, women were invariably selected for their existing leadership performance or potential (Calizo, 2011; Clarke, 2011; Gibbs, 2016; Harris & Leberman, 2012; Selzer et al., 2017; Vinas, 2018). The single exception to this was J. de Vries (2010) (and, by extension, and J. A. de Vries & van den Brink, 2016), who placed emphasis on a diverse and broadly representative cohort group, eschewing a “merit” based and competitive selection process.

Program length ranged from two days (Monks & Barker, 1999) to two years (Louw & Zuber-Skeritt, 2009). Six of the 12 programs ran for two weeks or less (Broyles, 2019; Debebe, 2009, 2011, 2017; Harris & Leberman, 2012; Kvach et al., 2017; Monks & Barker, 1999; Vinas, 2018), five programs ran between six and 12 months (Brue & Brue, 2016; Clarke, 2011; J. de Vries, 2010; J. A. de Vries & van den Brink, 2016; Gibbs, 2016; Selzer et al., 2017), and one program length was indeterminable (Calizo, 2011).

Five programs were embedded as part of a broader organisational strategy to address gender inequity in leadership, as explicitly discussed in the papers (Calizo, 2011; Debebe, 2009, 2011, 2017; J. A. de Vries & van den Brink, 2016; J. de Vries, 2010; Kvach et al., 2017; Selzer et al. 2017; Vinas, 2018). Of the remaining seven programs, five papers did not state whether the program was discrete or embedded with other gender equity practices (Broyles, 2019; Brue & Brue, 2016; Clarke, 2011; Louw & Zuber-Skeritt, 2009; Monks & Barker, 1999), and two papers made only vague references to wider gender equality strategies (Gibbs, 2016; Harris & Leberman, 2012).

Program evaluation

Table 4 lists the modes (internal/external), purposes (summative/formative), methods and outcome levels of the included evaluations. Six of the 12 women’s leadership programs were evaluated internally. Three programs were designed, delivered, and evaluated by the authors (J. de Vries, 2010; J. A. de Vries & van den Brink, 2016; Kvach et al., 2017; Monks & Barker, 1999), and three were evaluated by past participants or staff of the organisation (Broyles, 2019; Louw & Zuber-Skeritt 2009; Selzer et al., 2017). Evaluations were conducted externally in seven studies (Calizo, 2011; Clarke, 2011; Debebe, 2009, 2017; Gibbs, 2016; Harris & Leberman, 2012; Vinas, 2018). It was unclear whether evaluations were internal or external in two papers; one author was a former employee of the organisation in which the program was run (Brue & Brue, 2016) and another received research funding from the host organisation (Debebe, 2011).

Table 4. Evaluation purpose, methods and outcomes from 16 qualitative evaluations of women’s leadership programs.

Source	Data Collection Method	Data Collection Time Points	Evaluation Mode	Evaluation Purpose	Outcomes Measured
					1	2A	2B	3A	3B	4A	4B
					Reaction	Knowledge (subjective)	Knowledge (objective)	Behaviour/ expertise (subjective)	Behaviour/ expertise (objective)	System results/ performance (subjective)	System results/ performance (objective)
Broyles (2019)	Interviews with program participants, faculty and staff	x1 (Unclear how long after program completion)	Internal (Author is staff of the organisation)	Summative	x	x		x
Brue and Brue (2016)	Phone interviews with program participants	x1 (Between 2–4 years post-program)	Unclear (Author is former employee of the organisation)	Summative	x	x		x			x
Calizo (2011)†	Interviews with program participants and secondary informants	x2 program participants x 1 secondary informants (Unclear how long after program completion)	External	Summative	x	x		x			x
Clarke (2011)	Phone interviews with program participants	x1 (Unclear how long after program completion)	External	Summative	x	x		x			x
Debebe (2017)‡	(as above)	(as above)	External	Summative	x	x		x
Debebe (2011)‡	Interviews with program participants and secondary informants	(as above)	Unclear (Research funded by the organisation)	Summative	x	x		x
Debebe (2009)‡	Phone interviews with program participants	x1 (Unclear how long after program completion)	External	Summative	x	x		x
J. A. de Vries and van den Brink (2016)§	Interviews with program participants and secondary informants	x1 (Between 1–3 years post-program)	Internal (Program designed, delivered and evaluated by the authors)	Summative & Formative	x	x		x		x
J. de Vries (2010)§	Interviews with program participants and secondary informants	x1 (Between 1–3 years post-program)	Internal (Program designed, delivered and evaluated by the authors)	Summative & Formative	x	x		x		x
Gibbs (2016)	Secondary data (participant interviews)	x2 (First at 0–13 months post-program, second on average 5.5 years post-program)	External	Summative	x	x		x		x	x
Harris and Leberman (2012)	Phone interviews with program participants	x2 (Survey conducted 12 months post-program, interview time unknown)	External	Summative	x	x		x			x
Kvach et al. (2017)	Face-to-face interviews with program participants	x1 (Time intervals varied per participant)	Internal (Program designed, delivered and evaluated by the authors)	Summative	x	x		x			x
Louw and Zuber-Skeritt (2009)	Face-to-face interviews with program participants	x1 (5 years post-completion)	Internal (Authors are past participants/convenors and staff of the organisation)	Summative	x	x		x		x	x
Monks and Barker (1999)	Formal written evaluation and follow up survey	x2 (Written evaluation immediately post-program, questionnaire 1–2 years post-program)	Internal (Program designed, delivered and evaluated by the authors)	Summative	x	x		x			x
Selzer, Howton and Wallace (2017)†	Autoethnography	x1 (Unclear how long after program completion)	Internal (Authors are past participants/convenors and staff of the organisation)	Summative	x	x		x		x
Vinas (2018)	Face-to-face interviews with program participants	x1 (Time intervals varied per participant)	External	Summative	x	x		x			x
frequency of outcome level assessed					16/16	16/16	0/16	16/16	0/16	5/16	9/16

† reported same program.

‡ reported same program.

§ reported same program.

All papers employed evaluation for summative purposes only. All data were collected after program completion (i.e., none were collected pre-program), with nine studies collecting data at a single time point (Broyles, 2019; Brue & Brue, 2016; Clarke, 2011; J. de Vries, 2010; J. A. de Vries & van den Brink, 2016; Debebe, 2009, 2011, 2017; Kvach et al., 2017; Louw & Zuber-Skeritt, 2009; Selzer et al., 2017; Vinas, 2018). Four studies collected data at two time points (Calizo, 2011; Gibbs, 2016; Harris & Leberman, 2012; Monks & Barker, 1999). Data collection time points varied considerably between studies, ranging from immediately post-program (Monks & Barker, 1999) to 5.5 years post-completion (Gibbs, 2016). All but two studies (Louw and Zuber-Skeritt, 2009; Selzer, Howton & Wallace, 2017) included participants from multiple program cohorts resulting in considerable ambiguity in terms of the time that had elapsed between the completion of the program and data collection.

A total of eight studies used data gleaned exclusively from participant interviews, including face to face in three studies (Kvach et al., 2017; Louw & Zuber-Skeritt, 2009; Vinas, 2018) and via telephone in five studies (Brue & Brue, 2016; Clarke, 2011; Debebe, 2009, 2017; Harris & Leberman, 2012). Data triangulation featured in the evaluation of six programs, with studies eliciting responses from both participants and secondary informants such as supervisors, program administrators, instructors, or mentors (Broyles, 2019; Calizo, 2011; J. de Vries, 2010; J. A. de Vries & van den Brink, 2016; Debebe, 2011), and though autoethnography (Selzer et al., 2017). There were single cases of studies utilising written responses from participants (Monks & Barker, 1999), and secondary data from interview transcripts (Gibbs, 2016).

No formal models of program evaluation were used. Therefore, qualitative data from all papers were coded as per the seven outcome levels listed in Table 1. Subjective outcome levels (1, 2A, 3A, 4A) were addressed with greater frequency than objective outcomes levels (2B, 3B, 4B). Of the subjective outcome levels, all 16 papers addressed Reaction (1), Knowledge (subjective) (2A) and Behaviour/Expertise (subjective) (3A). System results/Performance (subjective) (4A) was addressed by 5 studies. The only objective outcome level addressed was System results/Performance (objective) (4B). Notably, nine of the 15 studies evaluated outcomes at this level, as indicated by references to promotions data. However, of these studies, three acknowledged that it was difficult (Clarke, 2011) or impossible (Calizo, 2011; Clarke, 2011) to establish a direct causal link between the leadership program and advancement to leadership positions.

Discussion

Despite the proliferation of women’s leadership programs globally, there have been limited attempts to investigate the impact of these programs. Fewer attempts have been made to examine how this impact is actually assessed, and this systematic review is the first to map the quality and nature of women’s leadership program evaluations globally. We identified 11 peer reviewed articles and five theses that described outcomes of women’s leadership programs. There was considerable heterogeneity in the methodological quality of the articles, in the women’s leadership programs, and in the quality and nature of the evaluations.

Almost thirty years ago, Gray (1994, p. 203) lamented that most of the literature on women’s leadership programs was “decontextualized, unreflective and pragmatic”. The state of the literature to date still warrants the same criticism. A large body of qualitative literature excluded from this review either simply described a program (13), or unreflectively stated the benefits of women’s leadership programs generally (20). Furthermore, eight papers evaluating women’s leadership programs were excluded due to extremely poor methodological quality, (as per Appendix B).

Of the studies included in the review, methodological shortfalls related to a lack of authors’ philosophical, cultural and theoretical statements, acknowledgement of the influence of the researcher on the research, and vice versa, and evidence of ethics approval. This is in line with previous studies examining program evaluation in women’s leadership (Tessens, 2008) and diversity management (Alhejji et al., 2016) which found only a quarter of papers explicitly stated a philosophical approach or theoretical perspective and the contribution of the paper to theory. An absence of women’s leadership theory in programs and evaluations studies is problematic. Echoing Gray’s (1994) criticisms, a-theoretical interventions have been accused of being cosmetic, fragmented and ineffective (Sandler, 2014), resulting in (at best) slow progress or (at worst) harm, in that they reinforce inequality (Janssens & Zanoni, 2014).

Women’s leadership program evaluations are conducted in multiple countries around the world, however US and Australian programs dominate the literature. Nine of the 12 programs were either conducted or co-developed in the US or Australia. This has considerable implications for our understanding of intersectionality in women’s leadership development, and how global economic and cultural complexities affect how women become leaders (Ngunjiri & Gardiner, 2017). Just as masculine theories of leadership development may not be applicable or beneficial to all women, leadership interventions perpetuating US or Australian cultural norms may be ineffective or inappropriate for women from other countries.

In addition to US and Australian domination, programs conducted in higher education, academic medicine and STEM comprise the bulk of the literature. This skew towards the public sector and science industries is congruent with the general leadership program evaluation literature (Voller, 2010), though multiple explanations from the leadership and diversity management literature are proposed for cross-sector differences in evaluation practices. An emphasis on evidence-based decision-making, obligations to demonstrate public investment value, and the academic outputs required of researchers (Hayward & Voller, 2010; Voller, 2010) may account for this discrepancy. Conversely, in the private sector, a lack of incentive to publish results (Voller 2010), commercial sensitivity (Hayward & Voller, 2010), different managerial priorities regarding diversity (Johansen & Zhu, 2017), or fear of unfavourable results (Alhejji et al., 2016), may explain a lack of published evaluations.

Programs varied considerably in terms of their aims, relationship to other gender strategies, length, participants, frequency of contact with cohort and instructors, and recruitment methods. Five programs were acknowledged as being embedded as part of a wider gender change strategy. The concurrent focus of J. de Vries (2010) and J. A. de Vries and van den Brink’s (2016) program on organisational level change was particularly noteworthy. While no programs were explicitly identified as being stand-alone, it is concerning that there was not more focus placed on the integration of women’s leadership programs with simultaneous meso- and macro-level strategies, as there is strong evidence that bundling interventions is the most effective way to tackle organisational problems generally (Becker & Gerhart, 1996; Kulik, 2014), and when addressing women’s leadership development specifically (Hopkins et al., 2008; Ngunjiri & Gardiner, 2017).

Limitations of the evaluation studies reflect those found in the general leadership development literature in terms of their summative focus, lack of longitudinal design, minimal focus on learning transfer, and an overreliance on self-reported data and subjective outcome levels (Voller, 2010). Leadership development occurs over time (Day et al., 2014), as does learning decay (Riggio, 2008), and training impact does not necessarily occur in a linear progression from input to results via knowledge and behaviour (Hirsh et al., 2011). As such, the evaluation of women’s leadership programs requires a long-term perspective (K. Ely et al., 2010). More than two-thirds of studies in this review evaluated a women’s leadership program at a single time point, leading to indeterminate conclusions about the sustainability of program impact.

In addition to being a continuous process, leadership development is also intensely personal (K. Ely et al., 2010), making participants logical sources of program feedback. However, self-assessments are notoriously subject to bias (Collins & Holton, 2004; Kruger & Dunning, 1999) and may be significantly different to outcomes reported by other stakeholders (Alhejji et al., 2016). Also, self-assessments are related only moderately to other measures of knowledge and performance (Ely & Sitzmann, 2007; Mabe & West, 1982) and are not intended to substitute for learning or behavioural assessments (K. Ely et al., 2010). As only six papers triangulated self-report data, it is possible that program outcomes may be overemphasised in some studies.

There are multiple complexities to consider when evaluating women’s leadership programs and multiple factors that may influence women’s advancement to leadership. The influence of factors such as “innate talent, individual motivation, luck, and competing home-life issues” (Kassotakis, 2017, p. 405) can fluctuate over time and be difficult to measure. These factors can also be difficult to disentangle from the complexities of evaluation, and link causally to program outcomes. Sampling issues in the reviewed evaluations further complicated this task. A vast majority of studies examined program experiences and outcomes across multiple cohorts and multiple program evolutions, using purposeful sampling techniques or giving little detail as to how participants were selected for the studies. As a result, it is difficult to determine which program cohorts or which versions of the program related to which outcomes, or whether a sampling bias towards those who had benefitted from the program magnified reports of program impact.

Clarke (2011, p. 508) recognised that “outcomes may also have been influenced by where participants were in their careers when they undertook the program”. Although it may be intuitively appealing to refer to promotions data as evidence for program effectiveness, it can be problematic to assume a link between a leadership program and the promotion of women. It could be argued that a leadership program had little or no impact on the career trajectories of those women who were selected for the program based on their existing or potential leadership abilities. Indeed, this was acknowledged by three of the nine papers that referred to promotions data when examining program outcomes. Of greater concern, when considering how many studies attempted to measure program outcomes at the system level (4A = 5/16; 4B = 9/16), and how few programs were explicitly embedded as part of a wider gender change strategy (5/12), is the assumption of a link between the development of individual women and organisational level change. Fundamentally, many academics argue that attempts to increase the “body count” (Due Billing & Alvesson, 2009) of senior women is a necessary but insufficient condition of organisational transformation towards gender equity (Due Billing & Alvesson, 2009).

A final concern regarding evaluations was that only eight were clearly conducted by researchers who were not involved or invested in the program or organisation. When considering the failures of multiple authors to address the influence of the researcher on the research (and vice-versa), the subsequent risk of author bias towards affirming their own program, the risk of participant response bias when reporting their experiences to internal evaluators, inflated participant self-assessments compared to others’ or objective assessments, and the conflation of promotions with program effectiveness, we return to Gray’s original criticism of the women’s leadership program literature. Apart from Calizo (2011), J. de Vries (2010), and Selzer et al. (2017) who highlighted the ineffectiveness of their evaluated programs, this systematic review concludes that qualitative evaluations of women’s leadership programs globally are largely affirming but unreflective.

These findings should be interpreted considering two main limitations. First, studies were restricted to those from the academic literature. As discussed, women’s leadership evaluations may go unpublished or be inaccessible for various reasons, and we therefore cannot claim to have exhausted the grey literature on women’s leadership program evaluation practices. Secondly, this review was limited to studies published in English. Due in part to this limitation, US and Australian leadership norms dominate the studies in this review and the importance of intersectionality in women’s leadership development was not fully examined.

The following recommendations to advance women’s leadership program evaluation practice and research emphasise the need for enhanced methodological rigour and realistic expectations regarding both the program and the evaluation. A greater focus on formative evaluation will provide valuable information regarding program development and improvement in addition to summative conclusions about whether the program worked. Data collection from multiple sources at multiple outcome levels and time points will allow for longitudinal and triangulated evidence of program impact. Evidence of participants’ knowledge and behavioural change resulting from the program should be sought via multi-source data collection before and after the intervention. Evidence should also be sought that participants advance more rapidly as a result of the program than do comparable women who did not complete it. The development of partnerships between external evaluators and internal stakeholders will aid in avoiding bias and potential conflicts of interest and provide more objective assessments of program outcomes.

Future research needs to consider and test the wider theories of women’s leadership that guide scholarship and understanding (van Eschet al., 2017). First, women’s leadership programs need to be viewed through a wider lens of intersectionality (Ngunjiri & Gardiner, 2017) to address the barriers to women’s leadership on a global level. Second, there is a dearth of theories focussed specifically on women’s leadership development (van Esch et al., 2017). Critical tests of theory are therefore vital to our understanding of women’s leadership. Opening the black box of evaluations, a theory of change plots a “logic model” of individual program steps and explains “how the intervention is expected to bring about the desired results rather than just describing the results” (Mayne, 2012, Section 3, para. 1). In future, approaches to evaluation that utilise a theory of change can be used to draw conclusions about whether and how a women’s leadership program contributed to observed outcomes.

As per the diversity management literature (Hirsh et al., 2011), academic debate is polarised between positivist and post-modern paradigms with vastly different perspectives and assumptions guiding research into women’s leadership development (Storberg-Walker & Natt och Dag, 2017). Multiplism is a “mixed method design strategy that stresses how validity of results can be enhanced via convergence of results from multiple methods, theoretical orientations, and political or value perspectives” (Ford & Sinha, 2008, p. 16). This paper has examined qualitative approaches to women’s leadership program evaluation. Future research examining quantitative and mixed methods approaches are required to further extend the evidence base for women’s leadership programs.

Multiple barriers to evaluation exist such as inadequate funding for rigorous evaluation, insufficient knowledge about administering an evaluation, and a lack of long-term evaluation priorities (Russon & Reinelt, 2004), reflecting a general perception that the “perfect” evaluation is unattainable (Hirsh et al., 2011). However, it is imperative that stakeholders place greater priority on the validity of their approaches to women’s leadership development and the evaluations of programs. Funding for rigorous evaluation should be factored into the initial development of a new program. This will place a high standard of evidence on providers to justify programs and demonstrate that an organisation’s investment pays off (Avolio et al., 2009; Riggio, 2008). It will also allow organisations to demonstrate that their efforts towards creating more gender equitable workplaces are legitimate. Currently, the global evidence of women’s leadership programs’ impact on individuals and organisations is far from conclusive.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data that support the findings of this study are available from the corresponding author, upon reasonable request.

Supplementary material

Supplemental data for this article can be accessed at https://doi.org/10.1080/00049530.2023.2213781