The effects of and support for anonymous job application procedures: evidence from a large-scale, multi-faceted study in the Netherlands

ABSTRACT

This study provides evidence from a large-scale, long-lasting, multi-faceted pilot with anonymized application procedures (AAP) in a Dutch city in 2016–2017. We evaluate effects of AAP on interview and hiring rates, compare two methods of anonymizing, and analyze attitudes related to AAP among hiring managers, applicants and the general public. Results show that minority applicants were relatively more likely to be invited for an interview after AAP were introduced. This difference was significant for applicants from the largest non-Western minority groups. Minority applicants’ hiring rates also improved, though only significantly so when comparing 2016 (not 2017) with the reference period (2015). Moreover, minority applicants, especially from the largest non-Western groups, were more likely to be invited when standardized anonymous application forms were used than when applications were manually anonymized. Finally, applicants and the general public were much more positive about AAP than managers. Many managers saw no need for AAP; they were often unconvinced that discrimination occurs in their organization. We discuss these results in light of the scarce and mixed evidence on AAP effectiveness, and reflect on important insights regarding conditions under which AAP can be implemented successfully and practical implications uncovered by our analyses of different actors’ views.

KEYWORDS:

• Anonymous job applications
• discrimination
• hiring
• evaluation
• support/opposition

Introduction

Field experiments have convincingly shown that ethnic and racial discrimination in hiring is widespread: minority applicants are less likely to receive an invitation for a job interview compared to equally qualified majority applicants (for reviews and meta-analyses see Baert 2018; Quillian et al. 2019; Thijssen et al. 2022; Zschirnt and Ruedin 2016). This is true despite the fact that equal opportunity legislation has been in place for decades. For example, although the European Union adopted antidiscrimination legislation in 2000 to ensure equal treatment of racial or ethnic groups (the Racial Equality Directive, 2000/43/EC), levels of discrimination remained rather consistent over time (Heath and Di Stasio 2019; Quillian et al. 2017; Zschirnt and Ruedin 2016). This has prompted debate about the effectiveness of legislation and possible additional interventions or policies to reduce labor market discrimination (e.g. Derous and Ryan 2019; Fibbi, Midtbøen, and Simon 2021).

Anonymized application procedures (AAP) are one potential way to reduce discrimination (e.g. Bertrand and Duflo 2017; Derous and Ryan 2019; Neumark 2018). Yet, thorough, quantitative empirical evaluations using (quasi-) experimental methods to assess the effectiveness of AAP remain remarkably scarce. Moreover, the few existing evaluations have yielded mixed results. Some found that AAP reduce inequalities in interview and hiring rates between majority and minority candidates (Åslund and Skans 2012; Bøg and Kranendonk 2011), whereas others found mixed effects (Krause, Rinne, and Zimmermann 2012a), null-effects (Hulsegge et al. 2023; Krause, Rinne, and Zimmermann 2012b for ethnic differences; Krause, Rinne, and Zimmermann 2014) or even that AAP reduce the chances of target groups¹ (Behaghel, Crépon, and Le Barbanchon 2015; Krause, Rinne, and Zimmermann 2012b for gender differences). Furthermore, existing evaluations have some important limitations regarding sample size and selectivity, their ability to assess the duration of effects, and the comparison of different methods of anonymizing. Moreover, they provide very limited insights in support for and experiences with AAP among different actors involved. Yet, such insights are crucial for understanding the conditions under which APP may be implemented successfully.

Against this backdrop, this study contributes to the literature by providing new evidence from an AAP pilot within the local administration of the Dutch city of The Hague, in 2016 and 2017. Leveraging several unique aspects of this quasi-experimental pilot, we build on the scarce existing empirical evidence on AAP pilots by not only assessing its effectiveness, but also providing insight in factors that are important for the successful implementation of AAP. Specifically, we contribute to the literature in the following ways. First, to our knowledge, this is the largest and longest-lasting pilot study on AAP to date. It covers a total of 7370 applications; considerably more than prior larger-scale pilots in Germany (3707 applications; Krause, Rinne, and Zimmermann 2012a), Sweden (3529 applicants; Åslund and Skans 2012), France (1268 applicants; Behaghel, Crépon, and Le Barbanchon 2015) and the Netherlands (2593 in Bøg and Kranendonk 2011; about 1650 in Hulsegge et al. 2023). That is important because larger numbers of applications are necessary to draw reliable conclusions about the effectiveness of AAP, which some prior studies were unable to do (Krause, Rinne, and Zimmermann 2014). Moreover, this evaluation spans a period of more than 2 two years with AAP. Prior studies typically covered periods ranging from several months up to a year (Behaghel, Crépon, and Le Barbanchon 2015; Bøg and Kranendonk 2011; Hulsegge et al. 2023; Krause, Rinne, and Zimmermann 2012a; 2014), while one earlier pilot covered about 20 months (Åslund and Skans 2012). Covering an extensive period allows us to examine whether AAP have lasting effects or whether effects fade over time. The latter might for example occur if the introduction of new hiring procedures is initially accompanied by a high organizational salience of non-discrimination that later decreases.

Second, different methods of anonymizing application materials exist, and may have different effects. During this pilot, two different ways of anonymizing were used: manually masking identifying characteristics in résumés and application letters versus using standardized anonymous online application forms. Therefore, an important contribution the present study makes to the literature is that it directly compares the effectiveness of different methods of anonymizing. Prior pilots were unable to make such direct comparisons because they used a single method of anonymizing; either masking or removing identifying characteristics from application materials (Behaghel, Crépon, and Le Barbanchon 2015; Bøg and Kranendonk 2011; Hulsegge et al. 2023; Krause, Rinne, and Zimmermann 2012b) or using anonymous application forms (Åslund and Skans 2012; Krause, Rinne, and Zimmermann 2014).²

Finally, prior evaluations of AAP focused predominantly on assessing their effectiveness in reducing unequal opportunities, and paid very limited attention to the views of and experiences with AAP among different groups of actors involved. Whilst some prior studies did survey hiring managers (Åslund and Skans 2012; Behaghel, Crépon, and Le Barbanchon 2015; Bøg and Kranendonk 2011; Krause, Rinne, and Zimmermann 2012a; 2014), applicants were rarely surveyed (but see Krause, Rinne, and Zimmermann 2012a; 2014), and to our knowledge, no prior studies surveyed the broader public. We argue that this is an important oversight because support for antidiscrimination initiatives is vital for the successful development and implementation of such interventions, and resistance can undermine their success (Avery 2011; Harrison et al. 2006; Iyer 2022). That is true both for attitudes among actors within organizations (i.e. employees, organizational leaders) and for public attitudes. Support within organizations has been identified as a key determinant of the efficacy of workplace diversity initiatives (e.g. Leslie and Flynn 2022; Wentling 2004). Likewise, support among the broader public can encourage employers to adopt workplace policies and create favorable circumstances for their implementation, whilst public opposition can negatively affect whether employers see initiatives as necessary and feasible and can therefore thwart such efforts (Gardberg et al. 2023; Kutlaca et al. 2020; Selvanathan, Lickel, and Dasgupta 2020). Moreover, employers may care about public opinion because (communication about) their efforts to increase diversity and equal opportunities or fight discrimination affect their public image (Jansen et al. 2021). Hence, analyzing to what extent actors within organizations and the broader public support AAP – and why – helps us understand what contributes to or hinders the adoption and successful implementation of AAP. In this study, we therefore surveyed hiring managers, applicants as well as the broader public. We assess how they feel about AAP, and also go beyond merely describing patterns of support or opposition by exploring what drives attitudes towards AAP. Building on prior research on attitudes towards workplace affirmative action and diversity policies (e.g. Avery 2011; Harrison et al. 2006; Iyer 2022; Scarborough, Lambouths, and Holbrook 2019) we examine the role of beliefs regarding the need for AAP, and which (positive or negative) consequences they may have. This allows us to shed light on the perceived need for and legitimacy of AAP, how to improve its implementation, and how AAP may affect the organization’s image.

Methods, data and measurements

The research context

The pilot was conducted in The Hague, the third largest city in the Netherlands. In 2021, 56% of its citizens had a migrant background. In 2015, the city council commissioned a field experiment investigating ethnic discrimination in the regional labor market, which provided clear proof of discrimination: job applicants without a migrant background were 1.5–1.8 times more likely to be invited for a job interview than those with a migrant background (Andriessen et al. 2015).

In response to these findings, the municipality started an AAP pilot in their own organization. As ethnic diversity is particularly low at higher job levels, the pilot was restricted to vacancies for medium and higher salary levels that were open to external applicants. The pilot was conducted between 1 January and 1 July 2016. Based on a positive evaluation, the city council implemented AAP as the standard procedure as of January 2017.³

Methods

To assess the chances of majority and minority applicants after the introduction of AAP, we use data on all applications during the first half of 2016, when the pilot started, and data from January to December 2017, when AAP had been implemented as the standard procedure. As APP were simultaneously applied in all departments of the organization, it was not possible to construct a control group during the same time period. Note that even if this would have been possible, there would have been a serious risk of contamination effects or treatment diffusion; the introduction of AAP received a lot of attention within the organization, and interactions between those involved in hiring decisions across control and intervention groups could not have been prevented. We therefore used the year preceding the implementation of AAP (2015) as our reference period. Our study covers 7370 applicants in total: 1576 in the control period (2015) and a total of 5794 during AAP (1880 in 2016; 3914 in 2017). The number of vacancies in the municipality was 36 during the control period in 2015, 57 during the first half of 2016, and 153 in 2017.

As our design is not fully randomized, we cannot strictly speaking draw causal conclusions about effects of AAP because we cannot completely rule out alternative explanations for changes, like changes in labor market conditions.⁴ We come back to this in the discussion section. Furthermore, to assess attitudes and experiences regarding AAP, we conducted surveys among hiring managers, applicants and the general public.

Different ways of anonymizing

We compare two different ways of anonymizing application materials. At first (i.e. in 2016 and between January and August 2017) résumés and application letters were checked and characteristics that could identify applicants’ gender, ethnic or migration origin and age were manually erased. As of September 2017, standardized anonymous online application forms were used: applicants were required to fill in an online form with standardized questions regarding their competences and experiences.⁵

Data and measurements

Likelihood of being invited for an interview or hired

We compare the likelihood to be invited for a job interview and to get the job for applicants with and without a migrant background during three time periods: (a) the first half of 2015, our reference period; (b) the first half of 2016, when AAP were introduced as a pilot; and (c) the year 2017, when AAP were implemented as the standard procedure. To estimate whether applicants had a migrant background, we used their first and family name, country of birth and nationality mentioned in the application materials, a picture if included, and language skills if mentioned. In case of doubt, additional information from public websites was used.⁶ We report separate results for applicants born in or whose parents were born in Turkey, Morocco, Suriname, or the former Dutch Antilles; the four largest groups with a non-western migrant background in the Netherlands.

Perspectives of managers and HR professionals

In the spring of 2018, an online survey was conducted among managers and HR professionals (henceforth: ‘managers’) who took part in the selection procedures. For every vacancy, one manager was listed in the HR system as the main responsible actor. We contacted 210 of them by e-mail, of which 128 participated in our survey (a response rate of 61%).

Perspectives of applicants and the general public

We contacted all 3064 job applicants who applied between July and December 2017. Of these, 1046 (34%) filled out our online survey. A quarter of them had a migrant background. In addition, an online survey was conducted among the general public in the region Haaglanden, which includes the city of The Hague and 8 surrounding municipalities. A representative sample of the regional population was drawn from the online panel of Kantar Public in April 2018. The sample was representative regarding gender, age (18–64 years old), educational level, social class, household size and (migrant) background. Of the 761 persons who were invited, 545 (72%) completed the questionnaire. The results were weighted by gender, age, educational level, social class, size of household and migration background.

Survey measurements

In the surveys, we included items referring to pro and con statements regarding AAP as well as open questions on perceived advantages and disadvantages. Furthermore, we constructed a scale for positive attitudes towards AAP, based on 4 items that were included in the surveys among managers, applicants and the public (Cronbach’s Alpha = 0.76, 0.85, and 0.86, respectively).⁷

Results

Share of minority applicants

Table 1 shows the number of vacancies and applicants in our study. Importantly, it also reveals that the share of applicants with a migrant background was larger after AAP were introduced than before. This is a relevant insight as many organizations implement more inclusive recruitment and selection methods because they want to attract a more diverse pool of applicants.

Table 1. Vacancies and migrant background of job applicants.

	Control period		Anonymous Applications
	Q1-2 2015		Q1-2 2016		2017
	No.	%	No.	%	No.	%
Number of vacancies	36		57		153
Number of job applicants	1.576	100%	1.880	100%	3.914	100%
Number of job applicants with a migrant background	380	24.1%	590	31.4%	1.095	28.0%
… of which those with a TMSA migrant background^1	187	11.9%	330	17.6%	642	16.4%

¹ Turkish, Moroccan, Surinamese or Antillean background.

Likelihood of being invited for an interview or hired

Invitation rates

We calculated the relative probability to be invited for an interview for majority applicants versus those with a migrant background. A relative invitation rate of 1 indicates equal probabilities, a rate higher than 1 indicates that majority applicants have a relatively higher chance to be invited. Figure 1(a) shows that, before applications were anonymized, majority applicants were more than twice as likely to be invited for an interview than minority applicants. The gap was even larger when we look specifically at applicants with a TMSA background. After AAP were implemented, there was still a significant gap, but the difference between majority and minority (particularly TMSA) applicants was much smaller. The increase in the chances for minority applicants (relative to majority applicants) between the non-anonymized applications in 2015 and AAP in 2016/2017 was only marginally significant (p < .10) but the improvement was statistically significant (p < .05) for applicants with a TMSA background.⁸ In short, minority applicants, particularly those from the largest non-Western migrant groups in the Netherlands, had a higher relative probability to be invited for an interview in AAP than in non-anonymized procedures. We found no significant differences between 2016, when AAP were first introduced, and 2017, indicative of rather lasting improvements.

Figure 1. a. Relative rates to get invited for a job interview. Note The difference in the relative invitation rates of majority versus minority applicants between non-anonymous application procedures (2015) and anonymous application procedures (2016 and 2017 combined, or 2017 only) was marginally significant (p < .10). The difference between 2015 and 2016 was not significant (p = 0.13). For the comparison between majority versus TMSA (Turkish, Moroccan, Surinamese or Antillean migrant background) applicants, the difference between non-anonymous (2015) and anonymous applications (2016 and 2017 combined, or 2016 only, or 2017 only) was significant (p < .05). In both cases, there were no significant differences between 2016 and 2017; b. Relative hiring rates. Note The difference in the relative hiring rates of majority versus minority applicants between non-anonymous application procedures (2015) and anonymous application procedures (2016 only) was marginally significant (p < .10). Differences between 2015 and 2016/2017 combined and between 2015 and 2017 were not significant. For the comparison between majority versus TMSA (Turks, Moroccan, Surinamese or Antillean migrant background) applicants, the difference between non-anonymous (2015) and anonymous applications (2016 and 2017 combined, or 2016 only, or 2017 only) were not significant. In both cases, there were also no significant differences between 2016 and 2017.

Hiring rates

Next, we calculated the relative hiring rates. Figure 1(b) shows that before applications were anonymized, majority applicants were almost 2.5 times as likely to be offered a job as minority applicants. After AAP were introduced, these inequalities clearly declined. In 2017, majority applicants were 1.6 times as likely to get a job offer as minority applicants. Moreover, in 2016, when the AAP pilot started, the relative hiring rate was close to 1, indicating that minority applicants’ chances of getting a job were almost equal to those of majority applicants. In that year, majority applicants were even slightly less likely to be hired than minority applicants, but this difference was not significant. Hence, in AAP, minority applicants had a relatively higher probability to get a job than in non-anonymized procedures. Although this reduction in inequality is substantial in size, it is marginally statistically significant for 2016 (p < .10) and not significant for 2017. This lack of significance may be due to the relatively small number of vacancies, mainly in the control period (2015). The results for majority versus TMSA applicants show similar patterns: although the level of inequality was clearly lower with AAP, this change is not statistically significant, likely also due to the smaller number of TMSA applicants.

Another way to analyze these data is to look at incremental chance to get a job once applicants have been invited for an interview. If we only look at applicants who passed this first selection phase, we found no significant differences between majority and minority applicants in the probability to get a job in any year. Hence, once applicants have been selected for an interview, there was no evidence of additional discrimination during the hiring phase. This is important because one of the most often heard arguments against AAP is that anonymous selection in the first phase would not be useful in the end, as applicants can still be identified as belonging to a minority group - and thus discriminated against - during the subsequent interview phase (e.g. Rinne 2018). Our findings show that, at least in this pilot, that was not the case; reducing discrimination during the first selection phase has a positive impact on the equality of opportunities in the hiring procedure as a whole.

Comparing anonymization methods

When AAP were introduced in 2016, résumés and motivation letters were anonymized manually. This method is time-consuming and potentially prone to errors. Later, standardized anonymous online forms were used in which applicants filled out their skills and competences. In the latter case, recruiters were presented with information about applicants in a more standardized, uniform way. As this facilitates a more structured, objective comparison – shifting the focus to skills and competences – we may expect comparatively less discrimination with the second method. Indeed, we found that with this method, minority applicants were relatively more likely to be invited for an interview. The difference in relative invitation rates between the two methods, although sizable, was not statistically significant for all minority applicants (p = .12), but it was significant for TMSA applicants (p < .01).⁹ When asked which method they preferred, managers and applicants had no strong preference for one or the other method.

Views and experiences regarding AAP

We present, and where possible compare, views and experiences of managers, applicants and the general public.

Managers lack enthusiasm for AAP

Overall, we found that managers do not have a very positive view of AAP. Their lack of enthusiasm is driven by their perception that there is no need to introduce AAP in order to remedy possible bias or discrimination in selection processes. Managers tend to severely underestimate the role that personal applicants’ attributes (like migrant background or gender) play in selection processes, and overestimate the extent to which they are able to select applicants without any bias.

Managers are rather negative about AAP

We asked managers about their general opinion of AAP (on a 5-point scale from very negative to very positive). Managers are strongly divided in their opinion: there were slightly more managers with a negative opinion (38%) than with a positive opinion (31%), and 32% had a neutral stance (Figure 2). An even more negative pattern emerged when we asked if they thought that AAP could prevent discrimination. Here, only one out of three managers agreed, whereas half of them disagreed (Figure 3). The mean score on the 4-item scale for attitudes towards AAP (with range 1 ‘very negative’ to 5 ‘very positive’) was 2.65, which is significantly lower than the neutral mid-point scale value of 3. Female managers, older managers, those with a higher education and those with a migrant background tend to have more positive attitudes towards anonymous application procedures than their counterparts. However, all of these differences are relatively small (compared to the variance we found among applicants and the general public) and non-significant.

Figure 2. Relative answer frequencies to the question ‘what is your general point of view of anonymous job applications’?

Figure 3. Relative answer frequencies to the survey statement ‘Anonymous job applications can prevent discrimination during the application process’.

Managers not convinced of potential benefits of AAP

We also asked managers about the specific pros and cons of AAP. We started with two open questions, where they could mention a pro and con of AAP. Remarkably, one out of four (26%) spontaneously mentioned that they saw no benefits of AAP at all. The most frequently mentioned benefits could be grouped into three categories. First, the most common answer, given by 27%, was that AAP reduce the impact of potential stereotypes and discrimination. Second, about one in ten (12%) mentioned that AAP lead to more equal chances. Third, roughly the same percentage (9%) mentioned that in AAP there is more focus on the objective qualities of applicants.

Next, we presented respondents with a list of positive and negative statements about AAP and asked whether they agreed or disagreed (5-point scale). Again, managers’ opinions are rather negative. For example, a sizeable majority (57%) disagreed with the statements that ‘anonymizing applications is needed to provide equal chances for everyone’ and ‘anonymized applications lead to a selection of surprising candidates’. Only 24% and 21%, respectively, agreed with these statements. Also, 47% and 48% of the managers respectively indicated that they do not believe that AAP lead to a greater likelihood for minority and elderly candidates to get invited for an interview, whereas only 37% and 39% indicated that they do. Finally, whilst the instrument of AAP is intended to promote a stronger focus on applicants’ qualities – instead of on their personal attributes – most managers do not perceive it as such. No less than 60% of them disagreed with the statement that AAP lead to more focus on applicants’ qualities, and only one in four agreed.

In short, managers were not convinced of the potential benefits of AAP. In particular, they did not endorse the claim that AAP would lead to more equal chances for minority candidates. Consequently, most managers (55%) believed that AAP are not necessary within their organization.

Managers perceived pragmatic and fundamental drawbacks of AAP

When asked about drawbacks of AAP, managers gave a range of different answers, that we grouped into three categories. First, relatively many managers (17%) spontaneously mentioned the lack of information. A related drawback, that it was difficult to make a correct assessment of applicants, was also mentioned quite often (by 16%). Second, managers referred to pragmatic drawbacks, for instance that AAP are more time-consuming and unwieldy (mentioned by 12%). Another rather pragmatic drawback that was mentioned by 12% was that it is impossible to completely anonymize applications. Third, managers referred to more fundamental problems, such as moral objections towards anonymizing personal information (mentioned by 11%). Finally, managers (11%) indicated that for some specific vacancies they would like to intentionally make a selection based on certain personal attributes (for instance when there is a gender disbalance in the team) but they considered that impossible in AAP. Note, however, that it is possible to take personal attributes into account after the first anonymized phase in the selection process.

This pattern of perceived drawbacks is also illustrated by other survey results. We presented the managers with a list of statements regarding specific potential drawbacks of AAP. A majority agreed with these negative statements. Over three out of four managers (77%) agreed with the statement that it is impossible to completely anonymize a résumé or motivation letter. Additionally, 61% agreed with the statement that AAP are useless, because applicants’ identity will be revealed during an interview. Also, 54% agreed with the statement that anonymizing applications does not help to reduce effects of stereotypes. Finally, other important limitations refer again to a lack of information and the wish to select candidates based on personal attributes: 57% and 56% respectively agreed with the statements that it is more difficult to find a candidate that fits within the team in AAP, and that AAP make it more difficult to knowingly strive for diversity.

In sum, managers see both pragmatic and fundamental drawbacks of AAP. Pragmatic drawbacks relate to the time investment and to the difficulty of completely anonymizing application materials, and lack of information about applicants. The latter concern partly indicated a pragmatic drawback, that could be solved when all relevant skills and competences of candidates are incorporated in the anonymized information. However, there is also a fundamental point here. Managers would like to have more personal information, for at least two reasons. First, managers would like to base their selection on such personal attributes because they are searching for a match between the candidate and the team or job. Second, sometimes managers deliberately use personal attributes of applicants in order to increase the diversity of the team. We will return to the relation between diversity and the implementation of AAP later on.

Managers perceived little need for AAP

Finally, we asked managers about diversity in their organization. About one in three disagreed with the statement that the organization’s workforce is a good representation of the city’s population. Nevertheless, 55% of the managers indicated that AAP are not necessary within their organization. Only 19% indicated that they are necessary, whereas one out of five had a neutral stance.

Job applicants more positive about AAP

When asked about their general opinion about AAP, half of the job applicants were either positive or very positive, whereas only one out of four was either negative or very negative (Figure 2). Women and those with a migrant background were significantly more positive: 54% of women were either positive or very positive compared to 47% of men. Among those with a migrant background 56% were (very) positive compared to 48% of those without a migration background. Overall, 48% viewed AAP as a good instrument to create equal chances on the labor market, whereas only 28% did not regard it as a good instrument.

The average score on the 4-item scale for positive AAP attitudes was 3.13. In a multiple regression analysis, controlling for gender, age, educational level, migrant origin and past experiences of discrimination, we found that gender, age and migrant origin were significantly related to AAP attitudes. Women and those with a migrant background were significantly more positive. Relative older (55 or older) and younger (40 or younger) were more positive compared to middle-aged applicants.

When asked about the specific pros and cons of AAP, applicants were also rather positive. We presented respondents with various statements referring to specific pros and cons, with 5 answer categories ranging from ‘totally disagree’ to ‘totally agree’. About half of the applicants agreed with statements referring to pros, whereas only a quarter disagreed. For example, 51% agreed (and 32% disagreed) that AAP can prevent discrimination (Figure 3). Also, 54% agreed (and 27% disagreed) with the statement that more attention is paid to the quality of candidates in AAP. Furthermore, 56% and 52% respectively agreed that minority and elderly applicants have better chances to get invited for an interview with AAP, whereas only, respectively, 21% and 28% disagreed.

At the beginning of the survey, we also asked applicants whether they could mention a benefit and a drawback of AAP in two open questions. Most spontaneously mentioned benefits could be grouped into 3 categories: ‘anonymized procedures limit the effect of prejudice and discrimination’, ‘in anonymized procedures, the focus is on applicants’ qualities’, and ‘it creates equal opportunities for everyone’. These top 3 answers were given by respectively 42%, 27% and 23% of the applicants. In line with the closed survey questions, they reflect the themes of ‘equality and non-discrimination’ and ‘objective selection, based on individual qualities’.

Applicants also perceive some limitations of AAP

Notwithstanding their overall positive view, applicants also perceive some drawbacks and limitations of AAP. In response to an open question, most of the spontaneously mentioned drawbacks concern the risk that ‘less information of the applicants is available’ (mentioned by 31%). Other drawbacks that were mentioned relate to the ‘impracticability to completely anonymize’ and that ‘during the interview the person is no longer anonymous’. Both these limitations were mentioned by 15% of the applicants. Practical burdens such as ‘time inefficiency’ and ‘lack of user friendliness’ were also mentioned, but by less than 10%. Finally, some applicants (10%) stress that for specific jobs a person with a ‘particular personal background’ is needed.

These perceived drawbacks are also reflected in responses to survey statements and questions about their experiences. Almost half of all applicants (44%) experienced that they were less able to create a distinct profile for themselves with AAP and indicated that important personal information was lost during AAP (49%). Three out of four applicants agree that ‘it is impossible to completely anonymize a résumé or motivation letter’. In addition, 53% agreed with the statement that ‘anonymizing applications are useless because during the interview the identity of the applicant is revealed’. Finally, 35% stated that AAP were more time-consuming.

Applicants had no strong preference for anonymized or non-anonymized procedures

Less than half of the applicants (40%) preferred non-anonymized procedures. They had no strong preference for AAP either: only 9% indicated that they would rather apply if the selection procedure was anonymous. Applicants with a migrant background were more in favor of AAP: only 32% preferred a non-anonymized procedure and 16% would rather apply if AAP was used.

Support for AAP among the general public

The general public in the region was, overall, rather positive about AAP. More than half of the public (53%) was positive or very positive, whereas only one out of five (21%) was negative (Figure 2). Respondents with a non-Western migrant background were (significantly) more positive about AAP than those without a migrant origin (Figure 4). But even among majority respondents, there was widespread support for AAP: half of those without a migrant background were positive, and less than one out of four was negative. Also, when asked specifically about the introduction of AAP by the municipality, most of the public was positive: respectively 25% and 30% were positive or very positive, whereas only 12% and 9% were negative or very negative, with 20% having a neutral stance.

Figure 4. Overall opinion towards anonymous job applications among the general public.

Much more support for AAP among those who experienced discrimination

Support for AAP was much stronger among those who previously experienced discrimination. We asked whether respondents ever experienced discrimination (in general, not only on the labor market) based on ethnic origin, age or gender. Among those who had, 74% were positive about AAP, whereas among those who had not this was only 48%. In a multiple regression analysis of the 4-item scale of positive AAP attitudes, including gender, age, educational level, migrant origin, and searching for a job, only gender and past experiences of discrimination were significantly related to attitudes towards AAP. Hence, those with a migrant background were significantly more positive about AAP, which is due to their higher level of experienced discrimination. Finally, women were more positive about AAP than men.

For some, AAP can be an additional reason to apply

Four out of ten respondents that were searching for a job at that time (41%), indicated that they considered AAP an additional reason to apply. For those who were not looking for a job, this question was more hypothetical. Here, 26% indicated that they would consider AAP an additional reason to apply. This percentage was higher among those who experienced discrimination, those with a non-Western migrant background, elderly (55+) and lower-educated people.

Strikingly different views on AAP among managers, applicants and general public

Overall, we found striking discrepancies between the perspectives of managers on the one hand and applicants and the general public on the other hand. The differences between the three groups in their mean score on the scale for positive AAP attitudes (range 1–5) were all significant. Managers were much more negative (mean score 2.66) compared to job applicants (mean score 3.13) and the general public, with the latter being the most positive (mean score 3.28). Note that all group means were significantly different from the neutral mid-point value of the scale (3.0), which again indicates that managers were overall negative (T-value = −4.1, p < 0.01) and job applicants and the general public were overall positive (T-values = 3.7 and 6.8, respectively, with p < 0.01).

Managers were much more reluctant about AAP and perceived more drawbacks. One important reason for this is that managers were much more confident that selection decisions are objective and not affected by irrelevant personal characteristics of applicants. Hence they saw less need to change selection procedures.

As illustrated in Figure 5, there are clear differences in perceptions of equality in selection processes. We asked managers, applicants, and the general public to what extent they believe that applicants’ (migrant) origin, age or gender affect selection decisions. Note that in fully objective, competence-based selection processes, applicants would only be evaluated based on their relevant knowledge, skills and competences, and attributes such as origin, age and gender would be irrelevant. Importantly, managers were rather convinced that such personal attributes do not affect selection decisions. Only 15% of the managers believed that applicants’ age has a lot of influence, and only 7% and 4%, respectively, believed that applicants’ origin and gender have a lot of influence. By contrast, job applicants and the general public were much more convinced that age, gender and origin can determine one’s chances during hiring procedures. This is particularly so with regard to age, but also regarding migrant origin and to a lesser extent gender. For instance, whereas only 7% of the managers believed that the selection process is affected by applicants’ migrant origin, 37% and 42% of the applicants and general public did.¹⁰

Figure 5. Percentage of respondents that believe that personal characteristics of job applicants (gender, age, migrant origin) have either ‘much influence’ or ‘very much influence’ on the selection process.

We see similar discrepancies in opinions about organizations introducing AAP. We asked whether the municipality sets a good example for the city and other employers by implementing AAP. Managers are divided on this issue: 34% agreed, 32% disagreed and 30% had a neutral stance. By contrast, job applicants and the general public were much more positive: roughly half of the applicants (48%) and the general public (53%) agreed that the municipality sets a good example. Only 27% of the applicants and 18% and the general public disagreed.

Conclusions and discussion

Below, we first summarize our findings about AAP effectiveness and the views and experiences regarding AAP of different actors involved. Next, we reflect on practical implications of these findings and what they teach us about conditions under which APP can be successfully implemented. Finally, we discuss the limitations of our study and directions for future research.

Are AAP effective?

This study revealed that minority applicants were more likely to get invited for an interview and to get hired after the implementation of AAP. The differences in relative invitation and hiring rates between non-anonymized and anonymized procedures were considerable, but not always significant, mainly due to the somewhat smaller number of non-anonymized vacancies and/or applicants with a non-western migrant background. Yet, particularly applicants from the largest non-Western migrant groups had a significantly higher relative probability to be invited for an interview after AAP were introduced. Hence, AAP seem especially effective in creating more equal opportunities for groups with a greater disadvantage in the labor market.

Regarding the chances of getting hired, we also found a considerably higher likelihood for minority applicants after the introduction of AAP, although this difference was only significant when comparing the non-anonymized procedures in 2015 with the anonymized procedures in 2016 (not those in 2017). An often-heard criticism regarding AAP is that they would simply delay discrimination, because after the first anonymized selection phase, applicants’ features (e.g. migrant origin, age, gender) are revealed during the interview (e.g. Rinne 2018). Indeed, some prior evaluations found an effect of AAP on the selection of candidates for an interview, but not on hiring rates (Åslund and Skans 2012 for ethnic origin). By contrast, we did find that chances for minority applicants to get hired were better after AAP were implemented, although the difference was only significant between 2015 and 2016. Moreover, looking at incremental chances of getting hired, we found no significant differences between the likelihood of success for minority and majority applicants once they were selected for an interview under AAP. Hence, our findings indicate that anonymizing the first selection phase does have an impact on final hiring outcomes.

Do AAP have lasting effects?

Improving upon previous studies, we monitored the outcomes of AAP over a long period of time to uncover whether effects are lasting or not. We found rather stable effects regarding the likelihood of getting invited for an interview over a two-year period (2016 and 2017). Regarding hiring rates, however, we observed smaller effects in 2017 compared to 2016. This may reflect that the initial enthusiasm and strong support from the municipal authority for AAP boosted minority applicants’ opportunities, but that this effect faded somewhat over time. Still, minority applicants’ chances were better in 2017 than in 2015, before AAP were implemented, so even if effects were more modest in the long run they did not disappear.

Do different methods of anonymizing matter?

Unlike prior evaluations, we were able to directly compare two different and commonly applied methods of anonymizing applications. We showed that minority applicants were more likely to be invited for an interview if anonymous online application forms were used compared to when application materials were anonymized manually, and this difference was significant for applicants from the largest non-western groups.

How do applicants and the general public view AAP?

Previous evaluations seldomly assessed attitudes and beliefs regarding APP among (potential) applicants and particularly the general public (but see Krause, Rinne, and Zimmermann (2012a), who surveyed applicants, though not the broader public). Our findings show that, overall, the general public and applicants are rather positive about AAP. Most of them believe that by implementing AAP the organization sets a good example. Moreover, support for AAP among the general public was much stronger among those who had experienced discrimination.

What about managers?

Among managers and HR professionals (henceforth: managers) we found much less enthusiasm for AAP. Whereas some are rather positive, there is a considerable share of managers who object to AAP. Many of them consider AAP unnecessary. This is likely related to the striking finding that managers are often convinced that traditional selection procedures are fair and objective: many believe that applicants’ migrant origin, age or gender do not affect the outcome of the selection procedure. That is at odds with the strong empirical evidence of discrimination in hiring, not only internationally (Baert 2018; Quillian et al. 2019; Thijssen et al. 2022; Zschirnt and Ruedin 2016), but also in the Netherlands (e.g. Blommaert, Van Tubergen, and Coenders 2014; Ramos, Thijssen, and Coenders 2021; Thijssen et al. 2021) and the region where this pilot took place (Andriessen et al. 2015). It is also remarkable given that the present study provided clear evidence of unequal opportunities for minority versus majority applicants in the organization where the surveyed managers work. This brings us to some specific recommendations.

Practical implications

Our findings have several practical implications regarding necessary conditions for the successful implementation of anonymous application procedures. First, our results revealed that many managers fail to see the need for any antidiscrimination initiatives; very few of them believed that applicants’ chances are affected by their migrant origin or gender. This is in line with recent studies showing that employers tend to underestimate discrimination and bias in hiring (e.g. SCP 2023). Scarborough, Lambouths, and Holbrook (2019) found that beliefs about discrimination as a cause of inequality are a strong predictor of support for diversity initiatives. Because support within the organization is crucial for the successful implementation of antidiscrimination and diversity interventions (Leslie and Flynn 2022), it is important to raise awareness of potential biases and discrimination in recruitment and selection among key actors if organizations want to successfully implement such interventions. Providing managers and recruiters with information on the need for workplace antidiscrimination and diversity initiatives – e.g. by sharing results of research revealing discrimination and unequal opportunities within their own organization, region, or country – may increase their support for AAP or other diversity initiatives. Moreover, sharing knowledge on effects of AAP may help, as we found that many managers doubt their effectiveness.

Second, we found that among applicants and the general public (including potential applicants) many believe that an organization sets a good example by implementing AAP, and some even consider AAP an extra reason to apply at this organization. Hence, anonymizing applications can affect employer branding, which is especially relevant in a tight labor market and for organizations interested in social responsibility (see Gardberg et al. (2023) for a discussion on the role of societal norms and acceptance as a policy diffusion mechanism). Moreover, this indicates that AAP can increase diversity in applicant pools, which is underscored by our finding that the share of minority applicants increased after AAP were introduced.

Third, results regarding relative invitation rates indicated that standardized anonymous online application forms are the most effective method of anonymization. As managers and applicants had no strong preference for one or the other method, this implies that standardized anonymous application forms – with formalized questions on skills and competencies – are to be preferred over manually anonymizing résumés and letters. Not only is manually anonymizing time-consuming and error-prone (Hulsegge et al. 2023), anonymous application forms with standardized questions also steer the focus of recruiters towards applicants’ skills and competencies, reducing the role of (implicit) negative bias based on personal and non-relevant characteristics (Derous and Ryan 2019).

Fourth, our analyses of the experiences of managers and applicants highlight that AAP should be designed diligently and tailored to fit the needs of specific organizations and vacancies. For example, a large share of managers felt they did not have enough information about candidates to make a good decision in AAP. Whilst AAP offer excellent opportunities to put applicants’ skills and competences at the center of selection processes (e.g. Rinne 2018), this does require careful consideration of which skills and competences are essential for a specific job and which indicators capture these. Ideally, this would be the shared responsibility of HR and the hiring manager, prior to any recruitment and selection process. Our results also underline that surveying managers and applicants can provide crucial insights in what their specific needs are and how these can be incorporated in the design and implementation of AAP within specific organizations.

Fifth, our study provided strong evidence of unequal opportunities between majority and minority applicants in non-anonymized procedures. This is in line with results from several prior AAP evaluations that likewise uncovered evidence of discrimination (Åslund and Skans 2012; Bøg and Kranendonk 2011, for interview invitations, not job offers; Krause, Rinne, and Zimmermann 2012a, for several sub-studies). There are, however, also evaluations that found little or no evidence of discrimination in the non-anonymized condition (Behaghel, Crépon, and Le Barbanchon 2015, among participating firms; Hulsegge et al. 2023; Krause, Rinne, and Zimmermann 2012b). We argue that it is important to take this into account when summarizing available evidence on AAP effectiveness, and that the combined evidence of AAP evaluations is best summarized as follows: AAP were generally found to reduce discrimination if there was evidence of discrimination in the non-anonymized condition. That is, AAP are unlikely to increase equal opportunities for minority groups when no significant group inequalities exist in the first place (c.f., Krause, Rinne, and Zimmermann 2012a, 2014). That may seem self-evident, but it is not always clearly articulated when the existing evidence on AAP effectiveness is discussed. A practical implication of this is that organizations looking to implement more inclusive forms of recruitment and selection can gain from a thorough problem analysis or examination of the causes of the lack of diversity or equality in their organization as a starting point. If, for example, the problem turns out to be that the applicant pool includes few target group members, focusing on making the recruitment stage more inclusive may make more sense than anonymizing or otherwise adapting selection procedures.

Finally, and relatedly, AAP are not a panacea to all obstacles to diversity, equity or inclusion (DEI) in organizations. Although AAP are a method to decrease the impact of bias on selection decisions by standardizing selection procedures and stimulating objective decisions based on candidates’ knowledge, skills and potential (c.f., Rinne 2018), they cannot erase all biases that may play a role during recruitment and selection. Furthermore, anonymizing applications may not always be equally feasible. Think, for example, of vacancies in the highest echelons where managers may more often have met potential applicants during their career. Hence, AAP are one of many tools that organizations can consider if they want to increase DEI. There are alternative ways to standardize selection procedures and increase their objectiveness, such as objectifying hiring criteria, using standardized (non-anonymized) application forms or assessments, or structured interview protocols (Derous and Ryan 2019). Other interventions to increase DEI include diversity quotas, audits/mystery guests, diversifying hiring committees, training recruiters/managers, and using video applications or algorithms (e.g. Derous and Ryan 2019; Scarborough, Lambouths, and Holbrook 2019).

Limitations and suggestions for future research

One limitation of our study is that we were unable to control for relevant job and applicant features other than migrant background, because no such information was collected for those who applied in 2015 and 2016. However, looking at our results, we see little reason to believe that group differences in how qualified applicants were may have distorted our conclusions. For instance, overestimation of the effect of AAP would occur if minority applicants were somehow relatively more qualified after AAP were introduced than before. We do not see any proof for this. In fact, it seems more likely that the pool of minority applicants would include more people who were slightly less qualified after AAP were introduced, as this pool had grown substantially and was presumably more heterogeneous. If that were true, we may even have underestimated effects of AAP. Still, future studies should aim to control for other applicant features.

Second, we were unable to use a fully randomized design, that is to randomly assign applications to an experimental (anonymized) or control (non-anonymized) condition. Completely randomized designs are widely considered the gold standard for experimental research (Bertrand and Duflo 2017, 380). In practice, however, very few of the existing pilots in this research field were able to use a completely randomized design (exceptions are Behaghel, Crépon, and Le Barbanchon 2015; Krause, Rinne, and Zimmermann 2012b). That is unsurprising, as full randomization in real-life hiring processes is difficult to achieve in a sensible manner. Existing studies that did use fully randomized designs also have some crucial limitations. For example, in the French pilot (Behaghel, Crépon, and Le Barbanchon 2015), firms recruiting via the public employment service were invited to participate in the experiment with anonymization. Only firms that agreed to participate were included in the pilot. Half of those received résumés that were anonymized; the other half received non-anonymized materials. Although this seems an elegant way to achieve randomization in the field, the authors provide convincing evidence that selectivity bias (partly) drove their unexpected results. There was little evidence of discrimination among participating firms (in the non-anonymized condition), but there were clear indications that discrimination did occur in firms that did not agree to participate. The authors found no effect of anonymization, which is unsurprising because there was little evidence of discrimination in participating firms to begin with. Importantly, it is unclear whether anonymization would have had an effect among the firms that did not participate. In a study by Krause, Rinne, and Zimmermann (2012b), two decision makers reviewed both standard application materials (for half of the applicants) and anonymized application materials (for the other half). Results provided no evidence that minority applicants were less likely to be invited than majority applicants (in the control condition). This may mean that no discrimination existed in this organization to begin with, which could explain the null effect of anonymization. It is also possible, however, that the fact that the decision makers reviewed both anonymized and non-anonymized procedures stimulated them to take extra care to avoid discrimination. Hence, these results might (partly) be driven by spillover effects or contamination (Banerjee and Duflo 2017). Summarizing, given the difficulties involved in achieving complete randomization with anonymization in field studies, a challenge for future research is to come up with fully randomized designs that avoid problems like selection bias, spillover or contamination. In lieu of that, quasi-experimental designs (e.g. before-and-after comparisons), if carefully designed and interpreted, may form a viable alternative to full randomization.

Third, our study revealed a lack of support for AAP among managers. Resistance among managers was also found to play a role in prior pilots in Sweden (Åslund and Skans 2012) and the Netherlands (Bøg and Kranendonk 2011). The latter study also found that ‘recruiters did not believe that they were discriminating’ (Bøg and Kranendonk 2011, 14) and the authors suggest this may have contributed to the discontinuation of that specific pilot. Skepticism among managers may also explain why several smaller pilots in other Dutch municipalities without thorough quantitative, evaluations were discontinued. As support within organizations is crucial for successful implementation (Leslie and Flynn 2022) future evaluations of AAP would benefit from also analyzing the views and perspectives of the actors involved to understand what drives their attitudes towards AAP.

Finally, as mentioned above, AAP is only one of the different possible ways to standardize selection procedures and increase their objectiveness. Alternatives include objectifying hiring criteria, using standardized (non-anonymized) application forms or assessments, or structured interview protocols (Derous and Ryan 2019). These might face less resistance from hiring managers, as they may be regarded as a less prescriptive initiative, which, compared to AAP, to a lesser extent ‘forces the hand of, and limits the discretion of’ decision makers (Harrison et al. 2006, 1014). Yet, we lack systematic experimental evaluations to assess and compare the effectiveness of and support for such initiatives. Future research may address this knowledge gap.

To conclude, as ethnic and cultural diversity in Western European societies continues to grow, particularly among recent graduates, it becomes increasingly important for employers to attract and select job seekers from minority groups. Anonymous applications are one of the available tools that, if implemented thoughtfully, firms and organizations may consider in order to achieve fair and objective selection processes and a more inclusive labor market.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by City of The Hague, the Netherlands.

Notes

1 These studies found evidence of positive discrimination in the non-anonymized conditions. Hence, anonymizing meant that positive discrimination was no longer possible, whilst it was in the non-anonymized procedures.

2 Krause, Rinne, and Zimmermann (2012a) consist of 9 separate pilots. Methods of anonymizing vary between but not within these separate tests.

3 Based on further positive evaluations, the municipality continues to use AAP as their standard procedure.

4 Regional unemployment decreased from 8.7% in 2015 to 6.3% in 2017, which could potentially cause a minor drop in discrimination.

5 Applicants could still upload a résumé and application letter. These were not used to select candidates for an interview, but were made available after this initial selection.

6 For 2015 and 2016, we used data from a research report by Motivaction (2016). For comparability, we applied the same procedures to assess applicants’ migrant background.

7 One item captured the overall opinion toward AAP (see Figure 2). The other three items covered specific aspects of AAP: whether AAP are considered a good tool to create equal chances on the labor market; whether they can prevent discrimination; whether respondents regard AAP as ultimately useless, because during the interview the identity of the applicant will be revealed.

8 Significance levels were calculated using logistic regressions comparing relative probabilities of majority and minority applicants.

9 The relative invitation rate for an interview for majority versus minority applicants was 1.72 with the manual AAP method and 1.29 with standardized online forms. For TMSA applicants the difference was larger, with relative invitation rates of 2.35 and 1.20.

10 There is remarkable consensus among the general public about the perceived influence of these personal characteristics during selection. We found no significant differences between men and women regarding the perceived influence of gender, and no significant differences between age groups and people with or without migrant background regarding the perceived influence of applicants’ age and migrant origin respectively.