Complexity embraced: a new perspective on the evaluation of organisational interventions

ABSTRACT

Organisational interventions are recommended to address the root causes of ill-health in organisations. Yet, the evidence for their effectiveness is inconclusive, likely because such interventions are complex, and their effectiveness depends on how and in which contexts the interventions are implemented. This makes organisational interventions challenging to evaluate. While multiple factors affecting implementation and intervention outcomes have been uncovered, it remains unclear which of them are necessary and which are sufficient to produce desired outcomes. To move forward, we argue that the field would benefit from using a theory of causation that better reflects that factors can combine in various ways, that there may be multiple paths to the same outcome, and that a factor can be necessary for bringing about an outcome and thus always leads to it, or sufficient, implying that multiple factors can independently lead to the same outcome. We believe that the use of evaluation designs that align with this type of causation, such as the configurational comparative methods in general and coincidence analysis in particular, will be a significant turning point for the field. The proposed paradigm will improve the precision of current frameworks and models for the evaluation and implementation of organisational interventions.

KEYWORDS:

• Evaluation
• causality
• coincidence analysis
• configurational comparative methods
• implementation
• methodology
• organisational interventions

Conducting organisational interventions is a way to ensure that insights from research on work and stress yield practical impacts through “planned, behavioral, theory-based actions that aim to improve employee health and well-being through changing how work is organized, designed, and managed” (Nielsen & Simonsen Abildgaard, 2013). Because organisational interventions have the potential to address the root causes of work-related ill health, they are currently the recommended approach in policies such as the Management Standards in the UK and systematic work environment management (SAM) in Sweden. However, meta-analyses and reviews summarising the increasing number of evaluations of such interventions often show mixed results, particularly for more distal outcomes, such as health and well-being (Aust et al., 2023; Fox et al., 2021). The variation across studies and outcomes is large, and overall, the impact of organisational interventions remains inconclusive. In this regard, it seems that the over a decade-old observation in a 2010 Work & Stress editorial still holds: despite their promise, organisational interventions fail to achieve their aspired results (Cox et al., 2010).

Yet the outlook is not all gloomy. The field has seen substantial progress in illuminating the former black box of factors that influence how and why organisational interventions succeed in bringing about changes in outcomes. A considerable body of research has contributed to the identification of such factors. A 2016 review described at least 47 factors (Havermans et al., 2016), related to the intervention itself (e.g. fit to the organisation), the process of implementing the intervention (e.g. manager support), and the context (e.g. concurrent changes). Yet, despite the multitude of potentially important factors, evaluations are often limited to focus exclusively on one or a few of such factors. Essentially, these evaluations test whether more or less of a factor predicts more or less of the outcome (e.g. Lundmark et al., 2017), addressing questions such as whether manager support or employee participation predict changes in intervention outcomes. But what questions should we be asking?

Towards necessary and sufficient questions

Reducing complexity by studying one or a few factors at a time does not acknowledge the reality of organisational interventions, and it means that vital questions are left unanswered. For example, while participation is considered vital for succeeding with organisational interventions, a realist synthesis concludes that participation itself depends on at least eight preconditions, such as outcome expectancy, managerial support, structural and financial resources, and the absence of an unfavourable internal event (Roodbari et al., 2022). Furthermore, whereas some conditions, such as available financial resources, may always be necessary for organisational interventions to succeed, others bring about the outcome only under certain conditions. Thus, not only are a substantial number of factors associated with the intervention, process, and context but those factors also combine to create configurations that affect chains of outcomes, resulting in considerable variations between studies (Nielsen, 2017).

The sheer number of factors and combinations prompts questions about the conditions under which they affect outcomes. For example, are all the preconditions for participation necessary, or is a smaller combination of them sufficient? In other words, does employee participation require high outcome expectancy and managerial support and structural resources and a lack of unfavourable internal events, et cetera? Or is it sufficient that one or another factor is present, implying that a precondition, such as managerial support, may be related to high participation in some cases but not in others when, for example, high outcome expectancy does the trick? Thus, might there be alternative pathways to the same outcome? What if some of these beneficial factors even reduce participation under certain conditions, such as when outcome expectancy is high but structural and financial resources are lacking? Rather than addressing complexity by reducing it through design and analysis, we propose that complexity could be embraced to allow questions such as these to be answered. In this way, the perplexing variations in the outcomes of organisational interventions could be untangled and explained.

Conceptions of causality in intervention research

One reason why vital questions related to the evaluation of organisational interventions remain unanswered may be that the field has traditionally favoured designs (e.g. the pretest-posttest control group design) and analytical approaches (e.g. regression-based methods that investigate covariation) that are better suited for addressing other types of questions, such as “Does intervention A cause outcome B?” (and possibly considering whether the relation between A and B depends on more or less of factor C). Yet this line of inquiry relies on the premise that it is possible to discern causal relationships through the comparison of two scenarios, one in which factor A is present and another, counterfactual situation in which A is absent but everything else is equal, thus forming a “parallel comparable world.” If outcome B happens when A is present but not in the comparable world in which it is not, a causal inference can be made. However, in evaluations of organisational interventions, it is often practically impossible to create appropriate comparison conditions, including finding suitable control groups, preventing contamination from the intervention group to the control group, and ensuring that the entire intervention group is exposed to the intervention; this is well established in the literature (Nielsen et al., 2010). These challenges are well known in the field, and alternative designs and analytical questions have been proposed in response (Nielsen et al., 2023), such as the adapted study design (Randall et al., 2005), multiple baseline design (Schelvis et al., 2015), and different ways of linking processes to outcome measures. We argue, however, that the problem is not primarily with the designs and methods but rather with the theory of causation that they build upon.

The proposition that causation can be inferred by creating and comparing “parallel comparable worlds” (either through experimental design or via analysis) is posited in the counterfactual theory of causation. Experiments represent the optimal designs for testing such causal relationships. Yet, while the design works well for evaluating relationships between two (or a few) factors (with others held constant), it is suboptimal for the more complex relationships typical of organisational interventions, with all the if, and, and or they come with. Representing this complexity requires another theory of causation, one that enables studying relationships such as those in which a factor sometimes but not always causes the outcome or in which the outcome can be achieved through various, independent pathways – that is, when factors can be necessary and/or sufficient. The regularity theory of causation does just that (Baumgartner, 2013).

The regularity theory of causation posits that complex relationships can be understood by investigating the regularity with which factors co-occur with the outcome. Three key ways in which factors can co-occur with outcomes are equifinality, conjunctivity, and asymmetry (so-called causal regularities). First, equifinality refers to regularly reoccurring relations that indicate the existence of different pathways to the same outcome. Thus, while managerial support may be proposed as a precondition for participation along with seven other preconditions (Roodbari et al., 2022), we do not know whether managerial support (or other preconditions) is necessary, i.e. always needed, or whether it can bring about the outcome (independently or in conjunction with other preconditions) but does not need to be present – that is, if other preconditions are sufficient for participation to be brought about without it. Thus, equifinality reflects causal pathways that supplement one another.

Second, conjunctivity is when some conditions (e.g. structural resources) produce the outcome only in combination with other conditions (e.g. managerial support). Conjuncts are similar to interactions in regression analysis (Thiem et al., 2016). However, whereas interactions capture complexity in terms of the joint effect of two to three variables, conjuncts reflect that factors need to be simultaneously present, such as when outcome expectancy and managerial support and structural resources are necessary for participation. Conjuncts can also include situations in which one factor is present and the other absent, such as when there is high outcome expectancy but a lack of structural resources. Importantly, conjunctivity can be combined with equifinality to model situations in which an outcome can come about by combinations of factors. Thus, conjuncts and interactions are similar but represent distinct theoretical and mathematical approaches to how complexity is modelled.

Third, asymmetry describes when the factors that are necessary or sufficient for success may not also be those that explain failure. Thus, even if a factor predicts intervention outcomes, its absence does not automatically imply intervention failure and vice versa (i.e. the lack of “negative factors” does not automatically mean that an intervention succeeds). For instance, while a lack of managerial support may be sufficient to disrupt participation, the presence of managerial support may not be a necessary condition in itself to result in participation. It may be necessary only in conjunction with other factors, and it may be a sufficient part of only one or more pathways.

In sum, although methodological challenges are a recurrent theme in the literature on organisational interventions, the theoretical underpinnings of the methods have received limited attention. There are alternative ways of viewing causality that better align with the complexity involved in organisational interventions by elaborating necessary and/or sufficient conditions. How, then, should interventions be evaluated to reflect this?

The promise of alternative methods to evaluate interventions

An option for evaluation that aligns with regularity theory and its notions regarding necessary and sufficient factors can be found in the family of configurational comparative methods (CCMs), such as qualitative comparative analysis (QCA) (Ragin & Zaret, 1983) and coincidence analysis (CNA) (Baumgartner & Falk, 2023). CCMs are case-oriented and can combine qualitative and quantitative data to explain the conditions under which organisational interventions bring about the intended outcome; that is, they explain the combinations of factors that are necessary and/or sufficient for achieving – or not achieving – specific outcomes (Fiss, 2007). These methods rely on Boolean rather than linear algebra. Whereas linear algebra uses linear equations, CCMs test and produce Boolean logic expressions, such as “IF (A OR NOT B) THEN C” (meaning that either the presence of A or the absence of B is sufficient for C) (Baumgartner & Falk, 2023). CCMs do not primarily provide solutions to a predefined equation. Instead, CCMs suggest equations that represent the causal regularities that can be identified in the data. Through metrics of how well each equation covers the patterns among the cases, the analysis provides insights into which combinations of factors are most common across cases and outcomes and explains the outcomes in the most consistent ways.

We argue that, by enabling the analysis of necessary and sufficient conditions across cases, CCMs offer an analytical approach that can advance the evaluation of organisational interventions and support the exploration of new questions that better reflect the ways in which the intervention, process, and context bring about intervention outcomes. Thus, CCMs offer a solution to a dilemma that has eluded organisational researchers for decades. Instead of making the complexity of organisational interventions manageable either by reducing the number of factors included in the analysis (e.g. using quantitative analysis to investigate generalisable patterns across cases) or by reducing the number of cases (e.g. performing qualitative case studies of cases as wholes), CCMs offer a middle way. They make it possible to analyse many cases and multiple factors to uncover patterns invisible in manual case comparisons or regression-based comparisons. CCMs enable a genuine case-based approach that can identify how multiple conditions associated with the intervention, process, and context combine to bring about outcomes in an organisational intervention. What can be expected of CCMs for theory and practice?

CCMs’ contribution to theory and practice

We believe that CCMs can contribute to theoretical progress by improving the precision of current frameworks and models for the evaluation and implementation of organisational interventions. In a call for theory development, Edwards and Berry (2010) noted that theories in organisational and management research tend to lead to propositions that are limited to simple relationships that, when tested, result in an inflation of theories rather than theory refinement. CCMs can contribute to such refinements.

Practically, CCMs can provide answers to questions that organisational actors typically ask. For example, knowing that there are eight preconditions for participation (Roodbari et al., 2022) can be daunting, whereas knowing which of these that make a difference is actionable. Similarly, it is helpful to know that one condition is required, whereas another is good to have but could be replaced with something else. Thus, information on necessary conditions informs what needs to be prioritised, awareness of alternative routes to outcomes offers flexibility and adaptability to local conditions, and asymmetries inform actions to reduce the risk of failure.

We do not suggest that CCMs replace regression-based methods. They rely on different and sometimes incommensurable schools of thought and mathematics and thus address fundamentally different questions (Thiem et al., 2016). There are possibilities in their complementarity. CCMs could be used to refine hypotheses of magnitude and offer nuance to findings from regression analyses that identify an average effect, uncovering, for example, asymmetry (Misangyi et al., 2017). The distinct approaches could also be used in sequence. Furthermore, CCMs could complement systematic reviews and meta-analyses of interventions (Roczniewska et al., 2023), helping to synthesise the complexities under which the main effects do or do not occur. As a CCM is an analytical method, it can be applied to various designs. We also foresee innovation in evaluation design if the regularity theory of causation is embraced.

So, let us ensure that, when we look back in ten years’ time, there has been substantial progress in the understanding of organisational interventions, with theories refined so that they describe (1) when and (2) what sort of effects can be expected (3) for whom (4) under which circumstances (5) what the specific processes responsible for such effects could be (Taris & Kompier, 2014). But let us not stop at that. Let us also systematically discover (6) the roads to failure as well as those to success and (7) alternative ways to achieve the same outcomes. There are now methods and conceptualizations of causation available that allow us to do all of the above, so let us embrace them to move the field of organisational interventions into the 2030s.

Disclosure statement

No potential conflict of interest was reported by the author(s).