Normal cyber accidents

ABSTRACT

Several of the most serious cyber incidents affecting critical infrastructure to date have been the result of collateral damage, indirect effects, malware that ‘escaped’ their intended target and/or incontrollable malware proliferation. This tendency has so far been under-explored in the International Relations (IR) literature, and its potential implications largely overlooked. By focusing on the role of socio-technical system dynamics, this article aims to contribute to advancing our understanding of collateral (incidental) damage and unexpected consequences connected to offensive cyber operations. More specifically, it introduces an analytical framework based on Normal Accidents (NA) theory. The framework highlights dynamics which make complex systems more difficult to analyse and more prone to cascading failures. Its application is explored using in-depth interviews and empirical case examples of large-scale cyber incidents. The results highlight the difficulty of achieving controlled and precise effects when disrupting components in complex systems. The article concludes with a discussion on the need for renewed attention to escalatory risks connected to destructive offensive cyber.

KEYWORDS:

• Offensive cyber operations
• Normal Accidents theory
• cyber incidents

Introduction

Narratives warning of a scenario in which directed offensive cyber operations induce deliberate catastrophic disruptions of critical infrastructure have figured prominently in public discourses on cybersecurity for the past 20 years (Lawson 2013; Valeriano and Maness 2015; Lawson 2019; Burton and Lain 2020; Burton and Christou 2021). However, despite the fear surrounding the prospects of deliberate disruptions of critical infrastructure operations, the empirical reality shows that several of the most serious cyber incidents affecting critical infrastructure to date have been the result of collateral damage, indirect effects, malware that ‘escaped’ their intended target and/or incontrollable malware proliferation. Well-known examples of this include the WannaCry ransomware, NotPetya, and the Kaseya incidents. This tendency has so far been under-explored in the International Relations (IR) literature, and its potential implications largely overlooked. What makes critical infrastructure operations vulnerable to these types of events, and how may disruptions of individual components in critical infrastructure-related systems lead to cascading and unexpected consequences?

By focusing on the role of socio-technical system dynamics, this study aims to contribute to answering these central questions and to advancing our understanding of collateral (incidental) damage and unexpected consequences connected to offensive cyber operations. More specifically, it develops an analytical framework based on Normal Accidents (NA) theory (Perrow 1984; Perrow 1999) which can be used to identify and understand socio-technical system conditions in critical infrastructure operations that could lead to cascading and unexpected consequences of component disruptions.

Importantly, the article does not mean to reproduce the idea that ‘cyber catastrophe/disaster’ is likely or ‘just around the corner’. Rather, it suggests that unexpected and/or unintended consequences of offensive cyber operations are more likely than sometimes assumed, and that the escalatory risks connected to these events should be taken seriously - even if the effects are not catastrophic.

The article begins with a review of the debates regarding offensive cyber operations within the International Relations (IR) security studies literature, followed by a literature review on critical infrastructure protection. Next, it introduces the theoretical approach of Normal Accidents and outlines how it can be adapted and applied to understand sociotechnical system vulnerabilities in critical infrastructure. The methods section details the case selection process and data used in the study. The analytical section draws upon case examples and interview data to explore the role of NA dynamics in large-scale cyber incidents affecting critical infrastructure. The article concludes with a discussion on the findings in the context of offensive cyber operations and risks of conflict escalation.

Offensive cyber operations in the literature

Offensive cyber operations (defined in this article as cyber operations with the aim to disrupt, deny, degrade or destroy systems) have been in the centre of scholarly interest since the emergence of cybersecurity on the IR security studies agenda. Initially, contributions came primarily from strategic studies perspectives and focused on offensive cyber in the context of (cyber)war (Arquilla and Ronfeldt 1993; Clarke and Knake 2010; Rid 2012). Theoretically, studies focused on translating classical security concepts such as coercion, deterrence and power to cyberspace (Libicki 2009; Nye 2010). Empirically, interest was centred on offensive cyber operations in practice, both in terms of (states) being subjected to them and using them. A multitude of studies have since then continued the tradition of discussing offensive cyber from a military, practical and state-centred perspective, focusing on how these capabilities can be (ethically) used, under what conditions, and to what end (Kello 2017; Ohlin, Govern, and Finkelstein 2015; Jenkins 2016; Dipert 2010; Allhoff, Henschke, and Strawser 2016). They have also focused on mapping how offensive capabilities are being developed and used in practice (Smeets 2022; Smeets and Lin 2018; Moore 2022). This has coincided with empirical developments such as the ‘defending forward’ and persistent engagement approach of the United States (since followed by similar approaches by other democratic states such as the United Kingdom) and increasing instances of states explicitly declaring that they have offensive capabilities.¹

A common argument within the strategic studies literature has been that democratic states need to develop offensive cyber capability for deterrence and/or strategic defensive purposes, especially in the face of the growing cyberthreat from states such as China, Russia, Iran and North Korea, who are continuously developing and launching offensive cyber operations to achieve various strategic and operational objectives. However, debates also highlight disagreements and varying opinions on central aspects, such as the utility of offensive cyber operations to achieve strategic aims compared to other tools, and the extent to which they can be controlled. For instance, while some have argued that using offensive cyber provides an ‘ideal’ way to wage war ‘justly’ (Jenkins 2016), others, like Smeets and Lin (2018) have recognised that collateral damage and a mismatch between intent and the actual damage caused by cyberattacks, has been a common pattern of offensive cyber operations historically (Smeets and Lin 2018, 104–105).

The cybersecurity/IR debates have gradually broadened to include perspectives beyond strategic studies. Since Hansen and Nissenbaum’s influential article ‘Digital Disaster, Cyber Security and the Copenhagen School’ (2009), several studies have used frameworks derived from securitisation theory (Buzan, Waever, and de Wilde 1998) to study how cybersecurity has increasingly been constructed as a national security problem. (Dunn Cavelty 2010; Christou 2019; Burton and Lain 2020; Burton and Christou 2021). These works have been complemented by studies focusing on language in relation to cybersecurity, famously identifying the use of ‘cyberdoom’ narratives (hypothetical catastrophic, and often war-related, scenarios of cyber disaster) in cybersecurity discourses, despite a persisting gap between these scenarios and the empirical reality of cyber incidents and cyber conflict (Eriksson and Giacomello 2007; Lawson 2013; Betz and Stevens 2013; Lawson 2019; Valeriano and Maness 2015).

More recent scholarly debates involve fewer instances of ‘cyberdoom’ language. Rather, there is a broad scholarly recognition that offensive cyber operations generally fall under the threshold of armed conflict (Jacobsen 2021; Moore 2022; Willett 2022). So far, the use of offensive cyber in Russia’s war on Ukraine has confirmed the view that these capabilities are primarily used in addition to conventional warfare, and with relatively limited effect (Willett 2022; Grossman et al. 2023; Microsoft 2022). The escalatory risks of offensive cyber have been the subject of more recent debates within the cybersecurity/IR literature. Connected to the notion that most offensive cyber activities fall below the threshold of armed conflict, it has often been assumed that the escalatory risks of offensive cyber (especially performed by democratic states) are low (Borghard and Lonergan 2019). This view is also reflected in ‘persistent engagement’approaches and policies of democratic states, prominent examples being the US and the UK (Gold 2020).

However, this assumption is also subject to increasing scholarly contestation. Authors have, for different reasons, begun to express concern that the escalatory potential of offensive cyber (even just in the form of persistent engagement and defending forward approaches) has been underestimated. Cyber policies of the US, and other democratic states have become more offence oriented over time (Valeriano and Jenson 2019; Gold 2020; Burton and Christou 2021). However, it appears to be challenging to achieve a controlled strategic impact through offensive cyber operations (Dunn Cavelty and Wenger 2022, 242). Assessing effects and potential side-effects of an exploit – thus ensuring proportionality and discrimination – is a challenging task, even for capable actors (Jacobsen 2021, 710). There is still a lack of international consensus for interpretation of these events (Willett 2022), and decision-makers may assess the same effects in different ways, depending on their socio-institutional context (Gomez and Whyte 2022). In this context, the perhaps most worrisome prospect is the potential for violent escalation - state actors responding to the effects of offensive cyber operations with kinetic means (Libicki 2012,78); accidental escalation – when one party takes action it thinks is non-escalatory but is considered so by another party; and/or escalation due to operational effects which are unintended (Lin 2012, 52).

Security studies, cyber and critical infrastructure protection

Critical infrastructure protection has played, and continues to play, a central role in the cybersecurity debates within International Relations and security studies. The significance of the subject has been reinforced by several empirical examples of cyber incidents involving critical infrastructure services in the last decade (however, largely without catastrophic consequences). These include the Ukraine power grid in 2015, the UK NHS in 2017, the Colonial Pipeline in 2021 and the Viasat hack in 2022.

Strategic studies debates have produced a plethora of contributions focusing on cyberthreats to critical infrastructure in connection to war and conflict (or scenarios/ideas of future war and conflict). This has included works focusing on cyber sabotage (see, for example, Kerigan-Kyro 2014), cyber terrorism (see, for example, Chen, Jarvis, and Macdonald 2014), and hybrid threats to infrastructure (for example, Rudner 2013; Lehto 2022). The Russian use of offensive cyber against Ukrainian critical infrastructure in 2022 has sparked a renewed intensity of interest in these focus areas (Willett 2022).

Scholars from constructivist and critical perspectives have focused on how cyberthreats to critical infrastructure (or prospects of them) are interpreted and framed, and how this relates to overarching security imaginaries (Dunn Cavelty 2010; Gjesvik and Szulecki 2023). Authors have, for example, highlighted how fears of catastrophic critical infrastructure breakdowns have been used to legitimize, and achieve acceptance, for securitising moves in the cybersecurity policy area (Dunn Cavelty 2008; Hansen and Nissenbaum 2009; Valeriano and Maness 2015).

An emerging interest in the intersection between critical infrastructure and politics has focused attention on the physical infrastructure upon which the internet relies, including land-based fibre optic communication cables, submarine cables and space infrastructure/satellites (Davenport 2015; Bueger and Liebetrau 2021; Franken et al. 2022). Empirically, this has included works on risks to critical infrastructure operations in the context of an increasingly militarised cyberspace, including dual-use technology - satellites and other infrastructure - which can serve both military and civilian purposes (Eriksson and Giacomello 2022, 100). Theoretically, it has coincided with a growing literature applying perspectives from STS (Science and Technology Studies) to IR cybersecurity issues. This literature emphasises the co-constitutive nature of technology and social practice, focusing on aspects such as the (un)intended consequences and (un)expected insecurities created by the entanglement of human and non-human agency (Liebetrau, Christensen, and K 2021). It rests in the assumption that technologies are not separate from politics, social construction and power relations, and that we need to study relations between technical and sociopolitical objects to understand how cybersecurity knowledge is formed in practice (Stevens 2018; Dunn Cavelty 2018).

Law and governance perspectives have provided a rich set of contributions to the literature, fuelled by important cybersecurity policy and regulatory developments relating to cybersecurity vis-a-vis critical infrastructure during the last decade - such as the EU NIS and NIS2 directives, the EU Cyber Diplomacy Toolbox and the NATO Tallinn Manual. Overall, this literature highlights that governance (and norms) of critical infrastructure protection from a cybersecurity standpoint are developing and advancing both vertically (at various levels of governance), and horizontally (transnationally and trans-sectorally) (Christou 2016; Michels and Walden 2018; Schmitz-Berndt and Cole 2022). Despite this evolution, fragmentation in governance and multidimensional socio-technical uncertainties persists (Carrapico and Barrinha 2017; Dunn Cavelty and Wenger 2022). These uncertainties are enhanced by militarisation and the expansion of emerging technologies such as AI (Riebe and Reuter 2019; Bonfanti 2022).

The consequences of the combined aspects of fragmentation in governance, transnational and complex critical infrastructure operations (Goldin and Mariathasan 2014) and militarisation, still needs further academic exploration, both empirically and theoretically. A central argument of this article is that sociotechnical systems theory provides us with fruitful tools to do so. This acknowledges the need to bridge between debates and insights from the security literature and the safety literature, the latter focusing less on antagonist threats and more on system-based dangers in cyberspace, including the ones stemming from complex systems (Michalec, Milyaeva, and Rashid 2022). The article builds on one socio-technical perspective in particular: Charles Perrow’s Normal Accidents (NA) theory.

Normal accidents theory

Normal Accidents (NA) theory can be considered a classic in the study of safety, technology and socio-technical systems. NA dynamics are present when (high-risk) systems have two simultaneous characteristics: complex interactivity and tight coupling between system components. When NA dynamics are present in a system, Perrow argued, accidents are (eventually) inevitable (Perrow 1999; Perrow 1984). Although these accidents may happen rarely, they will be exceedingly difficult to prevent. Before expanding on Perrow’s argument, we should first clarify some key terms.

From the NA perspective, a system can refer to a relatively small, closed computer system as well as a macro system comprising many sub-parts and systems (the organisation of organisations). System components are not just technical (e.g. individual pieces of hardware or software); they can also refer to human operators, procedures or administrative services (organisational components). Interactive complexity refers to the way these components interact and are tied together within the system, a condition where two or more component failures within the system can combine in unexpected ways to create consequences that the designers of the system could not have anticipated (Perrow 1999, 4).

Interactive complexity alone does not necessarily equal the potential for serious incidents. Another system characteristic is needed – tight coupling. Simply put, tight coupling refers to the interdependency and lack of redundancy (or slack) within the system (between components) – i.e. the inability to continue operations without failed parts and the inability to isolate failed parts from the other parts. The result of interactive complexity and tight coupling is that relatively simple failures in individual components can combine and cascade through the system in unexpected ways. Because the management of these systems requires increasingly complex organisational settings and technology (even the components that are supposed to make them safer), Perrow argued that attempts to fix the system through added components increases the interactive complexity and tight coupling of the system, making the system more prone to accidents (Perrow 1999, 5). Perrow emphasised the difficulty in analysing how failures in individual components may combine to cause cascading effects. Thus, operators or analysts of the system cannot be expected to anticipate how accidents or incidents in systems with NA dynamics will happen or develop.

Early scholarly criticism of NA focused on the technological determinism of the argument, which downplays the ability of organisational and human influence to reduce the danger of high-risk systems (La Porte 1994). Normal Accidents theory was especially contrasted with the perspectives from the High Reliability Organisation (HRO) project, which studied the practices of high-risk organisations that, while operating under NA conditions, experienced few incidents or accidents (La Porte 1996). However, authors have since then continuously pointed to the compatibility of the HRO and NA perspectives, suggesting that they are not mutually exclusive (Rijpma 1997; Brown 2018; Le Coze 2015).

Although Perrow early on realised the potential of extending his argument to the (then) nascent internet (Perrow 1999), he mainly, with some exceptions, applied the NA concept to micro or meso level systems (e.g. nuclear facilities or aircrafts). Contemporary scholarly contributions have directed more attention to how NA manifests at the organisational, political, and macro levels of high-risk systems. Le Coze (2021) argues, for example, for an expansion of the scope, scale, and timeframe of the original NA argument, including dimensions of scale of governance.

This article suggests that an adapted version of NA can be used to understand how disturbances in individual system components in the socio-technical systems underpinning critical infrastructure operations may result in cascading and unexpected net consequences. The adaptation of the original NA concept consists of an extended scope and greater focus on macro level dynamics and is in line with Le Coze’s argument (2015; 2021), which distinguishes between levels of governance in the NA application. This study applies three main analytical layers of NA application: technology, organisation and macro, and specifically adapts it to understand NAdynamics in modern critical infrastructure.

The first layer of analysis – technology – focuses solely on NA dynamics in technological systems (e.g. software or hardware). The second layer of analysis focuses on NA dynamics in socio-technical structures of organisations (including human cognitive aspects in connection to technology). The third layer of analysis focuses on NA dynamics at the macro level, including global interdependencies and flows of goods and services.

Methods and data

The article draws upon two main sources of data: interviews and primary/secondary documents/reports. The documents and reports primarily concern the case examples. The interview data is based on 14 interviews, of which 12 were in-depth. The interviewees, who were all allowed to stay anonymous, included senior strategic or operational cybersecurity experts/practitioners from four countries, and one EU agency (Sweden, UK, US, Switzerland, and the EC3/European Cybercrime Centre). Of these, about half were affiliated with the private sector (e.g. consultants or contractors specialising in cybersecurity and critical infrastructure protection), and half were affiliated with the public sector (e.g. national cybersecurity centres or CERTs/Computer Emergency Response Teams). Interviewees were selected based on their expertise on cyberattacks/cyber incidents affecting critical infrastructure, and on organisational and geographic diversity, although all were within the umbrella of the Euro-transatlantic zone. Interviews were conducted between 2020 and 2022. The interviews were structured like conversations that unfolded around the main questions (semi-structured). The main questions posed to the interviewees concerned their view on how and why cyber danger or vulnerability develops in critical infrastructure systems (at the technical, organisational and macro layer, respectively) and their view on/experiences with collateral damage and/or unintended consequences in cases of large-scale cyber incidents affecting critical infrastructure.

Analysis: normal cyber accidents

Technology

The first layer of analysis – technology – focuses on NA dynamics in code, systems or hardware (Table 1).

Interactive complexity and tight coupling on this layer could be exemplified by legacy code, systems and/or hardware underpinning critical infrastructure operations. Many modern critical infrastructure operations still rely on legacy in the form of outdated operating systems, legacy code that can no longer be supported by the software developer for patches, hardware no longer reproduced, or old code written in outdated code languages in the foundation of an Industrial Control System (ICS) (Walker 2020). Legacy code could be defined as an old source code or code base that is in some ways obsolete, unpatchable, or not supported by modern standard hardware or environments, but the system may still depend on it for the base layer on which other and more modern layers are added (Mullen 2016). These are often difficult to understand and change as they are ‘code[s] without tests’ or ‘unruly code bases’ (Feathers 2005, xvi). When the systems underpinning critical infrastructure are built on layers of legacy code and systems written in a variety of code languages with numerous add-ons to make them compatible, interactive complexity increases.

This interactive complexity makes it difficult to get a proper system understanding and enhances the risk of ‘human factor’ mistakes, including, for example, failing to segment components that should have been segmented, connecting a component to the internet that should not have been connected, or continuing to use an outdated component that should have been replaced. Such mistakes could be found and leveraged by threat actors or worming malware to traverse from one point in the system to another (potentially more critical). Moreover, dependence on each layer of legacy to the next (lack of redundancy, or alternative ways of operating) adds tight coupling, which means that the failing components on one layer may have disruptive consequences for the system as a whole.

Critical infrastructure operations often have little allowance for downtime, which further adds to the tight coupling component. Several interviewees, for example, noted that the constant requirement for availability of critical infrastructure services makes it difficult and expensive to replace legacy systems and hardware (Interviewee 2; Interviewee 6). Consequently, critical infrastructure systems are often used beyond their intended lifetime; in fact, as one interviewee noted, some of the devices used today were developed before the internet became ubiquitous (Interviewee 12).

The WannaCry ransomware is an example of a cyberattack proliferating widely due to the use of legacy systems. In 2017, WannaCry affected about 200,000 computers across more than a hundred countries, and caused system disruptions in one-third of the hospital trusts (NHS) in England, causing a national declaration of emergency in the UK (UK National Audit Office 2018, 5; Interviewee 8).WannaCry was not particularly sophisticated, but spread quickly and globally due to the lack of patching (security updates) of the known vulnerability which it targeted, and the use of systems beyond their intended lifespan. For instance, some of the NHS computers at the time still used Windows XP, an operating system so outdated that it no longer received patching updates (Interviewee 12). The lack of overview of legacy systems in the health sector also made it difficult for key actors, such as the UK National Cybersecurity Centre (NCSC UK), to estimate the spread of the vulnerability in the incident response efforts (Interviewee 13; Interviewee 14).

While cybersecurity maturity has generally increased in critical infrastructure sectors since the WannaCry ransomware attack in 2017, the larger problem of legacy (code, systems and/or hardware) in, or connected to, critical infrastructure operations remain, and could still be the source of NA dynamics and cyber incidents. NA dynamics could be caused by simple mistakes, such as the continued connection of a vulnerable legacy component which should have been disconnected, or the postponement of implementation of security measures. The vulnerability created by these mistakes, and the potential consequences of them, can still be difficult to detect before they are involved in incidents, for example, in the case of the WannaCry ransomware attack on the UK NHS. This is especially problematic in the context of the use of worm self-replicating malware designed to infiltrate a network and spread as far as possible (Interviewee 7; Interviewee 6; Interviewee 1).

Table 1. NA dynamics in the technological layer of critical infrastructure operations.

Layer of critical infrastructure operations	Interactive complexity	Tight coupling
Technology	Layered legacy code and combinations of old and new hardware	Legacy dependence and little/no allowance for downtime

Table 2. NA dynamics in the organisation/cognition layer of critical infrastructure operations.

Layer of critical infrastructure operations	Interactive complexity	Tight coupling
Organisation	Integration between organisational and technical components Lack of comprehensive cognitive understanding of system ecology	Lack of organisational slack/structural flexibility and redundancy Lack of distinction between important/critical and unimportant components

Table 3. NA dynamics in the macro layer of critical infrastructure operations.

Layer of critical infrastructure operations	Interactive complexity	Tight coupling
Macro	Inter-organisational dependencies	Dependence on delivery of supplies/services to keep system running/inability to operate without supply/service delivery Centralisation of services

Organisation/cognition

The second layer of analysis focuses on NA dynamics in socio-technical structures of organisation that underpin critical infrastructure operations (Table 2).

Interactive complexity at the organisational layer of critical infrastructure operations could be exemplified by the interactions and connections that exist between technical and non-technical components, or between the overall system and sub-system of an organisation. In big organisations, these interactions could be especially complex and multi-layered. Interview data highlighted the typical difficulty obtaining a comprehensive picture of all the components involved in large-scale operations and how they interconnect.

A component that is seemingly low risk can be important. Organizations and companies tend to have a bad habit of not maintaining a full inventory of assets. The old risk matrixes are outdated. Many of the incidents of the past, such as NotPetya, have been able to proliferate because of network components seen as ‘unimportant' or ‘low risk'. (Interviewee 9)

I think we are bad at seeing complexity and achieving a holistic understanding for these systems. (Interviewee 1)

The interactive connections and dependencies within and between systems are complex. The effects of these interactions will be clearest in the interfaces between competencies and between sectors. Everyone knows their piece, but no one sees the whole. (Interviewee 3)

People who are involved in the operation of a modern complex system are often experts in a certain aspect or part of that system, not understanding all the components and interdependencies of it. (Interviewee 5)

Although Industrial Control Systems (ICS) connected to critical infrastructure operations are usually better protected than others, these systems might be tightly coupled to (and therefore dependent on) less protected and more vulnerable systems, including, for instance, administration systems. The integration and tight coupling between administration and technical system components may result in unexpected net consequences from an initial, smaller disruption. Several interviewees mentioned the case of the Colonial Pipeline incident as an example of an accident where a cyberattack disruption of a single, seemingly quite unimportant administrative component of a critical infrastructure operation, led to a high-profile incident (Interviewee 7; Interviewee 6; Interviewee 1). The incident started with a ransomware attack (where the threat actor holds systems hostage until a fee is paid by the victim) in May 2021. The primary target of the attack was the billing infrastructure of the gas pipeline company. However, the inability to bill customers caused the executives to halt the operation of the pipeline entirely (partly due to legal reasons). The incident was followed by a Presidential declaration of emergency and Colonial Pipeline ended up paying the ransom.

In the colonial pipeline example, the ransomware didn’t reach all the way to the ICS; it only affected the office/administrative network. But they couldn’t operate without it. There are interdependencies between these systems and interface systems (that controls logs and logistics, for instance). If the support systems are out, it’s often difficult to keep operations going. (Interviewee 6)

You do not have to affect the operation of the ICS itself to cause severe damage or halt the operation. If you, for instance, have a loss of view and control because you lose a key admin system, you might not be able to run your operation or you might not be legally allowed to. That creates a vulnerability for indirect attacks. (Interviewee 1)

The case of the Colonial Pipeline incident illustrates how NA dynamics at the organisational/cognitive layer can create a condition where the disruption of seemingly non-critical components of a system (an admin system), may result in serious cascade effects – both in operational and political consequences.

Macro

The third layer of analysis focuses on NA dynamics at the macro level of critical infrastructure operations (Table 3).

Critical infrastructure services are typically dependent on a complex ecology consisting of a constant global flow of services from a variety of actors. Centralisation is often introduced to reduce cost and friction between entities, enhance alignment, and improve control (e.g. using the same vendor, service provider or administration system) (Interviewee 1; Interviewee 5). While centralisation can improve efficiency, it is also a source of tight coupling. The combination of a complex supply chain ecology and centralisation creates NA dynamics at the macro level. In this condition, disruption of a component in the supply chain could have negative effects for the whole system ecology. This could create a situation where malware proliferates beyond intent, or services are disrupted by mistake. However, these conditions could also be exploited by a cyberthreat actor to achieve proliferation by design, for instance, by targeting software which is distributed to a range of customers by a trusted and centralised source (Interviewee 4; Interviewee 1).

On 13 December 2020, FireEye announced the discovery of a highly sophisticated cyber intrusion that leveraged a commercial software application made by SolarWinds. When customers downloaded the Trojan Horse installation packages from SolarWinds, attackers were able to access the systems running the SolarWinds products (Orion) (Center for Internet Security 2021). The code created a backdoor to information technology systems used by the customers. The hackers used this backdoor to install more malware that gave them access to spy on thousands of companies and organisations, including government agencies (US Government Accountability Office 2021).

Unlike SolarWinds, the Kaseya incident was a ransomware attack (encrypting systems and requiring a fee for its release) deployed by a cyber-criminal group known as REvil/Sodinokibi. On 2 July 2021, customers of Kaseya’s Virtual System/Server Administrator (VSA), an on-premises product, reported that ransomware was being executed on endpoints. Although the attack directly affected fewer than sixty Kaseya customers, many of these customers provided services to other businesses, which meant that the attack indirectly affected more than a thousand companies (Kaseya 2021). One of these companies was COOP, a major supermarket chain in Sweden. Following the attack, which affected COOP’s payment system, COOP was compelled to enforce a nationwide lockdown of its 800 stores for almost a week. A later report from the firm that performed the incident response in this case highlighted the dependencies created using a centralised service (such as the VSA) without separation between client environments, where a compromise of the central server leads to downstream effects (Truesec 2021). This was also highlighted in the interview data as follows:

In the case of the COOP incident, there was not enough awareness on the dependence on the supply chain service. (Interviewee 6)

In the case of the SolarWinds’ incident as well as in the Kaseya incident, threat actors used the interactive complexity of the targeted systems to find and leverage vulnerabilities and get access. The tight coupling/centralisation of the service was then leveraged to achieve proliferation by design and create a ‘downstream’ effect.

Discussion

The previous section detailed how NA characteristics (the combination of interactive complexity and tight coupling) can manifest at three layers of critical infrastructure operations: technology, organisational and macro (see Table 4 below).

Table 4. NA dynamics in multiple layers of critical infrastructure operations.

Layer of critical infrastructure operations	System characteristic: Interactive complexity	System characteristic: Tight coupling
Macro	Inter-organisational dependencies	Dependence on delivery of supplies/services to keep system running/inability to operate without supply/service delivery Centralisation of services
Organisational	Integration between organisational and technical components Lack of comprehensive cognitive understanding of system ecology	Lack of organisational slack/structural flexibility and redundancy Lack of distinction between important/critical and unimportant components
Technology	Layered legacy code and combinations of old and new hardware	Legacy dependence and little/no allowance for downtime

As detailed above, there are several ways that effects of malware could cascade in unexpected ways due to NA dynamics in the socio-technical systems which underpins the operation of critical infrastructure. While NA dynamics in a system could be used deliberately by threat actors to achieve proliferation by design, these dynamics could also trigger proliferation beyond the control or intent of the threat actor. For example, according to interview data, this was likely the case of WannaCry.

My view on it was that it wasn’t the intent of the attackers to create something that would spread globally. Within the code there is a very subtle function which they could have easily made a mistake with, and that mistake was what made it spread globally. There is also the timing in how the attack was set up, which makes it look like it was maybe too quick for anyone to make the decision to attack the world. I suspect what they were trying to do was just to attack a single organization, and it might not have been a deliberate ransomware attack; they might have covered up for an attack against a payment system, for example. We have seen them do that kind of cover up attack before. But they set the thing loose and it spread across the globe. What they actually gained from it in bitcoin was rather small. (Interviewee 8)

A key aspect of NA dynamics is that they are not always clearly visible (as in the case of Colonial Pipeline, for instance). Consequently, there is a risk of threat actors mistakenly triggering NA dynamics in a system, especially since the NA dynamics could exist on multiple layers simultaneously (or in the interface between layers). The interview data suggests that risks of escalation exists whether the effects are intended by the threat actor or a result of mistakes (Interviewee 1; Interviewee 2; Interviewee 10; Interviewee 11). Several interviewees also mentioned that thresholds of escalation may vary from state to state.

We have seen malware get out of hand before on the internet and cause collateral damage, and I’m very concerned something like that will happen again and that a non-cyber response becomes necessary. (Interviewee 2)

Several countries now have doctrines in which they have stated that they might respond with kinetic power to a serious cyberattack. But different countries have different frameworks and mandates for escalation when it comes to cyberattacks, and different roles making key decisions. (Interviewee 10)

If a cyberattack leads to casualties, the response to such an event would likely differ depending on the country. In some countries such an event would likely escalate rather quickly. The question is how much time and effort will be spent on diplomacy? (Interviewee 1)

While most of the large-scale cyber incidents affecting critical infrastructure to date have been the result of malicious activity from groups based in non-democratic states, these risks are not exclusive to the offensive cyber-operations of these actors. The assumption of controllability and predictability of offensive cyber operations sits at the heart of ‘responsible behaviour’-approaches by national cyber forces of democratic states who are adopting ‘defending forward’ approaches, as reflected, for instance, in the UK National Cyber Force’s operational principles of accountability, precision and calibration (National Cyber Force 2023). The ability to prevent collateral damage resulting from offensive cyber operations is also a commonly held assumption within military communities, as highlighted by Smeets (2018):

..the belief held in many military quarters is that with sufficient testing and retesting prior to usage, offensive cyber operations can achieve a designed effect and minimize damage to entities that should remain unharmed (Smeets 2018, 104–105).

Democratic states will indeed aim to ensure non-escalatory and precise effects in their offensive cyber operations. However, this assumes the ability to correctly assess their impact on target systems, and the ability to correctly assess others’ interpretation of the effect. Secrecy surrounding states responses to offensive cyber activity could make it difficult to estimate interpretation and thus escalatory potential of offensive action in cyberspace – conducted ‘responsibly’ and with precise effects or not. The uncertainty and ambiguity surrounding cyberspace as a strategic context, including cognitive biases of decision makers and bureaucratic procedures may also increase risks of unintended escalation (Gomez and Whyte 2022; Dunn Cavelty and Wenger 2022).

When it comes to assessing actual impact on target systems, this may be exceedingly difficult to do if systems with NA characteristics are targeted (by mistake or design). In fact, according to NA-theory, it may be almost impossible to consistently and correctly assess the potential net effects of disturbing components in a NA system, because the possible ways in which failures in individual components of that system could interact and cause unexpected consequences are just too many.

Rather than understanding the tendencies of collateral damage from offensive cyber operations we have seen so far solely as a result of the lack of responsible design or intent, it should be considered that containing the effects of offensive cyber operations (especially if they trigger NA dynamics by mistake or design) may be more difficult than sometimes assumed. Risks for accidental triggering of NA dynamics and unintended consequences of offensive cyber (including unintended escalation) are compounded by increasing complexity of critical infrastructure systems, the proliferation of offensive tools (making these tools available for a wider range of actors), and the increased development of offensive cyber capabilities by both democratic and non-democratic states.

Conclusion

This article applied a ‘Normal Accidents’ perspective to advance our understanding of how disruptions of individual components in critical infrastructure-related systems may lead to cascading and unexpected consequences. It discussed how these socio-technical dynamics relates to risks of collateral damage from offensive cyber operations and unintended escalation. The article directs renewed attention to the subject of the escalatory potential of offensive cyber operations. While the escalatory risks of offensive cyber have been discussed in the literature (and often deemed low), this has mainly concerned risks of intentional escalation. Risks of unintended escalation based on mistakes and unexpected consequences of offensive cyber have been less explored, but constitute an important subject for further scholarly investigation. As this article has argued, the socio-technical systems-literature may provide us with useful frameworks and insights to theorise the unintended in the intersection between humans, systems and organisation.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 HCSS Cyber Arms Watch index, https://hcss.nl/cyber-arms-watch/.

 

Appendix

Table

Reference	Role/affiliation
Interviewee 1	State 1: Senior expert cybersecurity/critical infrastructure systems (private sector)
Interviewee 2	State 2: Senior expert cybersecurity/ critical infrastructure systems (private sector)
Interviewee 3	State 1: Senior expert cyber defence (private sector)
Interviewee 4	EC3/European Cybercrime Centre
Interviewee 5	State 1: Senior expert cybersecurity/critical infrastructure systems (private sector)
Interviewee 6	State 1: Senior expert cybersecurity/critical infrastructure systems (public sector)
Interviewee 7	State 1: Senior expert cybersecurity/critical infrastructure systems (private sector)
Interviewee 8	State 3: Senior cybersecurity researcher/malware expert (private sector)
Interviewee 9	State 4: Senior expert cybersecurity (private sector)
Interviewee 10	State 1: Senior expert cybersecurity (public sector)
Interviewee 11	State 1: Senior expert cybersecurity (public sector)
Interviewee 12	State 3: Senior expert cybersecurity/threat intelligence (public sector)
Interviewee 13	State 2: Senior expert cybersecurity (public sector)
Interviewee 14	State 1: Senior expert cybersecurity (public sector)