Evidence-based cybersecurity policy? A meta-review of security control effectiveness

ABSTRACT

Cybersecurity policy should guide firms towards implementing the most effective security controls and procedures. However, there is no authority that collects evidence and ranks cybersecurity controls by efficacy. The evidence needed by policymakers is distributed across academic studies and industry white papers. To address this gap, we conduct a meta-review of studies that empirically evaluate the efficacy of cybersecurity interventions. Attack surface management and patch cadence were consistently the first and second most effective interventions. Reduced cyber insurance claims frequency was associated with migrating to cloud email and avoiding specific VPN providers. Multi-factor authentication was effective in protecting individual accounts, although inconsistent MFA-implementation undermines efficacy when rolled out across an organisation. The evidence suggests effectiveness is driven by how a control is implemented more than by a binary yes-no regarding whether it is implemented. Thus, policy measures that mandate specific controls are unlikely to result in risk reduction. Instead, policymakers should aim to support organisations in administering security controls and making risk-based decisions. Successful examples can be seen in policy measures that improve the efficiency of patch management, such as funding for the US National Vulnerability Database, CERT/CC, and the Known Exploited Vulnerabilities catalog.

KEYWORDS:

• Cybersecurity policy
• science of cybersecurity
• security controls
• meta-review
• evidence-based policy

1. Introduction

Cyber risk concerns governments and organisations regardless of their geographic location, sector or size. This societal problem results from the reality that most critical infrastructure is owned and operated by private sector entities who make individual cybersecurity investment decisions based on the private entities’ risk tolerance (Carr 2016). The consequences of those private decisions are often borne by a much broader pool of victims (Anderson and Moore 2006). When disruptions meaningfully affect the availability of critical services, the public looks to governments to mitigate those consequences.

Given the societal problem, cybersecurity policy should improve the security posture of private sector firms. Yet the way forward for policymakers is less certain. To date, cybersecurity regulations typically avoid prescribing specific controls. Moore (2010) notes that:

[banking] regulations avoid technical prescriptions in favor of forcing compliance with organizational requirements. A process-based approach has the advantage of being less dependent on rapidly changing technologies, as well as making the job of compliance verification easier for regulators.

Turning to state-level cybersecurity regulations, Shackelford, Boustead, and Makridis (2023) show there is no consistent definition of ‘reasonable’ security across US states.

Exceptions exist in which policy attempts to guide private firms towards implementing security controls. The UK Government introduced the Cyber Essentials scheme, which defines concrete steps to achieve basic cyber hygiene (Such et al. 2019). Firms are required to implement Cyber Essentials in order to win certain government contracts. Although less formal, the Federal Trade Commission published a retrospective on enforcement actions, which resulted in ‘a partial definition of what constitutes “reasonable security”’ (Breaux and Baumer 2011). It is clear that the policy goal of guiding firms towards improved cybersecurity is incomplete.

Regardless of whether the intended audience is small businesses or public companies, policymakers should all agree that a more effective intervention should be implemented before a less effective one, all else being equal. For this reason, a ranking of cybersecurity controls by effectiveness would be a useful addition to a policymaker’s toolkit. Such a ranking would ideally be guided by scientific evidence.

Indeed, various government initiatives have been introduced to support the ‘Science of Security’ (Herley and Van Oorschot 2017). The US National Security Agency (NSA) awarded prizes to the best research in this space and now the NSA hosts an annual conference. Similarly, the US Department of Defence (DoD) commissioned the 86-page Science of Cyber-Security report (Office 2010). In a similar vein, the UK Government funded a research project to collect and classify the Cybersecurity Body of Knowledge (CYBoK) (Rashid et al. 2018). This stream of literature would ideally underpin evidence-based cybersecurity policy.

This motivates a meta-review that is tailored to cybersecurity policy. In particular, we focus on evidence that can help prioritise security controls. This differs from prior efforts that have, for example, focused on epistemic debates about what constitutes the science of security (Herley and Van Oorschot 2017) and the precise statistical models that have been used (Woods and Böhme 2021). Prior work cannot help a policymaker answer concrete questions such as which specific security measures, such as back-ups or multi-factor authentication, should be included in a checklist. It is even less clear how to set implementation details such as whether the second authentication factor uses SMS, a dedicated app or a hardware token.

In doing so, we organise the first half of our paper like a scientific article in that it has dedicated sections for related work (Section 2), methodology (Section 3) and results (Section 4). These sections do not advance an argument, but instead provide a structured and transparent approach to how we identified the scientific evidence. In Section 5, we reflect on how cybersecurity policy might be crafted in light of this evidence. We offer a conclusion in Section 6.

2. Related work

This section makes the normative argument that field studies involving real organisations represent the way forward for prioritising controls. In different ways, this argument has been made by various authors and working groups (Böhme 2010; Falco et al. 2019; Gollmann et al. 2015; Moore et al. 2017; Pfleeger 2012). Despite the enthusiasm, this type of study remains a ‘minority approach within the science of security’ (Woods and Böhme 2021). Before making the argument for empirical research, we first identify non-empirical approaches to generating security knowledge and explain why such approaches are inappropriate for the task of ranking the efficacy of security controls.

In a meta-review published at the most prestigious computer security conference, Herley and Van Oorschot (2017) conclude that ‘the security community is not learning from history lessons well-known in other sciences’. In particular, the authors discuss the limitations of using formal models, a core approach within computer science, as a method to ‘improve outcomes in the real world’ (Herley and Van Oorschot 2017). A simplistic summary is that computer scientists write flawless proofs of the security properties of mathematical models, but those models fail to map onto the real world. As Herley and Van Oorschot (2017) note:

A rigorous proof of security of a mathematical system allows guarantees about a real-world system only if the coupling between them is equally rigorous. We have seen repeated failure in poor connections between mathematical systems and real-world ones, and consequent failure of the latter to enjoy properties promised by the former.

For this reason, mathematical models are inappropriate for ranking the efficacy of security controls.

Hypothetical analysis represents another approach. To understand whether the UK cyber essentials scheme protects against cyberattacks, Such et al. (2019) construct a hypothetical set of four firms and their networks. The authors then randomly sample 200 vulnerabilities from a list of ten thousand real vulnerabilities, and evaluate whether the four hypothetical companies were potentially affected, which was the case for 137 vulnerabilities. Of these vulnerabilities, 98.5 per cent were either mitigated or partially-mitigated by the use of the Cyber Essentials security controls. Does this mean the controls are 98.5 per cent effective?

Unrealistic assumptions mean the real-world efficacy of Cyber Essentials is likely to fall well below mitigating 98.5 per cent of attacks. As acknowledged in the study, patches are assumed to be installed in a timely manner, which is often not the case (Such et al. 2019). An additional problem results from the researchers randomly choosing vulnerabilities, whereas real attackers are strategic. Real-world attackers typically select the subset of vulnerabilities that achieve an optimal trade-off between effort and compromise (Allodi, Massacci, and Williams 2022). For example, only around 5 per cent of vulnerabilities are ever exploited in the wild (Jacobs et al. 2021).

The core problem across both approaches is addressing ecological validity. Neither mathematical models nor hypothetical scenarios can easily capture how attackers behave in the real world. Field studies avoid this problem by studying real organisations which are (potentially) targeted by real attackers. For our purposes, such studies inform the relationship between security controls and the likelihood and impact of an incident, ideally by providing a ranking.

Although this seems like a foundational question within computer security, most review articles focus on other questions such as the impact of cyberattacks (Anderson et al. 2019; Schlackl, Link, and Hoehle 2022; Spanos and Angelis 2016), victimisation frequencies (Breen, Herley, and Redmiles 2022; Reep-van den Bergh and Junger 2018) and future research directions (Eling, McShane, and Nguyen 2021; Falco et al. 2019). An exception is provided by Woods and Böhme (2021) who collect studies with the goal of answering the question ‘Which security interventions effectively reduce harm?’ Discouragingly, the authors conclude that an answer is ‘unavailable based on current evidence’.

Our meta-review avoids this conclusion for three reasons. The first is that, in addition to academic studies, we collect industry reports that have access to data on whether firms suffered incidents and financial loss, whereas Woods and Böhme (2021) rely on academic studies that rely on the limited amount of incidents that are publicly reported. Second, we look for rankings that are replicated across multiple studies, acknowledging that no study will provide a definitive answer. Finally, three years have passed since Woods and Böhme (2021) collected their studies. As Table 1 shows, the majority of our studies were not identified in the prior work because they are either industry studies or were published after 2020. Collecting industry sources in 2023 allows us to rank security interventions by efficacy, although there are limitations in doing so. The next section describes our approach.

Table 1. The studies we collected via literature search.

Study (Year)	Outcome variable	Sample window		Type of Measurement
		Start	End
Industry
Marsh (2023)	claims	2010	2023	questionnaire
GallagherRe (2022)	claims	2021	2023	scans
AtBay (2023)	claims	June 2018	June 2022	mixed
Coalition (2023)	claims	January 2022	January 2023	scans
BitSight and Marsh (2022)	claims	2018	2021	scans
BitSight (2023)	incidents	Nov 2018	June 2021	scans
SecurityScorecard and Marsh (2022)	incidents	2019	2021	scans
Cisco (2021)	questionnaire	2021	2021	questionnaire
Arete and Cyentia (2022)	incidents	May 2019	December 2021	investigation
Meyer et al. (2023)^†	inbox activity	April 2022	September 2022	system data
IBM (2023)	incident cost	2022	2022	questionnaire
Academic
Gandal et al. (2023)	questionnaire	July 2020	March 2021	questionnaire
Doerfler et al. (2019)^‡	inbox activity	June 2017	May 2018	system data
Nagle, Ransbotham, and Westerman (2017)	security events	July 2014	April 2015	scans
Edwards, Jacobs, and Forrest (2019)	botnet infections	January 2015	January 2016	scans
Tajalizadehkhoob et al. (2017)	network abuse	March 2016	March 2016	scans
Bilge, Han, and Dell’Amico (2017)	device infection	Mid 2015	End of 2015	AV data
DeKoven et al. (2019)	device infection	June 2018	December 2018	network flow

Note: ^†all authors are Microsoft employees. ^‡two of the seven authors are Google employees.

3. Approach

Section 3.1 describes how we identified relevant studies. Section 3.2 explains how we extracted statistical results.

3.1. Search

To build a corpus of empirical studies, we identified research from three communities: academia, the insurance industry and the cybersecurity industry. We relied primarily on systematic reviews, identified in the previous section, to navigate the academic literature (Eling, McShane, and Nguyen 2021; Falco et al. 2019; Woods and Böhme 2021). For insurance, we searched for white papers produced by major insurers, InsurTechs, brokers and service providers. For the security industry, we conducted opportunistic searches, and asked a handful of practitioners whether they were aware of studies. Systematically mapping the various cybersecurity white papers across the $100 billion industry was deemed to be unfeasible given that they predominantly consist of marketing material and are rarely organised into journals.

Our inclusion criterion was that the study empirically investigated the statistical relationship between security controls and firm-level cyber risk outcomes in the real world. This resulted in 18 papers that are listed in Table 1. Essentially, a study must satisfy the following criteria: C1: data must be collected from real-world systems exposed to real threat actors; C2: the independent variables must be security controls, procedures or technologies; and C3: the dependent variable must be an outcome that matters to the organisation at large. The first criterion (C1) ensures ecological validity. The second criterion (C2) meant that we did not include studies that only investigated the cost of incidents. The final criterion (C3) excludes low-level technical outcomes, such as whether an anti-virus detects particular strains of malware.

3.2. Analysis

For each paper in our sample, we extracted metadata such as the data collection time window, the type of data collection, and the sample size. The sample size was inconsistently reported by the industry reports. We discarded studies from before 2017 as it is unclear whether the findings are still relevant in light of changes in the threat landscape, such as the ransomware epidemic. Some academic studies included data from 2015, as can be seen in Table 1.

To collect relevant evidence, we identified statistical tests linking security controls and procedures to cyber risk outcomes. We then extracted the statistical coefficient that described the relationship between a specific control and the dependent variable. This resulted in a list of 128 variables, each with a coefficient describing how it influences the security outcome.

We could not directly compare the results because the studies deployed different statistical designs and measurement approaches. To address this, we grouped similar variables, acknowledging that the tests and measurements never lined up perfectly. We looked at relative effect sizes rather than focusing on absolute effect sizes, which are difficult to compare when the precise details of the experiment do not match. This allowed us to identify controls that were consistently shown to be among the most effective in each study.

4. Results

Section 4.1 describes the design of the studies in our sample. Readers who are only interested in the results should skip to Section 4.2, which provides a summary of the effectiveness of different interventions.

4.1. Studies

Table 1 lists the studies in our corpus, eleven from industry and seven from academia. The industry studies typically modelled incidents at a firm-level, such as an insurance claim (multiple studies) or a ransomware incident (BitSight 2023). The IBM study focused instead on the cost of incidents (IBM 2023) while the Microsoft study looked at email compromise (Meyer et al. 2023).

Academics tend to focus on low-level indicators of compromise such as device infections (Bilge, Han, and Dell’Amico 2017; DeKoven et al. 2019) and network abuse (Nagle, Ransbotham, and Westerman 2017; Tajalizadehkhoob et al. 2017). These incidents may or may not cause actual losses at the firm. For example, devices in a network are regularly infected, but a firm’s security team may respond to the incident before there is a data breach or a disruption to business activities. In contrast, cyber insurance claims are by definition associated with a tangible loss, although in some cases the insurer can resolve incidents without a formal claim (Coalition 2023).

In terms of independent variables, most studies use network measurements to collect data about security posture. Some studies collect internal system data by partnering with vendors (Bilge, Han, and Dell’Amico 2017; Doerfler et al. 2019; Meyer et al. 2023). An academic study collects network flow data by partnering with the university’s IT department (DeKoven et al. 2019). A handful of studies collect self-reported data via survey instruments, which allow them to ask about organisational controls and procedures (Cisco 2021; Gandal et al. 2023). The main weakness of questionnaires is that they take time to fill out and rely on self-reported answers. Network scans solve both problems but can only collect data about externally-facing infrastructure such as web and mail servers. The trade-off is essentially to collect high-quality data about a subset of the network or low-quality data about the whole network.

4.2. Effectiveness

This section explores evidence on the effectiveness of different controls. It is organised according to high-level areas, such that the areas for which the clearest evidence is available are presented first.

4.2.1. Attack surface management

The organisation’s attack surface was the strongest explanatory variable when it comes to predicting cyber risk outcomes. The corresponding control is attack surface management, which captures a range of specific steps such as system hardening (Marsh 2023), closing ports (Nagle, Ransbotham, and Westerman 2017), complexity management (IBM 2023) and hiding version information (Tajalizadehkhoob et al. 2017). However, not all attack surface can be reduced without a corresponding contraction in business activity.

Irreducible attack surface derives from the organisation’s size and is arguably a fact of life. For example, Tajalizadehkhoob et al. (2017) found that the number of IPs and domains managed by shared hosting providers explained 71 per cent of the variance in network abuse (e.g. phishing domains managed by the provider), meanwhile adding tens of security indicators to their model only explained an additional 3 per cent of the variance. Similarly, GallagherRe (2022) found that revenue was the most important predictor of claims likelihood. These aspects of exposure cannot be meaningfully reduced without reducing business activity.

Reducible attack surface can be addressed via technical steps that do not necessarily reduce business activity, although they may create friction. For example, Marsh (2023) found that organisations that employ ‘hardening techniques’ were 5.58 times less likely to suffer a cyber insurance claim. The magnitude of this effect size should not be understated, given that the next highest effect size in their study was 2.92. A related approach is reducing system complexity; system complexity caused the highest increase in incident costs (IBM 2023).

The academic studies identified the link between system compromise and external network configuration. Nagle, Ransbotham, and Westerman (2017) found that the number of open ports on a Fortune500 network was a statistically significant predictor of multiple indicators of compromise in a longitudinal study. Similarly, organisations offering ‘risky services’ were more likely to suffer a botnet infection (Edwards, Jacobs, and Forrest 2019). In summary, an organisation’s attack surface is the strongest predictor of cyber incidents.

4.2.2. Patch cadence

The speed at which security patches are applied was reliably one of the most important predictors of security outcomes. GallagherRe (2022) found that patch cadence was the most predictive technographic variable in their study. BitSight and Marsh (2022) found that ‘Patching Cadence’ was the second strongest predictor of cyber claims from November 2018 to June 2021. The only variable that was more predictive was the scanning company’s high-level cybersecurity score, which incorporates patch cadence. Patch cadence being the second most predictive variable was replicated using scan data from another proprietary provider (SecurityScorecard and Marsh 2022). Along the same lines, Coalition (2023) found that policyholders with one unresolved critical vulnerability of any kind were 33 per cent more likely to file a claim. Marsh (2023) found that organisations that patch the high severity vulnerabilities within seven days were less likely to file a claim.

In order for a patch to be made available, the software must still be supported by the vendor. Software for which this is not the case is known as ‘End of Life’. Coalition (2023) found that organisations deploying End of Life software were 3.69 times more likely to suffer a claim, which had the highest odds ratio of all the variables tested in their study. This was also a variable in the model introduced by Gandal et al. (2023), although they did not report the individual effect size.

4.2.3. Multi-factor authentication (MFA)

Results on the efficacy of MFA differed in terms of whether the unit of analysis was an account or an organisation. Meyer et al. (2023) studied Microsoft Azure Active Directory accounts and found that MFA reduced the risk of compromise by 99 per cent (Meyer et al. 2023). Further, MFA via a dedicated app was shown to be more effective than SMS-based MFA. A similar study of Google accounts found that device-based MFA ‘prevents 94% of attacks rooted in phishing and 76% of targeted attacks’ (Doerfler et al. 2019).

Curiously, the efficacy in protecting individual email accounts did not translate into efficacy when Marsh explored the security of organisations (as opposed to user accounts). Marsh (2023) tested eleven security controls and implementing MFA was associated with the lowest reduction in claims likelihood. It remained the least effective, even after the authors grouped three sub-questions that concern the specifics of how MFA is implemented – typically grouping variables provides more predictive power. This finding could result from the univariate statistical test, which is vulnerable to spurious findings.

Highlighting the potential for confounding variables, Arete and Cyentia (2022) found that organisations that implement MFA were less likely to pay ransoms (just 69 per cent did) than organisations that did not implement MFA (81 per cent paid the ransom). This result is somewhat puzzling given that MFA has no obvious impact on the ability to recover and rebuild following a ransomware incident. One explanation is that adopting MFA is an indicator of a confounding variable, namely higher security maturity. This maturity also helps organisations to recover from ransomware incidents.

In weighing the evidence on MFA efficacy, all studies show that MFA reduces the frequency of either account compromise or insurance claims. However, the magnitude of this reduction varied across studies, which is worth further exploration. Meyer et al. (2023) find that MFA is a silver bullet preventing 99 per cent of attacks, yet Marsh (2023) finds that it is the least effective control. One explanation is that some efficacy is lost as MFA is rolled out as part of an organisation-wide policy, which further emphasises the importance of how controls are implemented and maintained. Another explanation is the difference in threat levels across the samples. For example, the Google data showed that SMS-based prompts are less effective against targeted attacks (Doerfler et al. 2019). It could be that Marsh, a global reinsurance broker, works with large organisations that face more sophisticated threat actors who have the resources to overcome MFA (e.g. SIM hijacking). This highlights how effectiveness should always be measured relative to a specific threat profile.

4.2.4 Monitoring

Monitoring activities typically cannot be observed based on network scans. For this reason, the only studies that explored the efficacy of monitoring relied on questionnaire data. Marsh (2023) found that ‘endpoint monitoring’ and ‘logging and monitoring’ were the third and fourth most effective controls when it came to reducing claims likelihood. Implementing another type of monitoring, Intrusion Detection and Prevention System, had middling efficacy. IBM (2023) include variables for four different kinds of monitoring. All are associated with a reduction in claims costs, with ‘AI, machine-learning-driven insights’ displaying the strongest effect size. These results support monitoring being an important part of cyber risk management, albeit less so than hardening and patch management. However, the evidence does not support endorsing a particular type of monitoring, let alone a specific product.

4.2.5. Cloud vs on-premises email servers

In contrast to the generic results on monitoring, two insurers provide specific findings on cloud migration. Organisations that run on-premises Windows exchange servers suffer a higher frequency of compromise than organisations that use cloud hosted mail servers. Coalition (2023) report that businesses with more than $100 million in revenue with on-premise Exchange are 260 per cent more likely to make a claim. This was confirmed by AtBay (2023) who show that firms running Microsoft Exchange have a claims frequency of 0.19 per cent compared to 0.14 per cent for Microsoft’s cloud email solution (Office365) and 0.07 per cent for Google’s cloud email solution. The baseline frequency was 0.12 per cent for all organisations (AtBay 2023). However, the positive impact of cloud migration is not completely clear. IBM (2023) found that cloud migration is associated with a greater cost of data breaches.

4.2.6. Network protocol configuration

Studies based on network scans link details about specific network protocols (e.g. TLS, SPF and DKIM) to security outcomes. When taken individually, these measurements provide limited predictive value (Tajalizadehkhoob et al. 2017). However, metrics that aggregate multiple indicators tend to explain a lot more variance. For example, BitSight (2023) found that their main score aggregating various indicators had the most predictive power. SecurityScorecard and Marsh (2022) found different results using their proprietary scores. In particular, network and DNS configuration were the fourth and fifth (of seven) most predictive variables, while a variable that aggregated indicators related to end-point device and software security had the most predictive power in their study (SecurityScorecard and Marsh 2022).

These findings support the idea that aggregating multiple indicators provides more predictive power, which was also used in academic studies with great predictive success. Unfortunately, this approach does not isolate the impact of specific configurations, which makes it difficult to derive normative recommendations. Perhaps the main takeaway is that security results from system administrators performing the unglamorous tasks that ensure services are securely configured and network infrastructure is up-to-date.

4.2.7. Identity and access management

Much like monitoring, it is difficult to collect data about this control via network scans. However, Marsh (2023) found that privileged access management was associated with the second highest reduction in claims frequency. IBM (2023) found a more moderate impact with ‘identity and access management’ reducing the average cost of a data breach, but only having the twelfth highest reduction. The description of this control is too high-level to make any strong recommendations, not least because it results from just two studies.

4.2.8. Virtual private networks

We found two strong but isolated statistical results about specific VPN providers. Coalition (2023) found that organisations with exposed Fortinet (a VPN provider) devices were three times as likely to file a claim. Similarly, BitSight (2023) found that organisations running PulseSecure Group (another VPN provider, acquired by Ivanti) software were 2.6 times more likely to suffer a ransomware incident. Notably, both studies report statistical results for specific VPN providers rather than the presence of a VPN, which means there is no support for generic recommendations about VPNs aside from ‘choose the right provider’.

4.2.9. Miscellaneous security controls

Thus far, we have reported on controls that were studied multiple times and that displayed high relative efficacy. However, the absence of evidence should be treated with caution, especially as some controls were never tested or tested just once or twice. We now report on the controls that were not ranked highly in terms of efficacy.

Marsh (2023) linked questionnaire responses to insurance claims, in such a way that higher numbers are associated with a greater reduction in claims frequency.¹ We have already discussed the impact of controls such as hardening (5.58), privileged access management (2.92), end-point monitoring (2.23) and MFA (1.44). In addition to these results, cybersecurity training (1.76) and email filtering (1.56) were associated with middling reductions in claims frequency. However, this study does not run any multivariate analysis, which makes it vulnerable to confounding variables.

IBM (2023) link 26 different variables to the cost of a data breach, many of which we have already discussed. Secure software engineering (the ‘DevSecOps approach’) was associated with the biggest reduction in incident costs, followed by ‘employee training’ and ‘IR plan and testing’. IBM (2023) explored novel variables related to governance, which cannot be captured by network scans. Organisational decisions like having an IR team, board-level oversight, and appointing a CISO were all associated with lower incident costs with relative rankings of 5th, 16th and 18th respectively.

Turning to specific incident response procedures, Arete and Cyentia (2022) found no statistical difference between the likelihood of paying a ransom between organisations who implement backups and those who do not. However, they do find a somewhat tautological result that firms with the ‘Ability to recover’ pay less often (65 per cent versus 84 per cent of incidents) and pay a lower share of the initial ransom demand (38 per cent versus 54 per cent) when compared to firms who do not have that ability.

Gandal et al. (2023) linked questionnaire responses on security controls to self- reported outcomes in terms of cyberattacks suffered in the last year. The authors do not report the effect sizes of individual controls. However, they do report on the effect of organisations implementing six ‘easy-to-implement’ controls that were shown to reduce victimisation from 80 per cent to 42 per cent. Those controls were: (1) a strong password policy; (2) keeping systems up-to-date as per manufacturer’s recommendation; (3) two-factor authentication; (4) end-point detection and response; (5) not using End-of-Life software; and (6) encrypting data, files and email.

4.3. Summary

The evidence suggests hardening and applying security patches are the most effective interventions. MFA, monitoring and identity management were also shown to be associated with lower incident frequency. The magnitude of the reduction due to MFA was puzzling – it is highly effective at protecting individual email accounts (Doerfler et al. 2019; Meyer et al. 2023) but this effect is less pronounced when MFA is rolled out across an entire organisation (Marsh 2023).

Insurers provide more actionable evidence. Two studies showed that migrating email to the cloud was associated with reduced attack frequency (AtBay 2023; Coalition 2023). There was also evidence that adopting VPN services provided by certain vendors (Fortinet and PulseSecure) was associated with a higher risk of compromise (BitSight 2023; Coalition 2023).

The evidence was limited regarding other areas of cybersecurity. Network configuration has some predictive power, particularly when aggregated into a single score. However, it is hard to isolate individual configuration choices that play a significant role. Turning to security controls, the picture that emerges is that controls such as password policy, access control, end point detection and encryption reduce the likelihood of an incident, although the size of the reduction is moderate. These controls were rarely studied because these controls are hard to detect via network scans.

The high-level takeaway is that how a control is implemented seems to matter more than whether it is implemented. Taking the two most predictive variables as an example, patch cadence captures how responsive administrators are in applying patches, and attack surface reduction involves efficiently managing the configuration of network infrastructure and end-user devices. Similarly, MFA is highly effective at protecting inboxes, but efficacy is lost when it is rolled out across an organisation, possibly due to inconsistent roll-out. This suggests security efficacy results from the specifics of configuring and maintaining systems, not the high-level decisions such as whether to adopt an off-the-shelf solution.

5. Discussion

This section reflects on the outlook for evidence-based cybersecurity regulation. Section 5.1 discusses how the available evidence could be used by policymakers. Section 5.2 identifies limitations in our meta-review. Section 5.3 describes policy measures that might improve the evidence base in the future.

5.1. Evidence-based cybersecurity policy

To ask how policymakers should use the evidence, it is useful to separately consider guidance on IT infrastructure, InfoSec products and security processes. The first consideration, IT infrastructure, follows CISA’s concept of Security by Design, in which technology is built to be secure without reliance on bolt-on security solutions. The second, InfoSec products, involves choosing the right off-the-shelf security solution, roughly what is provided by the cybersecurity industry. The third, security processes, involves implementing processes that use ‘security products effectively’ (Schneier 2000). These are non-exclusive routes to improving security.

Our review provides the least supporting evidence for policy guiding organisations towards specific InfoSec products. Put more simply, we found no evidence of a ‘silver bullet’ solution, not even MFA. In contrast, a successful example of evaluating risk mitigation products can be seen in medical science during the Covid pandemic. The world compared efficacy (97 per cent vs 91 per cent risk reduction) across different vaccines, with some estimates tailored to the specific strain of the virus.² Yet in cybersecurity it was rare to find estimates that could compare classes of security products, let alone studies that could isolate the efficacy of specific products. For this reason, there is no simple recommendation such as ‘organizations should be required to implement control X, Y and Z’.

There was evidence that would support guiding organisations towards more secure IT infrastructure, although it was only available for a handful of technology solutions. Specific VPN providers were associated with significantly higher rates of compromise and specific cloud services were associated with lower rates of compromise. Unfortunately, it is unclear how policy would actually take advantage of this given that the problem of assigning causality makes it difficult to definitively conclude that higher rates of compromise are caused by insecure software. This uncertainty makes it inappropriate for a central authority to advise against or endorse a specific product. For this reason, the problem may be better addressed by a software liability regime, in which a judge or regulator establishes clear rules for when a software vendor should be held liable for releasing insecure software (Ryan and Heckman 2003).

The most supporting evidence is available for policy measures that guide organisations towards following security processes such as patch and attack surface management, the two interventions with the most predictive power over risk outcomes. Patch management provides an interesting case study as there were significant policy measures that supported this process, which could well partly explain the efficacy of patch management.

The naive policy approach would have been to require firms to ‘patch all vulnerabilities within X days’ (Cormack and Leverett 2023). While this makes intuitive sense, it gets more complicated when considering technical details. A blanket mandate would have led firms to waste resources given that 95 per cent of discovered vulnerabilities are never exploited (Jacobs et al. 2021). Even the more nuanced mandate ‘patch severe vulnerabilities within X days’ would have led to inefficiencies because the prevailing proxy for severity was flawed (Spring et al. 2021).

Instead, various US Government entities invested in the information infrastructure that underpins vulnerability management. The CVE initiative that catalogues vulnerabilities is sponsored by the Department of Homeland Security (DHS) and the Cybersecurity and Infrastructure Security Agency (CISA). Similarly, first DARPA and then later DHS fund the CERT Coordination Center (CERT/CC), which supports vendors in responding to vulnerabilities and distributing patches. Notably, CERT/CC did not create dependency on government programmes as many large tech vendors went on to establish their own CERTs after working with CERT/CC (Sridhar et al. 2021). More recently, CISA launched the Known Exploited Vulnerabilities (KEV) Catalog, which helps organisations focus on the most important vulnerabilities. Via these three measures, policy directly supported the information structure that underpins patch management, which went on to become the second most effective control.

The success here could be a model for other policy measures. For network hardening, it would involve sharing information about specific configurations that were associated with past attacks. For example, Edwards, Jacobs, and Forrest (2019) show that enabling peer-to-peer file-sharing is associated with botnet compromise. A provocative question is whether there is scope for an institution (much like the National Vulnerability Database) that tracks configurations that are directly associated with increased rates of compromise. Here the goal would not be to track a binary variable (vulnerable vs not), but instead an ordinal variable (the most risky, second most risky and so on). An obvious candidate for a risky configuration is enabling Remote Desktop Protocol on internet-accessible devices, given this protocol is beloved by ransomware gangs. Communicating this kind of information would support system administrators in deciding which software and protocols end-users are allowed to install.

5.2. Limitations

Interpreting the evidence is challenging because of the diversity of data collection approaches, variables and statistical tests. We tried to distil findings that were consistent, but a number of limitations remain.

5.2.1. Comparing interventions

It is challenging to compare interventions across studies because they are defined differently in each study. To see this, consider that although reducing attack surface was the most effective intervention, the actual interventions differed across studies. Marsh (2023) asked a high-level question about whether firms implement ‘hardening techniques’, while the academic studies explored more specific steps such as closing ports (Nagle, Ransbotham, and Westerman 2017) or disabling P2P file-sharing (DeKoven et al. 2019). This creates uncertainty about which specific hardening steps organisations should take to follow the scientific evidence.

Another problem is that a given control may be implemented differently across organisations. For example, multi-factor authentication can be implemented with low friction if it is only required for a narrow set of purposes (e.g. to access network admin accounts), and authentication cookies have no expiry date. More secure configurations bring more friction, such as MFA being requested from more users and for many services and authentication cookies expiring on a daily basis. Variability in how controls are implemented makes evaluating effectiveness difficult, and may explain why MFA had lower than expected efficacy when rolled out across an organisation (Marsh 2023).

This variability can be contrasted against rigid and repeatable studies of vaccine effectiveness in which the interventions are manufactured in a consistent way, measured so that participants receive a consistent dose, and administered by a doctor with a uniform and reliable instrument (a needle).

5.2.2. Causality

Few of the reviewed studies were designed to test whether interventions caused better outcomes, which meant at best the studies established whether interventions were correlated with better outcomes. The most common statistical design was to calculate odds-ratios, which compare the likelihood of incidents between two populations, those who implement a control and those who do not. A causal interpretation of this evidence is potentially erroneous because the two populations may differ in more ways than simply whether they implement the control. For example, financial firms face a high rate of victimisation despite investing in the top controls. This is not because the controls do not work, but instead because financial firms tend to be more targeted by threat actors.

Nevertheless, the causation or correlation line of thinking can be taken too far. None of these studies provide watertight evidence, such as by conducting a randomised control trial, the gold standard in medicine. However, correlational evidence supported by technical understanding is a reliable way forward (Spring, Moore, and Pym 2017). Not only do reducing attack surface and patch cadence consistently show up as among the most effective interventions, but this supports reasoning from first principles via technical knowledge.

5.2.3. Searchlight effect

The studies in our sample do not necessarily focus on the most effective interventions. Instead, opportunistic researchers have studied the interventions that can be easily measured. As a result, many of the statistical results relate to how externally-facing infrastructure is configured, whereas few studies explored the impact of internal controls that are harder to detect without the cooperation of the firm under study. This creates a disconnect between the volume of evidence and the effectiveness of the control. For this reason, we recommend against reading too much into the absence of findings about a certain control. However, it is worth noting that externally-facing infrastructure is exactly what attackers see.

5.2.4. Historical validity

Better data-collection and statistical design cannot solve the question of whether the past resembles the future. For example, the advent of generative artificial intelligence allows threat actors to create realistic audio and visual content that impersonates corporate leaders (Chesney and Citron 2019). Yet none of the controls in our meta-review would mitigate this problem because deep fakes do not rely on compromising a firm’s network security. Instead firms need to be able to prove which corporate communications are authentic as part of a public relations response, which may involve watermarking technology. While there is no evidence that deep fakes represent a major risk driver at present, the issue highlights the problem of using historical studies to establish risk posture as adversaries change behaviour. Nevertheless, this limitation should not be overstated given that top controls such as patching and attack surface management have been around for decades.

5.2.5. Knowledge creation by for-profits

This meta-review includes evidence collected and curated by private sector firms, which raises questions about the implications on validity. In terms of how the findings were published, none of the industry studies are peer-reviewed in the conventional academic sense. Internal reviews are likely to focus on legal and reputational risk rather than scientific validity, although the validity could have implications for reputation. Further, it is difficult to evaluate methodology given that details are often omitted.

Another concern relates to why these studies were published. In some cases, the vendor evaluates their own product or service, such as the MFA studies using Microsoft and Google data, and this creates a publication bias – we are unlikely to see a study that found that the vendors’ own products are ineffective. In other cases, the bias is less clear. The bulk of the evidence was created by the insurance industry, but it did not probe the efficacy of insurance. Instead insurers found that certain VPN/cloud service providers were associated with higher/lower rates of compromise. In fact, publication may even create legal risk, such as if vendors sue for defamation related to assertions that their products are insecure.

Clearly, we should be concerned by potential biases associated with industry data. These potential biases make it surprising that the meta-review does not support specific products that can be purchased but instead processes that are hard to commercialise. The most effective steps can be taken by an IT team who follow publicly available security advisories and hardening guidelines. This outcome would be analogous to healthcare industry studies that recommend fresh air, exercise and a balanced diet rather than expensive medical treatments – that increases the authors’ confidence given the commercial interests at play.

5.3. Improving the science of security

A long-term goal would be to improve the evidence on efficacy, which could shape future policy formulation. Some of the problems are outside the purview of policy measures, such as the choice of statistical model. Data availability could, however, be improved via policy measures.

Academic studies have used data from two such initiatives. First, passing breach notification laws created the data set of breached firms that many academic studies rely upon (Woods and Böhme 2021). Second, commissioning national surveys enabled studies of firms in Israel (Gandal et al. 2023), Italy (Biancotti 2018), and the Netherlands (Dinkovay, El-Dardiryy, and Overvesty 2020). Such studies typically ask firms to self-report on which controls were in place and whether incidents occurred.

Broadly speaking, the authors believe policymakers need only ensure there is public data about variables of interest, and then academics hungry for data will do the rest. One potential direction is to create a public database with details of compromises related to ransomware victims, which would enable studies exploring which configurations and controls reduce ransomware frequency.

An academic created one such database by collecting press reports of ransomware victims. Open-sourcing this database spawned multiple studies (Jensen and Paine 2023; Rege and Bleiman 2020). A database curated by public agencies with greater visibility would lead to higher quality studies, given the limitations of publicly reported data (Laube and Böhme 2016). The US Cyber Incident Reporting for Critical Infrastructure Act of 2022 presents an opportunity to collect, sanitise, codify and make such information public. However, transparency must be balanced against the risk of disclosing individual identifiers that might result in reputational risk and further victimisation.

Turning away from academia, our meta-review shows the cyber insurance industry is beginning to produce statistical evidence. This occurred without policymakers mandating the sharing or creation of a public repository of claims data, both of which were early policy proposals (Woods and Simpson 2017). Insurance brokers such as Marsh and GallagherRe produced most of the studies (see Table 1). Brokers can collect claims across all the insurers they place business with, which means they essentially operate a repository by pooling claims data across insurers. The other insurance studies were produced by InsurTech startups (AtBay 2023; Coalition 2023). Both firms are less than eight years old, which shows historical data is not necessary to produce evidence. One interpretation is that the problem is not the availability of incident or claims data as was assumed in early policy discussions (Woods and Simpson 2017). Rather, the problem was that conventional insurers lacked expertise on how to collect data about security controls, or perhaps they were overly protective of the resulting insights.

6. Conclusion

In pursuit of evidence-based cybersecurity policy, our paper reviewed 18 studies exploring the link between security controls and cyber risk outcomes. We found little evidence regarding the efficacy of off-the-shelf security solutions, such as specific firewalls or antivirus products. Instead the evidence suggested that the most effective security interventions concern system configuration and maintenance. In particular, an organisation’s attack surface is the strongest predictor of cyber incidents. Attack surface can be reduced by a range of hardening measures. Patch cadence was the second strongest predictors of cyber incidents.

Translating this into policy-relevant conclusions, we found no evidence that would support mandating specific security controls, certainly no silver bullets. There is evidence that outcomes are what matter, such as high patch cadence and reduced attack surface, although we do not know the most effective way to get there. Finally, there is strong evidence that certain cloud vendors are associated with lower rates of cyber incidents, and specific VPN providers are associated with much higher rates – 300 per cent in one study. This suggests that secure-by-design is an important route to security, more so than bolt-on security solutions.

In addition to using mandates and/or nudges to guide organisations towards secure processes and outcomes, policy should improve the information available for organisations in hardening systems and managing security updates. Successful government-sponsored initiatives can be seen in CISA’s Known Exploited Vulnerabilities Catalog, the US National Vulnerability Database and CERT/CC. All three programmes help organisations to make risk-based decisions regarding patch management, which helped it become the second most predictive intervention in our meta-review. Exploring similar initiatives for attack surface management represents a promising direction.

There is also a role for governments in ensuring that evidence is available to formulate policy in the future. Policy measures that create public data about cybersecurity incidents will generate academic scholarship exploring the statistical relationships with controls and externally observable infrastructure. Thus, the main problem to solve is data availability. Solutions include commissioning surveys and broadening the public reporting regime. Finally, a positive finding is that insurance brokers and InsurTech startups have started publishing evidence based on cyber insurance claims data, a long-standing policy goal.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Notes

1 Concretely, the number is ‘the ratio of the conditional probability of a claim given a “no” response to the conditional probability given a “yes” response’ (Marsh 2023).

2 https://www.healthdata.org/research-analysis/diseases-injuries/covid/covid-19-vaccine-efficacy-summary