Does Implementation of Biomathematical Models Mitigate Fatigue and Fatigue-related Risks in Emergency Medical Services Operations? A Systematic Review

Abstract

Background: Work schedules like those of Emergency Medical Services (EMS) personnel have been associated with increased risk of fatigue-related impairment. Biomathematical modeling is a means of objectively estimating the potential impacts of fatigue on performance, which may be used in the mitigation of fatigue-related safety risks. In the context of EMS operations, our objective was to assess the evidence in the literature regarding the effectiveness of using biomathematical models to help mitigate fatigue and fatigue-related risks. Methods: A systematic review of the evidence evaluating the use of biomathematical models to manage fatigue in EMS personnel or similar shift workers was performed. Procedures proposed by the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) methodology were used to summarize and rate the certainty in the evidence. Potential bias attached to retained studies was documented using the Cochrane Collaboration's Risk of Bias tool for experimental studies. Results: The literature search strategy, which focused on both EMS personnel and non-EMS shift workers, yielded n = 2,777 unique records. One paper, which investigated non-EMS shift workers, met inclusion criteria. As part of a larger effort, managers and dispatchers of a trucking operation were provided with monthly biomathematical model analyses of predicted fatigue in the driver workforce, and educated on how they could reduce predicted fatigue by means of schedule adjustments. The intervention showed a significant reduction in the number and cost of vehicular accidents during the period in which biomathematical modeling was used. The overall GRADE assessment of evidence quality was very low due to risk of bias, indirectness, imprecision, and publication bias. Conclusions: This systematic review identified no studies that investigated the impact of biomathematical models in EMS operations. Findings from one study of non-EMS shift workers were favorable toward use of biomathematical models as a fatigue mitigation scheduling aid, albeit with very low quality of evidence pertaining to EMS operations. We propose three focus areas of research priorities that, if addressed, could help better elucidate the utility and impact of biomathematical models as a fatigue-mitigation tool in the EMS environment.

Key words::

• fatigue risk management
• biomathematical models of fatigue and performance
• shift work
• safety

Background

Fatigue-related cognitive impairment results from the interaction of multiple factors including sleep history, time awake, and time of day (1, 2). Studies of diverse shift worker groups show that work scheduling practices can create conditions that exacerbate the risk of fatigue-related cognitive impairment (3). The work schedules of Emergency Medical Services (EMS) personnel are notable for 12-hour and 24-hour shifts (4), which increases the risk of fatigue-related impairment and occupational injuries (5).

Biomathematical modeling is a tool for fatigue risk management in operations that involve shift workers who are at risk of negative safety outcomes. These models are widely used in aviation, rail, maritime, and other high-risk industries to estimate the risk of being impaired by fatigue (6). While a number of different biomathematical models of fatigue have been developed, they all generally take into account the relationships between factors that contribute to fatigue, including sleep history, time awake, and time of day (7). These models produce predictions of alertness, performance, or risk of impairment for given work/rest or wake/sleep schedules. They are therefore useful when making decisions regarding shift scheduling, including the duration and timing of duty periods and rest opportunities.

At their core, biomathematical fatigue models make use of equations capturing the temporal regulation of sleep and sleepiness (8, 9) and perform mathematical calculations on the established relationships between wake/sleep or work/rest schedules and two neurobiological processes that drive fatigue (7). One of these processes is the homeostatic process, which (through brain mechanisms that are yet to be fully elucidated) tracks sleep history and current time awake. The homeostatic process causes sleepiness when sleep and wake amounts are out of balance due to relatively excessive wakefulness (8). The other process is referred to as the circadian process, which (through the biological clock in the suprachiasmatic nuclei of the hypothalamus) drives wakefulness during the day and sleep at night (10). The interaction of the homeostatic and circadian processes, through the effects of sleep history and time awake on the one hand and time of day on the other hand, produces a net level of fatigue (11), which fluctuates over time yet manifests prominently during nighttime wakefulness and after sleep deprivation (9, 12). A key feature of biomathematical models is their ability to provide quantitative predictions of the relative risk of fatigue across hours and days of given wake/sleep or work/rest schedules (7). This is particularly important given that individuals' own ability to estimate their fatigue-related risk is generally poor (13, 14).

High-risk industries have used biomathematical modeling for over 30 years to predict fatigue risk (6, 15). The development of these models was accelerated in the 1990s when the U.S. Army and other groups around the globe increased use of modeling to simulate and investigate changes in human behavior under different sleep conditions (15). While different biomathematical models were developed and tested, all were based on the same fundamental concepts of the homeostatic and circadian processes and produced nearly identical predictions of fatigue risk (2, 16). Early versions of biomathematical models did not account for the cumulative build-up of fatigue across consecutive days of partial sleep restriction, which is a phenomenon that was discovered in the early 2000s (14, 17). Recent research addressed this issue (15, 18, 19), and modern-day biomathematical models can predict fatigue risk for a wide variety of wake/sleep and work/rest schedules (7). Current versions of these models provide a means of objectively estimating the potential impacts of fatigue on performance and safety, which can be used to help mitigate fatigue-related risks (20).

In shift work settings, the utility of biomathematical fatigue models is realized when making decisions about shift schedules (21). They are often used to adjust planned work schedules to reduce exposure to fatigue risk, especially in 24/7 operations (2, 6). In U.S. military applications and space flight, biomathematical modeling is used as a part of the mission planning process (22). The Federal Railroad Administration (FRA) mandates that modeling is employed in mitigating fatigue risk in passenger rail personnel schedules (23). The Federal Aviation Administration (FAA) recently employed biomathematical modeling in the development of updated duty and flight time regulations, and allows carriers to use biomathematical modeling as a key ingredient in proposals for “alternative methods of compliance” (i.e., operations that are outside the scope of the standard hours of service regulations but are demonstrably safer) (24). Commercial airlines in the U.S. routinely include assessments of fatigue risk produced from biomathematical modeling as part of trip schedule planning in operations (25).

The extent to which biomathematical models have been applied, tested, and evaluated in the EMS setting is unknown. Given significant growth and successful application of biomathematical modeling as part of fatigue risk management in safety-critical operations, a review of the evidence testing biomathematical models with EMS personnel or other shift workers is warranted. We sought to systematically review the evidence for the effectiveness of biomathematical models for fatigue mitigation. Our review was guided by the research question developed a priori by a panel of experts assembled to address fatigue risk management in EMS: “In EMS personnel or similar worker groups, does implementation of model-based fatigue risk management mitigate fatigue, fatigue-related risks, and/or improve sleep?” (PROSPERO 2016: CRD42016040112) (26).

Methods

We used a systematic review study design of 5 databases and one website: PubMed/Medline, the Cumulative Index to Nursing and Allied Health Literature (CINAHL), Scopus, PsycINFO, the Published International Literature on Traumatic Stress (PILOTS), and the publications section of the National Institute of Justice (NIJ) website. The details of our methodology, study protocol, and procedures for reviewing published and unpublished literature are described in a separate publication (27). In this paper, we describe the components of our protocol unique to this systematic review.

Study Design

We assessed publications that described use of randomized controlled trials, quasi-experimental studies (i.e., before and after designs) (28) and observational study designs. Publications that describe the use of modeling but include no intervention or quantifiable outcome were not considered to have met the inclusion criteria for this review.

Types of Participants

The definition of our target population was developed by a panel of experts and is inclusive of diverse shift worker groups: “EMS personnel or similar worker groups, defined as shift workers whose job activity requires multiple episodes of intense concentration and attention to detail per shift, with serious adverse consequences potentially resulting from lapses in concentration” (26). We excluded literature that involved non-shift worker “healthy volunteers” and other non-shift worker populations.

Types of Interventions

We retained studies that reported tests or evaluations of the effectiveness of a biomathematical model in the operational setting to address fatigue and fatigue-related risks. We excluded research if the aim of the study was to calibrate the biomathematical model rather than test the impact of the model on operational outcomes such as personnel performance or safety or patient safety.

Types of Outcome Measures

The outcomes of interest for our research question were selected a priori by a panel of experts prior to the execution of the search for literature (26). The panel reached agreement on seven outcome categories: patient safety, personnel safety, personnel performance, acute fatigue, sleep and/or sleep quality, indicators of long term health, and cost to the system. Outcome selection was guided by procedures proposed by the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) framework (27, 29).

Search Methods for Studies

A research librarian (PMW) performed searches of five bibliographic database products and one website. The methods of our search strategies are published in a separate publication (27). In that paper, we identify all sources searched, the search terms incorporated, and the description of search vocabulary. For the systematic review described here, the search incorporated multiple terms covering each of three concepts: emergency medical services and other critical shift-based occupations; fatigue, sleep, and sleep disorders; and models of multiple types and simulations. All searches included literature from January 1980 to September 2016. See Online Supplemental Material for search strategy details specific to this systematic review.

Data Collection and Selection of Studies

Screening

Two investigators (FOJ and LBW) independently screened titles and abstracts to identify potentially relevant publications. Three investigators (FOJ, LBW, and PDP) adjudicated disagreements based on the following inclusion criteria: a) the study describes the population of interest; b) the study describes use of a biomathematical model as the primary intervention of interest; and c) the title and/or abstract describes one or more outcomes of interest. The Kappa statistic was used to determine inter-rater agreement.

Full-Text Review

Two investigators (FOJ and LBW) worked independently to abstract key information from full-text articles. The abstracted information included study design, participant characteristics, intervention characteristics, comparisons, outcome measures, and key findings. The study's principal investigator (PDP) verified abstractions, and disagreements were handled with discussion. We systematically excluded non-peer reviewed literature including book chapters, conference abstracts, newsletters, and similar publications, dissertations and theses. Two investigators (FOJ and LBW) searched bibliographies to identify potentially relevant research.

Risk of Bias Assessment

Two investigators (FOJ and LBW) documented bias of the retained research across six domains with the Cochrane Collaboration's risk of bias tool for experimental studies (29). The Cochrane risk of bias tool outlines the assessment of bias across six domains: 1) selection bias (i.e., random sequence generation and allocation concealment); 2) performance bias (i.e., blinding of participants and personnel); 3) detection bias (i.e., blinding of outcome assessment); 4) attrition bias (i.e., incomplete outcome data); 5) reporting bias (i.e., selective reporting); and 6) other bias (i.e., other sources of bias not addressed in other domains). We addressed any disagreements with bias assessment by using discussion and consensus.

Statistical Analysis

Three investigators (FOJ, LBW, and PDP) used a system for categorizing the findings from retained research as favorable, unfavorable, mixed/inconclusive, or no impact (30). A description of this categorization methodology is published separately (27). Favorable was assigned when findings favored the intervention (biomathematical modeling). Unfavorable was assigned when findings did not favor the intervention. Mixed/Inconclusive was assigned when findings showed both positive and negative impacts on multiple components of an outcome or when the results reported were insufficient to determine the impact on an outcome (such as lack of adequate description of statistical analyses). No impact was assigned when authors determined the intervention showed no statistical and/or clinically meaningful impact on outcomes.

Quality of Evidence

Three investigators (FOJ, LBW, and PDP) used the GRADE framework to summarize and rate the quality of retained research (evidence) into a standardized evidence profile table (31). The evidence profile table presents the following information: number of studies per outcome; judgments about underlying quality of evidence (e.g., risk of bias, indirectness); statistical results; and a quality rating from very low, low, moderate, to high.

Reporting

We present the findings from this systematic review in accordance with the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement (32).

Results

The search strategy yielded n = 2,777 unique records (Figure 1). Two investigators (FOJ and LBW) independently screened the n = 2,777 titles and abstracts. The interrater agreement for inclusion/exclusion of titles and abstracts was substantial (Kappa = 0.80). Fifty-three records were judged potentially eligible based on title and abstract alone. Fifty-four records were reviewed following identification of one manuscript identified during bibliography searches as potentially relevant. We identified one paper, which involved non-EMS shift workers, and provided limited reporting on outcomes of interest to our systematic review (Table 1) (33). Key findings were abstracted and are reported in the Online Supplemental Material. Fifty-three records were excluded (with 3 duplicates detected), and reasons for exclusion organized in the Population, Intervention, Comparison, Outcome (PICO) format (See Online Supplemental Material).

Figure 1. PRISMA Flow Diagram for PICO#6 PROSPERO 2016:CRD42016040112.

Given that only one study met inclusion criteria (33), we provide an overview of this study and highlight key findings germane to our outcomes of interest. Moore-Ede et al. (33) briefly described a quasi-experimental study design (before/after design) (28) that used a biomathematical model called the Circadian Alertness Simulator (CAS) in an intervention for a commercial trucking organization. This evaluation was included in a larger investigation of the validity and applicability of the CAS model. In the intervention, truck dispatchers used the model by performing monthly analyses of model-derived fatigue estimates for every driver. The dispatchers used the model's output as a basis for making adjustments to the drivers' schedules in order to reduce predicted fatigue in day-to-day operations. This included adjustments of the start and end times of work for individual drivers, providing rest breaks which allowed two consecutive nights of sleep, minimizing night work, avoiding rapid rotations in the starting time of work, and reducing the number of consecutive shifts worked. Findings relevant to two of our seven outcomes of interest were reported: personnel safety and costs to the system.

Regarding the costs-related outcome, the study's authors reported a comparison of accident rates in the (3-year) period prior to the intervention with accident rates occurring during the intervention year. Findings show a reduction in the frequency and cost of truck accidents, as well as a reduction in CAS fatigue scores. Regarding the personnel safety-related outcome, the authors reported that truck accidents decreased 23.3% from 2.30 per million miles traveled during the 3-year period pre-intervention to 1.76 million miles during the year the intervention was implemented. Severe accidents (defined as those costing over $20,000) decreased by 55% from 0.20 per million miles prior to the intervention to 0.09 per million miles during the intervention. The mean cost per truck accident decreased 65.8% from $14,088 (Standard Error of Mean [SEM] $4,307) during the 3-year pre-intervention period to $4,820 (SEM $1,437; t-test, p < 0.05) during the intervention year. The reported mean cost of severe accidents decreased by 66.7% from $152,384 (SEM $40,841) per accident pre-intervention to $50,809 (SEM $6,080) per accident during the intervention (t-test, p < 0.05).

We detected serious risks of bias with the Moore-Ede et al. (33) study due to a lack of experimental controls, no indication of concealment of intervention, no indication of blinding participants or investigators, indirectness with EMS personnel, and potential for selective reporting of available outcomes. Our assessment of bias for Moore-Ede et al. (33) appears in Online Supplemental Material. Our quality assessment of this study was very low based on the GRADE elements of risk of bias, inconsistency, indirectness, imprecision, and other considerations (Table 2) (31).

Table 1. Synthesis of individual studies that test implementation of biomathematical models to determine impact on outcomes selected as important to this systematic review

Experimental Study Designs
			Important Outcomes
Author, Year	RefID PMID	Study Design	Patient Safety^*	Personnel Safety	Personnel Performance^†	Acute Fatigue^‡	Sleep and Sleep Quality^§	Long-Term Health^║	Cost to System
Moore-Ede, 2004	RefID-1691 PMID-15018271	Quasi-experimental	—	Favorable	—	—	—	—	Favorable

Findings are classified using a system adapted from Bolster and Rourke (2015) where interpretation of findings was classified as favorable if after implementation of the model, improvements in outcomes were observed. Other categories include: unfavorable, mixed/inconclusive, or no impact.

^* Includes quality of care.

^† Includes external subjective ratings of the study subject's performance including perceived satisfaction with the subject's performance.

^‡ Includes acute states of fatigue, sleepiness, and alertness.

^§ Includes sleep latency, total sleep time, recovery, and related measures.

^║ General wellness or well-being measures included.

Table 2. GRADE evidence profile table

Quality assessment
№ of studies	Study design	Risk of bias	Inconsistency	Indirectness	Imprecision	Other considerations	Impact / Effect	Quality	Importance
Patient Safety
0								—	IMPORTANT
Personnel Safety
1	Experimental study design^*	Serious^†,^‡	Not serious	Serious^§	Serious^║	Relevant data are held private and unpublished.^#	The number of truck accidents decreased 23.3% from 2.30 per million miles traveled during a three-year pre-intervention period to 1.76 million miles during the intervention year. The number of severe accidents decreased by 55% from 0.20 per million miles prior to the intervention to 0.09 per million miles during the intervention.	⊕◯◯◯ VERY LOW	IMPORTANT
Personnel Performance
0								—	IMPORTANT
Sleep / Sleep Quality
0								—	IMPORTANT
Indicators of Long-Term Health
0								—	IMPORTANT
Cost to System
1	Experimental study design^*	Serious^†,^‡	Not serious	Serious^§	Serious^║	Relevant data are held private and unpublished.^#	The mean cost per truck accident decreased 65.8% from $14,088 (SEM $4,307) during a three-year pre-intervention period to $4,820 (SEM $1,437; t-test p < 0.05) during the one-year intervention. The mean cost of “severity accidents” (over $20,000 cost) decreased by 66.7% from $152,384 (SEM $40,841) per accident pre-intervention to $50,809 (SEM $6,080) per accident during the intervention period (t-test p < 0.05).	⊕◯◯◯ VERY LOW	IMPORTANT
Acute Fatigue
0								—	IMPORTANT

EXPLANATIONS

^* Quasi-experimental study design.

^† Experimental: Lack of blinding of participants and/or personnel.

^‡ Experimental: Lack of blinding of outcome assessment.

^§ Shift workers other than EMS personnel.

^║ Sample size: a) one organization; b) unknown number of individual participants.

^# Potential for selective outcome reporting.

Discussion

Summary of Main Results

The results of this systematic review identified but a single peer-reviewed paper germane to our PICO question and inclusion criteria (33). The paper was judged to have serious risk of bias linked to study design, execution, incomplete reporting, and indirectness with EMS personnel. Findings were limited, yet favorable toward use of biomathematical models as a fatigue mitigation strategy targeting personnel safety and cost to system outcomes. The overall assessment of evidence quality was very low.

Discussion of Systematic Review Findings

To the best of our knowledge, no systematic reviews have been published on the effectiveness of biomathematical modeling–based interventions in reducing the risk or impact of observed fatigue-related impairment. Most of the research describing biomathematical models has reported on the refinement and validation of model mathematics and estimates (34–36) and simulations to estimate fatigue risk across schedules (37, 38).

Several papers retrieved in our search were judged relevant to the PICO question, as they reported on use of biomathematical modeling as a tool in the approach to fatigue mitigation in operational settings (39, 40). After further review, these studies were not retained because the investigators did not report on an evaluation of the effectiveness of the intervention. For example, McCormick et al. (39) and Tvaryanas et al. (40) reported on the use of biomathematical modeling to objectively compare schedule designs. Biomathematical modeling of simulated typical orthopedic surgery schedule rotations and remotely piloted aircraft crew, respectively, was determined useful for the objective comparison of predicted fatigue risk associated with several schedule designs. However, the authors reported no data pertaining to the impact of implementation of model-based interventions on outcomes from the operational environment.

The results of our systematic review show that only one paper, which describes the select findings of one study met our criteria for inclusion. This one paper reported on a limited number of outcomes of interest to our PICO question as part of a larger effort to validate a specific biomathematical model. The relevant findings are linked to an intervention that assessed the impact of a biomathematical model-based intervention. For this reason, the paper was retained. However, the reported findings were limited.

Minus this one paper describing one study, our review would have been classified as an “empty review.” One of the most important purposes of systematic review methodology is to identify gaps in the scientific literature and to make certain that if important questions are unsupported by published and peer-reviewed science that this state of affairs is rigorously confirmed and that this knowledge is disseminated. The PICO question underlying this systematic review was developed by a panel of multidisciplinary stakeholders and content experts (26). The panel was aware that biomathematical models are in active use in several high-risk operational domains (e.g., the commercial aviation industry) as a means of informing and evaluating scheduling processes and mitigating the risk of fatigue (6, 23–25, 37, 41). It is an established practice in most industries that interventions in active use by multiple organizations undergo some degree of scientific evaluation, but in the area of biomathematical fatigue modeling the published evidence is largely unavailable. Evaluations of biomathematical models may be withheld from the public domain if the model application failed (i.e., biased reporting), for proprietary reasons (e.g., in industry) and security reasons (e.g., in the military), or because they may have yet to be analyzed.

Systematic reviews that fail to identify research to address a well-formulated PICO question, known as “empty reviews,” are commonly reported in the scientific literature (42). In fact, the Cochrane Database of Systematic Reviews contains a growing number of empty reviews with over 100 reported by 2010. They consider empty reviews to be a methodologically sound resource for the advancement of science. Empty systematic reviews serve an important role in clarifying the current state of science for a given question (43). They help to lay the framework for what future directions are required through research. Empty reviews can serve as a call to action and a framework for funding agencies that need to be certain that the research being commissioned or funded in a competitive process will make a unique contribution to our understanding. Practices in healthcare are often supported by widely held beliefs and assumptions related to the scientific evidence supporting it. Empty or minimally sourced systematic reviews in these domains provide a wake-up call that can challenge dogma and call potentially harmful or wasteful practices into question. While an expanded scope to the PICO question posed in this review may have allowed the inclusion of other biomathematical modeling papers, they would have all lacked that critical element of evaluation of effectiveness. That component was seminal to the spirit of the original PICO question that guided this systematic review and staying the course was essential to remain faithful to the needs of the panel stakeholders even if only one study was ultimately deemed to meet the established inclusion criteria.

Even though this systematic review on the use of biomathematical modeling for mitigation of schedule-related fatigue risk was near-empty, biomathematical modeling is currently employed in a number of regulated operations, including rail and aviation (6), with the list of adopters growing steadily. Despite the increasingly wide-spread use of biomathematical modeling in safety-sensitive, 24/7 operations, however, studies of their effectiveness in controlled pre-post investigations or other scientifically rigorous study designs have not been published. In industries already employing modeling as a fatigue risk management approach, data sets exist that may be informative about the effectiveness of biomathematical model use. Given that such data sets would also include potentially sensitive data on work hours and operational performance, they are often considered proprietary and critical to corporate or organizational strategies and may therefore not be available for analysis and publication. Likewise, modeling efforts conducted by regulators and accident investigators are usually reported directly to individual organizations, and the data contained in these reports are cleaned of all confidential material or marked classified, and thus remain unpublished. As such, it is plausible that biomathematical modeling for managing fatigue risk in high-risk industries is more widely used and more thoroughly characterized than what is depicted in the peer-reviewed literature.

Prospective use of Biomathematical Models of Fatigue in EMS

The main utility of biomathematical models of fatigue is in their ability to include the interaction of factors that contribute to physiological fatigue that would be difficult and inefficient—if not impossible—to fully consider otherwise. Objective metrics estimating predicted fatigue risk allow for the comparison of scheduling solutions, and biomathematical modeling tools may be useful in prospective and retroactive analyses of planned shift schedules (41). This would be analogous to the approach described by the International Civil Aviation Organization, where biomathematical modeling may be used for predictive and reactive hazard identification in duty and flight schedules as part of a comprehensive Fatigue Risk Management System (44). However, biomathematical modeling-based solutions for addressing fatigue risk in EMS operations will require consideration for industry-specific shift work schedules, sleep patterns, and obstacles to obtain sleep. For example, use of biomathematical models with EMS-specific data may help set boundaries around expected sleep times (e.g., to occur before/after duty times, or during known periods of interim release) (6, 45).

Calibration of sleep estimates and model predictions to the EMS environment is important, given that for many personnel, on-duty sleep opportunities may be driven by task load rather than scheduled break opportunities. Moreover, it is estimated that one half of EMS personnel work 24-hour shifts and that about one third of personnel work multiple EMS jobs (4). Probabilistic estimates of likely sleep times for EMS personnel on typical duty schedules may be used to refine sleep estimates used in model-based analyses (25). Where the biological basis of fatigue risk includes sleep loss, this is a critical consideration for EMS personnel.

We propose 3 focus areas as research priorities that if addressed, could help determine the utility and impact of biomathematical models as a fatigue-mitigation tool in the EMS environment.

Research Priority 1: Which specific outcomes, if any, of the EMS operational environment are predicted by biomathematical modeling estimates? Addressing this research question would help to validate the utility of the modeling approach in EMS operations. Biomathematical models can estimate the physiological likelihood of fatigue related impairment, yet this may not directly relate to negative outcomes in the EMS work environment (7, 41). Planned, natural, or existing safety barriers (e.g., safety alarms, teamwork) may catch many operational errors before they occur or have a measurable impact. The added benefit of biomathematical modeling, or the unique contribution of a model, should be explored. A prospectively designed study could capture reliable sleep data and the occurrence of adverse events. A study that uses a quasi-experimental design (e.g., a before/after study design) would be more feasible and possibly more attractive to the EMS administrator than designs that require randomization to 2 or more conditions (28).

Research Priority 2: How does EMS personnel sleep duration vary by duty period duration, start time, and operation type? Given the known contribution of sleep loss to the risk of fatigue-related impairment, sleep estimates are an essential aspect for any modeling-based analysis and prediction (46). Sleep estimates for EMS must take on-duty sleep into account, especially considering the prevalence of 24-hour duty periods. Data from actigraphy- and/or sleep diary-based, naturalistic studies can inform the development of customized probabilistic distribution of sleep times across EMS duty period types (47), which would improve model based analysis of available schedules.

Research Priority 3: What is the unique contribution of biomathematical modeling to improvements in personnel sleep and organizational outcomes like safety incidents? Fatigue risk mitigation requires a multi-faceted approach or strategy. Seven commonly promoted elements of fatigue mitigation systems include: 1) fatigue management policies; 2) fatigue management with data collection and instillation of controls/countermeasures; 3) a fatigue reporting system for employees; 4) a process for investigation into suspected fatigue-related events; 5) education and training for employees and management; 6) an employer-supported process to diagnose and manage sleep disorders; and 7) a process for internal and external auditing of the fatigue mitigation system/program (48). The combination of these elements may have a positive impact on safety and other outcomes of interest. Carefully designed studies should determine the unique contribution of biomathematical modeling as a key component of a fatigue mitigation system.

Limitations

The low yield in research related to our PICO question raises a number of questions regarding the limitations of our study. First, our results may be related to the PICO question that guided our literature review. Poorly constructed research questions raise the risk that a team will review literature irrelevant to the prime focus of the systematic review and potentially miss or exclude literature germane to the population, intervention, comparisons, and outcomes of interest. The PICO question that guided this study was carefully constructed and framed in the PICO format (26). The question was judged by a panel of experts to have a high degree of relevance and clarity (27). The panel included individuals with direct involvement in the development, testing, and deployment of biomathematical models. We also reviewed the PROSPERO database of existing and/or previously completed systematic reviews and detected no overlap with our question. For these reasons, we believe that our PICO question was well constructed for purposes of a systematic literature review.

Second, our results may be related to the databases searched. The population, intervention, comparison, and outcomes contained within our PICO question are not isolated to one profession or occupation. Our search was therefore inclusive of multiple databases that may contain literature reporting on use of biomathematical models in diverse populations and settings. The databases searched included PubMed/Medline, the Cumulative Index to Nursing and Allied Health Literature (CINAHL), Scopus, PsycINFO, the Published International Literature on Traumatic Stress (PILOTS), and the publications section of the National Institute of Justice (NIJ) website. We provide a transparent summary of our search strategy in a separate paper (27), and we provide a detail list of key words and search elements used for each database (see Online Supplemental Material). While a search of other databases and with other search terms may have yielded additional research, we believe the likelihood of discovering additional research that meets our inclusion criteria is very low.

Third, the screening process for this review began with a pooled total of 2,777 from all databases. Per protocol, two screeners with content knowledge and experience with testing and application of biomathematical models evaluated each record independently and delivered a judgment of include or exclude. Their inter-rater agreement was substantial (Kappa = 0.80), and exceeded the initial screening agreement reported in previous systematic reviews on similar topics (49, 50). We also conducted a detailed search of bibliographies of the publications reviewed in full; we searched the grey literature (non-peer-reviewed publications); and we queried experts in the field to validate our findings. While it is possible that research related to our PICO question may have been overlooked, we believe that based on our methodology that the possibility is low.

Finally, we recognize that other investigators conducting the same search may have excluded the one paper retained in this systematic review. The one paper's reported findings may be described as minimal or incomplete, which makes it difficult to provide a comprehensive assessment of the impact that the biomathematical model had on outcomes. We chose to retain this paper because it met our criteria for inclusion: 1) the paper described a shift worker population (our population of interest); 2) the paper described the application of a biomathematical model (our intervention of interest); 3) the paper compared findings pre- and post-introduction of the biomathematical model (our comparison of interest); and 4) the paper reported on outcomes of interest to our search. We retained this paper because it was published in a peer-reviewed journal. Our search of bibliographies, the grey (non-peer-reviewed) literature, and consultation with experts in the field resulted in no additional published papers. We recognize that for the reasons previously outlined, the results and conclusions reached in our paper may differ from what others could have concluded given the available literature.

Conclusions

In this systematic review, we considered the scientific literature on biomathematical modeling as an intervention designed to mitigate the risk of shift worker fatigue-related impairment and/or to mitigate the likelihood of negative operational outcomes secondary to fatigue-related cognitive impairment. We discovered only one published study of non-EMS shift workers that met our inclusion criteria. While not very fruitful from the perspective of the evidence it yielded, this systematic review demonstrates convincingly the need for peer-reviewed and publicly available research in an area that has seen widespread commercial interest and investment. This is an important finding. For example, government agencies often expect systematic reviews to be completed on a topic prior to funding large-scale and expensive trials as a means of ensuring that the proposed research is relevant and not redundant. Despite very limited published evidence in real-world applications, biomathematical modeling may be a promising tool for fatigue mitigation (21) with considerable public safety and health implications. It is therefore important to document that the evaluation of this tool has been kept out of the public domain or has fallen short in regard to having been conducted in the first place. The results of this near-empty review thus represent a call to action to fill this important gap in the literature by publishing available studies and conducting new studies of the implementation of biomathematical models to mitigate fatigue and fatigue-related risks.

Disclosures

P. M. Weiss, P. D. Patterson, J. S. Higgins, and E. S. Lang have no financial or commercial interests to disclose. H. Van Dongen is a named inventor on a patent related to individualized fatigue modeling and on 2 patent applications currently pending in this area. Furthermore, H. Van Dongen's laboratory at Washington State University is funded by the National Science Foundation, Office of Naval Research, Federal Express Corporation, and the Institutes for Behavior Resources, to help develop and apply biomathematical and biostatistical fatigue models. H. Van Dongen receives no financial compensation for any patent licenses, and none of the funding, patent, or patent applications are contingent on the outcomes of this systematic review. F. O. James and L. B. Waggoner are employed by the Institutes for Behavior Resources, an independent nonprofit scientific organization that employs the SAFTE-FAST biomathematical modeling software in its fatigue risk management consulting services. The authors have no financial or commercial interests in the SAFTE-FAST product.