Is your training service resilient and postured to support organisational sustainment?

Figures & data

Figure 1. Examples of training System’s responses to perturbations.

Table 1. Examples of resilience definition increments.

Increment 1: Deal with changes as they occur while retaining original system’s design
Holling (1973)	‘ability to absorb change and disturbance to maintain the same relationships between populations or state variables’
Uday and Marais (2015)	‘ability to combine survivability and recoverability, to bounce back’
INCOSE Resilient Systems Working Group Rodríguez-Sánchez et al. (2021)	‘the ability to [continue to] provide required capability when facing adversity’
Increment 2: Prepare and deal with the changes while retaining original system’s design
Walker et al. (2004)	‘ability to absorb disturbances and reorganise while changing’
Laprie (2008)	‘persistence of dependability when facing changes’
Ayyub (2014)	‘ability to prepare and adapt to changing conditions in order to recover rapidly from disruptions’
Increment 3: Anticipate, prepare, and deal with the changes while retaining original system design
Hollnagel et al. (2006)	‘ability to create foresight, recognize, and anticipate in order to defend against the changing shape of risk before adverse consequences’
Tran et al. (2017)	‘ability to apply classical reliability methods, such as redundancy at the component level and use of preventative maintenance at the system level
Increment 4: Anticipate, prepare, deal, and learn from the changes (might alter original system design as part of lessons learnt)
Lengnick-Hall et al. (2011)	‘ability to effectively absorb, develop situation specific responses to, and ultimately engage in transformative activities’
Menéndez Blanco and Montes Menéndez Blanco and Montes Botella (2016)	‘ability to take both proactive and reactive measures including knowledge, learning, and innovation to balance efficiency and adaptability’
Increment 5: Anticipate, prepare, deal, and learn to live with uncertain changes (continuous system adaptation to new conditions)
Berkes (2007)	‘ability to learn to live with change and uncertainty; to nurture various types of ecological, social, and political diversity for increasing options and reducing risks; to increase the range of knowledge for learning and problem solving; and to create opportunities for self-organization, including cross-scale linkages and problem- solving networks’
Increment 6: Resilience is an emergent property of a continuously evolving system
Erol et al. (2010)	‘interaction of the characteristics and capacities of a system, which will eventually evolve in the case of a disruptive event’
Park et al. (2013)	‘the outcome of a recursive process that includes: sensing, anticipation, learning, adaptation where resilience is the emergent property of a system undergoing adaptive process—that is, not as something a system has, but a characteristic of the way it behaves’

Figure 2. Capability systems’ resilience framework (Jnitova et al. 2022).

Figure 3. Integrative resilience framework by Kantur and İşeri-Say (2012, 765).

Table 2. Survey target populations.

Target Population	Description
‘Desired Training System Performance’ Benchmark
Benchmarker	a key stakeholder(s) invited to establish a desired performance benchmark for their respective training system under survey
‘Actual Training System Performance’ Experience
Trainee	a training participant and a trained ‘product’ of their respective training system under survey
Instructor	an SME in a subject of associated skillset, a trainees’ manager and a ‘producer’ of a ‘trained product’; delivers training to trainees and assists Training Specialists with their tasks
Training Specialist	an SME, developer and manager of a training system under survey; creates, reviews, modifies and manages training, training policy, training workforce and training establishment(s), and assists Instructor’s training delivery
Workplace Supervisor	a supervisor of graduates [‘trained product’] from a training delivered by a respective training system under survey– ‘product’s manager and user’
Capability Manager	a manager of a capability serviced by workforce graduated from the training delivered by a respective training system under survey – ‘product user’

Table 3. Summary of TORS responses.

Application	Number of Responses
	Bench- markers	Trainees	Instructors	Training Specialists	Workplace Supervisors	Capability Managers	Total
Pilot	1	147	103	38	32	28	349
Verification	17	193	54	19	32	29	344
Broadscale Validation	5	238	76	35	233	30	617
International Validation	5	42	20	6	9	11	93
Total	28	620	253	98	306	98	1,403

Table 4. Issues identified in the ‘Pilot’ TORS design.

Issue	Examples	Consequences	Changes in Instrument
Lack of a standardised approach to the number, design and option selection	– the numbers of demographic questions per population vary from 8 to 11 – some multiple-choice demographic questions allow us to select multiple responses when others are limited to one-choice only ‒ the number of questions per attribute ranges from 3 to 16	– increased complexity of the response analysis – data analysis is less suitable for automation of reporting	– standardised nine demographic questions with single- response options – Resilience section modified to 9 questions per attribute
A large number of questions	the total number of resilience questions is 107	The average duration for survey completion exceeded 30 minutes from the maximum of 15–25 minutes maximum (expected in ‘pilot’)	The number of questions decreased to 54
“Likert Scale” had biases	the ‘not sure’ and mid-range options ‘sometimes’ were significantly more popular out of 5 response options	potential influence on survey response choices	– removal of ‘not sure’ option – adding ‘rarely’ option
Lack of trainee experience in the Defence training system	Initial training students have no exposure to transition to a new capability class	The more significant than expected number of trainees have commenced but have not completed their surveys	Reducing the number of resilience questions for trainees from 54 to 34 to remove those outside their experience
All resilience attributes are considered of an equal value	The quantitative data analysis algorithm does not distinguish between the attributes	Potential analysis error	Adding the attribute weighting section

Figure 4. Alpha reliability test for the ‘Pilot’ adaptive capacity set.

Table 5. Example of homogeneity analysis for the ‘Pilot’ application.

Questions/Topic	Trainee	Instructor	Training Specialist	Capability Manager	Workplace Supervisor
Grid question: In your opinion, does Defence adapt its training well to address new training requirements associated with:
new or updated capabilities	sometimes = 60%	sometimes = 53.2%	sometimes = 70.8%	sometimes = 53.1%	sometimes = 70%
new or updated equipment	sometimes = 59.1%	sometimes = 46.8%	sometimes = 79.2%	sometimes = 50%	sometimes = 90%
new or updated functionalities	sometimes = 62.2%	sometimes = 53.2%	sometimes = 62.5%	sometimes = 77.3%	sometimes = 70%
new or updated technologies	sometimes = 73.5%	sometimes = 59.7%	sometimes = 70.8%	sometimes = 81.5%	sometimes = 70%
new specialisations	sometimes = 53.3%	sometimes = 43.5%	sometimes = 54.2%	sometimes = 76%	sometimes = 90%
new positions	sometimes = 59.5%	sometimes = 54.8%	sometimes = 50%	sometimes = 71.4%	sometimes = 80%
anti-terrorism warfare	not sure = 60.4%	not sure = 51.6%	not sure = 56%	not sure = 45.8%	not sure = 50%

Figure 5. Example of ‘Pilot’ responses to a resilience question.

Figure 6. Two popular resilience attribute performance views from three applications.¹

Figure 7. Examples of frequency from deductive (upper) and inductive (lower) thematic analysis of freeform responses.

Figure 8. Example inductive code map for freeform response.

Figure 9. Example visualisations of performance ratings for extant training policy.

Figure 10. Example of comparative analysis of quantitative and qualitative responses.

Holling, C. 1973. “Resilience and Stability of Ecological Systems.” Annual Review of Ecology and Systematics 4 (1): 1–23. https://doi.org/10.1146/annurev.es.04.110173.000245.

 Google Scholar

Uday, P., and K. Marais. 2015. “Designing Resilient Systems-Of-Systems: A Survey of Metrics, Methods, and Challenges.” Systems Engineering 18 (5): 491–510. https://doi.org/10.1002/sys.21325.

 Web of Science ®Google Scholar

RSWG (INCOSE Resilient Systems Working Group) 2021 [ Accessed online 25 Aug 22 at URL: Resilient Systems (incose.org)

 Google Scholar

Walker, B., C. S. Holling, S. Carpenter, and A. Kinzig. 2004. “Resilience, Adaptability and Transformability in Social–Ecological Systems.” Ecology Society 9 (2): 10.5751/ES-00650-090205. 2. https://www.gbcma.vic.gov.au/downloads/wshop_resilience_reading/resilience_2.pdf. (Accessed August 28, 2020

 Web of Science ®Google Scholar

Laprie, J.-C., (2008). From dependability to resilience. 38th IEEE/IFIP Int. Conf. On dependable systems and networks, Toulouse, France.

 Google Scholar

Ayyub, B. M. 2014. “Systems resilience for multihazard environments: definition, metrics, and valuation for decision making.” Risk Analysis 34 (2): 340–355. https://doi.org/10.1111/risa.12093.

 PubMed Web of Science ®Google Scholar

Hollnagel, E., D. D. Woods, and N. Leveson, Eds. 2006. Resilience Engineering: Concepts and Precepts. Boca Raton, FL: Ashgate Publishing, Ltd.

 Google Scholar

Tran, H. T., M. Balchanos, J. C. Domerçant, and D. N. Mavris. 2017. “A Framework for the Quantitative Assessment of Performance-Based System Resilience.” Reliability Engineering & System Safety 158:73–84. https://doi.org/10.1016/j.ress.2016.10.014.

 Web of Science ®Google Scholar

Lengnick-Hall, C. A., T. E. Beck, and M. L. Lengnick-Hall. 2011. “Developing a Capacity for Organizational Resilience Through Strategic Human Resource Management.” Human Resource Management Review 21 (3): 243–255. https://doi.org/10.1016/j.hrmr.2010.07.001.

 Web of Science ®Google Scholar

Menéndez Blanco, J. M., and J. L. Montes Botella. 2016. “What Contributes to Adaptive Company Resilience? A Conceptual and Practical Approach.” Development & Learning in Organizations: An International Journal 30 (4): 17–20. https://doi.org/10.1108/DLO-10-2015-0080.

 Google Scholar

Berkes, F. 2007. “Understanding Uncertainty and Reducing Vulnerability: Lessons from Resilience Thinking.” Natural Hazards Review 41 (2): 283–295. https://doi.org/10.1007/s11069-006-9036-7.

 Web of Science ®Google Scholar

Erol, O., D. Henry, and B. Sauser 2010. 3.1.2 Exploring resilience measurement methodologies. Paper presented at the INCOSE International Symposium, Pretoria, South Africa.

 Google Scholar

Park, J., T. P. Seager, P. S. C. Rao, M. Convertino, and I. Linkov. 2013. “Integrating Risk and Resilience Approaches to Catastrophe Management in Engineering Systems.” Risk Analysis 33 (3): 356–367. https://doi.org/10.1111/j.1539-6924.2012.01885.x.

 PubMed Web of Science ®Google Scholar

Jnitova, V., M. Efatmaneshnik, K. Joiner, T. Ferris, and D. Wenzel. 2022. “Towards Mature Training System of Systems Desired Performance and Resilience.” In Proceedings of the SETE 2022 Conference, Barton, A.C.T., Australia.

 Google Scholar

Kantur, D., and A. İ̇şeri-Say. 2012. “Organisational resilience: A conceptual integrative framework.” Journal of Management & Organization 18 (6): 762–773. https://doi.org/10.5172/jmo.2012.18.6.762.

 Web of Science ®Google Scholar