Investigating mental workload caused by NDRTs in highly automated driving with deep learning

References

• Alexander AL, Wickens CD, Merwin DH. 2005. Perspective and Coplanar Cockpit Displays of Traffic Information: implications for Maneuver Choice, Flight Safety, and Mental Workload. Int J Aviat Psychol. 15(1):1–21. doi:10.1207/s15327108ijap1501_1.
   Web of Science ®Google Scholar
• Almogbel MA, Dang AH, Kameyama W. 2019. Cognitive Workload Detection from Raw EEG-Signals of Vehicle Driver using Deep Learning. In 2019 21st International Conference on Advanced Communication Technology (ICACT). p. 1–6. doi:10.23919/ICACT.2019.8702048.
   Google Scholar
• Bazilinskyy P, Petermeijer SM, Petrovych V, Dodou D, de Winter JCF. 2018. Take-over requests in highly automated driving: a crowdsourcing survey on auditory, vibrotactile, and visual displays. Transport Res Part F. 56:82–98. doi:10.1016/j.trf.2018.04.001.
   Google Scholar
• Borghini G, Astolfi L, Vecchiato G, Mattia D, Babiloni F. 2014. Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neurosci Biobehav Rev. 44:58–75. PMID: 23116991. doi:10.1016/j.neubiorev.2012.10.003.
   PubMed Web of Science ®Google Scholar
• Brookings JB, Wilson GF, Swain CR. 1996. Psychophysiological responses to changes in workload during simulated air traffic control. Biol Psychol. 42(3):361–377. PMID: 8652753. doi:10.1016/0301-0511(95)05167-8.
   PubMed Web of Science ®Google Scholar
• Craik A, He Y, Contreras-Vidal JL. 2019. Deep learning for electroencephalogram (EEG) classification tasks: a review. J Neural Eng. 16(3):031001. PMID: 30808014. doi:10.1088/1741-2552/ab0ab5.
   PubMed Web of Science ®Google Scholar
• Das Chakladar D, Dey S, Roy PP, Dogra DP. 2020. EEG-based mental workload estimation using deep BLSTM-LSTM network and evolutionary algorithm. Biomed Signal Process Control. 60:101989. doi:10.1016/j.bspc.2020.101989.
   Web of Science ®Google Scholar
• Fernández T, Harmony T, Rodríguez M, Bernal J, Silva J, Reyes A, Marosi E. 1995. EEG activation patterns during the performance of tasks involving different components of mental calculation. Electroencephalogr Clin Neurophysiol. 94(3):175–182. PMID: 7536152. doi:10.1016/0013-4694(94)00262-J.
   PubMedGoogle Scholar
• Fournier LR, Wilson GF, Swain CR. 1999. Electrophysiological, behavioral, and subjective indexes of workload when performing multiple tasks: manipulations of task difficulty and training. Int J Psychophysiol. 31(2):129–145. PMID: 9987059. doi:10.1016/S0167-8760(98)00049-X.
   PubMed Web of Science ®Google Scholar
• Galy E. 2018. Consideration of several mental workload categories: perspectives for elaboration of new ergonomic recommendations concerning shiftwork. Theor Issues Ergonom Sci. 19(4):483–497. doi:10.1080/1463922X.2017.1381777.
   Web of Science ®Google Scholar
• Gold C, Damböck D, Lorenz L, Bengler K. 2013. “Take over!” How long does it take to get the driver back into the loop? Proc Hum Factors Ergonom Soc Annu Meeting. 57(1):1938–1942. doi:10.1177/1541931213571433.
   Google Scholar
• Harmony T, Fernández T, Silva J, Bernal J, Díaz-Comas L, Reyes A, Marosi E, Rodríguez M, Rodríguez M. 1996. EEG delta activity: an indicator of attention to internal processing during performance of mental tasks. Int J Psychophysiol. 24(1-2):161–171. PMID: 8978441. doi:10.1016/S0167-8760(96)00053-0.
   PubMed Web of Science ®Google Scholar
• Hart SG, Staveland LE. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In: Hancock PA, Meshkati N, editors. Human mental workload. Vol. 52. North-Holland: Advances in psychology. p. 139–183.
   Google Scholar
• Jawabri KH, Sharma S. 2023. Physiology, Cerebral Cortex Functions. In StatPearls. Treasure Island (FL): statPearls Publishing.
   Google Scholar
• Jiao Z, Gao X, Wang Y, Li J, Xu H. 2018. Deep Convolutional Neural Networks for mental load classification based on EEG data. Pattern Recognit. 76:582–595. doi:10.1016/j.patcog.2017.12.002.
   Web of Science ®Google Scholar
• Kabilmiharbi N, Kamaliana Khamis N, Azila Noh N. 2022. Commonly Used Assessment Method to Evaluate Mental Workload for Multiple Driving Distractions: a Systematic Review. Iran J Public Health. 51(3):482–494. PMID: 35865045. doi:10.18502/ijph.v51i3.8924.
   PubMed Web of Science ®Google Scholar
• Kamaliana K, Deros B, Schramm D, Hesse B, Nuawi M, Ismail F. 2016. Subjective and indirect methods to observe driver’s drowsiness and alertness: an overview. J Eng Sci Technol. 11:28–39.
   Google Scholar
• Käthner I, Wriessnegger SC, Müller-Putz GR, Kübler A, Halder S. 2014. Effects of mental workload and fatigue on the P300, alpha and theta band power during operation of an ERP (P300) brain–computer interface. Biol Psychol. 102:118–129. PMID: 25088378. doi:10.1016/j.biopsycho.2014.07.014.
   PubMed Web of Science ®Google Scholar
• Ke Y, Jiang T, Liu S, Cao Y, Jiao X, Jiang J, Ming D. 2021. Cross-Task Consistency of Electroencephalography-Based Mental Workload Indicators: comparisons Between Power Spectral Density and Task-Irrelevant Auditory Event-Related Potentials. Front Neurosci. 15:703139. PMID: 34867143. doi:10.3389/fnins.2021.703139.
   PubMed Web of Science ®Google Scholar
• Kosti MV, Georgiadis K, Adamos DA, Laskaris N, Spinellis D, Angelis L. 2018. Towards an affordable brain computer interface for the assessment of programmers’ mental workload. Int J Hum Comput Stud. 115:52–66. doi:10.1016/j.ijhcs.2018.03.002.
   Web of Science ®Google Scholar
• LeCun Y, Kavukcuoglu K, Farabet C. 2010. Convolutional networks and applications in vision. In Proceedings of 2010 IEEE International Symposium on Circuits and Systems. p. 253–256.
   Google Scholar
• Lemm S, Blankertz B, Dickhaus T, Müller K-R. 2011. Introduction to machine learning for brain imaging. Neuroimage. 56(2):387–399. PMID: 21172442. doi:10.1016/j.neuroimage.2010.11.004.
   PubMed Web of Science ®Google Scholar
• Lin C-T, Ko L-W, Shen T-K. 2009. Computational intelligent brain computer interaction and its applications on driving cognition. IEEE Comput Intell Mag. 4(4):32–46. doi:10.1109/MCI.2009.934559.
   Web of Science ®Google Scholar
• Matthews G, Reinerman-Jones LE, Barber DJ, Abich J. 2015. The Psychometrics of Mental Workload: multiple Measures Are Sensitive but Divergent. Hum Factors. 57(1):125–143. PMID: 25790574. doi:10.1177/0018720814539505.
   PubMed Web of Science ®Google Scholar
• Mok B, Johns M, Lee KJ, Miller D, Sirkin D, Ive P, Ju W. 2015. Emergency, Automation Off: unstructured Transition Timing for Distracted Drivers of Automated Vehicles. In 2015 IEEE 18th International Conference on Intelligent Transportation Systems. p. 2458–2464.
   Google Scholar
• Müller AL. 2020. [Auswirkungen von naturalistischen fahrfremden Tätigkeiten während hochautomatisierter Fahrt ]. [[Ph.D. Thesis]]. [Darmstadt]: Technische Universität.
   Google Scholar
• Müller AL, Fernandes-Estrela N, Hetfleisch R, Zecha L, Abendroth B. 2021. Effects of non-driving related tasks on mental workload and take-over times during conditional automated driving. Eur Transp Res Rev. 13(1):16. doi:10.1186/s12544-021-00475-5.
   Web of Science ®Google Scholar
• Pakdamanian E, Hu E, Sheng S, Kraus S, Heo S, Feng L. 2022. Enjoy the Ride Consciously with CAWA: context-Aware Advisory Warnings for Automated Driving. In: Proceedings of the 14th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. New York, NY, USA: Association for Computing Machinery. (AutomotiveUI ‘22). p. 75–85. doi:10.1145/3543174.3546835.
   Google Scholar
• Radlmayr J, Weinbeer V, Löber C, Farid M, Bengler K. 2018. How Automation Level and System Reliability Influence Driver Performance in a Cut-In Situation. In: stanton NA, editor. Advances in Human Aspects of Transportation. Cham: springer International Publishing. (Advances in Intelligent Systems and Computingp. 684–694.
   Google Scholar
• Recarte MA, Nunes LM. 2003. Mental workload while driving: effects on visual search, discrimination, and decision making. J Exp Psychol Appl. 9(2):119–137. PMID: 12877271. doi:10.1037/1076-898X.9.2.119.
   PubMed Web of Science ®Google Scholar
• SAE. 2021. Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. SAE DocumentJ3016_202104, Ver. 4. SAE International. doi:10.4271/J3016_202104
   Google Scholar
• Sciaraffa N, Aricò P, Borghini G, Flumeri GD, Florio AD, Babiloni F. 2019. On the Use of Machine Learning for EEG-Based Workload Assessment: algorithms Comparison in a Realistic Task. In: longo L, Leva MC, editors. Human Mental Workload: models and Applications. Cham: Springer International Publishing. (Communications in Computer and Information Science). p. 170–185.
   Google Scholar
• Shahini F, Zahabi M. 2022. Effects of levels of automation and non-driving related tasks on driver performance and workload: a review of literature and meta-analysis. Appl Ergon. 104:103824. PMID: 35724471. doi:10.1016/j.apergo.2022.103824.
   PubMed Web of Science ®Google Scholar
• Shakouri M, Ikuma LH, Aghazadeh F, Nahmens I. 2018. Analysis of the sensitivity of heart rate variability and subjective workload measures in a driving simulator: the case of highway work zones. Int J Ind Ergon. 66:136–145. doi:10.1016/j.ergon.2018.02.015.
   Web of Science ®Google Scholar
• Smith ME, Gevins A, Brown H, Karnik A, Du R. 2001. Monitoring Task Loading with Multivariate EEG Measures during Complex Forms of Human-Computer Interaction. Hum Factors. 43(3):366–380. PMID: 11866193. doi:10.1518/001872001775898287.
   PubMed Web of Science ®Google Scholar
• Tao D, Tan H, Wang H, Zhang X, Qu X, Zhang T. 2019. A Systematic Review of Physiological Measures of Mental Workload. Int J Environ Res Public Health. 16(15):2716. PMID: 31366058. doi:10.3390/ijerph16152716.
   PubMed Web of Science ®Google Scholar
• Wagner J, Solis-Escalante T, Grieshofer P, Neuper C, Müller-Putz G, Scherer R. 2012. Level of participation in robotic-assisted treadmill walking modulates midline sensorimotor EEG rhythms in able-bodied subjects. Neuroimage. 63(3):1203–1211. PMID: 22906791. doi:10.1016/j.neuroimage.2012.08.019.
   PubMed Web of Science ®Google Scholar
• Wang Y-K, Chen S-A, Lin C-T. 2014. An EEG-based brain–computer interface for dual task driving detection. Neurocomputing. 129:85–93. doi:10.1016/j.neucom.2012.10.041.
   Web of Science ®Google Scholar
• Wickens CD. 2002. Multiple resources and performance prediction. Theor Issues Ergonom Sci. 3(2):159–177. doi:10.1080/14639220210123806.
   Google Scholar
• Young KL, Regan MA, Hammer M. 2003. Driver distraction: a review of the literature.
   Google Scholar
• Young MS, Brookhuis KA, Wickens CD, Hancock PA. 2015. State of science: mental workload in ergonomics. Ergonomics. 58(1):1–17. PMID: 25442818. doi:10.1080/00140139.2014.956151.
   PubMed Web of Science ®Google Scholar
• Zangi N, Srour-Zreik R, Ridel D, Chasidim H, Borowsky A. 2022. Driver distraction and its effects on partially automated driving performance: a driving simulator study among young-experienced drivers. Accid Anal Prev. 166:106565. PMID: 35032704. doi:10.1016/j.aap.2022.106565.
   PubMed Web of Science ®Google Scholar
• Zhang P, Wang X, Zhang W, Chen J. 2019. Learning Spatial–Spectral–Temporal EEG Features With Recurrent 3D Convolutional Neural Networks for Cross-Task Mental Workload Assessment. IEEE Trans Neural Syst Rehabil Eng. 27(1):31–42. PMID: 30507536. doi:10.1109/TNSRE.2018.2884641.
   PubMed Web of Science ®Google Scholar
• Zhao H, Ma J, Zhang Y, Chang R. 2022. Mental workload accumulation effect of mobile phone distraction in L2 autopilot mode. Sci Rep. 12(1):16856. PMID: 36207431. doi:10.1038/s41598-022-17419-1.
   PubMed Web of Science ®Google Scholar
• Zheng W-L, Lu B-L. 2015. Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks. IEEE Trans Auton Mental Develop. 7(3):162–175. doi:10.1109/TAMD.2015.2431497.
   Google Scholar
• Zhou Y, Huang S, Xu Z, Wang P, Wu X, Zhang D. 2022. Cognitive Workload Recognition Using EEG Signals and Machine Learning: a Review. IEEE Trans Cogn Dev Syst. 14(3):799–818. PMID: 36682107. doi:10.1109/TCDS.2021.3090217.
   Web of Science ®Google Scholar