Skip to main content
SpringerLink
Log in

Multi-modal physiological sensing approach for distinguishing high workload events in remotely piloted aircraft simulation

  • Research Article
  • Published:
Human-Intelligent Systems Integration Aims and scope Submit manuscript

A Correction to this article was published on 01 June 2021

This article has been updated

Abstract

Remotely piloted aircraft (RPA) operations are often characterized as highly taxing and dynamic. Physiological sensing technology can enhance personnel monitoring and training for these high-stress environments; however, work assessing the effectiveness of physiological sensors during RPA operations is limited. The proposed work tests two hypotheses: (1) physiological sensors can distinguish operator workload between scenario difficulty levels, and (2) the sensors can quantify the impact of RPA events on the operator workload. Twelve pilots completed RPA simulations at all three difficulty levels while physiological sensors collected electroencephalogram (EEG) and heart rate activity. Hypotheses were tested using mixed-effects models. Observed heart rate variability metrics did not differ among the three scenario difficulty levels except for LF/HF ratio. A 47% and 57% reduction in alpha band power was observed between easy and hard difficulty levels for the frontal and parietal channels, respectively. Abort and Reach objective events resulted in 0.2–0.3 dB lower beta activity and 66 ms increased heart rate, while losing sight of the objective (e.g., fog) had 0.72 dB increased high beta activity. Different physiological modalities (EEG and ECG) had varying effectiveness in distinguishing scenario difficulty and RPA events, suggesting a hybrid sensing approach may provide more insight than just using one modality. In conclusion, physiological sensing can distinguish operator response to scenario difficulty and events in high-fidelity RPA simulations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

De-identified datasets may be available upon institutional approval.

Change history

References

  • Abdelrahman AM, Bingener J, Yu D, Lowndes BR, Mohamed A, McConico AL, Hallbeck MS (2015) Impact of single-incision laparoscopic cholecystectomy (SILC) versus conventional laparoscopic cholecystectomy (CLC) procedures on surgeon stress and workload: a randomized controlled trial. Surg Endosc 30(3):1205–1211. https://doi.org/10.1007/s00464-015-4332-5

    Article  Google Scholar 

  • Afergan, D., Peck, E. M., Solovey, E. T., Jenkins, A., Hincks, S. W., Brown, E. T., . . . Jacob, R. J. (2014). Dynamic difficulty using brain metrics of workload. Paper presented at the Proceedings of the 32nd annual ACM conference on Human factors in computing systems

  • Akash K, Hu W-L, Jain N, Reid T (2018) A classification model for sensing human trust in machines using EEG and GSR. ACM Transactions on Interactive Intelligent Systems (TiiS) 8(4):1–20

    Article  Google Scholar 

  • Ayaz, H., Çakir, M. P., İzzetoğlu, K., Curtin, A., Shewokis, P. A., Bunce, S. C., & Onaral, B. (2012a). Monitoring expertise development during simulated UAV piloting tasks using optical brain imaging. Paper presented at the Aerospace Conference, 2012 IEEE

  • Ayaz H, Shewokis PA, Bunce S, Izzetoglu K, Willems B, Onaral B (2012b) Optical brain monitoring for operator training and mental workload assessment. Neuroimage 59(1):36–47. https://doi.org/10.1016/j.neuroimage.2011.06.023

  • Berka C, Levendowski DJ, Cvetinovic MM, Petrovic MM, Davis G, Lumicao MN, Zivkovic VT, Popovic MV, Olmstead R (2004) Real-time analysis of EEG indexes of alertness, cognition, and memory acquired with a wireless EEG headset. Int J Hum Comput Interact 17(2):151–170

    Article  Google Scholar 

  • Berka C, Levendowski DJ, Lumicao MN, Yau A, Davis G, Zivkovic VT et al (2007) EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviat Space Environ Med 78(5):B231–B244

    Google Scholar 

  • Billman GE (2013) The LF/HF ratio does not accurately measure cardiac sympatho-vagal balance. Front Physiol 4:26. https://doi.org/10.3389/fphys.2013.00026

    Article  Google Scholar 

  • Böhm B, Rötting N, Schwenk W, Grebe S, Mansmann U (2001) A prospective randomized trial on heart rate variability of the surgical team during laparoscopic and conventional sigmoid resection. Arch Surg 136(3):305–310

    Article  Google Scholar 

  • Borghini G, Astolfi L, Vecchiato G, Mattia D, Babiloni F (2014) Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neurosci Biobehav Rev 44:58–75. https://doi.org/10.1016/j.neubiorev.2012.10.003

    Article  Google Scholar 

  • Brouwer A-M, Hogervorst MA, Van Erp JBF, Heffelaar T, Zimmerman PH, Oostenveld R (2012) Estimating workload using EEG spectral power and ERPs in the n-back task. J Neural Eng 9(4):045008

    Article  Google Scholar 

  • Camm AJ, Malik M, Bigger J, Breithardt G, Cerutti S, Cohen RJ et al (1996) Heart rate variability. Standards of measurement, physiological interpretation, and clinical use. Eur Heart J 17(3):354–381

    Article  Google Scholar 

  • Cao X, MacNaughton P, Cadet LR, Cedeno-Laurent JG, Flanigan S, Vallarino J et al (2019) Heart rate variability and performance of commercial airline pilots during flight simulations. Int J Environ Res Public Health 16(2). https://doi.org/10.3390/ijerph16020237

  • Catherwood D, Edgar GK, Nikolla D, Alford C, Brookes D, Baker S, White S (2014) Mapping brain activity during loss of situation awareness: an EEG investigation of a basis for top-down influence on perception. Hum Factors 56(8):1428–1452

    Article  Google Scholar 

  • Causse M, Fabre E, Giraudet L, Gonzalez M, Peysakhovich V (2015) EEG/ERP as a measure of mental workload in a simple piloting task. Procedia Manuf 3:5230–5236. https://doi.org/10.1016/j.promfg.2015.07.594

    Article  Google Scholar 

  • Center, A. F. P. (2020). Military demographics: June

  • Charles RL, Nixon J (2019) Measuring mental workload using physiological measures: a systematic review. Appl Ergon 74:221–232. https://doi.org/10.1016/j.apergo.2018.08.028

    Article  Google Scholar 

  • Chen, J. Y., Barnes, M. J., & Harper-Sciarini, M. (2010). Supervisory control of unmanned vehicles. Retrieved from

  • Cinaz, B., La Marca, R., Arnrich, B., & Tröster, G. (2010). Monitoring of mental workload levels. Paper presented at the International Conference on e-Health. sn: IADIS

  • Cinaz B, Arnrich B, La Marca R, Tröster G (2013) Monitoring of mental workload levels during an everyday life office-work scenario. Pers Ubiquit Comput 17(2):229–239. https://doi.org/10.1007/s00779-011-0466-1

    Article  Google Scholar 

  • Delorme A, Makeig S (2004) EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods 134(1):9–21

    Article  Google Scholar 

  • Dong, S.-Y., Kim, B.-K., Lee, K., & Lee, S.-Y. (2015). A preliminary study on human trust measurements by EEG for human-machine interactions

  • Dussault C, Jouanin J-C, Guezennec C-Y (2004) EEG and ECG changes during selected flight sequences. Aviat Space Environ Med 75(10):889–897

    Google Scholar 

  • Dussault C, Jouanin J-C, Philippe M, Guezennec C-Y (2005) EEG and ECG changes during simulator operation reflect mental workload and vigilance. Aviat Space Environ Med 76(4):344–351

    Google Scholar 

  • Fu C, Zhang Z, He J, Huang S, Qiu J, Wang Y (2018) Brain dynamics of decision-making in the generalized trust game: evidence from ERPs and EEG time-frequency analysis. Acta Psychol Sin 50(3):317–326

    Google Scholar 

  • Harmony T a, Fernández T a, Silva J, Bosch J, Valdés P, Fernández-Bouzas A et al (1999) Do specific EEG frequencies indicate different processes during mental calculation? Neurosci Lett 266(1):25–28

    Article  Google Scholar 

  • Johnson RR, Popovic DP, Olmstead RE, Stikic M, Levendowski DJ, Berka C (2011) Drowsiness/alertness algorithm development and validation using synchronized EEG and cognitive performance to individualize a generalized model. Biol Psychol 87(2):241–250

    Article  Google Scholar 

  • Kaur A, Chaujar R, Chinnadurai V (2019) Effects of neural mechanisms of pretask resting EEG alpha information on situational awareness: a functional connectivity approach. Hum Factors 62:1150. https://doi.org/10.1177/0018720819869129

    Article  Google Scholar 

  • Kim HG, Cheon EJ, Bai DS, Lee YH, Koo BH (2018) Stress and heart rate variability: a meta-analysis and review of the literature. Psychiatry Investig 15(3):235–245. https://doi.org/10.30773/pi.2017.08.17

    Article  Google Scholar 

  • Klimesch W (1999) EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain Res Rev 29(2):169–195. https://doi.org/10.1016/S0165-0173(98)00056-3

    Article  Google Scholar 

  • Kumar N, Kumar J (2016) Measurement of cognitive load in HCI systems using EEG power spectrum: an experimental study. Procedia Comput Sci 84:70–78. https://doi.org/10.1016/j.procs.2016.04.068

    Article  Google Scholar 

  • Laborde S, Mosley E, Thayer JF (2017) Heart rate variability and cardiac vagal tone in psychophysiological research – recommendations for experiment planning, data analysis, and data reporting. Front Psychol 8:213. https://doi.org/10.3389/fpsyg.2017.00213

    Article  Google Scholar 

  • Liu, Y., Ayaz, H., Onaral, B., & Shewokis, P. A. (2015). Neural adaptation to a working memory task: a concurrent EEG-fNIRS Study

  • Liu Y, Ayaz H, Shewokis PA (2017) Multisubject “Learning” for mental workload classification using concurrent EEG, fNIRS, and physiological measures. Front Hum Neurosci 11:389. https://doi.org/10.3389/fnhum.2017.00389

    Article  Google Scholar 

  • Mark, J., Thomas, N., Kraft, A., Casebeer, W. D., Ziegler, M., & Ayaz, H. (2017). Neurofeedback for personalized adaptive training

  • McDuff, D., Gontarek, S., & Picard, R. (2014). Remote measurement of cognitive stress via heart rate variability. Paper presented at the Engineering in Medicine and Biology Society (EMBC), 2014 36th Annual International Conference of the IEEE

  • Mehler, B., Reimer, B., & Wang, Y. (2011). A comparison of heart rate and heart rate variability indices in distinguishing single-task driving and driving under secondary cognitive workload

  • Mehler B, Reimer B, Coughlin JF (2012) Sensitivity of physiological measures for detecting systematic variations in cognitive demand from a working memory task: an on-road study across three age groups. Hum Factors 54(3):396–412

    Article  Google Scholar 

  • Mognon A, Jovicich J, Bruzzone L, Buiatti M (2011) ADJUST: an automatic EEG artifact detector based on the joint use of spatial and temporal features. Psychophysiology 48(2):229–240

    Article  Google Scholar 

  • Moses, Z. B., Luecken, L. J., & Eason, J. C. (2007). Measuring task-related changes in heart rate variability. Paper presented at the 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society

  • Mullen, T. (2012). CleanLine EEGLAB plugin. San Diego, CA: Neuroimaging Informatics Tools and Resources Clearinghouse (NITRC)

  • Ramsey, L., Bloomer, J., Watz, E., & Neubauer, P. (2015). Pilot test of the Predator Research Integrated Networked Combat Environment Workload Indices via Neurobridge Kit. Warfighter Readiness Science And Technology Program

  • Rieger A, Fenger S, Neubert S, Weippert M, Kreuzfeld S, Stoll R (2014) Psychophysical workload in the operating room: primary surgeon versus assistant. Surg Endosc:1–9

  • Risser, M. R., & Ware, J. C. (1999). Driving simulation with EEG monitoring in normals and obstructive sleep apnea patients

  • Scannella S, Peysakhovich V, Ehrig F, Lepron E, Dehais F (2018) Assessment of ocular and physiological metrics to discriminate flight phases in real light aircraft. Hum Factors 60(7):922–935. https://doi.org/10.1177/0018720818787135

    Article  Google Scholar 

  • Schill, N. P., Leah, J., Gyovai, B. L., Joralmon, D. Q., Schneck, A. J., & Woudstra, D. A. (2014). Operational alignment in predator training research. Retrieved from

  • Shaffer F, McCraty R, Zerr CL (2014) A healthy heart is not a metronome: an integrative review of the heart’s anatomy and heart rate variability. Front Psychol 5:1040

    Article  Google Scholar 

  • Skibniewski FW, Dziuda Ł, Baran PM, Krej MK, Guzowski S, Piotrowski MA, Truszczyński OE (2015) Preliminary results of the LF/HF ratio as an indicator for estimating difficulty level of flight tasks. Aerosp Med Human Perform 86(6):518–523

    Article  Google Scholar 

  • Smith, M. E., & Gevins, A. (2005). Neurophysiologic monitoring of mental workload and fatigue during operation of a flight simulator. Paper presented at the Biomonitoring for Physiological and Cognitive Performance during Military Operations

  • Spüler M, Walter C, Rosenstiel W, Gerjets P, Moeller K, Klein E (2016) EEG-based prediction of cognitive workload induced by arithmetic: a step towards online adaptation in numerical learning. ZDM 48(3):267–278. https://doi.org/10.1007/s11858-015-0754-8

    Article  Google Scholar 

  • Sugimoto, I., Kitamura, K., Murai, K., Wang, Y., & Wang, J. (2016). Study on relation between operator and trainee’s mental workload for ship maneuvering simulator exercise using heart rate variability. Paper presented at the Systems, Man, and Cybernetics (SMC), 2016 IEEE International Conference on

  • Sztajzel J (2004) Heart rate variability: a noninvasive electrocardiographic method to measure the autonomic nervous system. Swiss Med Wkly 134(35–36):514–522

    Google Scholar 

  • Taelman, J., Vandeput, S., Spaepen, A., & Van Huffel, S. (2009). Influence of mental stress on heart rate and heart rate variability

  • Tarvainen MP, Niskanen J-P, Lipponen JA, Ranta-Aho PO, Karjalainen PA (2014) Kubios HRV–heart rate variability analysis software. Comput Methods Prog Biomed 113(1):210–220

    Article  Google Scholar 

  • Teo G, Matthews G, Reinerman-Jones L, Barber D (2019) Adaptive aiding with an individualized workload model based on psychophysiological measures. Human Intelligent Systems Integration. https://doi.org/10.1007/s42454-019-00005-8

  • Wang S, Gwizdka J, Chaovalitwongse WA (2016) Using wireless EEG signals to assess memory workload in the $ n $-Back task. IEEE Trans Human Mach Sys 46(3):424–435

    Article  Google Scholar 

  • Wang, M., Hussein, A., Rojas, R. F., Shafi, K., & Abbass, H. A. (2018). EEG-based neural correlates of trust in human-autonomy interaction

  • Wei Z, Zhuang D, Wanyan X, Zhang H, Liu C (2014a) A theoretical model of mental workload in pilots based on multiple experimental measurements. In: Harris D (ed) Engineering psychology and cognitive ergonomics: 11th International Conference, EPCE 2014, Held as Part of HCI International 2014, Heraklion, Crete, Greece, June 22-27, 2014. Proceedings. Springer International Publishing, Cham, pp 104–113

    Chapter  Google Scholar 

  • Wei, Z., Zhuang, D., Wanyan, X., Zhang, H., & Liu, C. (2014b). A theoretical model of mental workload in pilots based on multiple experimental measurements. Paper presented at the International Conference on Engineering Psychology and Cognitive Ergonomics.

  • Wilson GF (2002) An analysis of mental workload in pilots during flight using multiple psychophysiological measures. Int J Aviat Psychol 12(1):3–18. https://doi.org/10.1207/S15327108IJAP1201_2

    Article  Google Scholar 

  • Wilson GF, Fisher F (1995) Cognitive task classification based upon topographic EEG data. Biol Psychol 40(1–2):239–250. https://doi.org/10.1016/0301-0511(95)05102-3

    Article  Google Scholar 

  • Wilson, C., Ramsey, L., & Bloomer, J. (2016). Multimodal assessment of workload for predator training research. Interservice/Industry Training, Simulation, and Education Conference

  • Wu C, Cha J, Sulek J, Sundaram CP, Wachs J, Proctor RW, Yu D (2021) Sensor-based indicators of performance changes between sessions during robotic surgery training. Appl Ergon 90:103251. https://doi.org/10.1016/j.apergo.2020.103251

    Article  Google Scholar 

  • Yeo, L. G., Sun, H., Liu, Y., Trapsilawati, F., Sourina, O., Chen, C.-H., . . . Ang, W. T. (2017). Mobile EEG-based situation awareness recognition for air traffic controllers

  • Young MS, Brookhuis KA, Wickens CD, Hancock PA (2014) State of science: mental workload in ergonomics. Ergonomics 58(1):1–17. https://doi.org/10.1080/00140139.2014.956151

    Article  Google Scholar 

  • Yu D, Dural C, Morrow MMB, Yang L, Collins JW, Hallbeck S., . . . Forsman M (2017) Intraoperative workload in robotic surgery assessed by wearable motion tracking sensors and questionnaires. Surg Endosc 31(2):877–886. https://doi.org/10.1007/s00464-016-5047-y

  • Zhang W, Feltner D, Shirley J, Swangnetr M, and Kaber D (2016) Unmanned aerial vehicle control interface design and cognitive workload: a constrained review and research framework.

  • Zhang T, Yang J, Liang N, Pitts B, Prakah-Asante K, Curry R, . . . Yu D (2021) Physiological measurements of situation awareness: a systematic review. Humum Factors. https://doi.org/10.1177/0018720820969071

  • Zhao C, Zhao M, Liu J, Zheng C (2012) Electroencephalogram and electrocardiograph assessment of mental fatigue in a driving simulator. Accid Anal Prev 45:83–90

    Article  Google Scholar 

  • Zhou T, Cha JS, Gonzalez G, Wachs JP, Sundaram CP, Yu D (2020) Multimodal physiological signals for workload prediction in robot-assisted surgery. J Hum Robot Interact 9(2):Article 12. https://doi.org/10.1145/3368589

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to acknowledge the Air Force Research Laboratory’s Summer Faculty Fellowship Program and the Airman Systems Directorate for their support of this work. The authors would also like to thank research team members for all their efforts in conducting this study.

Code availability

Not applicable.

Funding

The authors would like to acknowledge the Air Force Research Laboratory’s Summer Faculty Fellowship Program for their support of the work.

Author information

Authors and Affiliations

  1. Purdue University, 315 N. Grant St., West Lafayette, IN, 47907, USA

    Denny Yu

  2. Air Force Research Laboratory, Warfighter Readiness Research Division, Dayton, OH, USA

    Chantale Wilson Antonik, Frederick Webber, Eric Watz & Winston Bennett Jr

Authors
  1. Denny Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chantale Wilson Antonik
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Frederick Webber
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eric Watz
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Winston Bennett Jr
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Denny Yu.

Ethics declarations

This research complied with the American Psychological Association Code of Ethics and was approved by the Institutional Review Board. Informed consent was obtained from each participant.

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This manuscript is submitted for the Special Issue on “Human-Autonomy Teaming in Military Contexts”

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, D., Antonik, C.W., Webber, F. et al. Multi-modal physiological sensing approach for distinguishing high workload events in remotely piloted aircraft simulation. Hum.-Intell. Syst. Integr. 1, 89–99 (2019). https://doi.org/10.1007/s42454-020-00016-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42454-020-00016-w

Keywords

Search

Navigation