The Loss of Manual Flying Skills in Pilots of Highly Automated Airliners

160 Pages·2009·10.16 MB·English

by  Nathalie Boston

Cranfield University

Matthew Ebbatson

The Loss of Manual Flying Skills in Pilots of
Highly Automated Airliners

School of Engineering

PhD
Cranfield University

School of Engineering

PhD Thesis

2009

Matthew Ebbatson

The Loss of Manual Flying Skills in Pilots of Highly Automated
Airliners

Supervisors: Dr. J. Huddlestone and Dr. D. Harris

Academic Years 2005 to 2009

© Cranfield University, 2009. All rights reserved. No part of this publication may
be reproduced without the written permission of the copyright holder.
“What is chiefly needed is skill rather than machinery”

Wilbur Wright
13 May 1900
Acknowledgements

I would like to thank all of those people at Cranfield and the contributing
organisations who made this work possible. Also, thank you to my friends and
family for their kind support and encouragement throughout. Most importantly,
thank you to Nathalie, for always having confidence in me.

2
Abstract

Anecdotal and subjective evidence suggests that the manual flying ability of pilots operating
highly automated aircraft is declining owing to a lack of opportunity to exercise such skills in
the modern air transport environment. However, there is a paucity of objective evidence to
support this safety concern. Consequently, the work presented in this thesis aims to provide
empirically derived data to evaluate the extent and causes of the speculated manual skills
decline and guide possible intervention strategies.

Initially a cognitive task analysis is undertaken to determine the cognitive demands of
performing manual flight in a large jet transport aircraft. Expert pilots report employing highly
refined mental models structures which enable them to predict the aircrafts performance
whilst causing minimal burden to their mental capacity. The study concludes that when
measuring manual flying performance careful consideration must be given to designing a task
which challenges both the cognitive and physical aspects of manual flying skill.

Secondly, relatively novel pilot performance measures based upon the frequency analysis of
control input data are evaluated. An empirical study finds that these techniques are both
reliable and sensitive to manual flying performance. Furthermore, when studying large
transport aircraft, such measures of the pilots control strategy are found to contribute valuable
information about performance which is missing when just traditional ‘outer-loop’ performance
measures are applied. The study concludes that these measures of control strategy are
valuable in evaluating manual flying performance.

Finally, the manual flying skills of a sample of pilots of highly automated aircraft are evaluated
on a challenging manual flying task. A significant proportion exhibit poor manual flying
performance as judged by a type rating examiner. Further analysis reveals that the
performance of the pilots is significantly influenced by the amount of recent manual handling
experience they have accumulated, rather than their longer-term manual flying experience.
Significantly, airspeed tracking ability is influenced which is cited elsewhere as a causal factor
in many manual flying skill related accidents. The results support the previous anecdotal and
subjective concerns relating to the loss of manual flying skills.

3
Publications Resulting from this Work

Journal Articles:

Ebbatson, M. (2006). Practice makes imperfect: common factors in recent
manual approach incidents. Human Factors and Aerospace Safety 6(3), 275-
278.

Ebbatson, M., Huddlestone, J., Harris, D. and Sears, R. (2007). The
Application of Frequency Analysis Based Performance Measures as an
Adjunct to Flight Path Derived Measures of Pilot Performance. Human
Factors and Aerospace Safety 6(4), 383-394.

Ebbatson, M., Harris, D., Huddlestone, J. and Sears, R. (2008). Combining
Control Input with Flight Path Data to Evaluate Pilot Performance in Transport
Aircraft. Aviation, Space and Environmental Medicine 79(10), 1061-1064.

Ebbatson, M., Harris, D., Huddlestone, J. and Sears, R. (2008). The
Relationship Between Manual Handling Performance and Recent Flying
Experience in Air Transport Pilots. Ergonomics. In review.

Conference Proceedings:

Ebbatson, M., Huddlestone, J., Sears, R. and Harris, D. (2006).
Loss of manual flying skills: Evaluating objective performance measures. 27th
Conference of the European Association for Aviation Psychology; September
24th – 28th; Potsdam, Germany.

Industry Presentations:

FORCE Safety Research for Flight Operations, Gatwick, March 2007
GAPAN Technical & Safety Committee, April 2008
RAeS Flight Crew Training Conference, September 2008

4
Table of Contents

1.1 Background..................................................................................................................11
1.2 Rationale for Research...............................................................................................13
1.2.1 Automated Flight................................................................................................13
1.2.2 Evidence for the Loss of Manual Flying Skills from Pilot Attitude Surveys....15
1.2.3 Evidence for the Loss of Manual Flying Skills from Accident Data................17
1.2.4 Evidence of Manual Flying Skill Loss from Experimental Work......................24
1.2.5 Adapting Training...............................................................................................26
1.3 Scope of Research......................................................................................................27
1.4 Research Objectives...................................................................................................28
1.5 Structure of the Thesis................................................................................................29
2.1 Introduction..................................................................................................................30
2.1.1 Models of Human-Aircraft Control.....................................................................33
2.1.2 Mental Models....................................................................................................34
2.2 Methodology................................................................................................................37
2.2.1 Cognitive Task Analysis.....................................................................................37
2.2.2 Participants.........................................................................................................38
2.2.3 Phase One – Introductions and Task Diagram................................................39
2.2.4 Phase Two – Knowledge Audit.........................................................................40
2.2.5 Phase Three – Simulation Interview.................................................................40
2.2.6 Treatment of Data..............................................................................................41
2.3 Results and Discussion..............................................................................................42
2.3.1 Task Diagram.....................................................................................................42
2.3.2 Cognitive Process..............................................................................................43
2.4 Chapter Conclusions...................................................................................................49
3.1 Introduction..................................................................................................................50
3.2 Performance Measurement in Flight Operations......................................................51
3.2.1 Flight Data Monitoring & Discreet Event Measures.........................................52
3.3 Performance Measurement in Research...................................................................54
3.3.1 General Properties of a Performance Metric...................................................55
3.3.2 Scalar Measures of Tracking Performance......................................................56
3.3.3 Advantages of Measuring Control Strategy...................................................57
3.3.4 Approaches to Measuring Control Strategy.....................................................59
3.3.5 Power Spectral Analysis....................................................................................62
3.3.6 An Alternative Characterisation of Control Strategy........................................64
3.4 Performance Measurement Literature Summary......................................................67
3.5 Study Aims & Objectives............................................................................................68
3.6 Method.........................................................................................................................69
3.6.1 Participants.........................................................................................................69
3.6.2 Equipment...........................................................................................................69
3.6.3 Task.....................................................................................................................70
3.6.4 Performance Measures.....................................................................................70
3.7 Results.........................................................................................................................73
3.7.1 Flight path tracking accuracy and smoothness (outer loop parameters).......73
3.7.2 Summary Frequency Measures of Control Strategy.......................................75
3.7.3 Frequency Band-Pass Measures of Control Strategy.....................................75
3.7.4 Descriptive Power of the Metrics......................................................................77
3.8 Discussion....................................................................................................................79
3.8.1 Measures of outer-loop parameter pursuit tracking.........................................79
3.8.2 Measures of control input strategy....................................................................80
3.8.3 Overview.............................................................................................................82
3.9 Chapter Conclusions...................................................................................................84
4.1 Introduction..................................................................................................................86
4.2 Study Aims and Objectives.........................................................................................86
4.3 Methodology................................................................................................................87
4.3.1 Research Setting................................................................................................87
4.3.2 Participants.........................................................................................................88
4.3.3 Ethical Considerations.......................................................................................89
5
4.3.4 Expert Observation............................................................................................89
4.3.5 Equipment...........................................................................................................90
4.3.6 Manual Handling Experience Measures (Independent Variables).................91
4.3.7 Manual Flying Task............................................................................................92
4.3.8 Experimental Procedure....................................................................................95
4.3.9 Flight Data Derived Performance Measures (Dependant Variables).............98
4.4 Analysis & Results......................................................................................................99
4.4.1 Demographic Data.............................................................................................99
4.4.2 Flight Data Performance Measures................................................................100
4.4.3 TRE Performance Assessment.......................................................................101
4.4.4 Validating the Objective Performance Measures against TRE Scores........106
4.4.5 Relationships between Manual Handling Experience and Performance.....111
4.5 Discussion..................................................................................................................115
4.5.1 Validation and Assessment of Sensitivity of Performance Measures..........116
4.5.2 Relationships between Manual Handling Experience and Performance.....119
5.1 General Discussion...................................................................................................123
5.2 Conclusions...............................................................................................................125
5.3 Suggested Future Research....................................................................................127
Appendix A...………………....................................................................................................134
Appendix B...………………....................................................................................................136
Appendix C...………………....................................................................................................141
Appendix D...………………....................................................................................................142
Appendix E...………………....................................................................................................143
Appendix F...………………....................................................................................................147
Appendix G...………………....................................................................................................149
Appendix H...………………....................................................................................................150
Appendix I...……………….....................................................................................................151
Appendix J...………………....................................................................................................153
Appendix K...………………....................................................................................................156

6
Table of Figures

Figure 1 - The HFACS model .................................................................................................................20
Figure 2 - Distribution of accidents by flight phase………………………………………………..22
Figure 3 - Structure of thesis....................................................................................................29
Figure 4 - Aircraft state transitions...........................................................................................31
Figure 5 - The series model of pilot control..........................................................................................34
Figure 6 - SME cognitive task diagrams...............................................................................................44
Figure 7 - Cognitive demands categories ............................................................................................45
Figure 8 - Example time series of control inceptor displacement....................................................60
Figure 9 - Transposition of time series data into the frequency domain........................................61
Figure 10 - Example Plot of Power Spectral Density distribution ...................................................63
Figure 11 - Illustration of frequency banding technique…………………………………………..64
Figure 12 - Illustration of Rantanen et al technique……………………………………………….65
Figure 13 - Averaged periodograms of Control Wheel Inputs..........................................................74
Figure 14 - Averaged periodograms of Control Column inputs.......................................................74
Figure 15 - Plot of partial Eta squared……………………………………………………………...78
Figure 16 - Front and profile views of Boeing 737-300…………………………………………...91
Figure 17 - Diagrammatic representation of the task scenario........................................................94
Figure 18 - FlightScape Insight Animation............................................................................................96
Figure 19 - FlightScape Insight Analysis...............................................................................................97
Figure 20 - TRE scoring distribution.....................................................................................................101

Table of Tables

Table 1 - Attributes of a performance metric........................................................................................55
Table 2 – ME and SDE for ILS outer control loop parameters........................................................73
Table 3 – Band-pass metrics for primary flight control Inputs during ILS .....................................76
Table 4 – Summery frequency metrics for Primary Flight Control Inputs during the ILS..........77
Table 5 - Selected battery of performance metrics.............................................................................84
Table 6 - Sample Demographic & Career Background.....................................................................99
Table 7 - Inter-Rater Reliability Assessment of TRE grading using Cohen's Kappa................102
Table 8 - Correlations between metrics and TRE score on the straight and level segment...103
Table 9 - Correlations between metrics and TRE score on the ILS segment ...........................104
Table 10 - Correlations between metrics and TRE score on the missed approach..................105
Table 11 - t-test results for performance metrics between TRE assigned groups....................108
Table 12 - Logistic Regression Model Parameters...........................................................................110
Table 13 - Logistic Regression Model Classification Rate..............................................................110
Table 14 - Correlations between number of manual flying hours and performance.................112
Table 15 - Correlations between recent manual flying exposure and performance ................113
Table 16 - Performance metrics summarised.........................................................................114
7
Glossary

ACTA Applied Cognitive Task Analysis
AAIB Air Accident Investigation Branch
ALAR Approach and Landing Accident Reduction (tool kit)
ATC Air Traffic Control
ATPL Air Transport Pilot Licence
ATQP Alternative Training and Qualification Programme
ATSB Australian Transport Safety Bureau
CAA Civil Aviation Authority (United Kingdom)
CATII Category Three Approach
CPL Commercial Pilots Licence
CSV Comma Separated Variable (file format)
CTA Cognitive Task Analysis
DFT Discreet Fourier Transform
DME Distance Measuring Equipment
EADI Electronic Attitude Directional Indicator
EFIS Electronic Flight Instrument System
EHSI Electronic Horizontal Situation Indicator
FAR Federal Aviation Regulations
FDM Flight Data Monitoring
FDR Flight Data Recorder
FMGS Flight Management and Guidance System
FMS Flight Management System
FODCOM Flight Operations Department Communication
FOQA Flight Operations Quality Assurance
FSF Flight Safety Foundation
FTD Flight Training Device
HFACS Human Factors Analysis and Classification System
HTA Hierarchical Task Analysis
Hz Hertz
ILS Instrument Landing System
IMC Instrument Meteorological Conditions
IOS Instructors Operating Station
IPC Instrument Proficiency Check
IR Instrument Rating
JAA Joint Aviation Authorities
JOT Jet Orientation Training
LOFT Line Orientated Flight Training
LPC Licence Proficiency Check
LR Logistic Regression
ME Mean Error
MSC Mean Power Spectral Density
NG Next Generation
NTSB National Transportation Safety Board
OPC Operational Proficiency Check
PF Pilot Flying
8