Table Of ContentTHE SIMPLIFICATION OF COMBAT AIR CREW
CRITERIA WITH SPECIAL REFERENCE
TO RANK-ORDER RATINGS
DISSERTATION
Presented in P artial Fulfillm ent of the Requirements
for the Degree Doctor of Philosophy in the
Graduate School of The Ohio State
University
tt? By
a*
Mr FRANCES ESTEP, B. S., M. A.
The Ohio "state University
1951
Approved by:
Adviser
PREFACE
The Personnel Research Center of Wayne University is one of the
several university groups currently performing contract research for
m ilitary agencies. Dr. Roger M. Bellows, D irector of the Center,
has facilitated the development of th is research study through making
data available from the Center and especially through his personal,
considerate, and patient attention to the demands of the project.
Dr. Carroll L. Shartle of The Ohio State University has been
adviser to the project, contributing his knowledge of governmental
and m ilitary research to the effective guidance of this study.
Dr. Robert J. Wherry and Dr. John R. Kinzer were consulted on
certain phases of the research, and both have criticized an early
draft of the dissertation, offering suggestions for its improve
ment. Mrs. Katherine Dockeray performed numerous kindnesses for
the w riter while the project was being carried on.
Appreciation is expressed to Rutledge Jay and to John Weaver
of the Personnel Research Center for assistance on statistic al
d etails.
- x -
86250,3
TABLE OF CONTENTS
Page
PREFACE i
CHAPTER I CRITERIA OF COMBAT PROFICIENCY............................. 1
CHAPTER II DESCRIPTION OF THE DATA........................................... lfc
The Crew Ratings............................................................. 17
Fam iliarity with the Crews Ratings................... 17
Individual Ratings........................................................ 19
CHAPTER III STATISTICAL ANALYSIS OF THE RELIABILITY OF
THE RATERS' RATINGS................................................... 21
CHAPTER IV FACTORIAL COMPOSITION OF THE VARIABLES. . . 50
CHAPTER V ANALYSIS OF OTHER STATISTICAL RELATIONSHIPS 57
CHAPTER VI CONCLUSIONS WITH SUGGESTIONS FOR FUTURE
RESEARCH............................................................................... 78
BIBLIOGRAPHY ......................................... 8l
AUTOBIOGRAPHY ....................................................................................................... 85
- ii -
LIST OF FIGURE AMD TABLES
Page
SAMPLE 3" X 5" CARD USED FOR RANKINGS............................ 18
CORRELATIONS BETWEEN EACH RATER'S RATINGS ON ALL
VARIABLES............................................................................................. 2h
CORRELATIONS BETWEEN PAIRS OF RATERS ARRANGED IN
ORDER OF MAGNITUDE OF THE CORRELATION COEFFICIENTS 25
MATRICES OF INTERCORRELATIONS BETWEEN RATERS FOR
EACH OF NINE VARIABLES, VARIABLE 1 ................................. 27
MATRICES OF INTERCORRELATIONS BETWEEN RATERS FOR
EACH OF NINE VARIABLES, VARIABLE 2 ................................. 29
MATRICES OF INTERCORRELATIONS BETWEEN RATERS FOR
EACH OF NINE VARIABLES, VARIABLE 3 ................................. 30
MATRICES OF INTERCORRELATIONS BETWEEN RATERS FOR
EACH OF NINE VARIABLES, VARIABLE k ................................. 31
MATRICES OF INTERCORRELATIONS BETWEEN RATERS FOR
EACH OF NINE VARIABLES, VARIABLE 5 ................................. 32
MATRICES OF INTERCORRELATIONS BETWEEN RATERS FOR
EACH OF NINE VARIABLES, VARIABLE 6 ................................. 3^
MATRICES OF INTERCORRELATIONS BETWEEN RATERS FOR
EACH OF NINE VARIABLES, VARIABLE 7 ................................. 35
MATRICES OF INTERCORRELATIONS BETWEEN RATERS FOR
EACH OF NINE VARIABLES, VARIABLE 8 ................................. 36
MATRICES OF INTERCORRELATIONS BETWEEN RATERS FOR
EACH OF NINE VARIABLES, VARIABLE 9 ................................. 38
MATRIX OF INTERCORRELATIONS BETWEEN VARIABLES
FOR EACH RATER, Rater 1 — Group Commanding
O fficer................................................................................................. kl
MATRIX OF INTERCORRELATIONS BETWEEN VARIABLES
FOR EACH RATER, Rater 2 — Squadron Commander (a) k2
MATRIX OF INTERCORRELATIONS BETWEEN VARIABLES
FOR EACH RATER, Rater 3 — Squadron Commander (b) ^3
MATRIX OF INTERCORRELATIONS BETWEEN VARIABLES
FOR EACH RATER, Rater H — Squadron Commander (c) kk
- iii -
Page
TABLE XVI MATRIX OF INTERCORRELATIONS BETWEEN VARIABLES
FOR EACH RATER, Rater 5 — Group Executive
O fficer......................................................................................... li-5
TABLE XVII MATRIX OF INTERCORRELATIONS BETWEEN VARIABLES
FOR EACH RATER, Rater 6 -- Group Operations
O fficer.....................................................• . . ........................ 1+6
TABLE XVIII MATRIX OF INTERCORRELATIONS BETWEEN VARIABLES
FOR EACH RATER, Rater 7 — Group Bombardier
O fficer......................................................................................... 1+7
TABLE XIX AVERAGE RELIABILITY COEFFICIENTS ARRANGED IN
ORDER OF MAGNITUDE ON ALL VARIABLES FOR EACH
RATER............................................................................................. 1+9
TABLE XX CORRELATIONS BETWEEN NINE CRITERION VARIABLES . 51
TABLE XXI AVERAGE RELIABILITY OF THE NINE CRITERION
VARIABLES ARRANGED IN ORDER OF MAGNITUDE OF
THE RELIABILITY COEFFICIENT.......................................... 53
TABLE XXII CORRELATIONS BETWEEN RATINGS ON EACH VARIABLE
AND THE TOTAL RATINGS WITH THE INDIVIDUAL
VARIABLE TAKEN OUT................................................................. 55
TABLE XXIII JUDGED FAMILIARITY OF EACH RATER WITH CREWS
ON EACH VARIABLE, Variable 1.......................................... 58
TABLE XXIV JUDGED FAMILIARITY OF EACH RATER WITH CREWS
ON EACH VARIABLE, Variable 2 .......................................... 59
TABLE XXV JUDGED FAMILIARITY OF EACH RATER WITH CREWS
ON EACH VARIABLE, Variable 3.......................................... 59
TABLE XXVI JUDGED FAMILIARITY OF EACH RATER WITH CREWS
ON EACH VARIABLE, Variable k.......................................... 60
TABLE XXVIII JUDGED FAMILIARITY OF EACH RATER WITH CREWS
ON EACH VARIABLE, Variable 6.......................................... 60
TABLE XXIX JUDGED FAMILIARITY OF EACH RATER WITH CREWS
ON EACH VARIABLE, Variable 7.......................................... 61
TABLE XXX JUDGED FAMILIARITY OF EACH RATER WITH CREWS
ON EACH VARIABLE, Variable 8.......................................... 61
TABLE XXXI RELATION BETWEEN FAMILIARITY WITH A VARIABLE
AND GOODNESS OF RATINGS ON THAT VARIABLE. . . . 63
- iv -
Page
TABLE XXXII COMPARISON OF RATERS ON SEVERAL INDEXES OF
GOODNESS OF RATING............................................................. 65
TABLE XXXIII AVERAGE RATER RELIABILITY COEFFICIENTS FOR
EACH VARIABLE FOR INDIVIDUAL RATINGS OF FIVE
MEMBERS OF THE BOMB TEAM............................................... 67
TABLE XXXIV COMPARISON BETWEEN RATINGS GIVEN TO FIVE
MEMBERS OF THE BOMB TEAM AND THE RATINGS
GIVEN TO THE CREW OF WHICH THEY WERE A
MEMBER........................................................................................ 69
TABLE XXXV RANK OF EIGHT LEAD CREWS AND THEIR AIRCRAFT
COMMANDERS............................................................................... 71
TABLE XXXVI RATER RELIABILITIES ON OVER-ALL CREW RATINGS
FOR EACH VARIABLE................................................................. 72
TABLE XXXVII COMPARISON BETWEEN RELIABILITY OF THE OVER-ALL
CREW RATINGS ON EACH VARIABLE IN TWO LOCATIONS 7^
TABLE XXXVIII INTERCORRELATIONS BETWEEN NINE VARIABLES USING
SUM OF RANKS BY FIVE RATERS.......................................... 75
TABLE XXXIX RELATION BETWEEN RATINGS GIVEN TO THE CREW ON
VARIABLE 9 AND THE RATINGS GIVEN TO INDIVIDUAL
CREW MEMBERS ON NINE VARIABLES . . . ..................... 76
- v -
THE SIMPLIFICATION OF COMBAT AIR CREW CRITERIA WITH
SPECIAL REFERENCE TO RANK-ORDER RATINGS
CHAPTER I
CRITERIA OF COMBAT PROFICIENCY
Among the several areas of contribution by psychologists to the
m ilitary program are the selection, classification, placement, and
training of personnel. Effective testing of research in these areas
depends upon identification of the successful performance behavior to
be duplicated by future groups of personnel. Selection and placement
techniques, such as psychological tests and other measurements, must
demonstrate an acceptable level of validity to be recommended for use.
The end evaluation of such techniques and procedures is th eir relation
ship to the successful accomplishment of the m ilitary goal. This goal
usually implies success in combat situations, although there are selected
instances which involve only non-combat activity.
During World War I psychologists applied the meager techniques then
available to the special problems of m ilitary classification and place
ment. The inadequacy of the instruments seemed rooted in th eir failure
to predict successful and unsuccessful performers in the designated
tasks. In the absence of a definition of the kind of job proficiency
sought, validation of the predictive devices was not possible. C riteria
had not yet been the object of a sim ilar volume of research then going
forward on the construction of psychological te sts. Jenkins has de
scribed the history of psychologists' experience with validation thus:
The events of World War I taught American psychologists
the necessity of validation. The experience of the next
two decades taught them much about the technique of
validation. It remained for World War II to drive home
to the psychologists at large the necessity for devoting
- 1 -
2
- -
much time and thought to the basis for validation.!
During World War II and the years since then, evaluation of the criterion
itse lf has been considered prerequisite to later phases of validation
research. If criterion data of a known and acceptable level of
dependability are not available for the particular sample of subjects
to be studied, the research project founders before it begins.
In practice, the adequacy of criteria varies. There were some
attempts during World War II to obtain measures of proficiency under
combat conditions but the worth of the criteria obtained was found to be
negated by circumstances beyond the control of the research workers.
There was no assurance that criteria gathered under combat conditions
would be collected system atically because of the pressure of other
more urgent activity. C riteria labelled as combat criteria may have
been made for seme other purpose, such as adm inistrative action, and
were not necessarily appropriate for psychological research. If the
criteria were to be used to evaluate training procedures, there was
sometimes a considerable delay before combat criteria could be obtained
for use in validation. In the meantime, both training requirements and
definition of combat proficiency may have changed. Such an example arose
during the la tte r part of World War II when the United States shifted
from defensive to offensive warfare.
In lieu of combat criteria, more available measures were used
without knowledge of th eir relationship -to the ultim ate goals of the
m ilitary establishment. C riteria which were utilized within the United
-'-John G. Jenkins, V alidity for what? Journal of Consulting
Psychology, 19^6, 10, 93.
3
- -
States bore an unknown relationship to measures of success of men who
were under enemy fire or who were living under stress conditions.
In the search fo r a criterion to use for the evaluation of
psychological techniques for combat air crews the greatest problem
seems to be to find the one which is the best measure of combat
effectiveness. Sisson has pointed out the general unsuitability of
common industrial criteria for m ilitary personnel research.1 Combat
efficiency is not only hard to define but it is also d ifficu lt to
reduce to quantitative indexes. He cites five industrial criteria
commonly used, which have few counterparts in m ilitary work, (l) An
index of output per unit of time is conceivably possible to tabulate,
but it is doubtful whether the number of shots fired or the number of
bombs dropped is related to the job of m ilitary personnel; in any event,
it seems that there are most likely a rtific ia l lim its to the amount of
"production" an individual could turn out. (2) An index of the quality
of production: Sisson commented, "In the job of destroying the enemy's
capacity to wage war, the absurdity of speaking in such terms as number
of rejected products, cost of spoiled work, or customer complaints is
sufficiently evident to need no further elaboration."2 (3) An index of
employee turnover cannot be based on voluntary quits as would be possible
in the civilian commercial establishment. (4) An index of employee satis
faction, or rather dissatisfaction, is available in great quantities, but
without much u tility for criteria. (5) An index of train ab ility is a
possible criterion for m ilitary personnel research, and was commonly used
1E. D. Sisson. The criterion in Army personnel research. In
George A. KeHy (Ed.), Hew Methods in Applied Psychology. College park,
Md.: University of Maryland, 19^7• Pp. 17-18.
2Ibid., 17.
- h -
in early phases of Army personnel research.
There are, according to Flanagan, four main categories of combat
criteria: (l) objective measures of combat proficiency, (2) records
of adm inistrative actions in combat units, (3) ratings based on direct
and systematic observation of combat effectiveness, and (4) ratings based
on general impressions, reports, and incidental observations.1
In the firs t category of objective records of combat proficiency,
the most readily accessible criteria, the AAF has used records of bomb
ing accuracy as obtained from strike photographs, records of airplane
accidents related to combat operations, records of numbers of enemy
planes shot down, and records of combat casualties.
In the second category, records of adm inistrative actions in combat
units, several criteria are widely used because of availability in spite
of being based upon the subjective judgments of superior officers. The
adm inistrative actions include promotions, special awards and decorations,
reclassifications due to anxiety reactions or lack of combat proficiency,
special duty assignments, such as to lead crews or for important combat
missions, official ratings of efficiency while assigned to a combat unit.
The third category of criteria includes ratings based on direct and
systematic observation of combat effectiveness. The evaluations include
ratings of navigational proficiency based on a study of the logs prepared
by the navigator on various missions, ratings regarding the success or
failure of bombing missions based on direct observation of the results or
1 John' C. Flanagan (Ed.). The Aviation Psychology Program in
the Army Air Forces, AAF Aviation Psychology Program Research Report No. 1.
Washington, D. C.: U. S. Government Printing Office, 19V8. Pp. 276ff.
