Table Of ContentINFORMATION TO USERS
This material was produced from a microfilm copy of the original document. While
the most advanced technological means to photograph and reproduce this document
have been used, the quality is heavily dependent upon the quality of the original
submitted.
The following explanation of techniques is provided to help you understand
markings or patterns which may appear on this reproduction.
1.The sign or "target" for pages apparently lacking from the document
photographed is "Missing Page(s)". If it was possible to obtain the missing
page(s) or section, they are spliced into the film along with adjacent pages.
This may have necessitated cutting thru an image and duplicating adjacent
pages to insure you complete continuity.
2. When an image on the film is obliterated with a large round black mark, it
is an indication that the photographer suspected that the copy may have
moved during exposure and thus cause a blurred image. You will find a
good image of the page in the adjacent frame.
3. When a map, drawing or chart, etc., was part of the material being
photographed the photographer followed a definite method in
"sectioning" the material. It is customary to begin photoing at the upper
left hand corner of a large sheet and to continue photoing from left to
right in equal sections with a small overlap. If necessary, sectioning is
continued again — beginning below the first row and continuing on until
complete.
4. The majority of users indicate that the textual content is of greatest value,
however, a somewhat higher quality reproduction could be made from
"photographs" if essential to the understanding of the dissertation. Silver
prints of "photographs" may be ordered at additional charge by writing
the Order Department, giving the catalog number, title, author and
specific pages you wish reproduced.
5. PLEASE NOTE: Some pages may have indistinct print. Filmed as
received.
Xerox University Microfilms
300 North Zeeb Road
Ann Arbor, Michigan 48106
LD390T 13-22112.
.G7
/
19^2 Huddleston, Edith Mary, 1918-
,H9 Measurement of writing ability at
the college-entrance level: objective
vs.. subjective testing techniques.
103p. form s,tables.
Thesis (Ph.D.) - N.Y.U., Graduate
School, 1952.
Bibliography: p .77-81.
C98314-6
Shelf Lf;,f
Xerox University Microfilms, Ann Arbor, Michigan 48106
THIS DISSERTATION HAS BEEN MICROFILMED EXACTLY AS RECEIVED.
roRK romsRsiT,
nNlvUtii’y heights
NEW YORK UNIVERSITY
MEASUREMENT OF WRITING ABILITY AT
THE COLLEGE-ENTRANCE LEVEL:
OBJECTIVE VS. SUBJECTIVE
TESTING TECHNIQUES
EDITH M. HUDDLESTON
A dissertation
in the Department of Psychology-
submitted to the faculty of the
Graduate School of Arts and Science
in partial fulfillm ent of the
requirements for the degree of
Doctor of Philosophy
April 1, 1952
ACKNOWLEDGMENTS
Without the capable and generous cooperation of a
number of interested persons the present study would have
been impossible. Dr. Scott B. Elledge, now Associate
Professor of English at Carleton College, worked closely
with the w riter in preparing and criticizing test m ateri
als and in developing the criterion data; his judgment
and insight, based on long experience in evaluation in
English composition, were an important contribution to
the character of the basic variables to be studied. For
technical consultation the w riter is indebted to the fol
lowing psychologists: Dr. Ralph W. Gilbert and Dr. Naomi
Stewart, the w riter's present and former advisers on the
faculty of the Graduate School of Arts and Science at New
York University; Dr. Ledyard R Tucker and Dr. William W.
Turnbull of the Educational Testing Service. Indebtedness
to all of the above is most gratefully acknowledged, as
well as to the many teachers and instructors whose enthusi
astic participation in furnishing criterion data consti
tuted the backbone of the study.
Edith M. Huddleston
Xoqt
it><]
TABLE OF CONTENTS
Page
A. Background of the Problem: A Survey of the Literature U
B. Purpose of the Investigation: To Determine the Value of
Objective and Subjective Testing Techniques in Measuring
A bility in English Composition 15
C. The General Study Plan: Use of Secondary School and Col
lege Freshman Classes for which Predictor and Criterion
Measures Can Be Obtained 17
D. Study I: Preliminary Investigation Involving Tryout of
Testing M aterial
(a) The Test Population 19
(b) Description of Variables 21
(c) Results 29
(d) Conclusions U8
E. Study II: Investigation Based on Study I, Designed to
Provide a More Definitive Answer to the Problem
(a) The Test Population 50
(b) Description of Variables 50
(c) Results 5U
(d) Conclusions 75
F. General Conclusions 76
G. Bibliography 77
Appendix I: Sample Questions in Objective English 82
Appendix II: Sample Verbal Items 86
Appendix III: Manual for the Grading of Style in the English
Composition Test 90
MEASUREMENT OF WRITING ABILITY AT THE COLLEGE-ENTRANCE
LEVEL: OBJECTIVE VS. SUBJECTIVE TESTING TECHNIQUES
A. Background of the Problem
One of the most fundamental controversies in the history of psy
chometrics has been the debate on the relative merits of objective vs.
subjective measuring techniques. The subjective, free-answer tests
have always had the advantage of apparent or "face" validity: the tra
ditional essay examinations are of the work-sample type, and require
the examinee independently to summon and organize his relevant knowledge.
Thus the essay test has been thought of as a "natural" task, allowing a
direct approach to important goals. But as far back as the eighteen-
r,
eighties it was realized that this otherwise ideal testing device was
beset with the p itfa ll of unreliability (17). Early studies (l!i, 5U)
showed that there was considerable discrepancy among teachers' marks,
and a vast array of later studies confirm this fact. In 192? Ruch pro
vided a table summarizing 285 coefficients of reliab ility for essay ex
aminations, with a median reliab ility of .59 (UU, p. 107). This corrob
orates the experience of later investigators. In 191*7 Adkins (1, p. 6)
wrote: "Essay tests, no matter what their merits may be, are commonly
considered im practical if the number of subjects is at all sizable, be
cause of the great difficulty in scoring them reliably and because of
the time required to score them." A review of the literature indicates
that the unreliability of essay examinations is most pronounced in the
area of English composition, while efforts to improve reliab ility for
achievement examinations covering specific content (e.g ., history or
physics) have been somewhat more successful. Stalnaker (3l*) who pro
5
posed an ••analytical" method of scoring essay tests as a means of rais
ing reliab ility , reported that for English composition the method is
unsatisfactory.
The objective test has its origins in this demonstrated need for
reliab ility . With the objective test the problem of "reader reliability"
(agreement of readers with one another) disappears, and the scoring of
the test is simply a matter of clerical accuracy. There s till remains
an error of measurement due to imperfect test reliab ility but it has
been demonstrated that the objective test is generally much more reli
able. A good objective test has a reliab ility of at least .8$, and fre
quently of .90 or more.
H istorically, however, the objective te st has been severely c rit
icized on the grounds that it presents the examinee with a task which is
artificia lly oversim plified. It has been charged that the examinee is
inadequately measured when he is required merely to choose his answer
from among a number of answer-choices which are set down for him. The
proponents of objective tests have been challenged to show that such
measuring devices can be inherently valid in spite of the apparent pre
digestion of much of the subject-m atter to be tested.
To a large extent this challenge has been satisfactorily met.
Objective-type questions have been developed which .do require thought
and organization on the part of the examinee; statistical analysis has
indicated that ab ilities tested by objective tests are frequently simi
lar to those which essay examinations aim to measure; in many cases
objective tests have been found superior because they can sample a
wider area of subject-m atter in a given length of time; and inevitably
6
the objective tests can be more reliably scored.
The testing of ab ility in English composition, however, is one
area in -which the dilemma has not been so easily resolved. The difficul
ties have been summarized in an article by Greene (2l)) in which he empha
sized the need for improvement in the validity and reliab ility of compo
sition scales of the essay type, but expressed unwillingness to accept
objective measurement as an alternative because most objective tests em
phasize mechanical skills at the expense of style and quality, and be
cause pupils may "respond correctly in objective tests to items which they
do not use correctly in their own expression."
Greene's view is typical of many others. Yet there has thus far
appeared no evidence to indicate that adequate essay tests can be devised.
While reader reliab ilities of .85 or higher have occasionally been re
ported (2, 12, 39, U8, 50, 57), the preponderance of the evidence indicates
that this is exceptional. The successes reported by Noyes (39) and by
Stalnaker (1*8, 50) are to be weighed against their contrary experience
(1*0, 1*1, 51, 52) and considered in the light of their conclusions, after
years of working with the problem, that no satisfactory solution had
been attained (3U, 1)0, 52). Likewise, the reader reliab ilities of .85
and .91) reported by Traxler and Anderson (57) are m itigated by the facts
that only two readers were involved and that the intercorrelation be
tween the two comparable essay tests was only .60. Traxler and Anderson
explained that "reliab ility of pupil performance is relatively low,"
but the hypothesis can be entertained that the readers may have reached
their highly consistent scores by ignoring certain aspects of pupil per
formance which should have been evaluated.
Probably there is no fund of experience in the field -which is
more extensive than that of the College Entrance Examination Board,
■which for years has put its best resources into the development of its
English Composition test; yet in the Board's annual report for 191*6
(10, p. 7) the executive secretary stated that "the problems involved
in developing a reliable essay examination are, if not unsolvable, at
least far from solved at the present time." For a long paper on a
single topic, the Board readers achieved a reader reliab ility of .55
•which is, as Stalnaker said, "about the same as the relationship be
tween height and weight" (52). Six other themes (1*0) were read with
reliab ilities, respectively, of .67, .66, .83, .69, .58, and .59.
Further unpublished data in the Board's files shows that for nine one-
hour examinations consisting of three or four short essay questions
each the reader reliab ilities for total score ranged from .68 to .89.
Reader Correlation with
Test R eliability S.A.T. Verbal
September 19l*l* .89
December 191*1* .85 .67
April 191*5 .69 .55
June 191*5 .81
September 191*5 .88
December 191*5 .82 .61
April 191*6 .68 .■LT\co
June 191*6 .69 .55
December 191*6 .68 .57
Note that these figures represent "reader" reliab ilities only; when the
(unknown) test unreliability is also taken into account, the to tal un
reliab ility is even greater. It is interesting to observe that the
correlations with the Verbal score of the Scholastic Aptitude Test were
almost as high as the reader reliab ilities.
Intercorrelations among the Board's essay questions tended to be
low. In view of the fact that the questions were designed to measure
comparable ab ilities, the low intercorrelations lead one to suspect that
the true test reliab ility was considerably lower than the reader re li
ability. The following data from the Board1s files refer^ to the essay
tests of September and December 19l*l* when the reader reliab ilities were
comparatively high. (The reader reliab ilities are shown in the diago
nals .)
Intercorrelations Among Questions
1 2 3 U
Question 1 (.81) .51* .1*9 .52
September
191*1* Question 2 (.82) .50 *56
Question 3 (.75) .53
Question 1* (.75)
1 2 3 1*
Question 1 (.80) .51* .1*8 .1*5
Question 2 (.66) .53 .1*1
Question 3 (.66) .39
Question 1* (.79)
The available evidence indicates that the experience of others
corroborates that of the Board. Starch and E lliott (51*) had ll*2 high
school English teachers each mark two compositions, and found varia-
