Comparison of test directions for ability tests

188 Pages·2016·2.12 MB·English

University of Iowa
Iowa Research Online
Theses and Dissertations
Spring 2010
Comparison of test directions for ability tests:
impact on young English-language learner and non-
ELL students
Joni Marie Lakin
University of Iowa
Copyright 2010 Joni Marie Lakin
This dissertation is available at Iowa Research Online: https://ir.uiowa.edu/etd/536
Recommended Citation
Lakin, Joni Marie. "Comparison of test directions for ability tests: impact on young English-language learner and non-ELL students."
PhD (Doctor of Philosophy) thesis, University of Iowa, 2010.
https://doi.org/10.17077/etd.nt67qao4.
Follow this and additional works at:https://ir.uiowa.edu/etd
Part of theEducational Psychology Commons
COMPARISON OF TEST DIRECTIONS FOR ABILITY TESTS: IMPACT ON
YOUNG ENGLISH-LANGUAGE LEARNER AND NON-ELL STUDENTS

by
Joni Marie Lakin

An Abstract

Of a thesis submitted in partial fulfillment
of the requirements for the Doctor of
Philosophy degree in Psychological and Quantitative Foundations
in the Graduate College of
The University of Iowa

May 2010

Thesis Supervisor: Professor David F. Lohman
1
ABSTRACT
Ability tests play an important role in the assessment programs of many schools.
However, the inferences about ability made from such tests presume that students understand the
tasks they are attempting. Task familiarity can vary by student as well as by format. By design,
nonverbal reasoning tests use formats that are intended to be novel. The popularity of nonverbal
reasoning tests has increased substantially in recent years because of the increasing number of
English-language learner (ELL) students in many U.S. school districts. Nonverbal tests are
thought to eliminate the need for language in test items and to reduce cultural content. Formats on
these tests are also assumed to be equally novel for all students. However, in at least one large
study, researchers found substantial differences between the average performance of ELL and
non-ELL Hispanic students on three of the most widely used nonverbal tests. Although these
differences might reflect real variation in cognitive development, they may also reflect
differences in knowledge of test formats and the testing practices used in U.S. schools.
In this study, I hypothesized that the score gaps between ELL and non-ELL students
might, in part, be due to differences in test familiarity and that providing directions that include
more practice and feedback might attenuate these differences. I drew from the research on
universal design, dynamic assessment, and cross-cultural testing to develop three different types
of directions with practice items. I then compared the effects of these three types of test directions
on students completing a nonverbal figure analogies test. Figure analogies tests are generally
among the best measures of reasoning abilities and are known to be quite difficult for young
students. All directions were provided using video with English and Spanish audio and minor
animations to concretize the instructions. The three types of directions were nonverbal-dynamic
directions, verbal-dynamic directions, and a control condition that used standard test directions.
The nonverbal-dynamic directions presented four practice problems that sampled the range of
items on the test. Oral instructions and feedback were minimal. The verbal-dynamic directions
presented the same four practice problems with more in-depth description and feedback. These
2
directions also described useful strategies for solving items. The standard test directions presented
two sample problems with minimal instruction and feedback.
The sample consisted of 882 students in 40 first- and second-grade classrooms in 8
schools. A hierarchical linear model was used to control for similarity among students nested in
classrooms and schools and to account for the assignment of treatment (type of directions) at the
classroom level. The model included tests for main effects and interactions among treatment,
ELL status, and grade. Results indicated that providing additional practice (the nonverbal-
dynamic directions) led to small gains in performance, but that the more extensive set of
directions (verbal-dynamic directions) were effective only for high-ability students. Contrary to
the hypotheses, there was no interaction of ELL status with treatment. An unexpected finding was
that use of teacher-read directions instead of video-based directions led to better performance for
second-grade students. I conclude that test directions are an important means for improving test
familiarity in young students, but that excessive standardization and lengthening of the directions
may hinder performance. I also conclude that the choice of practice items and feedback are
crucial considerations in the design of test directions.
Abstract Approved: _______________________________________
Thesis Supervisor
_______________________________________
Title and Department
_______________________________________
Date
COMPARISON OF TEST DIRECTIONS FOR ABILITY TESTS: IMPACT ON
YOUNG ENGLISH-LANGUAGE LEARNER AND NON-ELL STUDENTS
by
Joni Marie Lakin
A thesis submitted in partial fulfillment
of the requirements for the Doctor of
Philosophy degree in Psychological and Quantitative Foundations
in the Graduate College of
The University of Iowa
May 2010
Thesis Supervisor: Professor David F. Lohman
Graduate College
The University of Iowa
Iowa City, Iowa
CERTIFICATE OF APPROVAL
_______________________
PH.D. THESIS
_______________
This is to certify that the Ph.D. thesis of
Joni Marie Lakin
has been approved by the Examining Committee
for the thesis requirement for the Doctor of Philosophy
degree in Psychological and Quantitative Foundations at the May 2010
graduation.
Thesis Committee: ___________________________________
David F. Lohman, Thesis Supervisor
___________________________________
Stephen B. Dunbar
___________________________________
Judith E. Liskin-Gasparro
___________________________________
Walter P. Vispoel
___________________________________
Catherine Welch
To my granddaddy, W. M. (Mac) Schneider, for always being proud of me.
And to my parents, for nature and nurture.
ii
ACKNOWLEDGMENTS
I am sincerely grateful to my advisor, Dave Lohman, in particular for supporting
this research study and in general for six years of patient mentoring. I am also grateful to
my dissertation committee for their helpful feedback in developing this study.
I am also grateful to the many students, teachers, and administrators who
participated in this study. In particular, I owe a great debt to Cindy G., Gifted and
Talented Coordinator in the participating school district, for her incredible generosity of
time and energy. And I am grateful to the two translators who assisted with this study—
Antonio Heras and Angelique Dwyer—for sharing their expertise.
Several other individuals must be thanked for getting me through the Ph.D.
process. Margaret Beier helped me get into graduate school in the first place and
continues to be an invaluable mentor. Katrina Korb was always two steps ahead in the
Ph.D. program, so I never had to figure out the bureaucracy on my own. Emily Lai and
Jane Gressang have been my constant companions through the dissertation process. And,
finally, Kathy Schuh has been an unofficial advisor for the past five years. I am eternally
grateful to each of these people for their kindness and ready advice.
Finally, I am grateful to my family. None of this would have been possible
without the support of my parents, sisters, and Daniel. Thanks for letting me do what I
want to do.
iii
ABSTRACT
Ability tests play an important role in the assessment programs of many schools.
However, the inferences about ability made from such tests presume that students
understand the tasks they are attempting. Task familiarity can vary by student as well as
by format. By design, nonverbal reasoning tests use formats that are intended to be novel.
The popularity of nonverbal reasoning tests has increased substantially in recent years
because of the increasing number of English-language learner (ELL) students in many
U.S. school districts. Nonverbal tests are thought to eliminate the need for language in
test items and to reduce cultural content. Formats on these tests are also assumed to be
equally novel for all students. However, in at least one large study, researchers found
substantial differences between the average performance of ELL and non-ELL Hispanic
students on three of the most widely used nonverbal tests. Although these differences
might reflect real variation in cognitive development, they may also reflect differences in
knowledge of test formats and the testing practices used in U.S. schools.
In this study, I hypothesized that the score gaps between ELL and non-ELL
students might, in part, be due to differences in test familiarity and that providing
directions that include more practice and feedback might attenuate these differences. I
drew from the research on universal design, dynamic assessment, and cross-cultural
testing to develop three different types of directions with practice items. I then compared
the effects of these three types of test directions on students completing a nonverbal
figure analogies test. Figure analogies tests are generally among the best measures of
reasoning abilities and are known to be quite difficult for young students. All directions
were provided using video with English and Spanish audio and minor animations to
concretize the instructions. The three types of directions were nonverbal-dynamic
directions, verbal-dynamic directions, and a control condition that used standard test
directions. The nonverbal-dynamic directions presented four practice problems that
iv
sampled the range of items on the test. Oral instructions and feedback were minimal. The
verbal-dynamic directions presented the same four practice problems with more in-depth
description and feedback. These directions also described useful strategies for solving
items. The standard test directions presented two sample problems with minimal
instruction and feedback.
The sample consisted of 882 students in 40 first- and second-grade classrooms in
8 schools. A hierarchical linear model was used to control for similarity among students
nested in classrooms and schools and to account for the assignment of treatment (type of
directions) at the classroom level. The model included tests for main effects and
interactions among treatment, ELL status, and grade. Results indicated that providing
additional practice (the nonverbal-dynamic directions) led to small gains in performance,
but that the more extensive set of directions (verbal-dynamic directions) were effective
only for high-ability students. Contrary to the hypotheses, there was no interaction of
ELL status with treatment. An unexpected finding was that use of teacher-read directions
instead of video-based directions led to better performance for second-grade students. I
conclude that test directions are an important means for improving test familiarity in
young students, but that excessive standardization and lengthening of the directions may
hinder performance. I also conclude that the choice of practice items and feedback are
crucial considerations in the design of test directions.
v