The Empirical Selection of Anchor Items Using a Multistage Approach

117 Pages·2017·2.32 MB·English

UUnniivveerrssiittyy ooff SSoouutthh FFlloorriiddaa
DDiiggiittaall CCoommmmoonnss @@ UUnniivveerrssiittyy ooff
SSoouutthh FFlloorriiddaa
USF Tampa Graduate Theses and Dissertations USF Graduate Theses and Dissertations
June 2017
TThhee EEmmppiirriiccaall SSeelleeccttiioonn ooff AAnncchhoorr IItteemmss UUssiinngg aa MMuullttiissttaaggee
AApppprrooaacchh
Brandon Craig
University of South Florida, [email protected]
Follow this and additional works at: https://digitalcommons.usf.edu/etd
Part of the Educational Assessment, Evaluation, and Research Commons
SScchhoollaarr CCoommmmoonnss CCiittaattiioonn
Craig, Brandon, "The Empirical Selection of Anchor Items Using a Multistage Approach" (2017). USF
Tampa Graduate Theses and Dissertations.
https://digitalcommons.usf.edu/etd/6819
This Dissertation is brought to you for free and open access by the USF Graduate Theses and Dissertations at
Digital Commons @ University of South Florida. It has been accepted for inclusion in USF Tampa Graduate Theses
and Dissertations by an authorized administrator of Digital Commons @ University of South Florida. For more
information, please contact [email protected].
The Empirical Selection of Anchor Items Using a Multistage Approach

by

Brandon Craig

A dissertation submitted in partial fulfillment
of the requirements for the degree of
Doctor of Philosophy in Curriculum and Instruction
with an emphasis in
Measurement and Evaluation
Department of Educational and Psychological Studies
College of Education
University of South Florida

Major Professor: Eun Sook Kim, Ph.D.
John Ferron, Ph.D.
Jeffrey Kromrey, Ph.D.
Kristine Hogarty, Ph.D.

Date of Approval:
June 13, 2017

Keywords: DIF, differential item functioning, PROC IRT, Rasch

Copyright © 2017, Brandon Craig
TABLE OF CONTENTS

List of Tables ................................................................................................................................ iv

List of Figures ................................................................................................................................ vi

Abstract ......................................................................................................................................... vii

Chapter One: Introduction .............................................................................................................. 1
Summary ............................................................................................................................. 5
Problem ................................................................................................................... 5
Purpose .................................................................................................................... 5
Hypotheses .............................................................................................................. 6
Questions................................................................................................................. 6
Definitions of Frequently Used Terms ............................................................................... 6

Chapter Two: Literature Review .................................................................................................... 9
Overview ............................................................................................................................. 9
Differential Item Functioning (DIF) ................................................................................... 9
What is DIF? ........................................................................................................... 9
Why is DIF important? ......................................................................................... 13
When is DIF important?........................................................................................ 14
DIF from a Test Developer’s Prospective ............................................................ 16
Detecting DIF.................................................................................................................... 17
Mantel-Haenszel ................................................................................................... 17
Logistic Regression ............................................................................................... 19
Multiple Indicators, Multiple Causes Models ....................................................... 21
Item Response Theory .......................................................................................... 22
Anchor Items ..................................................................................................................... 25
Framework ............................................................................................................ 26
Anchor class .............................................................................................. 26
Anchor selection strategy .......................................................................... 27
Anchor method.......................................................................................... 28
Previously Studied Methods ................................................................................. 29
Iterative scale purification class ................................................................ 29
Constant anchor length class ..................................................................... 31
Anchor length............................................................................................ 31
AO vs. SA ................................................................................................. 31
Test statistic used ...................................................................................... 32
Need for Further Research .................................................................................... 33
i
Proposed Methods ................................................................................................. 34
Methodological Details of Prior Studies ........................................................................... 35
Sample Size ........................................................................................................... 36
Mean Group Difference ........................................................................................ 37
Number of Items on the Test ................................................................................ 37
Percentage of DIF on the Test .............................................................................. 39
Magnitude of DIF ................................................................................................. 39
Type of DIF........................................................................................................... 40
Balance of DIF ...................................................................................................... 41
Models Used to Generate Data ............................................................................. 41
Number of Replications ........................................................................................ 42

Chapter 3: Study Design ............................................................................................................... 43
Determining MPT and MTT ............................................................................................. 44
Anchor Selection Methods ................................................................................................ 44
C4-SA(MPT)......................................................................................................... 44
IF-SA(MTT) ......................................................................................................... 45
MS[C4-SA(MPT)] ................................................................................................ 47
MS[IF-SA(MTT)] ................................................................................................. 48
DIF-free Anchors .............................................................................................................. 49
Data Generation ................................................................................................................ 49
Statistical Software ............................................................................................... 49
Model Used to Generate Data ............................................................................... 50
Mean Group Difference ........................................................................................ 50
Test Length ........................................................................................................... 50
Item Parameters .................................................................................................... 51
DIF Magnitude ...................................................................................................... 51
Manipulated Variables ...................................................................................................... 52
Sample Size ........................................................................................................... 52
Percentage of DIF ................................................................................................. 52
Balance of DIF ...................................................................................................... 52
Number of Replications .................................................................................................... 53
Outcomes .......................................................................................................................... 53
False Positive Rate ................................................................................................ 54
True Positive Rate ................................................................................................. 54
Familywise False Positive Rate ............................................................................ 54
Anchor Contamination Rate ................................................................................. 54
Familywise Anchor Contamination Rate .............................................................. 54
Changes to Kopf et al.’s (2015b) Anchor Methods .......................................................... 55

Chapter 4: Results ......................................................................................................................... 58
True Positive Rates ........................................................................................................... 58
False Positive Rates .......................................................................................................... 59
Familywise False Positive Rates ....................................................................................... 62
Anchor Contamination Rates ............................................................................................ 62
Familywise Anchor Contamination Rates ........................................................................ 63
ii
Observed Anchor Lengths for IF Selection Methods ..................................................... 634

Chapter 5: Discussion and Conclusions ........................................................................................ 66
Discussion ......................................................................................................................... 66
Hypotheses ............................................................................................................ 66
Questions............................................................................................................... 66
Hypothesis 1.......................................................................................................... 67
Hypothesis 2.......................................................................................................... 67
Hypothesis 3.......................................................................................................... 68
Question 1 ............................................................................................................. 69
Question 2 ............................................................................................................. 71
Limitations ........................................................................................................................ 71
Conclusions ....................................................................................................................... 72
Recommendations for Further Research ........................................................................... 73

References ..................................................................................................................................... 75

Appendix A: Code for Data Generation ....................................................................................... 85

Appendix B: Code to Apply Anchor Selection Methods.............................................................. 90

Appendix C: Code to Apply DIF-free Anchors .......................................................................... 104

Appendix D: IRB Exemption Letter ........................................................................................... 107

About the Author ........................................................................................................................ 108

iii
LIST OF TABLES

Table 1: Contingency table for detecting differential item functioning in a dichotomous
item using Mantel-Haenszel ........................................................................................18

Table 2: Contingency table for detecting differential item functioning in a polytomous
item using generalized Mantel-Haenszel .....................................................................18

Table 3: Anchor class abbreviations and descriptions ...............................................................27

Table 4: Anchor selection strategies and descriptions ...............................................................28

Table 5: Study design for selected variables in reviewed studies ..............................................38

Table 6: Anchor lengths for IF-DIF-free anchor by percentage of DIF ....................................49

Table 7: Difficulty parameters used by Kopf et al. (2015b) and Wang et al. (2012) ................51
.
Table 8: Anchor lengths from applying different stopping criteria and anchor item
specifications on a five item test with 40% differential item functioning ...................56

Table 9: True positive rates by sample size, percentage of DIF, balance of DIF, and
anchor method ..............................................................................................................59

Table 10: False positive rates by sample size, percentage of DIF, balance of DIF, and
anchor method .............................................................................................................60

Table 11: Familywise false positive rates by sample size, percentage of DIF, balance of
DIF, and anchor method .............................................................................................61

Table 12: Anchor contamination by sample size, percentage of DIF, balance of DIF, and
anchor method .............................................................................................................62

Table 13: Familywise anchor contamination by sample size, percentage of DIF, balance
of DIF, and anchor method .........................................................................................63

Table 14: Mean anchor length by sample size, percentage of DIF, balance of DIF, and anchor
method .........................................................................................................................64

iv
Table 15: Minimum anchor length by sample size, percentage of DIF, balance of DIF, and anchor
method .........................................................................................................................65

v
LIST OF FIGURES

Figure 1: Uniform differential item functioning for a dichotomously scored item ...................11

Figure 2: Non-uniform differential item functioning for a dichotomously scored item ............12

Figure 3: Uniform differential item functioning for a four-level ordinal response item ...........12

vi
ABSTRACT
The purpose of this study was to determine if using a multistage approach for the
empirical selection of anchor items would lead to more accurate DIF detection rates than the
anchor selection methods proposed by Kopf, Zeileis, & Strobl (2015b). A simulation study was
conducted in which the sample size, percentage of DIF, and balance of DIF were manipulated.
The outcomes of interest were true positive rates, false positive rates, familywise false positive
rates, anchor contamination rates, and familywise anchor contamination rates. Results showed
the proposed multistage methods produced lower anchor contamination rates than the non-
multistage methods under some conditions, but there were generally no meaningful differences
in true positive and false positive rates.
vii
CHAPTER ONE:
INTRODUCTION
The Standards for Education and Psychological Testing describes bias as “construct-
irrelevant components that result in systematically lower or higher scores for identifiable groups
of examinees” (AERA, APA, & NCME, 1999, p. 76). One source of bias is when an item
performs differently for two groups of test takers after controlling for the construct ability of the
test takers. This source of bias is labeled differential item functioning (DIF). When a test
contains items with DIF, the results from the test may be biased, resulting in inaccurate estimates
of the test takers’ ability levels and compromising any conclusions that can be inferred from the
results of the test (Hidalgo, Galindo-Garre, & Gómez-Benito, 2015; Li & Zumbo, 2009).
Several techniques have been developed or used to detect DIF including Mantel-Haenszel
(Holland & Thayer, 1986), logistic regression (Rogers & Swaminathan, 1993), structural
equation modeling (Finch, 2005), and item response theory (Wang, 2004). One essential
component of these techniques is controlling for the construct ability of the test taker. However,
accurately estimating the construct ability can be difficult. Generally, the researcher does not
have a priori knowledge of the test taker’s ability level and must use the same test being
investigated for DIF to also estimate the construct ability of the test taker. This approach to
investigating DIF creates a problem. The researcher cannot accurately estimate the construct
ability of the examinees if the test contains items with DIF; however, the researcher cannot
1