Item Pocket Method to Allow Response Changing Item Pocket Method to Allow Response Review ...

34 Pages·2012·0.31 MB·English

Item Pocket Method to Allow Response Changing 1

Running head: Item Pocket Method to Allow Response Changing

Item Pocket Method to Allow Response Review and Change
in Computerized Adaptive Testing1

Kyung T. Han
Graduate Management Admission Council®

Correspondence may be sent to:
Kyung T. Han
Graduate Management Admission Council®
11921 Freedom Dr. Suite 300, Reston, VA 20190 USA
[email protected]
(Phone) [phone]
(Fax) [phone]

The views and opinions expressed in this article are those of the author and do not necessarily
reflect those of the Graduate Management Admission Council®.

1 Acknowledgements: The author wishes to thank Lawrence M Rudner, Fanmin Guo, and Eileen Talento-Miller of
Graduate Management Admission Council® (GMAC®) for their feedbacks and support. The author also is grateful to
Ernie Anastasio of GMAC®, Hua-Hua Chang of University of Illinois, Wim J. van der Linden of CTB/McGraw-Hill,
and Paula Bruggeman of GMAC® for review and valuable comments.
This is a revision of the original study that was presented at 2011 annual meeting of the National Council on
Measurement in Education (NCME). The paper received the Alicia Cascallar Award for Outstanding Paper by Early
Career Scholar from NCME.
Item Pocket Method to Allow Response Changing 2

Abstract

Most computerized adaptive testing (CAT) programs do not allow test takers to review and
change their responses because it could seriously deteriorate the efficiency of measurement and
make tests vulnerable to manipulative test-taking strategies. Several modified testing methods
have been developed that provide restricted review options while limiting the trade-off in CAT
efficiency. The extent to which these methods provided test takers with options to review test
items, however, still was quite limited. This study proposes the item pocket (IP) method, a new
testing approach that allows greater flexibility in changing responses, eliminating unnecessary
restrictions that prevent test takers from moving across test sections to review their answers. A
series of simulations were conducted to evaluate the robustness of the IP method against various
manipulative test-taking strategies. Findings and implications of the study suggest that the IP
method could be an effective solution for many CAT programs when the IP size and test time
limit are properly set.  
Keywords: computerized adaptive testing, test construction, test administration, response change
Item Pocket Method to Allow Response Changing 3

Item Pocket Method to Allow Response Review and Change
in Computerized Adaptive Testing

Computerized adaptive testing (CAT) is rapidly becoming a popular choice for test
administration, often favored by test developers and test takers alike (Poggio, 2010; Martineau &
Dean, 2010) because it delivers more accurate score estimates and requires relatively less testing
time than conventional paper-and-pencil-based tests (PBTs). Due to the item selection algorithm
that CAT programs employ, however, CAT usually does not allow test takers to review test
items and/or change their responses as they can on PBTs, since it relies on an interim score
estimate that is updated after each item administration. Previous research has shown that having
an opportunity to change their responses can help test takers reduce their anxiety and stress
levels during test administration (Wise, 1996; Stocking, 1997; Lunz, Bergstrom, & Wright,
1992; Papanastasiou, 2002), which can result in fewer mistakes made during testing and test
scores that may more accurately reflect test takers’ true proficiency (Papanastasiou, 2002). This
especially can be the case when the test is high stakes (Stocking, 1997) and/or tightly timed. Test
takers almost always prefer to have the option to change responses even if they do not always
exercise that option (Vispoel, Henderickson, & Bleiler, 2000). More importantly, research shows
that only a fraction of test takers actually benefit from response changing in terms of score
improvement (Benjamin, Cavell, & Shallenberger, 1984; Waddell & Blankenship, 1995; Wise,
1996).
Eliminating unnecessary causes of a test taker’s anxiety and stress during CAT
administration is an important step toward creating an ideal test environment. Reducing test
takers’ anxiety and stress by giving them sufficient control over the test administration is
Item Pocket Method to Allow Response Changing 4
critically important because a high level of test anxiety can contribute to increased measurement
errors: Providing test takers with a better testing experience is, educationally and morally, the
right thing to do.

Manipulative Test-Taking Strategies
One of the main practical objections to allowing for response changing in CAT programs
is that imprudently allowing test takers to change responses could open the door to systematic
test-taking strategies that could render CAT administration less efficient (Wainer, 1993; Wise,
1996) and/or result in biased score estimates (Vispoel, Rocklin, Wang, & Bleiler, 1999).
Wainer Strategy
For the manipulative test-taking strategy that Wainer (1993) introduced, a test taker could
intentionally answer all items incorrectly on the first round to make the CAT system administer
only easy items. After that, the test taker could go back to each item to review and change his or
her responses to get a perfect score on the second round. Simulation studies confirmed that this
so-called “Wainer strategy” could result in extremely large measurement errors with a
considerable risk for positively biased score estimates, especially for high-proficiency test takers
who are capable of implementing the strategy successfully (Wang & Wingersky, 1992; Gershon
& Bergstrom, 1995; Stocking, 1997; Vispoel et al., 1999; Bowles & Pommerich, 2001). In
practice, the Wainer strategy is difficult to implement successfully—test takers run the risk of
getting significantly underestimated scores if they fail to respond to all items correctly on the
second round (Wise, 1996). Notwithstanding the risks, studies with live data showed that high-
proficiency test takers had a good chance to profit from the Wainer strategy (Vispoel et al., 1999;
Wise, 1996).
Item Pocket Method to Allow Response Changing 5
Kingsbury Strategy
Another manipulative test-taking strategy for CAT involves examinee judgment on item
difficulty. As pointed out by Green, Bock, Humphreys, Linn, and Reckase (1984), test takers,
knowing that the current item’s difficulty depends heavily on the response made to the previous
item, could use the difficulty of the current item as a clue to the correctness of their response for
the earlier item. If a test taker thinks the current item’s difficulty is slightly higher than the
previous item, it would be reasonable for the test taker to think his or her answer for the previous
item was correct. On the other hand, if the current item seems easier than the previous item, it
may indicate that the previous item was answered incorrectly, and thus the test taker can go back
to the previous item and change the answer. In his simulation study, Kingsbury (1996) modeled
such a test-taking strategy based on two strong assumptions: (a) test takers were assumed to
make guesses if they saw an item whose difficulty was higher than their true proficiency by 1.0
theta unit or more, and (b) test takers also were assumed to go back to previous item and change
their responses if the next item’s difficulty was lower than the previous item by 0.5 theta unit or
more. The simulation result found that examinees could benefit from this test-taking strategy
depending on their proficiency level—the lower the examinees’ proficiency levels, the greater
the possible benefits they could see.
Generalized Kingsbury Strategy
Wise, Finney, Enders, Freeman, and Severance (1999) expanded Kingsbury’s test-taking
strategy so that test takers were assumed to have speculated on the difficulty level of the next
item not only for items with guessed responses, but for all previous items. Wise et al. (1999)
called this the “generalized Kingsbury” (GK) strategy and simulated it in more realistic
conditions where probabilities for correctly judging item difficulties derived from real data were
Item Pocket Method to Allow Response Changing 6
less than 1.0. The simulation results suggested small possible benefits to test takers from using
the Kingsbury strategy but the strategy offered no meaningful improvement in score estimates.
Vispoel, Clough, Bleiler, Hendrickson, and Ihrig (2002) used live testing data to examine the
effect of the Kingsbury and GK strategies on score estimates under various conditions and found
that for a majority of test takers in the study the Kingsbury and GK strategies were ineffective.
This was mainly because the accuracy of test takers’ item-difficulty rating was much lower than
what Kingsbury (1996) and Wise et al. (1999) assumed: Test takers were only 61% successful in
distinguishing the difficulty difference within each item pair. The study did find, however, that
some test takers were still able to improve their scores using either one of these strategies.

CAT With Restricted Revision Options
To minimize the effect of the manipulative test-taking strategies while still providing test
takers with reasonable options to review and change their responses, Stocking (1997) proposed
three different models for limiting test takers’ practices in item review and response change. In
the first model, test takers were allowed to change their responses at the end of the test, but were
limited to a maximum number of revisions. According to her simulation results, Stocking’s
model effectively reduced the impact of the Wainer strategy, and the conditional standard error
of measurement (CSEM) and bias were close to those with a zero-revision condition when the
number of revisions was limited to 2 out of 28 test items. When the number of allowable
revisions was greater than two, however, (in Stocking, 1997, the studied conditions were 0, 2, 7,
14, and 28 revisions for the 28-item test) Stocking’s model failed to control the effect of the
Wainer strategy.
Item Pocket Method to Allow Response Changing 7
In Stocking’s second model, the test consisted of multiple separately timed sections, and
test takers were allowed to revise their responses freely within each section. The simulation
results showed that administering the test in two separate sections substantially reduced the
effect of the Wainer strategy on the CSEM and bias. When the item revision option was
available with four or more separated sections, the effect of the Wainer strategy was almost
completely negated. Stocking’s third model, in which test takers were allowed to revise
responses only within each item set associated with the common stimulus, showed its robustness
against the Wainer strategy as well. The main disadvantage of Stocking’s third model, however,
was that test takers were not allowed to revise responses for discrete (i.e., non-set) items
(Stocking, 1997).
In their study using live test data, Vispoel et al. (2000) confirmed that Stocking’s second
model (the restricted review within each block) successfully reduced the possible effect of the
Wainer strategy. In addition, the majority of examinees (98.4%) in this study felt they had
adequate opportunity to review and change their responses. In the latter studies by Vispoel and
his colleagues, which also used live test data, the Kingsbury and GK strategies proved to be
ineffective when the restricted review was permitted (Vispoel et al., 2002; Vispoel, Clough, &
Bleiler, 2005).
Reviewing the overall results from Stocking (1997) and Vispoel et al. (2000, 2002, &
2005), the restricted review approach (especially Stocking’s second model) seems to be one of
several solutions that could allow test takers to change their response during CAT to a certain
degree without incurring unacceptable levels of sacrifice in measurement precision and
efficiency. A serious limitation of the restricted review approach, however, is that allowing test
takers to review and change their responses only within each section eventually causes them to
Item Pocket Method to Allow Response Changing 8
involuntarily surrender access to items in the current section in order to proceed to the next
section. If test sections are timed strictly and separately from each other as Stocking (1997)
originally proposed, test takers would not necessarily feel pressure to move on to the next section
before the section time expired. In a majority of operational CAT programs, however, small test
sections (or item sets) usually are not timed separately. As a result, every time test takers finish
one section (unless it is the last section), they must decide whether it is better to spend time
revisiting test items in the current section and improving their initial responses or use their time
to complete the remaining test sections. Being forced repeatedly to make such a decision, not
knowing exactly how much time will be needed to complete the remaining test sections, very
likely causes test takers the same kind of test anxiety as that observed among test takers in a
regular CAT administration with no response revision option.
Another downside of the restricted review approach is that it still does not allow test
takers to skip items (unless the items within each section are not adaptively administered). If test
takers want to proceed further, even within one section, they must answer every item because the
CAT program selects the following item based on the test taker’s initial response to the current
item. The inability to skip items may not be terribly bothersome for test takers because they can
simply answer randomly and move on to the next item, knowing they can return to the item
before advancing to the next section. In terms of measurement efficiency, however, an item
selection process that is based heavily on initial responses that do not necessarily reflect test
takers’ best effort could seriously erode CAT’s level of adaptiveness. Moreover, some test takers
might try different initial responses to find clues on correct response based on the difference in
item difficulty between item pairs, which was the source of concern for Kingsbury (1996) and
Wise et al. (1999). Vispoel et al. (2002, 2005) observed no meaningful gain from practicing
Item Pocket Method to Allow Response Changing 9
Kingsbury and GK strategies in their CATs with the restricted review option, but their finding
was based on results from low-stakes exams. In high-stakes CAT programs, test takers might be
tempted to follow the Kingsbury or GK strategies with the intent of improving their scores, and
CAT with the restricted review approach technically is vulnerable to successful implementations
of either of these strategies.

Item Pocket Method
To address the shortcomings of the restricted review approach, this study proposes a new
approach for allowing response change. This method, which hereafter will be referred to as the
“item pocket” (IP) method, provides test takers with item pockets into which they can place
items for later review and response change. Test takers can skip answering items by putting them
in the item pocket. Once an item is placed in the item pocket, a test taker can go back to it
anytime during the test until the test taker submits his or her final answer for the item. For
example, in the CAT interface shown in Figure 1, a test taker is reviewing Item 4 among the
three items in the item pocket. If a test taker wants to take Item 4 out of the item pocket, she or
he ‘confirms’ the final answer for it. Once removed from the item pocket, Item 4 cannot be
placed back in. Test takers must confirm final answers for any items in the item pocket in order
to empty the item pockets before the test time expires or face the prospect that any remaining
items in the item pocket will be counted as incorrect responses. CAT developers can determine
the item pocket size based on the test length and time limit (more discussion on the item pocket
size follows later in this paper). The item pocket size was five in the example shown in Figure 1.
As for CAT item selection, only items outside the pocket (in other words, items with final
responses) are included in the interim score estimation procedure.
Item Pocket Method to Allow Response Changing 10
Insert Figure 1 about here

The IP method has several advantages over the restricted review approach. First, there is
no restriction on the number and range of items that can be revisited for changing responses. In
contrast to the restricted review approach, the IP method eliminates the need for CAT sections to
break into smaller separately timed sections, and test takers can return to any item as long as the
item is in the item pocket. Even if a test taker removes an item from the item pocket because it is
full and the test taker wants to add a new item, she or he would still have control over which item
to remove from the item pocket. From a psychological point of view, this is a significant
improvement. Test takers’ feeling of loss of control over the test, a major source of their anxiety
and stress (Stocking, 1997; Olea, Revuelta, Ximénez, & Abad, 2000), potentially may be
reduced with the IP method.
Another merit of the IP method is that test takers are not forced to provide an answer just
to move forward but instead can skip items simply by adding them to the item pocket (as many
as the item pocket size allows). The reduced anxiety by not being forced to answer each item
before proceeding to the next would be one of the IP method’s possible direct psychological
benefits, but the IP method’s psychometric benefits are also worth mentioning. In the IP method,
test takers’ initial responses for the items in the item pocket, including skipped items, have no
impact on item selection. The CAT system excludes items in the item pocket when computing
interim score estimates. Therefore, CAT item selection is always based on test takers’ final
responses, and as a result, CAT’s level of adaptiveness can be retained effectively. Since test
takers cannot change their answers once they are finalized—in other words, once items are