Table Of ContentVolume 9, Issue 2, 2015 [ARTICLE]
E P - H
VALUATING A RE SESSION OMEWORK
E S
XERCISE IN A TANDALONE
I L C
NFORMATION ITERACY LASS
Joseph E. Goetz In this study, researchers evaluate a homework
Rice University exercise assigned before a standalone
information literacy session. Students in a
Catherine R. Barber Master of Education program completed a
University of St. Thomas, Houston worksheet using the ERIC database thesaurus.
The researchers conducted pre- and posttests
within a single library session to assess student
learning, using a control group for comparison.
The treatment group did not demonstrate better
thesaurus skills than students who had regular
library instruction alone, but results pointed the
way to targeted improvements of pre-session
learning materials. This approach could inform
other information literacy homework
applications such as flipping the classroom.
176
Goetz & Barber, Evaluating a Pre-Session Exercise Communications in Information Literacy 9(2), 2015
INTRODUCTION evaluate the impact on student learning of
completing a pre-session exercise, assigned
Librarians teaching standalone “one-shot” as independent homework, in addition to in-
instruction sessions may feel forced to session instruction on database search skills
choose among full content coverage, student and strategy. Given the approach and
practice, and assessment. Extending student institutional context, the researchers saw the
learning time by assigning activities outside following constraints as imperative:
of class can lessen the constraints of that
dilemma. But demonstrating the value of The librarian would retain
those learning activities can present control over the administration
instructional librarians with an additional and collection of assessment
challenge. In the traditional one-shot format, materials, including an in-class
course instructors expect students to pretest and posttest.
develop their skills independently after a The assessment would compare
the effects of pre-session
self-contained library session. How should
librarians make the case for assigning an homework on the treatment
activity before the session? group versus the control group.
Treatment and control groups
The argument to expand single-session would be established based on
information literacy instruction should rely pre-existing course sections
on evidence of student learning. This article rather than individual random
describes the evaluation of a pre-session assignment.
homework activity’s impact on student A posttest performance task
learning compared with in-class instruction would assess students’ skill
alone using pretest and posttest assessments application in an authentic
administered within the standalone class. In database search scenario.
keeping with its traditional one-shot context,
this article does not assume that librarian As with the authors of previous studies
instructors can reliably appropriate time (Bryan & Karshmer, 2013; Hufford, 2010),
outside the session to conduct assessments. the researchers had trouble finding cases in
The researchers—the information literacy the information literacy instruction literature
librarian and the educational research that fully applied to their own situation. In
program director—sought to answer this response, the researchers analyzed the
and other constraints with a site-specific literature for the ways librarians and their
approach to assessment. They found that collaborators responded to methodological
despite limiting conditions, valid assessment constraints: namely, the timing of pre- and
data can point the way to iterative posttests, the ethics and composition of
improvement of instructional practices and control groups, and the use of performance
student learning. assessments in information literacy. Overall,
methodological strategy provided a useful
framework for applying the arguments and
LITERATURE REVIEW
lessons of previous researchers to a
specially adapted assessment project.
In this study, assessments were used to
[ARTICLE]
177
Goetz & Barber, Evaluating a Pre-Session Exercise Communications in Information Literacy 9(2), 2015
Pretest and Posttest Timing efforts by the librarians were required to
Pretests and posttests are commonly prevent drop-offs in the teaching faculty’s
employed tools for assessing student administration of tests and surveys (p. 466).
learning from library instruction (Hufford, Bryan and Karshmer (2013) also
2010, p. 140), but librarians face choices in experienced low participation by
when to assign these tests, and challenges in instructional faculty; in response, the
ensuring their completion. Assigning authors visited each class in person to
pretests and posttests outside the library administer pretests and posttests (p. 580).
session can both save time for instruction The need for this recourse points to the use
and affect the focus of the assessment. For of instructors’ class time as a challenge in
example, Choinski and Emmanuel (2006), terms of both logistics and scalability.
not wanting to lose “even a minute” of
instructional time, had instructors assign one Control Groups: Whether and How
-minute paper assessments as extra-credit Librarians have not always seen control
homework rather than an in-class activity groups as necessary or desirable for
(p. 151). In addition to making more time assessment. Barclay (1993) pointed out the
available, assigning pretests and posttests practical and ethical difficulties of creating
before and after the session may help assess control groups in library research, and
students’ longer-term development, as argued that it is better to dispense with them
discussed by Pierce and Fox (2012, p. 4). than not to do research at all. On a practical
Carter (2002), despite having two sessions level, different library instruction sessions
of a freshman seminar class available to are frequently too dissimilar to each other to
teach research skills, arranged for a pretest serve as valid control and experimental
to be administered during academic elements in a single research project; Carter
orientation and a posttest at the end of the (2002), responding to practical difficulties,
semester (p. 38). Similarly, Swoger (2011) used pretests and posttests without control
described pretests and posttests as part of a groups to measure student learning and
semester-length assessment cycle. improve the efficiency of class time.
Ethically, control groups may seem to call
Conversely, conducting assessments outside for one group of students to be taught less
the library session can lead to difficulties well than another. Bryan and Karshmer
controlling the process. Brooks (2013) (2013) addressed this dilemma by teaching
blamed low student motivation for poor both groups the same content with only
response rates on voluntary posttests differing methods of instruction; using a
emailed to students. Portmann and Roush control group allowed them to compare
(2004) named “student apathy” as a “fatal outcomes. When the superiority of either
flaw” to their research design (p. 464), and instructional method is still unclear, the
pointed to the need for grades to increase potential of long-term benefit can outweigh
student motivation. Still, tests administered the risk of using multiple approaches.
during instructors’ class time can also face Nevertheless, librarians might hesitate to
interference from course priorities and other use techniques to benefit future students at
factors. Hsieh and Holden (2010) noted that the possible expense of those present.
“consistent and persistent” communication
[ARTICLE]
178
Goetz & Barber, Evaluating a Pre-Session Exercise Communications in Information Literacy 9(2), 2015
Whatever the benefits of using a control across different departments, and either
group, educational researchers have long assigned within the session or as homework.
recognized the frequent impracticality for In the worksheet, students were asked to
their field of individual random group find a source, write a citation for it and write
assignment. Campbell and Stanley (1963) several sentences evaluating its
evaluated a range of models for performing appropriateness. This task directly
quasi-experimental research, in which connected the instruction session’s learning
individual random assignment or some other objectives with students’ research work,
requirement for true experimental research calling for open-ended responses that
does not take place. Among those models, allowed students to demonstrate applied
the nonequivalent control group design understanding. Teaching a semester-long
allows for division of treatment and control science information literacy course, Johnson
subjects by pre-existing groups, such as et al. (2011) assessed students’ growth with
course sections. However, the model calls a variety of performance-based homework
for a pretest to strengthen the evidence that assignments and exam questions together
these groups are not significantly different with surveys and citation analysis. Such
from each other in the area being measured approaches may seem best suited to
(p. 47-48). Campbell and Stanley argued extended course formats, but as Bluemle et
that the nonequivalent control group design, al. (2013) pointed out, “carefully designed”
while not truly experimental, is “well worth assessments can elicit performances of
using in many instances” and more secure higher-order tasks within short time frames
(all else being equal) from threats to internal (p. 300), meaningfully adding to the
and external validity than pretest-posttest assessment picture for an instruction
arrangements that forgo using a control program based on standalone sessions.
group (p. 47).
METHOD
Performance Assessment
Performance assessment, as advocated by Participants
Wiggins and McTighe (2005), gives Study participants (N = 138) were graduate
students a chance to demonstrate a students in a Master of Education (M.Ed.)
transferable understanding of skills that program that provided additional training in
goes beyond recall-based knowledge (p. 153 counseling, curriculum and instruction, dual
-155). In the context of library instruction, language/bilingual education, educational
Oakleaf (2008) contrasted performance diagnostics, educational leadership,
assessments with fixed-choice tests, exceptionality/special education, or reading.
pointing out the advantages of assessments Regardless of their area of specialty,
that “reinforce the concept that what students completed two action research
students learn in class should be usable seminars that prepared them to analyze,
outside the classroom” (p. 239). As one plan, and conduct educational research. The
example of performance assessment in a study took place in the context of the first
standalone library session, Bluemle et al. action research seminar, which involved a
(2013) described a “Source Evaluation single, librarian-led, in-person group session
Worksheet” that could be used in classes on information literacy for educational
[ARTICLE]
179
Goetz & Barber, Evaluating a Pre-Session Exercise Communications in Information Literacy 9(2), 2015
research. Students were informed on all to assess key learning outcomes in a brief
written materials that their responses (if timespan. For the performance activity,
handed in) would be analyzed confidentially participants read a research question
to improve library instruction; additionally, scenario and a sample thesaurus entry; they
students in the experimental group were were then asked to advise an imaginary peer
asked to give explicit consent for their work on developing a search strategy with
to be analyzed. selected thesaurus terms. For the searching
exercise multiple choice items, a total score
Measures was calculated, with possible scores ranging
All participants completed three measures: from 0 to 6. Cronbach’s alpha was
pretest, posttest, and post-session survey calculated as a measure of internal
(see Appendix A). The pretest was a two- consistency; for this six-item posttest
part measure that rated students’ self- measure, reliability was again lower than
assessed familiarity with library research desired (α = .37). Therefore, in addition to
processes (six items, using a 4-point rating looking at the six-item total score, student
scale with 1 = Not at all familiar; 4 = Very performance on individual items was also
familiar) and their knowledge of specific evaluated. For the performance-based
resources (six items, using a multiple-choice searching exercise activity, the authors
format). This provided both subjective and independently coded participants’ open-
objective information about students’ ended responses according to a four-point
information literacy baseline skills. A total rubric, with 4 indicating the highest level of
score for familiarity was calculated for each proficiency with the thesaurus and 1
participant, with possible scores ranging indicating the lowest level of proficiency
from 6 to 24, and for knowledge with (see Appendix B). Inter-rater agreement was
possible scores ranging from 0 to 6. lower than expected (Kappa = .42), though
Reliability for each measure was calculated better than chance (60% agreement across
using Cronbach’s alpha; the familiarity four categories). Discrepancies among
measure had acceptable internal consistency codes were discussed and resolved, resulting
(Cronbach’s α = .77), while the knowledge in a final set of codes used in the data
measure’s internal consistency was much analysis.
lower than expected (Cronbach’s α = .38),
suggesting that knowledge of these The final measure that all participants
resources was not a unitary construct. Thus, completed was a seven-item post-session
for pretest knowledge, both the total score survey that assessed participants’ opinions
and the individual item scores were about the relevance, value, and convenience
examined. of the library session. The post-session
survey included three open-ended items
After receiving instruction about search about the aspects of the session that
strategies and the use of the ERIC thesaurus, participants found most valuable, the topics
students completed a posttest on those that participants still had questions about,
topics. This assessment included six and any suggestions participants had for the
multiple-choice items and a performance librarian. Each item was examined
activity, both developed by the researchers independently.
[ARTICLE]
180
Goetz & Barber, Evaluating a Pre-Session Exercise Communications in Information Literacy 9(2), 2015
In addition, treatment group participants scheduled class times. Treatment group
completed a pre-class homework exercise participants turned in their completed
(see Design and Procedure section below). homework activity handout to their
The librarian scored successful completion instructor, who submitted all handouts to the
of this pre-class homework exercise on a librarian conducting the session. Students in
simple three-point scale with 3=successful the session received a handout containing
completion, 2=partially successful the pretest, posttest and post-session survey.
completion and 1=unsuccessful completion. Then the librarian administered the pretest.
The fact that students in the treatment group
Design and Procedure had already completed the pre-session
A quasi-experimental (non-equivalent homework exercise before taking the pretest
control group) design was used to determine may seem a threat to the pretest’s validity;
whether exposure to a pre-session however, the homework exercise was
homework activity on the ERIC thesaurus carefully designed so that students would
would be associated with better learn to use the ERIC thesaurus nearly
performance on an in-session research exclusively of other library skills. The
activity. Participants were non-randomly researchers intended to thereby leave
assigned to one of two groups: the treatment unaffected the general library familiarity
group (which received the pre-session and knowledge that the pretest measured in
homework activity) and the control group order to assess the groups’ similarity.
(which did not receive the pre-session
homework activity). Research instructors Instruction focused on how to access, search
were invited to incorporate the pre-session and manage library resources in education.
homework activity into the lesson plan After hearing about the library’s resources
during the class prior to the library session. in education and how to find them, students
Those instructors who volunteered provided viewed a demonstration of keyword
the pre-session homework activity handout, searching. Then the librarian lectured on the
including instructions (see Appendix C), to purposes and structure of the ERIC
their students, who constituted the treatment thesaurus, including the elements of a
group; students whose instructors did not thesaurus entry and the differences between
volunteer constituted the control group. All searching with subject terms and with
other aspects of the two groups’ library general keywords. Participants then
instruction were identical, with the two completed both the searching exercise
library instructors having carefully multiple choice items and the searching
coordinated lesson plans and presentations; exercise activity in 10 minutes. In the last
however, differences between groups due to section of instruction, the librarian gave
differences in instructors, location, etc., students methods for managing research
cannot be ruled out. information, including note-taking strategies
and a demonstration of bibliographic
Participants attended one of five 90-minute management software. Participants then
library sessions as part of their class offered feedback through a post-session
requirements; for most students, these survey. As they handed in their written
sessions fell outside their regularly- work, students received a handout with
[ARTICLE]
181
Goetz & Barber, Evaluating a Pre-Session Exercise Communications in Information Literacy 9(2), 2015
descriptions and locations of key resources members of the treatment group only).
discussed during the session. Second, independent groups t tests
compared the treatment and control group
RESULTS means on the main outcomes of interest
(posttest searching exercise multiple choice
Data analysis involved three steps. First, and posttest searching exercise activity).
means and standard deviations were Given the low reliability of the posttest
calculated for each group (treatment and multiple choice score, chi-square analyses
control) and the entire sample for these were also performed on the correct vs.
variables: pretest familiarity, pretest incorrect response frequencies for each
knowledge, posttest searching exercise item. Finally, participants’ responses to the
multiple choice, and posttest searching post-session survey were summarized.
exercise activity. In addition, frequencies
were calculated for correct vs. incorrect A scoring of the pre-session homework
responses to each pretest knowledge exercise on a 3-point scale showed high
question, each posttest multiple choice rates of successful or partially successful
question, each level of proficiency completion among members of the
demonstrated in participants’ responses to treatment group, with 61% achieving full
the posttest searching exercise activity, and success and 20% achieving partial success.
degree of successful completion of the pre-
session homework exercise (among Independent t tests revealed no difference
between groups in terms of their pre-session
TABLE 1—ASSESSMENT MEANS AND STANDARD DEVIATIONS
Item Treatment (N) Control (N) All (N)
Pretest Familiarity
Mean 12.92 (42) 12.75 (93) 12.80 (135)
Stand. Dev. 3.89 3.65 3.56
Pretest Knowledge
Mean 3.33 (43) 3.09 (92) 3.16 (135)
Stand. Dev. 1.08 1.48 1.37
Posttest Searching
Exercise Multiple
Choice
Mean 3.83 (42) 2.93 (92) 3.22 (134)
Stand. Dev. 1.41 1.45 1.49
Posttest Searching
Exercise Activity
Mean 2.29 (31) 2.30 (77) 2.30 (108)
Stand. Dev. 1.10 0.78 0.88
[ARTICLE]
182
Goetz & Barber, Evaluating a Pre-Session Exercise Communications in Information Literacy 9(2), 2015
familiarity, t (133) = 0.26, p = .79, or group (86% correct) and the control group
knowledge, t (133) = 0.94, p = .34 (see (54% correct): χ2(1) = 13.04, p < .001.
Table 1). However, given the low reliability
of the knowledge measure (α = .38), correct Posttest searching exercise activity
vs. incorrect responses to individual responses showed no difference, t (106) = -
knowledge items were examined with chi- 0.04, p = .96 (see Table 1). Thus, the
square analyses. The only significant pretest hypothesis that the treatment group would
difference observed between the two groups demonstrate better performance on a
was on the first knowledge question: “the searching exercise was partially supported.
list of subject terms is called…,” with more
treatment group participants (93%) than The results of the post-session survey
control group participants (61%) responding indicated that participants generally
correctly: χ2(1) = 11.72, p < .001. (Note that perceived the session to be very relevant,
Yates’ correction for low cell size was used that they felt moderately prepared to
for this analysis.) Since the first knowledge perform research, and that attending the
question was the only one to deal directly session was moderately to very worthwhile.
with the content of the pre-session Database search skills and citation
homework exercise and questions on other management software usage were frequently
aspects of library use showed no significant mentioned as valuable elements of the
difference, these findings were helpful for session. Common suggestions included
demonstrating that the treatment and control handing out an outline during the session
groups did not differ in their general and being able to follow along on a
familiarity with library and research computer.
procedures in ways that could influence
their performance on the searching exercise CONCLUSIONS
measures.
Assessment results demonstrated that while
Independent t tests revealed a significant students in the treatment group had a high
difference between treatment and control rate of success completing the pre-session
groups in terms of their posttest searching homework activity, that advantage did not
exercise multiple choice scores, t (132) = translate to overall better performance on
3.35, p = .001 (see Table 1). However, as the in-class searching activity compared
noted above, the low internal consistency of with students in the control group. The
this measure made it an unstable estimate of researchers can make reasoned guesses as to
student performance. An analysis of why this occurred. Perhaps the homework
individual posttest questions revealed the exercise’s mainly procedural activities did
main source of this difference to be question not lead to transferable understanding of
1, regarding keyword selection. thesaurus structure as called for in the in-
Specifically, of six chi-square analyses class searching activity. Thus, students
comparing the percentages of correct completing the pre-session homework
responses, only the analysis of question 1 exercise might have located a narrower or
(about keyword selection) showed a related term in a thesaurus entry, for
significant difference between the treatment example, without understanding what those
[ARTICLE]
183
Goetz & Barber, Evaluating a Pre-Session Exercise Communications in Information Literacy 9(2), 2015
structural elements meant in context. Acting such a limited assessment time frame point
on this hypothesis, the researchers could to the potential of quasi-experimental
develop more robust explanations of approaches in evaluating pre-session library
thesaurus structure for the pre-session instruction tools. The use of a control group
homework exercise, perhaps including made it possible for the researchers to
taxonomic diagrams or sample thesaurus evaluate the pre-session homework exercise
entries with readily-understood terms from independently of the impact of classroom
everyday life, to better build on students’ instruction, which was identical for both
previous learning and lead to greater gains groups. One application of this study’s
in understanding. approach would be to help develop the
assessment of “flipped” information literacy
It is difficult to make a case for improved classrooms, which assign homework before
student skills in keyword selection based on a session in order to focus class time on
responses to a single multiple choice active learning experiences. Researchers
question in the posttest. Nevertheless, the assessing student learning in flipped
researchers could consider what features of classrooms have compared the outcomes of
the keyword selection part of the homework students in classes that flip with students in
exercise might have allowed students to classes that do not (Arnold-Garza, 2014,
build on their prior knowledge to develop a p.19). While these results may point to the
transferable skill, and how such features benefit of active learning in the classroom,
could be used in other parts of the exercise. they cannot indicate which elements of the
flipped learning experience (including pre-
Students’ low performance in the posttest session instructional videos, for example)
assessment across both treatment and had the most benefit. By iteratively building
control groups might indicate that this on the use of quasi-experimental methods to
assessment should be revised to better evaluate pre-session exercises, a flipped
measure and reinforce student learning. The classroom project could better its position to
researchers could develop more multiple increase student learning and impact
choice questions and gather feedback on the academic culture.
questions’ clarity and perceived difficulty.
The searching activity could be revised in
REFERENCES
light of demonstrated student difficulties
such as not knowing the meanings of terms
Arnold-Garza, S. (2014). The flipped
in the sample thesaurus entry or not being
classroom teaching model and its use for
willing to engage with the assigned “email
information literacy instruction.
to a friend” genre. Such a revised posttest
Communications in Information Literacy, 8
assessment could lead to even more targeted
(1), 7–22.
improvements of the pre-session homework
exercise.
Barclay, D. (1993). Evaluating library
instruction: Doing the best you can with
DISCUSSION what you have. RQ, 33(2), 195–198.
The insights gained through this study in Bluemle, S. R., Makula, A. Y., & Rogal, M.
[ARTICLE]
184
Goetz & Barber, Evaluating a Pre-Session Exercise Communications in Information Literacy 9(2), 2015
W. (2013). Learning by doing: Performance Hufford, J. R. (2010). What are they
assessment of information literacy across learning? Pre- and post-assessment surveys
the first-year curriculum. College & for LIBR 1100, Introduction to Library
Undergraduate Libraries, 20(3-4), 298–313. Research. College & Research Libraries, 71
doi:10.1080/10691316.2013.829368 (2), 139–158. doi:10.5860/0710139
Brooks, A. (2013). Maximizing one-shot Johnson, C. M., Anelli, C. M., Galbraith, B.
impact: Using pre-test responses in the J., & Green, K. A. (2011). Information
information literacy classroom. The literacy instruction and assessment in an
Southeastern Librarian, 61(1), 41-43. honors college science fundamentals course.
Retrieved from College & Research Libraries, 72(6), 533–
http://digitalcommons.kennesaw.edu/seln/ 547. doi:10.5860/crl-166
vol61/iss1/6
Oakleaf, M. (2008). Dangers and
Bryan, J. E., & Karshmer, E. (2013). opportunities: A conceptual map of
Assessment in the one-shot session: Using information literacy assessment approaches.
pre- and post-tests to measure innovative portal: Libraries and the Academy, 8(3),
instructional strategies among first-year 233–253. doi:10.1353/pla.0.0011
students. College & Research Libraries, 74
(6), 574–586. doi:10.5860/crl12-369 Pierce, R., & Fox, J. (2012). Vodcasts and
active-learning exercises in a “flipped
Campbell, D. T., & Stanley, J.C. (1966). classroom” model of a renal
Experimental and quasi-experimental pharmacotherapy module. American
designs for research. Chicago: R. McNally. Journal of Pharmaceutical Education, 76
(10), 1-10. doi:10.5688/ajpe7610196
Carter, E. W. (2002). “Doing the best you
can with what you have”: Lessons learned Portmann, C. A., & Roush, A. J. (2004).
from outcomes assessment. Journal of Assessing the effects of library instruction.
Academic Librarianship, 28(1), 36–41. Journal of Academic Librarianship, 30(6),
461–465. doi:10.1016/j.acalib.2004.07.004
Choinski, E., & Emanuel, M. (2006). The
one-minute paper and the one-hour class: Swoger, B. J. M. (2011). Closing the
Outcomes assessment for one-shot library assessment loop using pre- and post-
instruction. Reference Services Review, 34 assessment. Reference Services Review, 39
(1), 148–155. (2), 244–259.
doi:10.1108/00907320610648824 doi:10.1108/00907321111135475
Hsieh, M. L., & Holden, H. A. (2010). The Wiggins, G. P., & McTighe, J. (2005).
effectiveness of a university’s single-session Understanding by design. Alexandria, VA:
information literacy instruction. Reference Association for Supervision and Curriculum
Services Review, 38(3), 458–473. doi: Development.
10.1108/00907321011070937
[ARTICLE]
185
