2013 Hsieh, Dawson, and Carlin. This is an
Open Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 2.5 Canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/),
which permits unrestricted use, distribution, and reproduction in any medium,
provided the original work is properly attributed, not used for commercial
purposes, and, if transformed, the resulting work is redistributed under the
same or similar license to this one.

Abstract

Objective – Librarians at Rider University attempted to
discern the basic information literacy (IL) skills of students over a two year
period (2009-2011). This study aims to explore the impact of one-session
information literacy instruction on student acquisition of the information
literacy skills of identifying information and accessing information using a
pretest/posttest design at a single institution. The research questions include:
Do different student populations (in different class years, Honors students,
etc.) possess different levels of IL? Does the frequency of prior IL
Instruction (ILI) make a difference? Do students improve their IL skills after
the ILI?

Methods
– The
librarians at Rider University developed the test instruments over two years
and administered them to students attending the ILI sessions each semester. The
test was given to students as they entered the classroom before the official
start-time of the class, and the test was stopped five minutes into the class.
A pretest with five questions was developed from the 1st ACRL IL Standards. A
few demographic questions were added. This pretest was used in fall 2009. In
spring 2010, a second pretest was developed with five questions on the 2nd
ACRL IL Standards. Students of all class years who attended ILI sessions took
the pretests. In 2010-2011, the pretest combining the 10 questions used in the
previous year was administered to classes taking the required CMP-125 Research
Writing and the BHP-150 Honors Seminar courses. An identical posttest was given
to those classes that returned for a follow-up session. Only the scores from
students taking both pretests and posttests were used to compare learning
outcomes.

Results
– Participants’
basic levels of IL skills were relatively low. Their skills in identifying
needed resources (ACRL IL Standards 1) were higher than those related to
information access (ACRL IL Standards 2). Freshmen in the Honors Seminar
outperformed all other Rider students. No differences were found in different
class years or with varying frequencies of prior IL training. In 2010-2011,
students improved significantly in a few IL concepts after the ILI, but overall
gains were limited.

Limitations – Many limitations are present in this study, including
the challenge of developing ideal test questions and that the pretest was
administered to a wide variety of classes. Also not all the IL concepts in the
test were adequately addressed in thesesessions.
These factors would have affected the results.

Conclusions – The results defy a common assumption that students’
levels of IL proficiency correlate with their class years and the frequency of
prior ILI in college. These findings fill a gap in the literature by supporting
the anecdote that students do not retain or transfer their IL skills in the
long term. The results raise an important question as to what can be done to
help students more effectively learn and retain IL in college. The authors
offer strategies to improve instruction and assessment, including experimenting
with different pedagogies and creating different posttests for spring 2012.

Introduction

Rider University, located
in Lawrenceville, NJ, is a private, coeducational university with 5,500
students, offering 69 undergraduate programs in business administration,
education, liberal arts, the sciences, fine and performing arts, counseling,
and leadership, plus 25 Masters level degrees. Librarians at the Franklin F.
Moore Library (also known as the Moore Library) have established an active
library instruction program, working with teaching faculty to integrate
information literacy (IL) into their courses for the past decade. Following the
emphasis placed on assessment by the Middle States Commission on Higher
Education, the accrediting body for Rider University, the Moore Librarians have
been involved in assessment since 2002.

The learning objectives for
information literacy are based on the Association of College and Research
Libraries (ACRL) Information Literacy Competency Standards for Higher Education
(American Library Association, Association of College and Research Libraries,
2000). This study reports on the Moore Library’s assessment program that
measured students’ IL levels in two academic years (2009-2010 and 2010-2011) on
the first two ACRL IL standards, which include the same IL learning objectives
for students at Rider University. These objectives contain the basic
information literacy competencies and are appropriate for lower division
undergraduates:

1.The information
literate student determines the nature and extent of the information needed.
Students will identify a variety of types and formats of potential sources of
information.

Most of the ILI sessions
occur in the Library’s two computer labs. The topics of the assignments and
areas of study range widely from business to humanities, social sciences,
sciences, and technology. The librarians teach sessions for their liaison
departments and share teaching responsibilities for the core Research Writing
and the Honors Seminar classes.

The current study assessed
the knowledge of students of all class years for the first two IL objectives in
the academic year 2009-2010. For the 2010-2011 academic year, identical pretests
and posttests combining these two IL objectives were used to assess the impact
of ILI on students’ learning in the Research Writing course. Our research
questions for this study include:

Do students’ possess different levels of knowledge and skills for
the two IL objectives?

Do different student populations [freshmen (1st year), sophomores
(2nd year), juniors (3rd year), seniors (4th year),
Honors students, and students in different areas of study] possess
different levels of IL?

Are students’ performances on the IL pretest associated with the
frequency of prior ILI received?

Do students improve their IL skills after the ILI?

Literature Review

Several approaches to
assessing undergraduate students’ acquisition of information literacy skills
have been documented. This literature review will discuss these different methods
used, including the pre and posttest system used by the authors.

Authentic assessment
depends on students’ actual performance on tasks such as an annotated
bibliography, submitted research papers, bibliographies, and worksheets as
discussed by Oakleaf (2011) and Brown and Kingsley-Wilson (2010). Rubrics are
designed to assess these types of documents and provide a systematic way to
determine how well students have achieved the learning objectives. McCulley
(2009) and Rogers (2001) discuss “Reflective Learning” where students think
about their processes of learning through the use of portfolios and journals as
a way to help them become better learners.

Standardized Testing

Other types of assessment
include standardized testing formats used to assess baseline competencies among
undergraduate students. The Educational Testing Service (ETS) and Kent State
University have developed standardized tests for measuring student information
research skills using scenarios: the iSkills exam (Katz, 2007; Katz &
Macklin, 2007), and Project SAILS, (Radcliff, Salem, O’Connor, Burhanna, &
Gedeon, 2007; Rumble & Noe, 2009) respectively. These tests allow
large-scale aggregation of data amongst many institutions, but can be
expensive, time consuming, and would be difficult to use for pretesting and
posttesting. However, these tests can be used to assess gains over time to
determine trends.

Pre and Posttest Methods

It is evident that the
library literature details a wide variety of mechanisms for assessing IL
skills. Moore Librarians selected the pretest and posttest method for obtaining
a sense of students’ understanding of different resources and search strategy
skills and confirm or disprove anecdotal evidence of such skills. These tests
are easy to construct using online forms via Google Docs; they can be
administered quickly, and data they generate are easily downloaded and analyzed
(Hsieh & Dawson, 2010). This procedure allows librarians to document
students’ IL skills and to measure gains over time.

There are instances of the
use of pretests and posttests in credit courses. In a one-credit course taught
at Texas Technology University, students were given a pretest at the start of
the semester and a posttest at the very end of the semester. Both tests were
identical and some questions had multiple answers. Despite IL skills taught
over a 14 week period, students did not do as well as expected in the posttest
(Hufford, 2010). In another example, a three-credit class conducted at the
University of Rhode Island used pretests and posttests to determine student
learning of Boolean operators (Burkhardt, 2007). The librarians were
disappointed in the small increase between these two test scores. Gandhi (2004)
detailed the assessment of a five-session model for library instruction and
found students learned more than after a one-shot session. However, this may
not be a very practical approach because of the many demands on the librarians’
times. Further, faculty members are usually very reluctant to give up class
time for such sessions.

About 85 % of 60- to
90-minute course-integrated ILI sessions taught by the Moore librarians at
Rider University are single-sessions, typical ILI offered in many academic
libraries in the United States (Merz & Mark, 2002).Most of the literature reviewed for this
paper involves one-shot instruction sessions. A survey developed by librarians
at the University of Northern Texas used a software system that tracked
websites used by students during their assessment; four questions were used as
a pretest and posttest. This survey demonstrated that ILI helped students and
provided information on weaknesses such as subject searching in the online
catalog (Byerly, Downey, & Ramin, 2006). A review paper describing the
pretest/posttest techniques raised concerns about using identical sets of
questions for both tests, and the problem with the span of time placed between
these two tests as the major factor in determining retention (Emmett &
Emde, 2007). At Cornell University, the posttests indicated improvement in IL
skills but the authors stated that a posttest later in the term would be needed
to determine the amount of retention of the material (Tancheva, Andrews &
Steinhart, 2007). Julien and Boon (2004) did just that. The posttest was given
immediately after a library session and a post-posttest given to students three
to four months later. Students showed a decline from the posttest to the
post-posttest. This indicates that information is not retained well, and
suggests the need for reinforcement of IL concepts throughout the semester.

Fain (2011) outlined a
five-year longitudinal study using pretests and posttests administered by
teaching faculty instead of librarians. The author enlisted the help of
Psychology faculty for the statistical analysis of the data, similar to what
Moore Librarians have done. In addition, the study emphasized the impact of the
assignment on teaching information literacy skills, i.e., if journal articles
are required but not books, then assessment questions related to using an
online catalog or types of books will not be taught by librarians. This would
affect the outcomes of any IL assessment that asks about resources such as
books and skills using the online catalog.

Surveys have frequently
been used to assess students’ opinions or ask about their satisfaction with IL
instruction (Matthews, 2007). Freeman and Lynd-Balta (2010) and Knight (2002)
described studies using pretests and posttests to assess students’ confidence
levels in information literacy skills. However, these types of studies do not
demonstrate students’ knowledge or capability to apply learned IL skills. As illustrated
by Dawson and Campbell (2009), computer and information literacy skills are not
equivalent. Students may exhibit confidence in their search skills because of
their familiarity with Google and social media. This may be a consequence of
students confusing their computer skills with information literacy skills.

Method

Participants

Participants were
undergraduate and graduate students at Rider University sampled from the fall
2009 semester through the spring semester of 2011. For the first year of the
study (academic year 2009-2010), all students attending ILI sessions in the
Moore Library computer labs were assessed to establish a baseline IL level of
all students. In the second year (academic year 2010-2011), instead of testing
students in all ILI sessions, the librarians narrowed the study population to
studentsin the Research Writing and
the Honors Seminar courses. This was done because the IL objectives matched the
courses’ objectives well. In addition, 7 of the Research Writing instructors
requested a follow-up session after the first ILI, allowing the use of
posttests in 15 classes to determine learning outcomes from their previous ILI.

Numbers of participants are
shown in Table 1. Participants for the fall 2009 and spring 2010 semesters
included all who received ILI at the Moore Library. Of these 1,986 students,
560 were freshmen, 310 were sophomores, 420 were juniors, 338 were seniors, 313
were graduate students, and 45 were reported as “other.” Students in subsequent
cohorts (fall 2010 and spring 2011) included students from core writing courses
only. Students not in the Honors program were enrolled in Research Writing
(CMP125) course. The fall 2010 cohort consisted mainly of sophomores (129 of
177 students), whereas freshmen composed the majority in the spring 2011 cohort
(362 of 436 students).

Students in the
Baccalaureate Honors Program (BHP) have GPAs of 3.5 or better. They scored
higher in the standardized college entrance exam Scholastic Aptitude Test (SAT)
with Critical Reading and Math scores of at least 600 and Writing score of at
least 550 (BHP, n.d.). The Honors students typically take the BHP-150 Honors
Seminar in the spring semester during their freshmen year and therefore, almost
all (121/122) were freshmen. This group did not receive follow-up sessions.

Information Literacy
Instruction Sessions

All sessions took place in
the computer labs of the Moore Library and were conducted by one librarian.
Classes typically included up to 25 students. The basic IL concepts in the
first two IL objectives are applicable in any ILI. Some concepts might be introduced
more thoroughly than others in a session depending on the requirements of the
assignments and students’ topics. For example, when an assignment called only
for journal articles, librarians would demonstrate searching for articles in
selected databases but not searching for books in the online catalog. Book
sources might be merely mentioned in such a session. On the other hand, some
concepts and skills such as search logic, methods of searching for articles
using the library subscription databases, and locating the library’s
periodicals were emphasized in almost all sessions.

The IL content the
librarians taught was not limited to the IL concepts represented in the tests.
For each session, the librarians provided handouts that included a combination
of content outlines and step-by-step instructions for their sessions. In summer
2010, the Library began subscribing to LibGuidesTM for research
guides, with Moore librarians gradually developing their instruction content in
these online guides and using them in ILI. Librarians’ teaching styles varied
and individual librarians engaged students and faculty differently in their
sessions. Nevertheless, most ILI consisted of lecture, search demonstration,
and hands-on time during which librarians monitored and coached students in
researching their topics. In the 2010-2011 academic year, of the 20 CMP-125
faculty members, 7 requested a follow-up session after the first ILI to allow
students more instruction, hands-on time, and coaching from the librarians. As
the result, 15 of the 39 classes (38%) had follow-up sessions.

Time is at a premium in the
ILI sessions if both instruction and hands-on time are to be included,
therefore, the assessment instrument had to be short and easily accessible to
students in the library labs. In fall 2009, an online test with five
multiple-choice questions (see Appendix A) was developed to assess the IL
abilities of students on the first IL objective – identifying a variety of
sources. The test was developed according to the best practices guidelines for
generating tests/surveys outlined by Radcliff, Jensen, Salem, Kenneth, and
Gedeon (2007) and by adapting test questions used elsewhere (Burkhardt, 2007;
Goebel & Mandeville, 2007; Mery, Newby, & Peng, 2011; Schroeder &
Mashek, 2007; Staley, Branch, & Hewitt, 2010). It was piloted on student
workers at the Moore Library to ensure that the language in the test was clear
to college level students. The librarians installed the test online using Google
Docs. In spring 2010, a second test (see Appendix B) was developed in the same
fashion for the second IL objective on constructing search queries.

In the second year of the
study, the librarians aimed to measure student learning in the ILI sessions by
developing identical pretests and posttests. The two sets of questions used in
2009-2010 were combined into a single set of questions used in 2010. Following
use of this 10-item test in the fall of 2010, the Moore librarians shared the
fall 2010 results with a group of teaching faculty and received feedback on the
test in January 2011. Consequently, in spring 2011, the wording of several
questions was modified to make them clearer without changing the IL concepts
assessed. The changes are noted under each question in the Appendices.

In addition to the
questions regarding IL objectives, demographic questions were included on all
versions of the first- and second-year tests. These questions included the
course number, class year (freshmen, sophomore, junior, senior, graduate
student, or other), major area of study (humanities, business, education,
science, social science, undeclared, or other), the number of prior library
instruction sessions attended (on spring 2010 and later versions), and a
four-digit identifying code (e.g., ID number) for matching pretests and
posttests in the fall 2010 and spring 2011 cohorts.

Procedure

As students arrived for
their sessions at the library class labs, they were instructed to take the
online pretest. Students had until five minutes into the scheduled session time
to take the test. Those arriving after that time did not take the test.
Students then completed the ILI session and departed. In the fall of 2010 and
spring of 2011, 7 out of 20 instructors of Research Writing (CMP125) classes
had their classes return for a follow-up library session to receive additional
instruction and hands-on time. These students again were tested as they arrived
for their session and up to five minutes into the scheduled session time. The time
from the first session to the follow-up session varied, but averaged
approximately three weeks.

Design & Analysis

Analyses comprised a series of one-way and factorial analyses of
variance ANOVAs for the focal hypotheses. The REGW-q multiple comparison procedure was used. In addition, chi-squared
tests of association and McNemar’s Test were used for analyses involving
nominal scale measures. The Type I error rate for all tests was .05. Eta, a
measure of nonlinear relationship, was used as a measure of effect size.

Independent variables were
class year (freshmen, sophomore, junior, senior, graduate student), area of
study (humanities, science, social science, education, business, undeclared,
other), course type (core, Honors), IL objective (1, 2), and test (pretest,
posttest). Additionally, the number of prior IL sessions was used as a
correlate of performance for certain cohorts who were asked to report this
information. The dependent variable was the number of questions answered correctly
(of 5) for each learning objective.

Results

In this section, a
discussion of the impact of the question revision in spring 2011 leads to the
combined data that revealed findings of the pretests in both years.
Performances on the two IL objectives were examined separately. Comparisons
were made for the cohorts by semester, class year, course (including Honors
program), major area of study, and frequency of prior ILI. Following the
pretests results were the comparisons of students’ matching pretest and
posttest records in fall 2010 and spring 2011. Findings include students’
learning outcomes on the whole, by objectives and by questions.

Impact of Revised
Questions in Spring 2011

After receiving
feedback from some class faculty on the test results, six questions were
revised (Q2, 3, 5, 6, 7, and 10) in spring 2011 to make the questions clearer
to students without changing the IL concepts (changes are indicated in the
Appendices under each original question). Scores for the original test and
revised test were compared. Only the questions with scores that varied largely
from the previous year were examined for the impact of the revision (see Figure
1). Only questions 2 and 3 yielded results warranting notice.

The decline in
scores by almost 10% in Q2 indicated that the change from “A library’s
database/index” to “A database such as Academic Search Premier” did not help
more students to choose this answer. The reason behind the change was that
perhaps some students did not understand the database/index reference, but they
might be familiar with “Academic Search Premier.” The lower scores in the
revised version indicated that probably fewer participants recognized this
database’s name and therefore fewer chose it.

Question 3 asked
which source one should use to search for background information on an
unfamiliar topic. The question changed from “What's the first thing you should
do to get started?” to “What's the best way to get an overview of this topic?”
The correct answer changed from “Find out some basics on watersheds from an
encyclopedia” to the same answer but included both print and online versions.
The revision moved 16% more participants in the spring to choose “encyclopedia
(online or print),” but the majority (52.8%) still chose the Web for their
answer. The preference of most participants remained the same in both
semesters. On the whole, the minor revisions to the questions in spring 2011
had minimal impact on the results because the highest scores received by the
participants – Q4, Q2, Q1, Q5 in this order, and the lowest scores – Q6, Q8,
Q10, were the same in both semesters and not altered by the question revision. For
this reason, subsequent analyses combined data from the two versions of the
test.

Pretest

IL Objectives. The pretest data revealed that the participants performed significantly
better on IL Objective 1 (Q1-Q5) than on Objective 2 (Q6-Q10), F(1,629) = 143.01, p < .001, eta = .43.
This effect of objective did not interact with year, major, or cohort.

Course comparisons.Participants were coded as
students in an Honors course, the Research Writing course (CMP-125), any other
undergraduate course, or a graduate course. For Objective 1, Honors students
scored higher than the undergraduate students, F(3,1780) = 4.14, p =
.006, eta = .08. Graduate students
scored equivalently to all other groups. For Objective 2, the Honors students
scored higher than all groups, F(3,1555)
= 16.59, p < .001, eta = .18. In addition, the CMP-125
students scored lower than all other groups. Descriptive statistics for the
course groups by objective are shown in Figure 2 and Table 2.

In fall 2010 and
spring 2011, those completing both objectives in the pretests included only
CMP125 students (n = 611) and Honors
students (n = 68). The difference in total
number correct for the 10 questions was significant, F(1,677) = 26.67, p <
.001, eta = .19. The Honors group [M (mean) = 5.32, SD (Standard Deviation) = 1.88] scored higher than the CMP125 group
(M = 4.16, SD = 1.75).

Comparisons by major area of study.Areas of study were
self-identified by students but the areas do not necessarily correspond to
their schools or their academic departments. For example, psychology and
counseling are under the School of Education but students in these programs may
consider social sciences as their area of study. With the broadly defined
areas, most participants of the tests were in business, followed by social
sciences and education. The percentages of participants representing sciences,
the humanities, undeclared and other were all in the single digits.

There was a
significant effect of area of study for Objective 1, F(6,1777) = 7.544, p < .001, eta = .16. Multiple comparisons indicated that business,
undeclared, and other categories scored lower than did the other four groups.
There was no difference across majors for Objective 2. For the CMP125 students
in fall 2010 and spring 2011 who took the tests for both Objective 1 and
Objective 2, there was a significant effect present, F(6,673) = 5.33, p <
.001, eta = .21.The humanities and
science students scored higher than the business and other students. Descriptive
statistics are shown in Figure 3 and Table 3.

Frequency of Prior ILI.The question regarding the
frequency of prior ILI from Rider University first appeared in the test in the
spring of 2010 and was included in the following year. From the self-report of
the participants, 62% of freshmen, 43% of graduate students and 20% – 30% of
undergraduates other than freshmen never had a prior ILI session (see Figure
4). The number of prior ILI sessions was not significantly correlated with any
outcome variable (i.e., numbers correct for Objective 1, Objective 2, or
combined).

Table 3

Number of Correct
Responses by Major Area of Study (N = Number)

Mean

SD

N

Humanities

5.04

1.89

45

Science

4.89

2.01

71

Business

3.99

1.71

225

Education

4.21

1.62

123

Social Science

4.50

1.84

132

Undeclared

4.09

1.66

45

Other

3.61

1.70

38

Figure 3

Mean number correct on the
combined pretests by major area of study. Error bars denote 95% confidence
intervals.

Percentage of students with
each indicated number of prior ILI session by class year.

Figure 5

Percentage of
correct responses for each question on the pretest and posttest 2010-2011.

McNemar's Tests were
performed to determine whether accuracy rates varied across tests (pretest vs.
posttest) for each question. Significant increases in accuracy were seen only
for questions 6 (subject search in the catalog, p = .003) and 8 (truncation, p
< .001), even though the posttest scores for both questions were still rather
low. These increases were evident in both semesters. The differences between
the pretests and posttests for the other questions were not significant (see
Figure 5).

For the first IL
objective on identifying a variety of sources (Q1 to Q5), participants
performed the best in differentiating scholarly journals from popular magazines
(Q4). They did relatively well on the purposes of the catalog and the library’s
databases (Q1 & Q2). More than half of the population knew how to find the
library’s full-text journals (Q5). Less than half of the participants would use
encyclopedias to search for background information of an unfamiliar topic (Q3).
Most chose to use a Web search engine for this purpose.

Among the five
questions on Objective 2, participants performed the best on the combination
use of Boolean AND and OR (Q7) with around 40% accuracy rate. They received the
lowest scores on using subject search to find books on the critiques of
Shakespeare’s Romeo and Juliet in
the online catalog (Q6).More of themwould search by
title and by keyword, and very few chose the correct answer, “By subject.” The
posttest scores of this question improved significantly from 10% to 20%.

The tests revealed
further that a great majority of participants were unfamiliar with the use of
truncation (Q8) and the Boolean connector OR (Q9). Very few would consider
using books for a reliable and thorough history on a common topic (Q10). Most chose
to use scholarly journals.

Discussion

Do students’ possess different levels of knowledge and skills for the
two IL objectives?

Participants had higher
scores on the first IL objective than the second IL objective, indicating that
although a majority of Rider students could identify a variety of sources, few
could construct their searches efficiently using these resources.

Do different student populations (class
years, Honors students, area of study)possess different levels of IL?

Faculty members
often assume that students have received their IL training before entering
their class, and expect students to know how to search for information
(Kolowich, 2011; Lindsay et al., 2006). The finding that upperclassmen
performed no differently than their lower division counterparts defies the
assumption and raises an important question about the long-term effectiveness
of the ILI that students receive in and prior to college.

The Honors group,
composed mainly of second semester freshmen, demonstrated higher IL levels than
their peers. It is worth noting that two other studies found positive
correlations between students’ IL levels and their GPAs (McDermott, 2005;
Silvernail, Small, Walker, Wilson, & Wintle, 2008). Do these findings
suggest that the Honors students are efficient in doing research and would not
need IL training? The average pretest scores of 67% on the first objective and
38% on the second objective for this group suggest that they have ample room
for improvement, especially on the second objective (search queries), and could
benefit from IL instruction. The findings of Johnson, Anelli, Galbraith, and
Green (2011) agree with those of the present study: their Honors students
demonstrated the same problems as the others in locating the library’s
resources.

As explained earlier, the
areas of study were broadly defined and do not necessarily correspond to the
participants’ majors. The findings revealed that the humanities and science
students scored higher than business students. Additional research is needed to
determine whether students perform better on discipline-specific IL concepts
than on general IL concepts. One business librarian found no such correlation
(Campbell, 2011) for her business students. In the future, other researchers
may wish to investigate students’ IL in relation to their majors.

Are students’ performances on the IL pretest associated with the
frequency of receiving prior library instruction?

Participants did
report having multiple ILI sessions during their undergraduate years at Rider
University. Whereas the majority (63%) of freshmen reported having no prior ILI
at Rider, by senior year nearly 80% had experienced one or multiple ILI
sessions. The finding of no correlation between the frequency of participants’
prior ILI and pretest scores suggests that students might not develop IL skills
through the “law of exposure” (Matthews, 2007). What they learn may not be
retained for or transferred to another research experience. Julien and Boon
(2004) reported a similar finding of students not retaining their IL learning
three or four months after the ILI. Numerous other studies also came to the
same conclusion that it is erroneous to assume that one or more ILI sessions in
the early college years will prepare students well for higher levels of
research work, and students do not become IL proficient from the single-session
ILI (Stec, 2006; Lechner, 2005; Johnson, 2011; Mokhtar & Majid, 2011; Wong
& Webb, 2011). It can be inferred from these findings that more intensified
IL training than the current single-session model needs to take place during
the four-year period for students to retain basic IL skills.

Do students improve their IL skills after the
ILI?

The pretest data
revealed that the IL level of all students was relatively low. The overall
differences between the pretests and posttests in the CMP-125 students were not
significant. This supports other findings that students’ learning from
single-session ILI is limited (Mokhtar & Majid, 2006; Portmann & Roush, 2004; Hsieh & Holden, 2010). While the credit
IL course is not the preferred or the primary way for IL training on most
college campuses (Tancheva et al., 2007), some librarians have embedded
themselves in the classroom over a period of time (Steiner & Madden, 2008),
or have actually gained grading power (Coulter , Clarke, & Scamman, 2007).
Moore Librarians need to investigate other means for working more closely with
the professors to integrate IL into their courses in the future in order to
increase the short- and long-term impact of ILI.

Reflections on constructing test items

Analysis of
responses to several individual test questions provided significant insight
regarding students' knowledge and misconceptions. More participants chose to
use “a web search engine for a complete list of references on the topic”
instead of “an encyclopedia” to find background information on an unfamiliar
topic (Q3). The answer for encyclopedia was revised in spring 2011 to include
online encyclopedias. As in the previous semester, the majority of participants
chose to use the Web over an encyclopedia. Many faculty members agreed with
students and considered a Web search engine a better tool than encyclopedias
for background information. Even though librarians prefer encyclopedia sources
for their reliability, it is hard for the library reference sources to compete
with the easy access of Web search engines. This preference of users, including
faculty members, for using the Web over traditional reference sources is an established
trend that was documented a decade ago in Rockman’s (2002) study. Considering
the development of the Web sources over the past decade, the Web could be
considered acceptable for this question on most topics, if not all.

Participants’ accuracy rates for the Boolean operators (AND/OR) in Q7
and Q9 were in 30% to 40% range. Biddix, Chung and Park (2011) observed that
library databases, with their subject, thesaurus and Boolean operators search
systems, are too complicated and problematic for students. Burkhardt (2007) was
dissatisfied with students’ improvement on Boolean operators over a
three-credit IL course. The authors of the current study agree with Kolowich
(2011) that the majority of students do not understand search logic and would
have great difficulty finding good sources. To master the basic search logic,
students need to learn the operators’ functions correctly and need practice to
reinforce their learning.

Few students reported they
would search for critiques of literary works by subject in the catalog as
indicated in Q6. Rider students were not exceptional when compared with those
in the study of Byerly et al., (2006) where only 1.6% of students chose the
correct answer for the question on subject search. Many more studies in the
literature also found students’ lack of knowledge on subject or controlled
vocabulary (Matthews, 2007; Brown & Krumholz, 2002; Lindsay et al., 2006;
Riddle & Hartman, 2000). In discussions with Rider faculty members about
the question, some of them disagreed with the importance that librarians tend
to place on subject searching; they preferred keyword searches instead. As
experienced researchers in their fields, these faculty members may know how to
use the appropriate keywords to find relevant items with considerable
efficiency. They are also motivated to spend the time to sort through large
numbers of returns for their studies. But the same knowledge and search mode
cannot be expected of most freshmen and sophomores. Even though librarians
would like to teach students about the concept of subject and controlled
vocabulary, various factors, including ingrained personal preferences and
habitual research patterns may make learning this concept a challenge.

Few participants had prior
knowledge about the use of truncation (Q8, Figure 7). Students in other
institutions also had trouble with this concept (Matthews, 2007; Furno &
Flanagan, 2008). Even though participants improved significantly in the
posttest, it remains the case that a minority (30%) scored correctly on the
question after the ILI.

More participants
chose journals than books as their source even when books may have been more
appropriate (Q10). Other researchers also noticed that college students are
overlooking the value of books and printed materials (Rockman, 2002; Head, 2012)
and do not understand the limitations of scholarly journals (Furno &
Flanagan, 2008; Schroeder & Mashek, 2007; Stec, 2006).

The finding that
students were weak in constructing search queries prompted the Moore librarians
to spend more time explaining the search concepts in the sessions. Vocabulary
may play a part in students’ understanding. Defining terms such as “truncation”
or avoiding library jargon may improve students’ search skills. Video clips and
tutorials were included in the class research guides to help students learn and
review the search logic and processes. Knowing that those students who had
prior ILI sessions might not remember or transfer that knowledge in more
advanced classes, some librarians used the inquiry method to determine what
these upper class students might already know about IL. If a majority of
students could answer the questions correctly, then the librarians would skip
teaching those concepts. The assessment findings also helped librarians work
more closely with the class faculty to include IL concepts in their
assignments.

Limitations

Moore librarians,
in teaching the course integrated one-session ILI, face serious limitations and
obstacles that the teaching faculty does not. Professors in a variety of
disciplines request ILI and have different requirements for their assignments.
Further, even though the librarians were aware of the common IL objectives for
students as well as the items on the tests, they necessarily taught to the
assignments, not to the tests. Some IL concepts received greater emphasis than
others during the ILI sessions. This lack of uniformity and control by
librarians in teaching IL is common for the single-session ILI in most
colleges. It would help if, in the future, librarians record which IL concepts
they teach in each session. This will allow for more precision in assessment by
relating what is taught to what students learn in the sessions.

Owing to the
limited class time in the session, the test instruments were very brief and
used only multiple-choice questions. The number of questions was not large
enough to provide a comprehensive picture of students’ IL skills.
Multiple-choice questions are limited when it comes to assessing participants’
higher order thinking skills (Oakleaf, 2008).
Nevertheless, the format is still widely used by researchers and educators because
access, data gathering, tabulation, and analysis are comparatively easy (Hsieh
& Holden, 2010). As suggested by Suskie (2007) and Oakleaf (2010), there is
no single instrument that can provide a comprehensive picture of IL competency.
When practicable, it is best to use multiple instruments– at least enough to
supplement the perspective gained from a single assessment tool. These could
include students’ reflections on their learning, one-minute papers and
librarians’ reviewing of students’ papers to evaluate students’ application of
research concepts and methods in actual productions. In addition, performance
measures, such as timing how long it takes students to complete searches, may
be useful, even though this method would be highly sensitive to individual
differences and learning styles.

The tests used in
the current research were developed over the course of two years. The
participants taking each test were the students in the ILI sessions of each
semester. There is a certain limitation in making comparisons among the
different cohorts because each cohort may have experiences that are different
from the other cohorts (Furno & Flanagan, 2008). Nevertheless, the tests on
the same concepts over the semesters captured accumulated snapshots of data that
revealed the strengths and weaknesses of not only the specific cohorts at
specific moments in time, but also over the longer-term, in this case two
years.

The librarians
found it challenging to develop perfect questions. Even though the librarians
encouraged students to use reference resources that are considered more
reliable than the freely available but, qualitatively, highly inconsistent
sources on the Web (Q3), and they would also like students to learn about
subject searching in the catalog because it is an efficient search method (Q6),
these questions could arguably have more than one correct answer depending on
user preferences and topics. The minor revision in the test questions in spring
2011 may have also affected test results albeit the impact seemed
insignificant. The authors continue to improve the assessment instrument,
including developing questions with multiple correct answers and the
opportunity to select multiple responses for each question. These changes will
reduce the impact of guessing and increase the psychometric quality of the
items. The librarians also intend to develop alternate versions of the test so
different pretests and posttests can be employed. This will reduce the effect
of memory for prior responses on performance and provide a better estimate of
learning.

Conclusions

Despite the
described limitations, the brief tests yielded meaningful information about the
students’ IL abilities for the Moore librarians, allowing for adjustment of IL
instruction and future assessment.
Among the findings, the IL program reached over 70% of the undergraduates
beyond their freshmen year. Students’ IL levels on the first two IL objectives
were relatively low, but significantly higher for IL Objective 1 than for IL
Objective 2. This means their skills
in identifying needed resources (ACRL IL Standards 1) were higher than those
related to information access (ACRL IL Standards 2). More importantly,
students’ basic IL levels correlate positively with their academic levels (e.g.
the Honors group), but not with their class years (e.g. freshman, sophomore) or
with the number of prior IL instruction they received. The pretests and
posttests of the Research Writing classes revealed that students’ gains over
one-session ILI were limited. Participants did improve significantly after the instruction on two IL concepts:
truncation and subject searching in the catalog.

The finding that
students did not improve their IL skills significantly via single-session ILI
is not new to the literature. On the other hand, the literature contains very
few investigations into students’ longer term learning outcomes in IL. This
study shows participants not demonstrating progress in IL despite possibly
multiple prior ILI over a span of years. This finding suggests that students in
all class years (including graduate students) need continued reinforcement of
basic IL concepts and skills.

The findings raise
the important question as to what can be done to help students learn and retain
IL more effectively in college. More – and multiple – teaching strategies,
including a combination of online, face-to-face, embedded librarian, credit
course, and curriculum mapping may be considered in the library’s future
instruction program as resources are made available. The librarians have also
considered other curricular changes such as providing students and/or teaching
faculty with answer sheets with rationales for each response for continual
learning. In addition, the Moore librarians planned with professors in
experimenting different pedagogies to reinforce student learning of IL in the
following semester. One had classes preview IL research guides and take a
graded quiz before ILI, another included interactive learning activities during
the ILI, and the other had multiple ILI and follow-up sessions. Comparisons of
these approaches will help determine which teaching methods might be the most
effective means for helping students improve their IL skills.

Acknowledgements

The authors wish
to thank the instruction librarians at the Franklin Moore Library of Rider University
for their support and dedication in administering the assessment instruments to
students in the information literacy instruction sessions in the past few
years. These colleagues include Diane Campbell, Robert Congleton, Melissa
Hofmann, Katharine Holden, Robert Lackie, Marilyn Quinn and Sharon Yang. We
also wish to express our appreciation to the Moore Library Assessment Committee
for the ideas its members offered during the design phase of this research
project and for its support throughout, and to colleagues Kathy Holden for
proofreading and to Hugh Holden for polishing the manuscript.

Hufford, J. R.
(2010). What are they learning? Pre- and post-assessment surveys for LIBR 1100,
introduction to library research. College
& Research Libraries, 71(2), 139-158. Retrieved 26 July 2013 from http://crl.acrl.org/content/71/2/139.full.pdf

Schroeder, R.,
& Mashek, K. B. (2007). Building a case for the teaching library: Using a
culture of assessment to reassure converted campus partners while persuading
the reluctant. Public Services Quarterly,
3(1-2), 83-110.

To find the
critiques on William Shakespeare’s play Romeo and Juliet, in the Online
Catalog, I would do a search:

a. By title

b. By keyword

c. By subject

d. By author

e. Don’t know

* Question changed to “What is an efficient way to find critiques on
William Shakespeare’s …” in S6.

7.Which is the correct search strategy to
combine terms with the operators (AND, OR)?

a. Death penalty or capital punishment and women

b. Death penalty or (capital punishment and women)

c. (Death penalty
or capital punishment) and women

d. (Death penalty and women) or capital punishment”

e. I don't know

* Question changed to
“Which search statement is correct when you search for information on the topic
‘Should women be exempt from death penalty?’” in spring2011.

8.Truncation is a library
computer-searching term meaning that the last letter or letters of a word are
substituted with a symbol, such as “*” or “$”. A good reason you might truncate
a search term such as child* is that truncation will

a. limit the search to descriptor or subject fields

b. reduce the number of irrelevant citations

c. yield more
citations

d. save time in typing a long word

e. I don't know

9.In order to find more documents on my topic I
can include synonyms in my search statement. To connect those synonyms in my
statement, I use:

a. AND

b. +

c. NOT

d. OR

e. I don’t know

10.Choose the best place to find a reliable and
detailed history of television in the US for a research paper.

a. Book

b. Website

c. Magazine/newspaper

d. Scholarly Journal

e. I don’t know

*Question changed to “Choose the best place to find a reliable and
thorough history ….” in spring 2011.