2013 Faricy-Beredo.
This is an Open Access article distributed under the terms of the Creative
Commons-Attribution-Noncommercial-Share Alike License 2.5 Canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/),
which permits unrestricted use, distribution, and reproduction in any medium,
provided the original work is properly attributed, not used for commercial
purposes, and, if transformed, the resulting work is redistributed under the
same or similar license to this one.

Abstract

Objective - To use a multi-dimensional approach to evaluate the
effectiveness of a nursing information literacy program (Pathways to
Information Literacy) delivered to undergraduate nursing students. Assessment
sought to track progress in both affective and cognitive spheres.

Design – This program evaluation focuses on thePathways to Information Literacy (PIL)curriculum, which was delivered from
1988-1992. It consisted of 6 hours of librarian-delivered instruction, divided
over 4 sessions. To evaluate the impact of this curriculum, the authors
gathered five different data sets: informal feedback; the results of a survey
measuring the affective domain of confidence; the results of a longitudinal
cohort survey of graduates; and two different sets of data gathered from
distinct samples but utilizing the same information literacy assessment tool.
All five data sets served the greater purpose of assessing students’ mastery of
information literacy.

Setting - An undergraduate Bachelor of Science in Nursing (BSN)
program within a state university, the University of Northern Colorado.

Subjects - In general, the subjects were different cohorts of
nursing students between 1988 and 1992. Class size hovered at just over 100.
For most measures, response rate was high and the dropout rate was low. It is
worth noting that one of the assessments was a longitudinal cohort survey of
graduates. As the mobility of graduates often decreases the number of
responses, the omission of the response rate for this measure is of concern.

Methods
- The methods are one of the most striking aspects of this study. The
authors employed no less than five methods of assessment:

From 1988-1992, investigators gathered informal feedback from both
students and faculty members about the written assignments of the PIL
program. The specific method for gathering feedback was not reported.

From 1988-1990PILstudents in their junior year took
pre- and post- PILprogram
confidence surveys. The survey tool, which was included, contained 6
Likert-like questions which assessed their affective domain of confidence
as related to their ability to perform information literacy related tasks,
such as using a bibliographic index.

During the 1990/91 academic year the authors administered a
general, university-wide information literacy assessment tool, which was
included. It measured both self-perceived progress and objectively
measured skill attainment in the cognitive domain. The survey tool lacked
validation, but had been previously published in the library literature (Greer, 1991). Students from the PIL
program were extracted from the data for comparison to the general student
population.

Both PIL (1990/91) and non-PIL (1988/89) cohorts completed a
longitudinalpost-graduation
survey which was included. Comparison of the two cohorts aimed to examine
the effect of the PILprogram on
subsequent scholarly professional activities.

While the methodologies were all tied to the overall
purpose of program evaluation, they were not tied to specific pedagogies or
content units.

All outcomes supported the positive effect of the PILprogram. The soft technique of
gathering informal feedback from students and faculty resulted in positive
feedback. Faculty reported that their students became independent in
information-seeking and the quality of their papers increased. Students also
reported that the assignment and instruction gave them confidence and that the
written assignments were a nice break from the traditional examinations. The
affective confidence survey noted a substantial improvement: pre-program only
26% reported confidence when performing information-seeking strategies compared
to 76% post-instruction. When the information
literacy assessment toolwas administered both to 68 PIL participants and to
208 general students, the PIL students both believed themselves to be more
successful and demonstrated greater knowledge. 70% of PILstudents answered CD-ROM index questions correctly, compared to
49% of general students. When the same tool was given to only PIL students as a
pre-and post- test, it showed statically significant increases in the use of the library
and mastery of several specific search techniques: p<0.05 for 7 of 20
measures. The post-graduation survey showed that 45% of the PIL students had
engaged in some scholarly activity, as compared to 10% of non-PILgraduates.

Conclusion - The authors concluded that the multidimensional
assessment efforts delivered a comprehensive view of the effectiveness of the
program, demonstrating student benefit in cognitive (knowledge attainment) and
affective (confidence levels) domains as well as in subsequent professional behavior.

Commentary

A multidimensional evaluation of a nursing information-literacy program
(MENILP) (Fox, Richter & White, 1996) contains a
description of the Pathways to
Information Literacy (PIL) program. This program consisted of four
instruction sessions, which were integrated across two required nursing courses
in the junior year of study. Though the instructional pedagogies of the
sessions were not detailed, the authors did provide the goals for the
instruction and mentioned several assessment methods. The goals for the
instruction included:

develop
an understanding of library organization and services

acquire
skills in forming research questions and locating and evaluating accurate,
relevant information for problem solving

The first information literacy instruction session
targeted the skills of information location and synthesis. Mastery of the
course was measured by an exam and an assignment which required the students to
synthesize general sources to create a topical summary of a condition and its
treatment. Session two introduced more advanced and nursing-specific research
techniques such as locating and using specialized handbooks and citation indexes.
Librarian-led group discussion surrounding conflicting research findings was
utilized as a synthesizing methodology. Session three presented the process of
topic selection and research question refinement. It offered the students
in-class time to work on their topics using a topic selection grid tool.
Session four focused on computer search strategies, including CINAHL on CD-ROM,
which was a relatively new addition to the library’s collection. Students were
taught effective search strategies for CINAHL and asked to demonstrate their
knowledge by submitting a print out of their strategy. A detailed description
of the program’s rationale and creation can be found in an earlier work by the
same authors (Fox, Richter, & White, 1989).
Though well done, that project description has not had as great an impact as
MENILPwhich focuses on program
evaluation.

The program evaluation plan for the PIL program
pre-dated many of the current, widely-used frameworks for program evaluation,
such as the Framework for Program Evaluation
in Public Health(Centers for Disease
Control and Prevention, 1999). The authors did however utilize an
appropriate evaluation framework, selecting Staropoli
and Waltz’s model (1978) which is
specific to health education programs.The model takes a familiar five question format, asking first, who will
be involved in the evaluation? The faculty and the librarian.
What are the purposes in conducting an evaluation? To
evaluate the effectiveness of the program and measure the degree of skill
acquisition. What is to be evaluated: curriculum, objectives, faculty or
students? Students’ mastery of information literacy in the
cognitive and affective domains. How is the evaluation to proceed?The authors summarize the past practices and
detail plans for on-going evaluation “every two-three years” (Fox et al., 1996, p. 184). While this older
model pre-dates the now commonly seen emphasis on evidence, the authors were
ahead of their time in broadly defining their purpose as creating “[o]bjective procedures [which] should be used to facilitate
the collection of dependable, unbiased data… to determine value” (Fox et al., 1996, p. 184).

MENILP significantly impacted subsequent
scholarship.A three-pronged approach
using the cited reference search functions of ISI Web of Knowledge, CINAHL, and
Google Scholar identified 39 subsequent citations spanning 17 years. Although
most of the publications were within the field of Library and Information
Science, approximately 25% were from Nursing/Allied Health. Google Scholar, as
expected, identified the greatest diversity of formats (books, thesis, articles
and reports) as well as the most non-English language publications, six in
total (Kleibel & Mayer, 2005; Meneses
Placeres, 2008a, 2008b; Qun, 2011; Nodarse Rodríguez, 2005; Sundin, 2003).

Figure 1

Number of MENILP citations by year

The five-prong evaluation strategy was one of the main
drivers of the article’s popularity.Each of the evaluation methods designs can be appraised in terms of
evidence quality. The informal feedback, while illuminating, did not have the
power to influence practice. The other four methodologies involved either
survey or tests; none of which utilized validated tools. The full text of the tools
was included so the reader is able to make some judgments
about the content and face validity of the measures. When the information
literacy assessment toolwas used as
a pre- and post-test, the results
were determined statistically to be significant, with reported p values. When
it was used to compare PIL students to the larger under-graduate population,
there are concerns. The PIL students differed significantly from the general
student population. No controls were put in place for the nursing students’
higher average GPAs, greater average age and their skewed gender, though the
authors noted these differences. The confidence survey’s descriptive statistics
were impressive but are not analyzed for statistical significance. The
post-graduation survey suffers from the same limitations. It does however have
the most interesting design of all the methodologies: a cohort, longitudinal
survey. When considering an evidence base, this type of study design is
considered strong and is especially well-suited to look at questions of
etiology (MacKibbon, Wilczynski, & Eady,
2009). In this design, the PIL program was the exposure which was
controlled for in two different cohorts of graduates.The PILgraduates, when compared to their non-PIL peers using simple descriptive
statistics, reported more professional reading and greater participation in
scholarly activity. Fox and colleagues do not discuss the internal/external
validity, reliability or limitations of their methodologies at any point.

Though the individual results dim under an analysis of
their strength of evidence, they do synergistically work together to form a larger
preponderance of evidence. When considered as a whole, the methodologies create
a body of data that delivers its own internal triangulation of results and meet
the authors’ objective of a collection of data which determines value. The
thoroughness and ultimate success of this work as a program evaluation goes a
long way in explaining why this article has had such an impact. It was of
particular relevance to the subsequent researchers who were picking up on the
growing evidence-based trend that was occurring both in program evaluation and
in nursing practice.

These subsequent researchers have used and continue to
use this work in divergent ways.The
multiple methodologies and conclusions are often considered in piecemeal
fashion. This study provides support or evidence for:

the
definition of information literacy (IL)

the
positive effect of IL instruction in general as well as within the
affective and cognitive domains

While not as note-worthy within the library
literature, several works from the nursing perspective addressed the strength
of the collaboration between librarians and nursing faculty described inMENILP (Carter-Templeton,
2011; Honey, 2007; Schloman, 2001).The library literature was much more likely
to focus on the aspect of curriculum integration.

Evidence-Based Practice

Obviously the world has changed since 1996. Of
tremendous import to the practice of health science librarianship has been the
advent of evidence-based practice (EBP). Though EBP was not an explicit concern
of Fox et al., their work does eloquently speak of a nurse’s need to be able to
acquire and appraise information, which are critical steps in the EBP process.
By clearly addressing these concerns their study drew the interest of later EBP
investigators (Durando & Oakley, 2005; Jacobs et al., 2003; Rush, 2008;
Shorten, Wallace & Crookes, 2001; Urquhart, 1998).

A multidimensional evaluation of a nursing information-literacy program could have been sprawling mess of an article. It has the clear aim of a
program evaluation but presents five different evaluation strategies, spanning
a five year period. The evaluation strategies themselves vary from simple
informal feedback to a complex cohort design. It tries to measure both
affective and cognitive changes. With such a broad focus in terms of time,
methodology, and approach it certainly had the potential to fail. Instead it is
a powerful, enduring article with a global impact. Its forward-thinking focus
on evaluation and evidence continues to serve the needs of scholars both inside
and outside of librarianship. It is well-deserving of the descriptor “classic”.