Reference Management Software: a Comparative Analysis of Four Products

Copyright 2011, Ron Gilmour and Laura Cobus-Kuo. Used with permission.

Abstract

Reference management (RM) software is widely used by researchers in the health and natural sciences. Librarians are often called upon to provide support for these products. The present study compares four prominent RMs: CiteULike, RefWorks, Mendeley, and Zotero, in terms of features offered and the accuracy of the bibliographies that they generate. To test importing and data management features, fourteen references from seven bibliographic databases were imported into each RM, using automated features whenever possible. To test citation accuracy, bibliographies of these references were generated in five different styles. The authors found that RefWorks generated the most accurate citations. The other RMs offered contrasting strengths: CiteULike in simplicity and social networking, Zotero in ease of automated importing, and Mendeley in PDF management. Ultimately, the choice of an RM should reflect the user's needs and work habits.

Introduction

Reference management is one of the most complicated aspects of being a researcher. The tedium of formatting references based on a variety of citation styles has made the reference manager (RM) an essential tool for scholars at all levels. The variety of RMs available makes it difficult to know which tool to select. Although there are a number of RMs on the market, we selected the following for analysis: CiteULike, RefWorks, Mendeley, and Zotero. These RMs are well known within the scientific community (Hull et al. 2008; Norman 2010; Duong 2010; Mead and Berryman 2010). We chose RefWorks as our only representative of the more traditional style of RM. For an analysis of the desktop-based EndNote, see Hensley (2011).

RMs serve a variety of functions. Generally, we would expect an RM to be able to:

Not every RM offers all of these features. Neither RefWorks nor CiteULike, for instance, can extract data from PDF files. Generally, online-only tools, such as CiteULike, tend to have poorer integration with word processing software than tools that employ a desktop client.

The strengths and weaknesses of the different RMs often reflect the original impulse behind their creation. In our analysis, CiteULike scored quite low in regard to the production of formatted citations. It describes itself as "a free service to help you to store, organise and share the scholarly papers you are reading." Citation formatting, presumably, is not its raison d'être. RefWorks is the only one of the four products analyzed that specifically mentions "writing" on its homepage.

Literature Review

The research landscape has changed tremendously since the first RMs were developed in the 1980s. Since then, we have moved from the PC revolution of the 1980s to the World Wide Web and finally Web 2.0 (O'Reilly 2005). Although the traditional RMs were desktop tools that required floppy-disk drives, they had the same major features that we currently expect of RMs: storing references, organizing references, and creating bibliographies in a variety of styles with word processing integration (Warling 1992).

In today's world of open-source and Web 2.0 start-ups, there is a corresponding emergence of RMs that are tailored to the demands of users with different needs and expectations. The web has changed from a platform where one author produces content for many to a space where anyone can be a content provider. The newer RMs, such as CiteULike, Zotero, and Mendeley, all created since 2004, have incorporated social, collaborative features (Hull et al. 2008; Mead and Berryman 2010; Duong 2010; Pucket 2010). This allows a user to share a personal library within a private or public group and to decide at what level to collaborate and be "discovered" by other researchers. Users may also search for citations in the collective library that are similar to those stored in one's own library. This creates a resource discovery role for the RM, letting researchers expand their bibliographic reach and perhaps interact with other researchers in their field. In addition to these social elements, the newer RMs provide the features expected in a traditional reference manager.

Some have argued that it is not the users themselves who have changed, but their workflow (Mead and Berryman 2010). The all-too-familiar scenario as discussed in the literature depicts the researcher with many PDFs stored in various places who needs a tool to simply upload the documents and pull the citation information into their RM product of choice (Mead and Berryman 2010; Barsky 2010). Both Mendeley and Zotero provide this service, although extracting metadata is not a perfect science. Metadata inconsistency is a direct result of there not being a dominant standard for metadata style (Hull et al. 2008). Mead and Berryman (2010) observe that when critical mass is reached with a tool such as Zotero or Mendeley, the community's collective wisdom will assist in capturing correct metadata.

With all this in mind, the question becomes: What are the primary and secondary needs of the user based on workflow? There are seasoned researchers who are quite attached to their "1.0" RMs, such as EndNote or RefWorks, and may not feel the need to change their ways. There are new researchers who are more flexible in their work habits and may be more willing to learn new RMs that provide Web 2.0 functionality and PDF features. Finally, there are students who just need to generate a bibliography. As the traditional RMs respond to the changing needs and workflow of users while the "2.0" RMs continue to improve and compete for market share, it will be interesting to see which tools are able to "have it all" with low cost and maximum functionality for the user. At present, we are in a transitional state in which users may be working with both "1.0" and "2.0" tools to have the best of both worlds (Mead and Berryman 2010; Cordón-García et al. 2009). As a way of addressing these issues, we have systematically tested and reviewed four RMs with the hopes of offering some guidance in making the best decision based on one's own unique needs and workflow.

Product Information

RefWorks (www.refworks.com), developed in 2001 by a business unit of ProQuest, is a web-based RM that requires a fee-based license. Individuals can purchase a subscription, but institutional accounts provide more options and features.

CiteULike (www.citeulike.org), developed in 2004 and sponsored by Springer Science and Business Media, is a free online RM. CiteULike's primary intention is to act as a social networking tool by letting users search for related articles or helping users to connect with researchers who have similar interests.

Zotero (www.zotero.org), developed in 2006 by George Mason University's Center for History and New Media (CHNM), is a free, open source plug-in for the Firefox browser. Once installed, the user simply clicks on an icon in the address bar to save the citation information in one's library without navigating away from the web page. At present, Zotero has an alpha version of an independent desktop client, Zotero Standalone, which will work on Mac OS, Windows, and Linux, offering independence from the Firefox browser.

Mendeley (www.mendeley.com), was developed in 2008 by a Web 2.0 start-up. Mendeley offers a free package with the option to upgrade for more individual and shared storage space. Mendeley was modeled after Last.fm. Like the social networking music site, it provides recommendations based on one's interests (Fenner 2008). It offers both a desktop and web version, with Mendeley Web giving users access to social features such as sharing references with other users or discovering research trends (Barsky 2010; Fenner 2008).

Getting Data into the Reference Manager

From a Bibliographic Database

In order to see how different reference managers interact with different bibliographic databases, we selected seven bibliographic databases and two articles from each to use in our testing. The seven databases were: arXiv, BioMed Central, PLoS, EBSCO's CINAHL, Ovid's Medline, ScienceDirect, and PubMed. These databases were chosen because they are considered highly relevant for researchers in the health and natural sciences. A complete list of these articles by source database can be found in Appendix 1.

Each article was located in the bibliographic database and its metadata imported into each of the RMs. We tried to find a direct method of import, but when this was not possible, we used an indirect method. A "direct" method is one in which a link, bookmarklet, or icon is clicked to transfer the data into the reference manager. An "indirect" method is a two-step process of exporting the record from the bibliographic database (usually in RIS or BibTeX format) and then importing that record into the RM. This follows the usage of Cordón-García et al. (2009).

Let us first examine the process of getting bibliographic data into the RM. Factors considered include:

Table 1 summarizes the data importing compatibilities between the four reference managers and the seven bibliographic databases. In this list, Zotero stands out as having direct downloading compatibility with the largest number of bibliographic databases. When Zotero identifies a web site as a provider of bibliographic data, it displays an icon in the Firefox address bar corresponding to the source type. Clicking this icon instantly downloads the data into Zotero. Ovid proved troublesome with Zotero, which insisted on interpreting an Ovid RIS record as a reference to a web page rather than a journal article. It therefore ignored such information as volume, issue, and page numbers.

RefWorks performed well in cases where direct importing was available, but indirect importing was often puzzling, requiring some trial and error. This was especially true with arXiv and PubMed. RefWorks, a commercial product, provides easy direct importing from other commercial products, but not from free resources such as PubMed and arXiv.

Mendeley also performed well. Its bookmarklet did not work with ScienceDirect, EBSCO, or Ovid, but data was easily imported through an intermediate format.

CiteULike performed poorly in regard to data importing. Automatic importing is accomplished through a bookmarklet. It was completely ineffective with PubMed, which does not allow records to be downloaded as RIS or BibTeX. [Erratum]

From a PDF

Both Mendeley and Zotero have the ability to acquire metadata from a PDF document. This is of great benefit to researchers who may have previously relied on a library of PDFs stored on a hard drive. The documents can simply be dragged into the Mendeley or Zotero client window. Mendeley will automatically try to locate metadata. Zotero must be asked to do so explicitly by right clicking on the item and selecting "Retrieve metadata for PDF." Limited testing suggests that Mendeley is far more likely to be able to acquire PDF metadata than Zotero.

RM Database Features and Functions

Organization

Reference managers offer a number of ways to organize references. Generally, these fall into the categories of the old-school "folder" approach to organization and the more "Web 2.0" approach of tagging. Of the products tested, only RefWorks does not explicitly allow tagging, although a skilled user could edit the "descriptors" field to include his own taxonomy of tags. Only CiteULike lacks an explicit "folders" feature, but folder functionality could be emulated by either judicious use of tags or by creating groups. In addition, Mendeley allows users to search the full text of one's PDF library, which has been argued to be a more straightforward method of organizing and discovering information, thereby, eliminating the need for folders or tags (Crotty 2009).

Searching and Sorting

All of the products tested allow searching within one's library of references, including searching of user-created notes. CiteULike and Mendeley also let users search the entire universe of references in their respective databases, making them tools for resource discovery. These two products also allow users to assign importance ratings to references and to use these as a sorting criterion. This feature is best developed in CiteULike, where users can rate references on a five-point scale ranging from "I don't really want to read it" to "Top priority!" CiteULike highlights frequently tagged articles in its "CiteGeist" list. This is certainly not a mission-critical feature, but it may enhance the value of CiteULike as a current awareness tool. Zotero and RefWorks allow searching only within one's own library, but offer field-specific searching.

Annotation

All of the products tested allow users to add notes to their references, and, if sharing is supported, to make those notes available to collaborators. Mendeley offers the important additional capability of letting users add notes directly to a saved PDF. This unique feature is very useful for researchers with large PDF libraries. It can be especially helpful when dealing with large documents, where it might be important to associate notes with particular parts of the text. CiteULike, in keeping with its more "social" orientation, lets users post public comments on references.

Data Migration

Movement of data between various reference managers is generally very easy, thanks to standardized metadata formats, especially RIS and BibTeX. These two formats can be both exported and imported by all of the RMs examined. Formats other than RIS and BibTeX are supported by some of the products. CiteULike is unique in offering automatic syncing with a Delicious account, while Zotero allows data interchange with Firefox bookmarks, and Mendeley supports syncing with a Zotero account. Importing and exporting of data provides security as a user can download a backup of his library. It also offers flexibility by facilitating migration to a new product or enabling a user to use two RMs in concert.

Data Storage

CiteULike offers unlimited, free storage. RefWorks also offers unlimited storage, once one has paid for the product. Zotero and Mendeley offer limited free storage (100 MB and 1 GB, respectively), with the option to purchase additional storage if it is needed. Citations themselves do not take a great deal of memory, but associated PDFs can consume memory very quickly.

Sharing

All four of the products under consideration offer some capability for sharing part or all of one's reference library with other users of the product. RefWorks, as a commercial product, has a distinct disadvantage: one can easily invite a new collaborator to open an account with CiteULike or another free product, but asking him to subscribe to RefWorks would be a serious imposition. Perhaps for this reason, RefWorks is the only one of the products that does not offer the ability to create personal profiles or to locate researchers with similar interests. RefWorks offers an add-on called "RefShare" for institutional subscriptions only, which allows one to let others see a set of references, but in a completely non-interactive way. In order to truly share libraries of references, one would have to create a new account and share that username and password with others (a capability that exists without paying extra for RefShare).

"Social" features of the other three managers are similar to each other, mimicking the functionality of popular social web applications such as Facebook and Delicious. All allow users to create groups and to invite others to join those groups in order to share references. "Sharing," in this context, means that group members can edit the RM database. These groups may be designated as public or private, with the later being limited to invited members. Mendeley is unique in allowing only a certain number of members in private groups before one must pay for additional membership slots. All allow the creation of personal profiles, which may include CVs, research interests, etc. Mendeley and Zotero offer the most robust suites of social features, with the ability to send messages to other members and the ability to search and browse for public groups.

Citing References and Creating Bibliographies

In addition to their organizational functions, most RMs are able to produce formatted citations. This functionality is a boon to scholars who find themselves spending far too much time navigating the intricacies of multiple citation styles. When this feature works correctly, it can save a great deal of time. Even when it works poorly, it is usually faster to edit an automatically generated bibliography than to produce a bibliography by hand.

Most of these products support a huge number of citation styles. CiteULike is deficient in this respect, offering only sixteen styles. Zotero and Mendeley come with a certain number of styles installed, but allow additional styles to be downloaded easily. The flexibility offered by having a large numbers of styles at one's disposal is especially desirable in the sciences, where, unfortunately, each journal is a standard to itself.

In order to assess the quality of the various RMs, we used the data that had already been imported for the data gathering evaluation described above. We selected five citation styles commonly used in the sciences that were supported by all four of the RMs. For each style, we produced a reference list using each of the RMs. No modification was made to any data within the RM. Bibliographies produced by CiteULike contained only twelve references, rather than the total fourteen, since there were certain databases, PubMed and EBSCO respectively, from which data could not be automatically imported.

Once these bibliographies were produced, we counted the number of errors (E) as well as the number of perfect citations (P). These results are shown in Table 2.

CiteULike (N=12)

Mendeley (N=14)

RefWorks (N=14)

Zotero (N=14)

ACS

E

19

15

13

11

E/N

1.58

1.07

0.93

0.79

P

0

0

2

7

AMA

E

42

16

5

20

E/N

3.5

1.14

0.36

1.43

P

0

0

10

4

APA

E

33

22

8

17

E/N

2.75

1.57

0.57

1.21

P

0

3

6

5

IEEE

E

50

38

23

23

E/N

4.17

2.71

1.64

1.64

P

0

0

2

1

Nature

E

29

15

7

20

E/N

2.42

1.07

0.5

1.43

P

0

2

9

3

Avg. E/N

2.88

1.51

0.8

1.3

Table 2. Total errors (E), errors per citation (E/N), and number of error-free citations (P). At the bottom of each column, E/N is averaged over the five citation styles.

In terms of citation accuracy, RefWorks emerges as a clear winner (E/N=0.8), with Zotero (E/N=1.3) and Mendeley (E/N=1.08) lagging some distance behind. Citations from CiteULike (E/N=2.88) were disappointing.

The kinds of errors produced were fairly consistent, especially within a particular citation style. Perhaps the most common of all errors involved the use of journal title abbreviations. RefWorks is the only product tested that seemed to even attempt to correct abbreviations of journal titles. Errors in the capitalization of article titles (sentence case vs. title case) were also common. Reference managers generally do not change article title capitalization and simply output whatever was initially imported into the database. Article titles were usually downloaded in sentence case, which is favored by all the styles tested except ACS, so this was often not a problem, but by accident rather than design. It was also common for RMs to ignore rules about how many authors should be listed in cases where there are many. IEEE, for instance, lists only the first three authors with "et al." This rule was applied inconsistently or not at all by all four of the products tested.

Questions about whether to include a URL, and what that URL should look like, are so complex that the APA actually produced a flowchart to guide users through various situations (APA 2009). It is not surprising that RMs had a hard time with this, often including proxied URLs, which would only be of use to other members of the author's institution; or overly long URLs that would be unacceptable in print.

For the sake of uniformity in our testing, the data stored in the RM was left exactly as imported. A more realistic workflow would be to briefly look over the data when it is imported, and edit it as necessary. This would result in improved citations, especially with reference to URLs.

Word Processor Integration

Being able to create bibliographies is only half of the citation process. Ideally, a reference manager should also be able to assist with in-text citations, a task that requires the RM to work with the word processing software. Three of the four RMs tested offer word-processor integration. All require the download of a plug-in. Mendeley and Zotero both offer word-processor integration for Windows or Mac OS, and for either Microsoft Word or OpenOffice. The operation of these systems is similar, each adding a citation toolbar (Windows) or menu (Mac OS) that includes commands such as "add citation" and "insert bibliography." We tested these systems on Word 2008 for Mac and found them easy to use and effective.

RefWorks offers a program, Write-N-Cite, which integrates only with Microsoft Word, on either Windows or Mac OS. The earlier version of Write-N-Cite, which is the only one available for Mac OS, is very confusing and prone to crashing. A newer version, for Windows only, has solved some of these problems and offers some additional features.

Discussion

Our study is based on a small sample of citations, without a formal statistical analysis. Most of these were traditional journal articles, those being the most common type of citation in the scientific literature. We did not, for instance, test the ability of these products to grab information from web sites. Nor did we examine all features of the products, limiting our analysis to areas of core functionality that most RMs share.

Another limitation of our analysis stems from the fact that the origin of errors in the bibliography can come from multiple sources. For example, if a journal title appears in all caps, this may be because the bibliographic database provided it in that form rather than because of any deficiency in the RM. As mentioned above, we did not make any alterations to the imported data before generating the bibliography.

Table 3 provides an admittedly subjective rating of various functions of the RMs. We scored CiteULike poorly relative to the other products, but it does have the advantage of ease of use. Mendeley has the highest cumulative score and offers unique features for the management of a PDF library and advanced annotation features. RefWorks offers the most accurate bibliographic output, but scores low in other areas such as sharing and word processor integration. Zotero's score is close to that of Mendeley, and it is especially strong in being able to directly import information from a number of sources.

CiteULike

Mendeley

RefWorks

Zotero

Import from bibliographic databases

2

3

4

5

Gather metadata from PDFs

N/A

5

N/A

2

Organize citations in RM

4

5

5

5

Annotate citations

4

5

4

4

Share library

4

4

1

3

Exchange data with other RMs

5

5

4

5

Format citations in multiple styles

1

3

5

3

Integrate with word processors

N/A

5

2

5

Sum

20

35

25

32

Table 3: Our ratings of the feature sets of the four RMs on a scale of 1 to 5, with 5 being best.

Conclusions

Choosing the best reference manager will be dependent on the needs and workflow of the individual user. Each of these tools has its own strengths and weaknesses that will be of greater or lesser importance to different groups of users. One's choice will inevitably be influenced by computing environment. Some libraries, for instance, may restrict the software that may be downloaded to a desktop computer, making a primarily web-based solution like CiteULike or RefWorks the most practical choice (although Zotero and Mendeley both include a web component).

New users should consider the amount of time that they are willing to devote to learning a new RM. RefWorks, while it produces very accurate bibliographies, has a steeper learning curve than the more "2.0" products. One might also consider the future disposition of data. Is this a "one off" project, where the reference list will be disposable at the end of the semester, or will the library of references be the basis for ongoing research? In the former case, an easy-to-use product would probably be best. In the latter case, factors such as storage capacity, annotation features, and searching capabilities may weigh more heavily.

Users differ in the degree of importance that they attach to flawless bibliography generation. If initial accuracy of citations is paramount, then the financial and temporal commitments required by RefWorks may be worth it. If the user is comfortable editing the bibliography (or, better, the records in the RM database), then one of the simpler, free tools may be a better choice.

In an institutional context, the issue of support becomes more important (Hensley 2011). RefWorks, as a commercial product, comes with a certain amount of technical and instructional support. Of course, this benefit comes with a considerable price tag. The various free products do not offer this, but most have enthusiastic user communities whose help can be sought through message boards, mailing lists, etc.

Reference managers continue to evolve, so any decisions may be temporary. The product could vanish, or newer ones with better features could appear. It is good to keep abreast of changes in this area and enjoy the process of experimenting, while taking portability and backup into consideration.