The success of citizen science in producing important and unique data is attracting interest from scientists and resource managers. Nonetheless, questions remain about the credibility of citizen science data. Citizen science programs desire to meet the same standards of credibility as academic science, but they usually work within a different context, for example, training and managing significant numbers of volunteers with limited resources. We surveyed the credibility-building strategies of 30 citizen science programs that monitor environmental aspects of the California coast. We identified a total of twelve strategies: Three that are applied during training and planning; four that are applied during data collection; and five that are applied during data analysis and program evaluation. Variation in the application of these strategies by program is related to factors such as the number of participants, the focus on group or individual work, and the time commitment required of volunteers. The structure of each program and available resources require program designers to navigate tradeoffs in the choices of their credibility strategies. Our results illustrate those tradeoffs and provide a framework for the necessary discussions between citizen science programs and potential users of their data—including scientists and decision makers—about shared expectations for credibility and practical approaches for meeting those expectations.

This article has been corrected here: http://dx.doi.org/10.5334/cstp.91.

While government-based natural resources monitoring is notoriously hard to fund and implement owing to a variety of political and practical challenges (Biber 2013), recent reviews suggest that citizen science has great potential to meet monitoring needs cost effectively (Aceves-Bueno 2015; Berkes 2009; Connors et al. 2012; Danielsen et al. 2007; Roy et al. 2012). Consequently, increasing numbers of citizen science programs and projects are striving to meet agency needs for monitoring data. For example, the US National Oceanographic and Atmospheric Administration has formed a citizen science community of practice (NOAA 2014), and the Scottish Environmental Protection Agency has published a guide describing when and how to use citizen science for environmental monitoring (Pocock et al. 2014). A recent survey of 83 citizen science projects found that more than 40% had generated data that were used by natural resource managers (Aceves-Bueno et al. 2015). Yet, while citizen science is gaining legitimacy in decision making and in mainstream science, questions about the credibility of citizen science results are still common (e.g., Henderson 2012; Nature 2015; Wiggins et al. 2013).

Potential audiences such as scientists and resource managers must recognize the credibility of citizen science in order to use its data. But citizen science is tremendously diverse, and credibility standards vary by project. Thornton and Leahy (2012) suggest that the typical academic signals of trust, peer review, and quality assurance are not sufficient for building trust in citizen science data. However, while citizen science groups have been urged to explicitly consider credibility (Wiggins et al. 2013), practical guidance for what such consideration should entail is still limited.

Credibility is broadly defined as the quality of being believable or worthy of trust. For science (citizen or otherwise), credibility has both technical and social components. The field of science and technology studies (STS) consistently highlights the socially and politically contextualized nature of any scientific endeavor (e.g., Fischer 2002; Jasanoff 2004). New forms of science, including many kinds of citizen science, are not judged simply on technical attributes. They also require new social practices encompassing the governance, execution, vetting, and use of new science (Funtowicz and Ravetz 1993). Therefore, a citizen science group may require many pathways to achieve the necessary credibility to imbue trust in its data (Lawrence 2009). The ways in which citizen science groups, like any scientists, perceive the social and technical challenge of credibility will impact the data-quality practices they put in place.

Data users within a regulatory or management setting look for signals of credibility accompanying traditional academic science, such as peer review or investigator reputation (Keller 2009). While rarely explicitly discussed, these signals are an important element in arenas where legal or constituent defensibility is a requirement (Shapin 1995). It remains unclear whether, when presented with citizen-generated data, resource managers should seek the same, similar, or different signals of credibility, and the answer is probably context-dependent.

The credibility challenge specific to citizen science

While all science can face challenges to its credibility, the specific context of citizen science, along with external assumptions about citizen science, can make establishment of its credibility particularly difficult. Implementing science projects with volunteers, often outside traditional science institutions and typically with limited resources, contributes to the credibility challenge. Preconceptions about citizen science also can be an issue. For example, citizen science data can be distributed, hard to access, and may have incomplete metadata, presenting obstacles for their use (Dickinson et al. 2010; Gouveia et al. 2004). Many of these issues can be overcome by using sophisticated methods of data analysis, however (Bird et al. 2014).

Another common assumption is that volunteers collecting data have an agenda that will impact their ability to record objective scientific observations (Gouveia et al. 2004, Nature 2015). While this is a valid concern for any type of science (Jentoft et al. 1998; McCormick 2007), volunteers can learn to eliminate bias through training and experience (Trumbull et al. 2000). A related assumption is that volunteers collecting data are untrained or unable to follow complex scientific protocols (Gouveia et al. 2004). However, comparative studies have demonstrated that volunteer capabilities can be equal to professionals (e.g., Gillett et al. 2012; Jordan et al. 2012). In addition, some citizen science projects are designed specifically to tap into the expertise outside of professional science, as with collaborative fisheries research (Aswani and Lauer 2006; Dale and Armitage 2011) or ethnobotanical studies (Kimmerer 2012; Oguamanam 2004).

Precedents for credibility in citizen science

Some national-level communities of practice have set precedents for expectations of credibility-building activities in citizen science. For example, the water quality community long ago embraced citizen science as an effective means to keep track of the many water bodies important to local communities and the national water supply. In response, the US Environmental Protection Agency provides methods manuals, official protocols, and protocol certifications, and it endorses groups that follow its guidance (US EPA 2014). Statewide networks have followed suit, such as the Surface Water Ambient Monitoring Program (SWAMP) in California, which provides technical assistance, training, quality assurance support, equipment loans, and communications help (SWRCB 2016).

America’s longest-running citizen science project, the Cooperative Weather Observer Program, was established in 1890 before the distinction between volunteer and professional scientist was commonly made (NWS 2015). Volunteer weather observer data are used by current weather forecasting models and help to calibrate technologies such as remote sensing and automatic weather stations. The high quality of these data are consistently proven by the accuracy of weather predictions at the local scale, and they allow for detecting more localized microclimates than other technological approaches (Doesken and Reges 2010).

The Data Observation Network for Earth (DataONE) includes citizen science as one of its main stakeholders when developing data management plans. Its work addresses common problems for all types of science: Dealing with different types of data, standards for format and metadata, and provisions for archiving, access policies, and eventual transition or termination of data. DataONE describes data management as a process that must occur at each stage of inquiry throughout the lifecycle of a scientific project. The DataONE citizen science working group agreed that the staged approach is appropriate for citizen science but has added some special prescriptions about the particularities of involving volunteers regarding data quality, data usability, and data accessibility (Wiggins et al. 2013). Adherence to this guidance is one way to gain credibility.

The birding community–in particular, the data platform eBird–is often held up as the gold standard in citizen science credibility (Connors et al. 2012; Newman et al. 2011). eBird’s enormous data set allows statistical methods to detect and accommodate volunteer error. For example, detected spatial or temporal bias might result in the project directing volunteers to cover less popular times or spaces or for its analysts to statistically weight observations (Kelling et al. 2013). Large data sets also allow a high degree of filtering focused on suspect data–i.e., first-year participants, people who submit data erratically, or participants who are known to submit erroneous reports (Sullivan et al. 2014). Big data metrics also allow for spatiotemporal modeling to flag and verify observations that are outside the expected patterns (Hochachka et al. 2012).

Finally, a growing number of cases show that citizen science volunteers can maintain community data standards through self-policing. This is especially true in the open source software community, which has demonstrated benefits to science from crowdsourcing which far outweigh the costs. For example, the crowdsourced mapping platform OpenStreetMap is stronger than professional platforms and can address more kinds of geographic questions, largely owing to interactions among volunteers and the culture of correcting one another (Newman et al. 2010). These large-scale projects showcase a set of well-respected strategies for gaining credibility within citizen science.

Here we examine credibility-building strategies across citizen science projects with different structures, priorities, and volunteer bases to illuminate what it means to balance a need for credibility with the realities of a volunteer-based program. By organizing practical approaches into a framework, we show pathways that projects can follow to achieve greater credibility and to combat monolithic assumptions about the nature of citizen science (Silvertown 2009). An understanding of how citizen science projects explicitly pursue the goal of credibility can be useful to both producers and users of citizen science products.

Methods

Our study focuses on state waters in the central coast region of California, stretching from Pigeon Point to Point Conception up to three nautical miles offshore, a region that is home to a network of 29 marine protected areas. The Marine Life Protection Act (MLPA) mandates monitoring to inform adaptive management of the network, and the state is meeting this mandate through a public-private partnership. The monitoring framework, adopted as state policy (OST 2014a), explicitly acknowledges the potential role for citizen science. Several citizen science groups participated as partners in early monitoring efforts, but many others are operating in the region. Understanding the credibility-building strategies of these citizen science projects is one piece of a broader effort to understand the region’s citizen science capacity and its potential to inform decisions about California’s oceans (OST 2014b).

To inventory project strategies we first compiled a roster of citizen science groups through a comprehensive Google search followed by references from our initial contacts (snowball-style), yielding a total of 30 groups. Once we were not receiving any new suggestions we felt satisfied that our roster was complete. However, we recognize that citizen science groups and projects within groups may come and go over time.

We then conducted semi-structured phone interviews, each lasting between 30 and 90 minutes, with project coordinators from each of the 30 groups. We also held seven focus groups with ten of the groups that were willing and able to organize in-person meetings (Elkhorn Slough groups participated in a combined focus group). Protocols for interviews and focus groups focused on program goals and ways that leaders guide their programs to meet those goals, including tradeoffs, challenges, and priorities. (Our interview protocols are available in Supplementary Materials.)

We transcribed and coded the interview and focus group data using the Dedoose software package. We used a grounded theory analysis, which relies on several rounds of coding and annotation to identify key themes emerging from the data (Charmaz et al. 2000). This process highlighted data credibility–with related concepts of rigor and trust–as an important challenge that citizen science groups face in developing partnerships with managers, scientists, and other potential users of their data. Although every project coordinator referenced the concept of credibility during initial conversations, we asked several coordinators additional questions about data quality to provide greater detail for our analysis of credibility strategies.

Next we held a two-day workshop with representatives of 18 of the 30 identified citizen science groups and six resource managers identified through interviews as key data users. Many workshop sessions directly focused on credibility and built on results from the interviews and focus groups. Each discussion group had at least two note takers, and notes were later coded and analyzed along with our interview and focus group results. Workshop discussions reinforced the importance of credibility strategies and helped to build our credibility strategy framework.

Finally, the authors inductively developed a credibility strategies framework. We did not seek universal agreement on the categories used in the framework; rather, the categories were developed internally as a result of our grounded theory analysis. However, while opinions about appropriate categories of credibility strategies may vary, we are confident that our data accurately reflect practices of the citizen science projects that we examined. Results (see Table 1) were vetted by workshop participants and by other project leaders who were not able to be present.

Table 1

Summary of credibility-building strategies and related context of 30 citizen science groups working in the Central Coast of California. Symbols in each column are explained in detail in the text, but each activity column was either Y/N for yes or no regarding whether the activity exists within the project or H/M/L/N for high/medium/low/no indicating the level of the activity. Each context column is Y/N for yes or no in answer to the question, S/M/L for small/medium/large depicting the size of a program component, or G/I for group or individual activity.

Credibility-building Strategies

total

Context for Strategies

early actions

in the field

in the office

prior expertise

training

science advising

ranking system

in-person oversight

re-training

technological aids

Validation of observations

cross comparison

publication

management use

Quality assurance

sole source of data?

institutional affiliation

size of volunteer pool

group vs. individual

time commitment

Beach Watch

N

H

Y

N

N

optional

N

Y

N

N

Y

N

5

Y

Y

L

G

M

BeachCOMBERS

N

H

Y

N

N

optional

N

Y

Y

Y

Y

N

6

Y

Y

M

G

M

Beachkeepers

N

N

N

N

N

N

N

N

N

N

Y

N

1

Y

N

L

G

L

Black Oystercatcher Monitoring

Y

N

Y

N

N

N

N

N

N

Y

Y

N

4

Y

N

M

G

M

Blue Water Task Force

N

L

Y

N

Y

N

N

N

Y

Y

Y

Y

7

N

N

L

G

M

CA King Tides

N

N

N

N

N

N

Y

N

N

N

N

N

1

Y

N

L

I

L

CCFRP

N

L

Y

N

Y

N

N

Y

N

Y

Y

N

6

N

Y

L

G

M

CWC (First Flush)

N

M

Y

N

N

N

Y

N

N

N

Y

Y

5

N

N

M

G

L

CWC (Urban Watch)

N

M

Y

N

maybe

optional

Y

N

N

N

Y

Y

7

N

N

S

G

L

Elkhorn Slough (otters)

Y

L

Y

N

N

N

Y

N

Y

Y

Y

N

7

N

Y

S

G

M

Elkhorn Slough (algae)

Y

L

Y

N

Y

N

N

N

N

Y

N

N

5

N

Y

S

G

M

Elkhorn Slough (nestboxes)

N

M

Y

N

N

N

N

N

N

Y

Y

N

3

N

Y

S

I

H

Elkhorn Slough (shorebirds)

Y

L

Y

N

Y

N

N

N

Y

Y

Y

N

6

N

Y

M

G

L

Grunion Greeters

N

M

Y

N

maybe

N

N

N

Y

Y

Y

N

5

N

Y

L

G

M

iNaturalist

N

N

N

Y

N

N

Y

Y

N

Y

Y

N

5

N

Y

L

I

L

Jellywatch

N

N

Y

Y

N

N

Y

Y

n/a

Y

Y

N

6

Y

Y

L

I

L

Leatherback Watch

N

N

Y

Y

N

N

Y

Y

N

y

N

Y

6

N

N

S

I

L

Lighthawk

Y

N

Y

n/a

n/a

n/a

Y

n/a

n/a

Y

Y

n/a

5

Y

N

S

I

M

LiMPETS

N

M

Y

Y

Y

Y

N

Y

Y

Y

Y

N

8

N

Y

M

G

M

Marine Debris Action Teams

N

M

N

N

Y

N

N

Y

Y

Y

Y

N

6

N

Y

M

G

M

Marine Debris Tracker

N

N

N

N

N

N

Y

Y

N

Y

Y

N

4

N

Y

L

I

L

Monterey Bay NMS VMP

N

M

Y

N

maybe

N

Y

N

N

Y

Y

Y

7

N

Y

L

G

H

Morro Bay NEP VMP

N

L

Y

N

Y

N

Y

N

Y

Y

Y

N

6

N

Y

M

I

M

MPA Watch

N

L

Y

N

N

N

Y

Y

n/a

N

N

N

4

Y

N

M

I

H

Phytoplankton Monitoring Program

N

M

Y

N

N

N

N

Y

Y

Y

Y

N

6

N

Y

M

I

L

REEF

N

L

Y

Y

N

optional

N

Y

Y

Y

Y

N

8

N

N

L

I

L

Reef Check CA

Y

M

Y

Y

Y

Y

N

Y

Y

Y

Y

N

10

N

N

L

G

H

Seabird Protection Network

N

L

Y

Y

N

N

N

Y

N

Y

Y

N

6

N

N

L

G

M

Shorebird Monitoring (Morro Bay)

N

L

Y

N

Y

N

N

N

N

Y

Y

N

5

N

N

M

G

L

SPLASH

N

L

Y

N

N

N

Y

Y

N

Y

Y

N

6

Y

N

L

I

L

average:

% employing strategy

20

73

83

23

40

20

43

50

37

80

87

17

5.5

Results

From our survey of 30 citizen science groups we identified 12 distinct strategies for demonstrating credibility. We included only activities that constitute a formal component of a project and which project leaders reported built credibility (future studies will have to verify with data users how much the strategies actually worked). We did not evaluate or include in our framework such informal factors as developing close personal ties with data users or personal reputation of project staff. While factors such as these may be important in developing credibility, they were not amenable to analysis by our methods and would require further study.

We grouped credibility strategies around three stages of a research project: Planning (early actions), data collection (in the field), and data analysis and interpretation (in the office–see Table 1). Explanations of each strategy shown in the table are provided in the following section. We also identify and discuss five contextual and programmatic factors that may be related to patterns observed in the credibility strategies of each of the projects. Note that group leaders did not feel that doing more strategies was a good way to increase credibility (no one, for example, strove to achieve all 12 options). Therefore, this chart should not serve as a checklist of prescribed credibility-building strategies that a project “must” follow.

Strategies for demonstrating credibility

Our framework for establishing credibility follows the suggestion from the DataONE Data Lifecycle concept (Strasser et al. 2012) to separate strategies by stage of the research process in which they occur. We identified three important stages that should be addressed and then identified common credibility-building strategies within each stage.

Early actions

Early actions are steps taken to increase credibility before any data are collected. They include working with volunteers to ensure that they can successfully complete required tasks and structuring methods to best answer the scientific questions at hand. Decisions about how to apply these strategies have direct consequences for the kinds of science that volunteers can do and the availability of qualified volunteers. For example, more stringent expectations of volunteers allow for more complex protocols but decrease the size of the potential volunteer pool.

Prior expertise – The expectations that project leaders have of volunteers—in terms of skills or knowledge—when they join the program might set a barrier to entry. In the table, “yes” indicates that a project has a formalized minimum standard that volunteers must meet in order to participate.

Training – Projects often train their volunteers on the project protocol and logistics and sometimes offer substantial training in the form of classes, readings, and online materials. In the table, an “H” designation represents required training with a substantial time investment (a course or apprenticeship stage), while “none” indicates that training is not required. “L” and “M” fall in between.

Science advising – Scientific advice during the project development stage can help strengthen methods, tailor data to their intended use, and ensure standard practices in the field. Advice may come in the form of a partnership with a university lab, a science advisory team, or other formal arrangement. A “yes” in the table refers to any of these options.

In the field

Credibility strategies implemented while volunteers are collecting data tend to focus on individual data point quality. Two of these strategies, in-person oversight and technological aids, involve real-time mediation of data collection. These strategies can take advantage of a diverse volunteer pool by leveraging skills from long-term volunteers, local experts, and those savvy with technology to assist the new or less-skilled volunteers in learning. Other strategies in this category track volunteer learning and give formal credit to volunteers as they improve the quality of their data.

Ranking system – Volunteers join projects at different levels and add skill through experience in a project. Increased numbers of designated “experts” can instill trust in the project’s data. Projects designate experts as they cross hurdles such as tests, tenure in the program, or trainings attended (any of which are indicated as a “yes” in the table).

In-person oversight – Many data errors happen in the field. To address this, some projects designate staff, science partners, or “expert” volunteers to directly oversee data collection (indicated as a “yes” in the table).

Retraining – Opportunities for continuing education can advance volunteer skill through classes, online trainings, readings, and other training opportunities. In the table, “yes” indicates that continuing education is required, “optional” indicates that it is available but not required, and “no” indicates that no further training is available.

Technological aids – Challenging forms of data collection can be simplified and streamlined using technology. For example, technology can automate location recording, photo-based validation, or water quality sensing. “Yes” in the table indicates that technology simplifies data collection in some way.

In the office

We observed the highest number of credibility strategies in the later stages of project implementation. Most of these strategies are designed to improve the reputation and therefore the credibility of the project as a whole. Here, opportunity for outside review plays a key role in contextualizing citizen science among a community of scientific peers through publishing peer-reviewed articles, enrolling data users, and employing disciplinary best practices.

Validation of observations – Many projects conduct checks for human error and answer questions about species identification or other difficult evaluations. Validation can range from ensuring that data sheets are complete and legible to including statistics-driven flagging of possibly incorrect data and expert identification of voucher samples (all indicated as “yes” in the table).

Cross-comparison – Side-by-side comparisons of citizen science data with data collected by trusted professionals can document credibility of methods and data while demonstrating that volunteers can collect data accurately. Cross-comparison requires an existing data source with which to compare. “Yes” in the table indicates that a program participated in such a comparison.

Publication – Academic peer review puts the research through the same gauntlet of critique as research conducted by professional scientists. Publications may be written by project staff or by other scientists using the data. A less common, but growing, strategy is peer-reviewed and published data sets. Any variety of publication is indicated as “yes” in the table.

Management use – Managers who use citizen science data to inform their decision-making are expressing trust in the data. “Yes” in the table indicates that managers used data produced by the citizen science group.

Quality assurance protocol – For some topics, standard quality assurance protocols are a required part of scientific practice in order to calibrate methods, technology, and practice over time. For citizen science, these protocols also certify volunteer capability in addition to the methods. “Yes” in the table indicates that a QA protocol is required.

Program structure and context

Columns at the far right of the table show some factors that project staff suggested might be related to a project’s choice in employing a particular mix of credibility strategies.

Sole source of data

The “sole source of data?” column in the table refers to whether data provided by the group are available through any other source. If a group produces the only data on a given subject, then the acceptable standard of quality may be lower than if there are many established groups in the field (Aoki et al. 2009). For instance, sightings data from JellyWatch are often the only indication of ephemeral jelly blooms that otherwise receive little research attention or funding but are of high concern for ocean health (Condon et al. 2013). In another example, MPA Watch tracks recreational uses of marine protected areas, a topic for which there are no standard quality assurance protocols, so in-house data quality checks are the best they can do. Most of the projects in our census that utilize few credibility strategies are also the sole source of the kind of data they are producing. These projects are trying something new, and therefore certain strategies for credibility that require scientific partnerships or comparisons are not yet available to them. Therefore, their data have may great value by virtue of the fact that they exist – in the future such projects may increase their data quality with greater time and resources, but for potential data users today, it’s these data or nothing.

Institutional affiliation

The “institutional affiliation” column in the table refers to whether a group is officially affiliated with a larger institution such as a university, government agency, or museum, and therefore may benefit from institutional support. According to participants at our workshop, affiliation often comes with support for grant writing and management from a budget office, statistics software and expertise from partner departments, and communications support from a news office. They also suggested that larger institutions may have an established reputation in the scientific community which furthers the credibility of any citizen science taking place under the umbrella of the larger institution.

The projects with the most credibility-building activities were not affiliated with a university, government agency, or museum (most notably REEF and ReefCheck, both diver-volunteer programs). Some leaders of these groups reported feeling the need to strengthen their program in response to external assumptions of questionable credibility. Some leaders for projects with an intermediate number of credibility activities felt that institutional support alleviated the pressure to add more.

Size of volunteer pool

The “size of volunteer pool” column in the table characterizes the number of volunteers in three categories (Small < 20 people), Medium = 20–100 people, and Large > 100 people) This number matters especially for credibility strategies focused on individual volunteers, such as training or in-person oversight, because more volunteers increase the resources needed to implement these strategies.

The three projects in our census with the most credibility-building activities have large volunteer pools. During focus groups, volunteers frequently discussed the need to accommodate a volunteer pool with a range of capabilities. A larger volunteer pool likely includes a wider range of volunteer capabilities, and therefore demands more credibility strategies to accommodate the lower-performing volunteers.

Group versus individual

The “group vs. individual” column in the table refers to whether volunteers collect data in organized groups or as a solo activity. Solo activities are often designed to take advantage of a volunteers’ normal routines, such as taking walks on the beach or SCUBA diving on a family vacations. The data-collection activities of these projects could be fundamentally altered by in-person oversight. They also make group activities such as trainings more difficult to coordinate. In these cases, credibility must come from strategies focused on the project as a whole more than from strategies focused on individual volunteers.

All but two of the projects in our sample that are designed for solo data collection are app-based. With the exception of REEF, these projects also employ few credibility-building activities. They generally prioritize fun in order to recruit and retain volunteers, so they minimize the number of tasks that are tedious, often by building them into an app. The app-based platforms help with data management and ensure that more complex data are consistently measured (like GPS coordinates), demonstrating the important intersection between technology and citizen science outcomes.

Time commitment

Finally, “time commitment” in the table refers to the time required of the typical volunteer: “L” refers to one-time events; “M” refers to a medium time-commitment by volunteers such as a short season; and “H” refers to a large commitment such as year-round data collection. One might expect that programs requiring only a low time commitment would avoid resource-intensive credibility strategies focused on individual volunteers, as the expected return on that investment would be low.

Time commitment expected of volunteers varies widely, sometimes within a single project. Some participants might come to all possible events while others pick and choose depending on their schedule. Our census did not reveal a solid connection between average time commitment and the number of credibility-building activities; variation within a single project might explain why.

Two contrasting examples

To further illustrate credibility-building strategies we next describe two projects with a similar number of strategies but very different approaches based on the focus of their work and the volunteer experiences that they strive to create. This comparison helps to demonstrate the importance of scientific context and project structure in understanding how citizen science groups approach the challenge of signaling credibility. The examples also highlight some of the tradeoffs involved in deciding how many and which kinds of credibility-building strategies to employ.

iNaturalist

iNaturalist, based at the California Academy of Sciences, relies upon broad participation from a wide variety of people to create a global database of biodiversity observations. Through its web- and app-based platform, anyone can sign up, take a picture of any organism, and report it, regardless of expertise. Most contributions come from smartphones, whose photos automate and standardize location and time records. Users can also set up projects within iNaturalist, asking additional questions about observations such as weather conditions or animal behavior.

iNaturalist uses five strategies to demonstrate credibility: Ranking system, technological aids, validated observations, publication, and management use. The large number of observations cultivated through a large volunteer base tease out the signal from the noise and data can be compared from different volunteers at the same location. Verification also happens through discussion of observations among users. When no designated expert is available to definitively identify an observation, consensus among at least two-thirds of identifiers can elevate an observation to “research grade.” All such data are published to the Global Biodiversity Information Facility (GBIF), from which researchers around the world frequently download data for biodiversity analyses.

Observational data is where much of ecological science begins, and volunteers gain skill through participation. The leaders of iNaturalist are open about the limitations of opportunistic observations submitted by users but maintain their importance in global ecology. Volunteers are encouraged to be honest about their capabilities – i.e, to report observations as unknown rather than guessing at identifications. In addition, the social nature of the platform allows users with more expertise to lend credibility to the data by identifying unknown observations and verifying others. Developers are currently working on features that will support additional kinds of data collection such as transects or effort reporting.

Because iNaturalist relies on a large number of volunteers distributed throughout the world, checks on credibility are loaded at the end of the data-collection process. Rather than ensuring that all observations are accurate, the system works to identify high quality data from within the larger pool of contributions. This means that many observations will never be used. However, it also means that anyone with access to the requisite technology can participate regardless of prior knowledge, and that anyone can take advantage of the educational opportunities that iNaturalist provides through access to a community of online experts. The program requires very little commitment from volunteers and does not need to invest heavily in individual volunteers to sustain the model.

BeachCOMBERS

Conversely, BeachCOMBERS begins to build credibility at the front end of the project by investing heavily in individuals through training and one-on-one attention, relying on a carefully structured protocol and experimental design to ensure success. BeachCOMBERS volunteers walk designated beaches and record detailed information on beach-cast birds, mammals, and tar balls. They also help local researchers with short-term projects, recording other kinds of data or taking samples from the birds. The bird data are used primarily by the project to write scientific publications focusing on birds as an indicator of ocean health.

BeachCOMBERS employs six credibility strategies: Training, science advising, retraining, validation, cross-comparison, publication, and management use. Joining BeachCOMBERS requires first attending an 80-hour course including classroom lessons and in-the-field practice sessions. Volunteers must commit to participating for at least two years, a barrier that significantly limits the number of potential volunteers. However, volunteers tend to stay longer than their initial commitment and report a sense of camaraderie with their training cohort.

BeachCOMBERS doesn’t rely solely on strict training of participants, however; it also includes a series of checks on data once they’ve been collected. Volunteers have close connections to local experts and program leaders, so they can send pictures of their observations when they have questions. The science advisory team and neighboring beach-cast bird citizen science programs stay in close contact to maintain a high level of review for program, methods, and future opportunities. This broader community has helped to pioneer the technique of using beach cast birds as indicators of ocean health, and thus has gained credibility in creating useful, novel methods for better understanding and monitoring the Pacific Ocean while creating a highly structured and rigorous program.

Discussion

DataONE recommends that all research should address credibility at each stage of the scientific process (Wiggins et al. 2013). Examining our table of findings indicates that this advice is both tractable and is actually heeded across a broad range of project structures: 87% of projects do at least one credibility-building activity in the “early actions” category; 87% do at least one “in the field,” and 97% do at least one “in the office.” However, some stages are easier than others for implementing credibility strategies, reflected in the popularity of strategies in “early actions” and “in the office.”

Indeed, few citizen science groups in our sample have implemented significant strategies for credibility in the field. Programs that do commented on the commitment involved in sending staff out with each group of volunteers. Often projects do not have sufficient staff to go around. To address this problem, one project leader emphasized “I’m only a phone call away” should volunteers at different sites have questions at the same time. Another group considered the idea of transitioning to app-based data reporting to improve reporting accuracy, but realized that the cost of the technology and the investment in smartphone training for its mostly older volunteers would be prohibitive.

One set of credibility activities seems to be favored by a majority of programs: Training, scientific advising, publication, and management use. These are likely the most logistically feasible strategies, and some offer benefits in addition to credibility. For example, scientific publications and use of data for management motivate volunteers who can see the results being used in a tangible way.

These additional benefits also demonstrate the potential for feedback loops linking credibility strategies. For example, if managers use the data, thus raising the profile and impact of the program, more volunteers may be motivated to join. With increased volunteer demand, the program can institute stronger training or requirements for joining. These same managers might also motivate, or directly request, more directed scientific review of the methods or data interpretation to better fit their agency’s needs. Increased credibility through all of these activities might then enroll more managers to use the data, completing the feedback loop.

In short, the number and types of credibility-building activities both influence and are influenced by program structure. This constitutes a tradeoff in decision-making and program planning that project leaders must balance.

Conclusions

Our investigation identified twelve strategies to ensure and/or demonstrate the credibility of citizen science. According to our interviews, some of these strategies mirror the precedents set by established, high-profile programs such as eBird and the Cooperative Weather Observer Program; some mirror industry standards; and others worked with existing resources to create a uniquely tailored credibility-building system.

While it is important to pursue standards related to credibility in citizen science, it is crucial that we avoid monolithic thinking in this endeavor. Each group should consider its context and goals in deciding what strategies to employ. In particular the number of volunteers, the group nature of the activity, and the time commitment expected of participants in the program each play an important role. For example, a training requirement would drastically alter the iNaturalist program. For BeachCOMBERS, however, training is essential for development of adequate expertise and the program’s culture. Building credibility, which has both technical and social components, is a dynamic process with built-in feedback loops.

Just as citizen science projects must balance priorities internally, they also must work with potential users to establish shared expectations around credibility. Ultimately credibility is in the eyes of the data user, and regular communication as part of a relationship is critical in navigating the tradeoffs associated with employing credibility strategies. Explicitly planning credibility expectations and performance helps move past the simplistic question of whether or not citizen science is credible into how it can be credible and for what purpose.

In our workshop, citizen science leaders expressed a desire to be held to the same credibility standards as academic science. Thus the practices and expectations described will become increasingly important as the popularity of citizen science continues to grow and citizen science becomes a more common approach to understanding the world around us.

Supplementary Materials

Supplementary material relating to this article is available at http://dx.doi.org/10.5334/cstp.6.s1

Ethics and Consent

The authors strove to ensure that all procedures contributing to this work comply with the ethical standards of the Helsinki Declaration of 1975, as revised in 2013. All participants in this research were informed at the outset about the goals of the project; the risks and expectations of participation; procedures related to gathering, storage, and use of data; and their ability to withdraw from the project at any time. A copy of the information provided to project participants can be accessed at www.oceansciencetrust.org/wp-content/uploads/2015/08/Citizen-Science-and-MPA-Monitoring-focus-group-handout.pdf.

Acknowledgements

The authors would like to acknowledge our wonderful participants for giving their time for interviews, focus groups, and taking us into the field with their groups. We also acknowledge our helpful advisory group: H. Ballard, F. Shilling, O. Boyle, J. Freiwald, and M. Hall-Arber. This research was funded by the David and Lucile Packard Foundation.

Competing Interests

The authors declare that they have no competing interests.

1Aceves-BuenoE.Citizen Science as an Approach for Overcoming Insufficient Monitoring and Inadequate Stakeholder Buy-in in Adaptive Management: Criteria and EvidenceEcosystems201549350610.1007/s10021-015-9842-42AokiP.M.A Vehicle for Research: Using Street Sweepers to Explore the Landscape of Environmental Community ActionProceedings of the SIGCHI Conference on Human Factors in Computing Systems200910.1145/1518701.15187623AswaniS.LauerM.Incorporating Fishermen’s Local Knowledge and Behavior into Geographical Information Systems (GIS) for Designing Marine Protected Areas in OceaniaHuman Organization20066518110210.17730/humo.65.1.4y2q0vhe4l30n0uj4BerkesF.Evolution of co-management: role of knowledge generation, bridging organizations and social learningJournal of environmental management2009905169270210.1016/j.jenvman.2008.12.0015BiberE.The Challenge of Collecting and Using Environmental Monitoring DataEcology and Society201318410.5751/ES-06117-1804686BirdT.J.BatesA.E.LefcheckJ.S.HillN.A.ThomsonR.J.EdgarG.J.Stuart-SmithR.D.WotherspoonS.KrkosekM.Stuart-SmithJ.F.PeclG.T.Statistical solutions for error and bias in global citizen science datasetsBiological Conservation201417314415410.1016/j.biocon.2013.07.0377CharmazK.DenzinN.K.LincolnY.S.Handbook of qualitative researchThousand Oaks20008CondonR.H.Recurrent jellyfish blooms are a consequence of global oscillationsProceedings of the National Academy of Sciences of the United States of America201311031000510.1073/pnas.12109201109ConnorsJ.P.LeiS.KellyM.Citizen Science in the Age of Neogeography: Utilizing Volunteered Geographic Information for Environmental Monitoring Citizen Science in the Age of Neogeography: Utilizing Volunteered Geographic Information for Environmental MonitoringAnnals of the Association of American Geographers20121026374110.1080/00045608.2011.62705810DaleA.ArmitageD.Marine mammal co-management in Canada’s Arctic: Knowledge co-production for learning and adaptive capacityMarine Policy201135444044910.1016/j.marpol.2010.10.01911DanielsenF.Increasing conservation management action by involving local people in natural resource monitoringAmbio200736756657010.1579/0044-7447(2007)36[566:ICMABI]2.0.CO;212DickinsonJ.L.ZuckerbergB.BonterD.N.Citizen Science as an Ecological Research Tool: Challenges and BenefitsAnnual Review of Ecology, Evolution, and Systematics201041114917210.1146/annurev-ecolsys-102209-144636Available at: http://www.annualreviews.org/doi/abs/10.1146/annurev-ecolsys-102209-144636 [Accessed July 20, 2011]13DoeskenN.RegesH.The Value of the Citizen Weather ObserverWeatherwise2010636303710.1080/00431672.2010.51960714FischerF.Citizens, experts, and the environment: The politics of local knowledgePlanning Theory & Practice20023224925515FuntowiczS.O.RavetzJ.R.Science for the post-normal ageFutures199325773975510.1016/0016-3287(93)90022-L16GillettD.J.Comparing volunteer and professionally collected monitoring data from the rocky subtidal reefs of Southern California, USAEnvironmental monitoring and assessment2012184532395710.1007/s10661-011-2185-5Available at: http://www.ncbi.nlm.nih.gov/pubmed/21739280 [Accessed July 19, 2012]17GouveiaC.Promoting the use of environmental data collected by concerned citizens through information and communication technologiesJournal of environmental management20047121355410.1016/j.jenvman.2004.01.00918HendersonS.Citizen science comes of ageFrontiers in Ecology and the Environment201210628310.1890/1540-9295-10.6.28319HochachkaW.M.Data-intensive science applied to broad-scale citizen scienceTrends in ecology & evolution2012272130710.1016/j.tree.2011.11.00620JasanoffS.States of knowledge: the co-production of science and social order2004Routledge10.4324/978020341384521JentoftS.McCayB.J.WilsonD.C.Social theory and fisheries co-managementMarine Policy1998224–542343610.1016/S0308-597X(97)00040-722JordanR.C.Evaluating the performance of volunteers in mapping invasive plants in public conservation landsEnvironmental Management201249242543410.1007/s00267-011-9789-y23KellerA.C.Credibility and Relevance in Environmental Policy: Measuring Strategies and Performance among Science Assessment OrganizationsJournal of Public Administration Research and Theory200920235738610.1093/jopart/mup00124KellingS.eBird: A Human / Computer Conservation and ResearchAI Magazine201325KimmererR.W.Searching for synergy: integrating traditional and scientific ecological knowledge in environmental science educationJournal of Environmental Studies and Sciences201226LawrenceA.The first cuckoo in winter: Phenology, recording, credibility and meaning in BritainGlobal Environmental Change200919217317910.1016/j.gloenvcha.2009.01.00627McCormickS.Democratizing Science Movements: A New Framework for Mobilization and ContestationSocial Studies of Science200737460962310.1177/030631270707659828NewmanG.International Journal of Geographical User-friendly web mapping: lessons from a citizen science website2010February3741(2012)29NewmanG.The art and science of multi-scale citizen science supportEcological Informatics201163–421722710.1016/j.ecoinf.2011.03.00230NOAACitizen Science Community of Practice2014Available at: http://www.oesd.noaa.gov/leadership/citizen_sci.html [Accessed August 28, 2014]31NWSNWS Cooperative Observer Program2015Available at: http://www.nws.noaa.gov/os/coop/what-is-coop.html [Accessed October 20, 2015]32OguamanamC.Localizing Intellectual Property in the Globalization Epoch: The Integration of Indigenous KnowledgeIndiana Journal of Global Legal Studies200411213516910.2979/GLS.2004.11.2.13533OSTCentral Coast MPA Monitoring Plan2014aOakland, CA, USA34OSTCitizen Science & Ocean Resource Management in California: Guidance for forming productive partnerships2014bOakland, CA35PocockM.J.O.Choosing and Using Citizen Science: a guide to when and how to use citizen science to monitor biodiversity and the environment2014WallingfordOxfordshire36RoyH.E.Understanding Citizen Science and Environmental Monitoring2012WallingfordOxfordshire37ShapinS.Cordelia’s Love: Credibility and the Social Studies of SciencePerspectives on Science19953338SilvertownJ.A new dawn for citizen scienceTrends in ecology & evolution20092494677110.1016/j.tree.2009.03.01739StrasserC.Primer on Data Management: What you always wanted to know2012Available at: http://www.dataone.org/sites/all/documents/DataONE_BP_Primer_020212.pdf40SullivanB.L.The eBird enterprise: An integrated approach to development and application of citizen scienceBiological Conservation2014169314010.1016/j.biocon.2013.11.00341SWRCBSurface Water Ambient Monitoring Program2016http://www.waterboards.ca.gov/water_issues/programs/swamp/ [Accessed January 24, 2016]42ThorntonT.LeahyJ.Trust in Citizen Science Research: A Case Study of the Groundwater Education Through Water Evaluation & Testing Program 1JAWRA Journal of the American Water Resources Association20124851032104010.1111/j.1752-1688.2012.00670.x43TrumbullD.J.Thinking Scientifically during Participation in a Citizen-Science ProjectScience Education200084226527510.1002/(SICI)1098-237X(200003)84:2<265::AID-SCE7>3.3.CO;2-Xhttp://dx.doi.org/10.1002/(SICI)1098-237X(200003)84:2<265::AID-SCE7>3.0.CO;2-544US EPAO.Water: Monitoring & AssessmentAvailable at: http://water.epa.gov/type/watersheds/monitoring/dir.cfm [Accessed June 16, 2014]45WigginsA.Data Management Guide for Public Participation in Scientific Research2013