Part of successful endorsement partnership is holding your system to a standard

Several of the projects that won grants through the Digital Media and Learning competition offer centralized online platforms for educational programs that take place in geographically distributed physical classrooms. The principles used among projects that fit this profile, and especially the transitions they made as they shifted from intended to enacted practices, illustrate lessons learned about how to confront the challenge of running this type of badge system.

The Planet Stewards badge system offers a prototypical example of a distributed badge system, where learning experiences are facilitated by local teachers spread across the country.

Planet Stewards is a badge-driven learning program that guides high school science students through exploration of careers in five categories of Earth sciences. It is a partnership effort by the National Oceanic and Atmospheric Administration (NOAA) and Boise State University’s 3D GameLab, an online learning platform designed around quest-based learning. 3D GameLab developers, headed by Lisa Dawley of GoGo Labs and Chris Haskell of Boise State, partnered with NOAA to offer badges based on NOAA’s educational content.

In the Planet Stewards system, like many distributed recognition schemes, site teachers are responsible for administering the program to their own classes of students. Much of the content is available through quests on the 3D GameLab online platform, but individual teachers are responsible for customizing the curriculum to match their classroom objectives, assessing student work and issuing badges.

Distributed badge systems are especially susceptible to questions about the validity of the learning claims made by their badges, because of this separation between the organization who is listed as the issuer and the person who performs the assessments. Carla Casilli’s post on “what we talk about when we talk about validity” expands on the validity concern that has been a common thread of many skeptical stances toward badges. She talks about validity as a multifaceted interplay between system credibility and reliability, pointing out that these issues are not unique to any educational credential. Systems not granted credibility by presumption, surface inspection, or reputation could earn credibility through reliable assessment of the learning claims they make.

This potential for doubt is further intensified at Planet Stewards because of the NOAA branding of the badges, represented by the “externally endorsed” category of the principle “seek external backing of credential.” (See: forthcoming case study on difficulties of securing outside endorsement ) While NOAA branding possibly adds reputational credibility based on its broad name recognition, NOAA and 3D GameLab felt from the beginning of the project that they needed to protect the validity of the badges by ensuring the participating teachers were well trained to offer instruction and assessment.

Planet Stewards had to design practices to ensure that badge earners were held to the high standards NOAA desired. This became one of the main system design challenges Planet Stewards faced, and it impacted the team’s decisions around both recognition and assessment practices.

The team from NOAA and 3D GameLab decided that the need for consistent assessments across many classrooms required teacher training and that the ability to offer the Planet Stewards curriculum would be limited to teachers with the training. They proposed a solution tailored to the capabilities of the 3D GameLab platform:

Groups of quests that are associated to awarding the badge will be “locked.” It will require the teacher to complete a pre-training in the use of a quest group, as well as a verification that they are a certified teacher in order to unlock access for her class of students. (DML Stage 2 Proposal)

Certification of teachers would then open up the ability to “clone” the Planet Stewards quests, customize them to the needs of a particular classroom, and then to issue badges. This represents the principle “have experts issue badges” in limiting issuing to teachers who have passed a training threshold.

Implementing the training practices first occurred in Spring 2013 with a pilot group of 20 teachers interested in running the Planet Stewards program. Speaking from 3D GameLab’s perspective, Lisa Dawley described one of the goals of the training program as ensuring badge integrity, saying “when you’re a platform and you’re issuing badges on behalf of someone else, [where the assessment isn’t automatic] you want to make sure that the assessments are valid.” The training also ensures that teachers will be able to support their students’ use of the quest platform, help them work through the content and assignments, and adequately assess the learning claimed by the badges.

While teachers may be initially attracted to either the NOAA content or the 3D GameLab quest-based learning methods, they need to learn about both in order to run a successful program in their classrooms. Through the training program, teachers learn about quest-based learning, instructional design for quests, and how to integrate activities in the online platform into their classes. Dawley says, “We first pull them into this piece, then we introduce them to the Planet Stewards curriculum.”

There remain challenges for the badge system’s training practices. The team determined that the pilot round of training with 20 teachers went well and immediately made plans to follow up with larger groups in the summer and fall. Dawley has an eye to scaling the training system as the next large challenge to face, saying “My next question is ‘ok this works with 20, what happens when you try to scale to 5000, and we haven’t addressed that yet’” (DML Initial Interview). The question remained when the DPD Project checked in with Planet Stewards again in October 2013. As Dawley explains, this “limits the scaling of the Planet Stewards platform, because they can’t just say ‘Kids, go play Planet Stewards. They have to be trained’” (DPD Follow-up Interview). That fall, 70 teachers went through a third round, but the Planet Stewards team is still considering how to scale the teacher training portion of the badge system even more, recognizing that teachers’ ability to attend training sessions synchronously and the associated costs are obstacles to overcome.

For the teachers who went through the training, their user accounts in 3D GameLab gain the technical privileges necessary to run the program. Though not initially intended, Planet Stewards expanded their certification to include awarding teachers an OBI-compliant badge to recognize them as a NOAA-certified educator, representing the principle “recognize educator learning.” This carries NOAA logo and may prove to be an attractive credential itself. The Planet Stewards team felt that offering a badge for teachers also made the training requirement more clear in terms of representing an actual certification and endorsement to be able to offer the curriculum.

The Planet Stewards badges make claims about student learning that may be valuable to various audiences, including colleges and potential employers. They show evidence of career exploration as well as what science education standards were targeted by the learning activities.

In these conversations about the badges, if questions of badge validity arise, Planet Stewards and its badge earners can point not only to the evidence contained in the student’s badges but also to the teacher certification badge. The training program serves to assure NOAA’s stakeholders that its name will be associated with high quality learning, and it helps earners reinforce the value of their experience in whatever contexts they choose to present their badges to others.

Loves open education, Open Badges, free culture, Progress of the Useful Arts and Sciences, people-powered politics, and local food production. Coordinator for the badges Design Principles Documentation Project at Indiana University.

Regardless of where you start, it’s more than likely you’ll end up somewhere other than your intended destination. That’s okay. Systems are living things, and your badge system needs to be flexible. You must embrace a bit of chaos in its design.
-Carla Casilli, Director of Design + Practice at the Badge Alliance