Starlite Astronaut Academy in conjunction with NASA Robotics Badges

Preface

The information contained in this project appendix was gathered from the original project proposal that was funded in 2012 and interviews with project leaders through the end of 2014. This appendix and the overall design principles database from the DPD project does not reflect further evolution of the project or developments after the final interview that took place in 2014. As of the final interview, Starlite Academy had issued badges and developed a functioning badge system. Based on this information, we have classified this badge system as an implemented (rather than a partial or suspended) badge system.

Summary

In an initiative made in conjunction with NASA, Project Whitecard Inc. collaborated with the Center for Educational Technologies (CET) at Wheeling Jesuit University to construct a badge system for middle to high school students with the goal of promoting skills and competence in the fields of science, technology, engineering, and math (STEM). The project designed a badge system to build upon their existing programs of robotics-related educational projects and activities, offering a new way to recognize students for their skills and achievements. As part of the core team, the badge system assembled project managers from NASA Education’s three “technology-focused education projects… Learning Environments and Research Network (LEARN), Learning Technologies Project (LTP) and the NASA Educational Technology Services Project (NETS)” (DML Stage 1 Proposal). To this effort, the project aimed to extend its reach and enhance the educational experience in the STEM subjects with digital badges. The badge system later transitioned its focus and took the form of the Starlite Astronaut Academy that embodies the spirit of the original proposal. Built with Drupal, the badge system enables students to learn through video-game based simulations of space-related topics.

Based on their pre-existing curricular activities, the project originally organized the plans for badges around two major themes of communication and collaboration. The badging effort originally articulated that “[t]he NASA Communication Badge focuses on the 21st Century Skill of Communication using robotics content with students while concentrating on articulating their knowledge through listening, writing, and presenting using a variety of media and technologies” (DML Stage 1 Proposal). While the Communication Badge places an emphasis on leveraging a variety of methods and technologies to understand other perspectives and articulate one’s message, the Collaboration Badge involves team projects in which students carry out a goal or mission by working collaboratively on objectives such as guiding a robot through roadblocks or building a robotic crane (DML Stage 1 Proposal). The project designated Collaboration Badges for students who work together in groups and Communication Badges for individual students, promoting skillsets that can translate across school and professional settings. Since then, the initiative has been focused on developing a virtual simulation of space-related projects, awarding badges for learners’ achievements in the system.

In addition, the effort aligned its badges to the leveled structure of the Common Core State Standards, indicating the topic or skill of focus. Further, the project issues badges not only to students but also to educators. After educators gain mastery over the curriculum content, they guide students in the badge system to develop skills and competence in working with robotics. Although the project planned to draft a set of badges to issue to educators, it awards the same set of badges to both students and educators. The badge system offers educational opportunities and nurtures learning and curiosity in the sciences.

Evolving Practices and Design Principles

What follows is a list of practices as they relate to the general and more specific design principles in each category of practice. The headings name a (a) General Principle, (b) Specific principle, (c) Specific practice. Below each heading, we indicate what stages this principle appeared in the project: as an (a) intended practice, (b) enacted practice, or (c) formalized continuing practice.

Design Principles for Recognizing Learning

In conjunction with NASA, the badging effort aligned its badges to internal and national standards. Additionally, the project issues the same badges to both students and educators, and certified educators can guide students in the badge system and award them badges. Once badges are earned, students can keep them forever, as they last for an indefinite duration. The project independently accredits the badge credentials and leverages them to identify students with relevant skillsets and abilities.

Align badges to standards > Use internal and national or international standards > CC and Next Generation Science Standards (NGSS) Alignment

Intended, enacted, formal.

In addition to internal standards, the project aimed to align badges to the Common Core State Standards for science and Next Generation Science Standards (NGSS). The project explained, “We are waiting for the release of the post-draft Next Generation Science Standards. We have a two-tiered approach: that we will develop capability to examine a badge, even if it was issued in the past, and assess its ability to fit into the Common Core or NGSS” (DPD Initial Interview). Given the completed construction of the badge system, the project aligns to the Common Core and NGSS as well as a process of review that involves the project-developed standards. Alignment with the Common Core boosts the value of the credential and the capacity to communicate with those outside of the issuing community.

The initiative intended to issue badges to both students and teachers. The project explained:

Summer of Innovation students and educators who complete five NASA robotics activities such as End Effector, Rover Races, and Remote Vehicle, mixed with learning about the Spirit, Opportunity, and Curiosity Mars exploration missions will earn NASA Robotics Collaboration Level 1 badge. Students who blog about their experience, including images or YouTube videos, will also earn the NASA Robotics Communication Level 1 badges (DML Stage 2 Proposal).

To this end, the effort planned to make available online activities related to NASA subjects for educators to implement in the classroom. The initiative aimed to offer online activities that are designed 1) with alignment to standards and exposed criteria for easy uptake, and 2) the system will allow educators to create badge opportunities both within NASA and without with an eye to credentials while remaining accessible.

As enacted in Starlite Academy, teachers and students earn the same badges. The project pointed out thatthe badge system excludes under-13-year-old badge earners for purposes of child protection and COPPA requirements. However, NASA’s activities are widely and freely available to learners of all ages. Although Starlite Academy in conjunction with NASA offers badges through the Mozilla backpack to students age 13 and older, the project also offers certificates as records of participation in their activities.

The badging effort aspired to validate badge credentials before issuing them to potential earners. The project asserted, “Classroom educators and youth group leaders will evaluate achievement levels by students involved in class or group activities. Individuals may be evaluated by “mentors” – advanced students or educators who have achieved highly in that area.” (DML Stage 2 Proposal). The initiative aimed for a community certified expert awards the badge credential, and badge metadata would exist to further validate the credential.

In specific, the badging initiative envisioned that qualified educators would validate NASA activity badges; these may be educators who are certified through institutions of higher learning, through state or national education board certification, mentors who are more experienced in the field (e.g., upper level university students), or actual NASA or professional community practitioners (e.g., working engineers). The NASA education department already has formal educational activity review procedures in place to assure content accuracy, appropriate assessment, and target audience engagement. The project described:

NASA activities have clearly stated objectives. These are met through creation of deliverables or through participation in distance communication activities. Educators sign off on participating students achieving satisfactory levels. A living searchable website of NASA badges will be created to which objectives or achievement criteria will be added, so that anyone may see requirements to achieve a badge, and also see related badges (DML Stage 2 Proposal).

The project further noted that its educational activities are written by qualified educational developers working with experts, and the activities are subject to a review process for content accuracy, assessment techniques, and target audience appropriateness and engagement value. In the Starlite Astronaut Academy, however, the project did not proceed in this direction due to changes in the organizational structure.

Determine appropriate lifespan of badges > Never expires > Permanence

Intended, enacted, formal.

The project envisioned the capacity for credential permanence, stating that “[o]nce a badge is in a backpack, it won’t be removed unless deleted by the owner. . . Even if projects change over time, earned achievements aren’t removed” (DML Stage 2 Proposal). The badge credential is permanent in the Starlite Academy badging system.

The project sought accreditation of badges through collaboration with external organizations. It outlined in the proposal that the badges would be gear-shaped and focused on topics such as Robonautics and Mars Exploration. The proposal described that it “will select its badgeable activities and provide the badge criteria and levels associated with each activity or activity cluster, and participate in the graphic design of the badges” (DML Stage 2 Proposal). The plan included that, as learners earn badges, the gear-shaped badges interlock with one another when grouped together. The proposal further explained, “It’s hard to imagine a higher endorsement value than NASA, but there are respected robotics communities such as FLL and Botball and youth groups such as Girl and Boy Scouts, 4-H, Girls and Boys Clubs of America, YMCA, and YWCA” (DML Stage 2 Proposal). The badge system aimed for backing by accredited agencies to increase the breadth of credentials that were offered.

In the badge system, the project moves beyond partnering with accredited institutions and actually have independently accredited credentials.They asserted,“Our goal is to have accredited badges” (DPD Initial Interview). As of the DPD follow-up interview, in the Starlite Astronaut Academy, the project has yet to continue talks with other organizations but is open to future collaboration.

Design Principles for Assessing Learning in Digital Badge Systems

Originally, the project intended to build an online assessment system that would measure the progress of students and employ summative evaluations that yielded meaningful results. The activities are aligned to the Common Core State Standards and follows the national leveled progression of the framework. In the implemented badge system, the effort carries out an automated scoring system to support students as they engage in research, science and engineering activities.

The badging system planned for badges to recognize a spectrum of learning that range from social to STEM skills, proposing the development of badges based on the themes of collaboration and communication. The project indicated, “We plan to assess how well students are collaborating. We are planning to have activities that require teammates” (DPD Initial Interview). The Starlite Astronaut Academy assesses how well students collaborate before issuing badges designed to encourage teamwork.

Enhance validity with expert judgement > Use computer scoring system and Use a combination of human and computer experts> Badges are validated by an automated system

Not intended, enacted, formal (Use Computer Scoring Systems)

Intended, not enacted, not formal (Use a combination of human & Computer Experts)

The badging effort intended for experts to review student performance and validate the claims made about learning by badges. They detailed, “Student performance on NASA activities will be assessed by supervising educators if receiving credit for a class activity, group leaders for a youth group activity, and by mentors recruited from the Robotics Course graduates, other NASA educator communities, and high level student ‘mentors’” (DML Stage 2 Proposal). Moreover, the project planned for experts to evaluate criteria regarding skills and qualifications. The badging initiative explained, “NASA activities have clearly stated objectives. These are met through creation of deliverables or through participation in distance communication activities. Educators sign off on participating students achieving satisfactory levels.” (DML Stage 2 Proposal). The project further delineated the process:

Classroom educators and youth group leaders will evaluate acceptable achievement levels by students involved in class or group activities. Individuals may be evaluated by “mentors” – advanced students or educators who have achieved highly in that area. A Mentor Training Blog will assist mentors in giving constructive feedback. All mentors will require two certified references (e.g., from teachers or principals) (DML Stage 2 Proposal).

As of the DPD follow-up interview, the Starlite Academy badging system has not put into place a formal educational activity review procedures in place to assure content accuracy and engaging activities. Instead, they use automated computer scoring systems.

Use leveled badge systems > Competency levels > Leveled badges with metadata and alignment to the natural leveled structure of Common Core

Intended, enacted, formal.

The badging effort envisioned a leveled badge structure that represents students’ progression of learning. They outlined the various levels of badges:

Minimal levels of communication, like participation in a teleconference, would receive Level 1 Communication badges. More advanced final products, like posting comments with documentation and/or images from a completed activity to a blog would receive a Level 3 badge, and finally publishing a Do-It-Yourself podcast or posting a YouTube video on Robots in Space Exploration would receive a high level badge, e.g., Level 5. Level 1 badges are proposed for single and short-term activities. They are easy enough to achieve that they motivate students to seek more advanced badges (DML Stage 2 Proposal).

In the Starlite Academy, the project levels the badges as participation and performances increase. The effort considers the varying amounts of time and effort required to complete different activities and distinguishes this difference with the badge levels. The badging system notes that various activities carry specific degrees of weight or value, and higher level badges take into account the greater amount of time and effort invested.

The project aimed to align its assessment activities to the Common Core, NGSS, and internally developed standards. Starlite Academy described,“What that means is if you have an activity and you are a badge issuer, you will be able to generate conditions for success, and very technically use Drupal to post your activity database – which we will do – because what we plan to do is show those activities to anybody with portable code which is actually coming from the Drupal server” (DPD Initial Interview). In the badge system, issuers generate conditions for success. The badge system further explained:

We want to put badge levels and taxonomies together with the Common Core and Next Generation Science Standards and allow badges to be searchable by standards and STEM topics… We plan to put together a Common Core and NGSS matrix for our badge. And… we can implement the matrix retroactively and pull the badges back. What this involves doing is having the ability to store that information but also to know the Common Core. Where we are in this process right now is we have decided to author a template as completely as possible for describing one of our badge activities including at the top, the learning assessment and the badge activity (DPD Initial Interview).

From the original plans, the Starlite Astronaut Academy carries out its assessment processes in agreement with the standards as intended in the proposal.

Design Principles for Motivating Learning

This initiative gives students the chance to interact with robotics and other STEM activities, learn skills that are not necessarily taught in school, and gives students credentials that they have accomplished and learned these skills. Students who collect badges will be further encouraged to pursue them by accessing new activities and opportunities for earning badges. Badges also signify to future employers the skills that the individual have. Students will be provided greater access to STEM internships as well as visibility to employers as a result of these badges. The initiative is hoping that this badge system will pique students’ interest in pursuing STEM careers that are based on the technical skills that students have acquired.

Set goals > Display of goal trajectory > Outlining future goals

Intended, enacted, formal.

The project intended to outline future goals for potential badge earners. The initiative explained, “As students work up to highest levels, they will be prompted with additional activities of that type and shown how close they are to achieving top levels” (DML Stage 2 Proposal). As in the Starlite Astronaut Academy, the badge system still presents students with concrete goals to develop their competence in STEM subjects.

Provide incentivess > New activities > Badges as gatekeeper connecting learners to more activities and building a STEM Pipeline

Intended, enacted, formal.

The project envisioned badges as acting as gatekeepers that open up more opportunities for STEM learning. In their DML Stage 2 Proposal, the badge system asserted:

Accumulating high level badges indicates a level of achievement that NASA can use to assist in its selection of competitive interns. Mentors will be selected upon application and review of subject-specific entries. For instance, a student who has posted several communications blogs and has responded in a professional manner to blog entries of other students may earn a Mentor badge after completing online training involving simulations of different types of blog entries. Previous NASA educational activity reviewers and graduates of the NASA ePDN Robotics courses are good candidates for the first round of mentors. A certain number of entries as a mentor would qualify educators to earn different levels of NASA Robotics Educator badges (DML Stage 2 Proposal).

The badging effort intended for badge credentials to serve as an enabler for further advancement within the issuing community. In the badge system, certain badges “unlock” new privileges to badge earners. The project considers the possibility of leveraging badges to identify and recruit potential interns for NASA. It also views credentials as a marker for the issuing organization to discover individuals who possess specific skills relevant to that organization. From the initial badge system proposal, the Starlite Astronaut Academy provides certificates of accomplishment that link to related and additional challenging activities, encouraging badge earners and their friends to continue developing their STEM skills.

The badge system aspired to develop different types of assessments, which would have an impact on the motivation of students. Project Whitecard CEO, Khaled Shariff, articulated, “At the time we submitted our proposal, we were going to use online assessment for quizzes. We said we would evolve the assessments as we started opening up to more complex uses of technology…. So we took a big leap forward with the Rover Activity. It does have a quiz, but also an activity. What we are working toward is trying to understand how to use a 3D environment that’s very hands-on in meaningful summative assessment related to virtual gameplay” (DPD Initial Interview). In this way, summative assessment practices become more interactive.

The effort indicated that many of its activities can be readily assessed by educators. The project described the different kinds of assessment used in NASA challenges and contests:

NASA has a series of engineering design challenges/contests that ask student teams to apply engineering process skills to research and solve authentic challenges – e.g., to design a sunshield or mirror for the James Webb Space Telescope. Assessment and feedback come at multiple levels in multiple forms. The top 20 teams are promoted to the next level, where they are partnered with engineering mentors who provide feedback as they help with presentations and improvements in the proposed designs. The final projects of these top teams are judged by experts using carefully prepared rubrics. Thus, students engaged in these challenges receive continuous formative feedback and final summative assessments. We are in the process of creating a series of badges for this particular challenge, as a model for other NASA real world challenges (IUDPD).

In the enacted Starlite Academy, the project integrates primarily automated assessment, based on the quality and significance of the activity. It has yet to integrate experts in the assessment process.

Recognize identities > Roles within a system > Identity

Intended, enacted, formal.

The project planned to highlight differing identities and modes of participation within the badging system. They described, “Top level achievers will be invited to become mentors,” and with badges “[s]tudents can start identifying themselves as mentors” (DPD Initial Interview). Moreover, the credentials link to related activities that can assist students in their continued development of skills that would qualify them for STEM careers. They specified that “[c]arefully written badge criteria will help students realize their qualifications and complete more thorough resumes for applying to any type of career” (IUDPD). The badging system envisioned that the badges would promote greater self-efficacy, intended to foster independence and competency in targeted skills. In the project effort, badges still promote self-efficacy in the targeted areas. The Starlite Astronaut Academy badge system provides a clear list of learning objectives and accomplishments that can be shared. The badges can enable earners to see themselves potentially in professional roles related to the fields of focus.

Provide incentives > Prizes > Offer prizes for earning badges

Intended, not enacted, not formal.

The project effort aimed to enable students to win prizes as a result of earning badges. They noted that badges in a resume can clearly define students’ skills and time & effort invested in the activities through NASA badge certificates. As of the DPD follow-up interview, the Starlite Academy badging effort has yet to issue badges to winners of challenges or contests.

Engage with the community > Involvement in digital community > Student and teacher badges

Intended, enacted, formal.

The effort planned forteachers and students to earn badges at the same time. The project meant for the badges to encourage a community of learners and build a strong support system. They envisioned that the badges would ultimately have led up to a Master Educator Badge and elevating their status to that of a mentor. The badging system worked to design badges for educators that could include reflections on teaching certain badge activities. The incorporation of teacher badges can motivate educators in completing badge activities and guiding students through the learning process. The Starlite Astronaut Academy badge system, however, awards only the same set of badges to educators as their students.

Provide incentives > Internships > Privileges

Intended, not enacted, not formal.

The effort aimed to motivate students with badges by providing access to internships. Additionally, they saw the potential of badges to motivate earners by demonstrating evidence of participation in professional development. In specific, they asserted:

Accumulating high level badges indicates a level of achievement that NASA can use to assist in its selection of competitive interns. Mentors will be selected upon application and review of subject-specific entries. For instance, a student who has posted several communications blogs and has responded in a professional manner to blog entries of other students may earn a Mentor badge after completing online training involving simulations of different types of blog entries. Previous NASA educational activity reviewers and graduates of the NASA ePDN Robotics courses are good candidates for the first round of mentors. A certain number of entries as a mentor would qualify educators to earn different levels of NASA Robotics Educator badges (DML Stage 2 Proposal).

In describing their use of badges to determine internship candidates, the project stated,“There was excitement about the potential of badges from the highest level at NASA. They were looking for kinds of prizes to give to people who won badges. What could we do to evaluate potential interns six months down the line through badges?… If you went and got all of NASA’s badges, that makes a pretty interesting candidate for NASA” (DPD Initial Interview). The Starlite Astronaut Academy badging project discussed the possibility of selecting future interns for their organization, but the effort has yet to continue development on this front.

Design Principles for Studying Learning

At the time of the DML Stage 1 proposal, the project did not articulate any specific intention to conduct research on the badging system. However, as part of the Starlite Astronaut Academy, the badge system expressed the capacity to use Google analytics to measure the interactions and participations of users. In this way, the effort follows the pathways of users and observes the choices they make (DPD Follow-up Interview).

Regardless of where you start, it’s more than likely you’ll end up somewhere other than your intended destination. That’s okay. Systems are living things, and your badge system needs to be flexible. You must embrace a bit of chaos in its design.
-Carla Casilli, Director of Design + Practice at the Badge Alliance