Cooper-Hewitt DesignPrep Badges

Preface

The information contained in this project appendix was gathered from the original project proposal that was funded in 2012 and interviews with project leaders through the end of 2014. This appendix and the overall design principles database from the DPD project does not reflect further evolution of the project or developments after the final interview that took place in 2014. As of the final interview, Cooper-Hewitt had issued badges and deeveloped a functioning badge system. Based on this information, we have classified this badge system as an implemented (rather than a partial or suspended) badge system.

Summary

Smithsonian’s Cooper Hewitt and National Design Museum and technology partner LearningTimes proposed to build the DesignPrep badge system (which at the beginning was called Design Exchange). Their program is a digital badge system that has been integrated into Cooper-Hewitt’s DesignPrep program, which engages underserved high school students in New York City in design activities. The program provides youth the opportunity to develop design, collaboration, and presentation skills through participating in activities focused around fashion, architecture, and 3D design. While the activities within DesignPrep focus on specific types of design, DesignPrep also aims to teach “soft skills” such as leadership, collaboration, and presentation skills while showing youth possible career paths in design they may not have considered.

Participants have the opportunity to build portfolios of their work as they progress through the different activities. While these portfolios show final products and accomplishments, DesignPrep badges recognize the processes and pathways a student went through to create that final product. The DesignPrep badges serve to show employers and colleges that the learners possess valued “soft skills” that they need to successfully participate in and progress through advanced design – or other career – programs. DesignPrep uses badges to communicate skills and abilities that grades or transcripts do not highlight, capturing the learner’s progression through the design process.

DesignPrep lets students “collect…skills in a way that is presentable” to design schools and potential employers” (DPD Follow-up Interview). The program aims to teach and recognize the skills that are valuable to these different parties.

Timeline

When DML announced the Badges for Lifelong Learning competition in 2012, Cooper-Hewitt and National Design Museum saw it as an opportunity to expand their DesignPrep program by integrating digital badges into their existing curricula. Initially, the badges focused on discrete design skills, providing learners the opportunity to design both the image of the badge and the pathway to earning it. However, initial feedback from the students indicated that they wanted the badges to “do something;” learners were unsure why they would want to earn a digital badge (DPD Follow-up Interview). Rather than earning the plethora of “game-like” badges that were initially available, learners were interested in “pre-professional” badges that would help them be recognized for their skills and accomplishments. In an early event when learners had the opportunity to earn badges with no stakes attached, only a few of several hundred students opted to earn a digital badge. However, in a later event when learners were told that the digital badges would be the primary criteria on which privileges would be awarded, most of the participants worked to earn digital badges.

As of early 2014, DesignPrep Badges is in the process of moving to a new platform that is more closely tailored to the practices that eventually emerged (DPD Follow-up Interview). The project is expanding and working with employers and colleges to find out what a digital badge needs to communicate in order to be formally recognizable by these outside institutions. The project is focused on building a rich learning ecosystem in which youth can learn life skills that will help them be successful, whether they decide to pursue a career in design or another field.

Evolving Practices and Design Principles

What follows is a list of practices as they relate to the general and more specific design principles in each category of practice. The headings name a (a) General Principle, (b) Specific principle, (c) Specific practice. The paragraphs below each heading detail the project’s (a) intended practice, (b) enacted practice, and (c) how that practice relates to the specific and general principles.

Design Principles for Recognizing Learning

The Cooper-Hewitt DesignPrep badge system developed numerous features to display, support, and provide benefits to badge earners. A major decision was to develop the badges in collaboration with both potential earners and major design institutes in order to ensure that their badges would provide relevant skills, experiences, and knowledge for future designers. Badges were developed to promote and reward the acquisitions of these features.

While DesignPrep is supported by Smithsonian’s Cooper-Hewitt and National Design Museum, the project intended for employers and colleges to recognize digital badges as credentials of accomplishments within the design field. Part of this effort involves collaborating with external organizations. For example, they are partnering with Hive New York to provide youth with some design project experiences. The badges were initially earned as badges in a game are earned, and often represented discrete skills for small accomplishments. Feedback from earners indicated that they wanted the badges to “do more;” they wanted them to be markers that helped them earn privileges. Feedback from outside institutions such as colleges and employers indicated that they wanted students to have worked on “soft skills” such as presentation, collaboration, and leadership skills. While the project did integrate soft skills such as leadership and collaboration into their original design, badges around the concepts felt to provide the strongest opportunities were prioritized in later iterations.

DesignPrep used this feedback to redesign their badges and the pathways to the badges to highlight and recognize these skills. To emphasize that opportunities are connected to achievement, they also attached some internal privileges to the badges, making them the primary criteria for consideration to participate in coveted activities such as internships and presenting their design knowledge at the White House. Because Cooper-Hewitt is already a well known and well-connected institution, the project anticipated that encouraging outside entities to recognize (and presumably offer privileges or opportunities for) the badges would not be a difficult process. While the project is having conversations with outside entities such as the Alberta College of Art + Design (ACAD), these outside entities have not yet formally recognized digital badges.

However, the project views their talks with ACAD as progress, not a roadblock. They are confident that the badges will be recognized, and are thus tailoring the badge pathways, criteria, and information to the needs of both the learners and the external entities.

As stated above, in their initial proposal DesignPrep put more emphasis on discrete skills, but knew they wanted to foster and assess “soft skills” as well. It quickly became clear that while not all learners that participated in the program wanted to be designers, many wanted to learn the skills involved with design and collaboration.

Badges were redesigned to recognize the processes and skills involved in the design process, and highlighted things that the student-curated portfolios did not. After receiving feedback from learners and outside entities, the badges were refined again to carry more weight, earn privileges, and recognized skills valued by external entities. To give the badges weight, the work completed in the activities was ultimately judged by design experts working for the Cooper-Hewitt Design Museum programs (DPD Follow Up Interview). However, in the process of completing the work, the project encouraged peers to provide feedback to one another. This peer-to-peer interaction only occured in face-to-face settings, but they may expand that to online interactions in their platform redesign.

The process of receiving feedback from peers and experts gives the learner the opportunity to hear different perspectives as they go through the revision process, ultimately making their final product more appealing to external entities. This process also allows skills such as communication and collaboration to be recognized and showcase to external viewers.

Use badges as a means of external communication of learning >Badges as communication

Intended, enacted, not yet formalized.

The Cooper-Hewitt Design Prep team intended a badge ecosystem through which learners would be able to display their badges in a variety of contexts. This led to investigating the development of how to utilize badges as evidence of learning for AICAD institutions. David Rios described this investigation as a way to develop the following system:

Students need a way to collect these badges in an environment that allows for sharing. What we’re seeing is that here’s a home for these badges,but it’s hard for users to be able to share them out. We’re trying to develop astructure where students can share them out likea digital resume (DPD Follow Up Interview)

This practice, at the time of the follow up interview, was envisioned as a form of displaying badges to interested institutions as evidence of relevant design skills but has not been formally developed into the badge system.

David Rios said that a major focus of the final and continuing direction for the system is that “students are aware of the pathways they can take” (DPD Follow-up Interview). Halima Johnson elaborated that the badges will be defined around skills, which will show students how their different experiences connect, and give them useful credentials for applying to colleges:

We’re not badging that workshop as “you developed these specific skills by working with an architect one day,” but rather we’re breaking down bite size pieces, so we identified some skills and competencies that we value as a design museum….and we’re looking at those as the basis of our badging system. So instead of getting broad and wildly diverse badges… and having the students not see the connection between those elements, [we’re] showing them that within each of the workshops that they’ve participated in, they’re working on some of the same skills. We’re pulling out those different items, and those are what we’re building the badges around. (DPD Follow-up Interview)

Design Principles for Assessing Learning in Digital Badge Systems

One of the major features of the Cooper-Hewitt badge system was its iterative nature, and how students were involved in providing continual feedback on the program. The result was a system that changed significantly between initial intentions and the various iterations that followed, especially in assessment. Program staff, trained to assess for the badges offered in Cooper-Hewitt’s various design programs, formed the core of the assessors, but the programs made an effort to bring in peer feedback as well, in order to introduce students to how assessment practices are integrated into collaborative teamwork in the design world.

Involve students at a granular level > Learning pathways and badge design > Students help design badges

Intended, enacted, not formalized.

Cooper-Hewitt initially intended for students to design the badges and “badge structures” (pathways) that they would work toward earning, but as the program ran its first iterations of student programs where earning badges were a possibility, they found that students needed better understanding of what digital badges were and how they would be valuable. Team member Halima Johnson noted, that “they understood the idea of a badge, but they didn’t understand why digital badges… How is it going to be used, who is looking at it, why should they be invested in it?” (DPD Follow-up Interview). The program turned to focus more on using badges to promote what they “are really good at,” college and career readiness.

Enhance validity with expert judgment > Use a combination of human and computer experts > Badges validated by experts, peers, or a computer scoring system, depending on the type of badge earned

Intended, not enacted. not formal (Use a combination of human and computer judgment)

Not intended, enacted, formal (Use human experts)

Cooper-Hewitt intended to assess student achievement with a variety of methods, including input peer, expert, and computer systems and over time pared back the methods as badge definitions iterated. As the team reported in an interview with HASTAC, “Our badges rely heavily on iteration, by testing through application and then reviewing and refining an idea in a further application” (HASTAC Q&A). Computer scoring here involved attendance measures. As initially enacted in the DesignPrep program, program staff (experts) perform the summative assessments that are involved in making the final decision to issue a badge, and the involvement of other forms of assessment have been scaled back.

The Cooper-Hewitt team intended a tiered badge system that would allow multiple levels of engagement and interaction to be awarded. This principle was and continues to be intended to create a flexible program that recognized a students level of competence in design at multiple levels. Halima Johnson described this process:

We have many students that come and go; you don’t have to complete every badge to be successful. some people only come for 1-day programs. wanted to give each student a way to interact with us, all the way up to our scholars who are very interested in studying design formally…(DPD FOllow Up Interview).

As of January 2014, the Cooper-Hewitt team was in the process of developing this practice more formally but had not yet decided on how to make assessing different levels of achievement a formal practice in the badge system.

Use formative functions of assessment> Peer feedback > Students get formative feedback

Not intended, enacted, not formalized.

The Cooper-Hewitt design team initially conceived validation of performance on the expert level, because staff had the opportunity to get specific training in how to assess work for issuing badges (DPD Follow-up Interview). One important element of the system is to simulate how assessment practices work in design employment, part of which involves feedback from peers. Originally, the system intended to make peer feedback an important part of participants’ assessments, but they altered the purpose to let peers provides formative feedback rather than make any final decisions about badge issuing (DPD Follow-up Interview).

Formative feedback was enacted on the personal level between peers. This shift in conceptions was relevant to the DesignPrep project in part due to the demands for the development of peer to peer assessment skills inherent in the learning and development of designer skills and the difficulty of building a culture that relies on students’ peers for the summative functions of assessment.

Design Principles for Motivating Learning

The iterative nature of the Cooper-Hewitt DesignPrep badge system allowed for the development of numerous possibilities for badge earners. Feedback from students facilitated the development of new opportunities for badge earners as well as displayed the potential for badges to provide new outcomes for completing the learning objectives required by the badge.

Cooper-Hewitt has an established practice of awarding mentor internships in design fields to their design scholars. In their initial grant proposal, they described that “Consideration for the Cooper-Hewitt Scholars program— a rigorous program for an intimate group of students to concentrate on design as a profession and secure design internships” (DML Stage 1 Proposal) would be among possible incentives connected to earning badges.

As the program developed, varying levels of mentorships and related practices were considered in order to reward a larger pool of students, though no systematic method of organizing these opportunities and connecting them to badges has yet been formalized. The potential rewards for badges ranges from learning experiences in exhibit design, individual meetings with designers, and potential inclusion into future design scholar candidates. The program is still considering how to formally develop these practices in terms of awarding a leveled privilege to a particular badge level into future iterations of its badge system design.

Give badges for small accomplishments to hook in learners > Initiating interest with badges for small accomplishments

Intended, enacted, not yet formalized.

As part of the tiered system of badges, the Designprep team intended to develop inclusive badges for small levels of engagement. These badges were intended to provide connections and insights into further opportunities and privileges for learning design (DPD Follow Up Interview). As of the January 2014, the exact system for rewarding these badges had not been fully developed.

Build outside value for badges > Evidence for outside opportunities > Badges as evidence for school admission or employment

Intended, enacted, formal

Badges were intended to recognize industry-specific skills as well as 21st Century skills that would be valued by prestigious design schools and employers. These badges were developed in collaboration with AICAD in order to potentially facilitate the process of admission into prominent design institutions. As of the DPD follow up interview, the DesignPrep team was currently developing badges to make the evidence of these opportunities more explicit to badge earners.

Recognize identities>Roles within a system>Participating as designers

Intended, enacted, formal

As stated, badges provided recognition of skills and abilities relevant to design practices. Earning a badge was intended to enable empowerment of learners by recognizing design scholars as experts capable of participating within design communities and new experiences. As the system developed, the DesignPrep team enabled earners of badges to participate in activities and roles that enabled learners to view themselves as active designers and scholars of design.

Utilize different types of assessment>Peer, Expert, and Computer>Motivating further learning

Intended, not enacted, not formal

As discussed in the assessment section, Cooper_Hewitt intended to use multiple types of assessment practices including peers, computers, and human experts. As the system developed, however, the use of multiple types of experts for awarding badges was not substantively developed into the system in any officially recognized practice. Of the three different types of assessment practices, only the use of human experts were developed in any officially recognized capacity. The other forms of assessment were shifted down towards a more individualized, informal processes for motivating further learning.

Set goals > User created badges> Earning the opportunity to design badges

Intended, not enacted, not formal

The Cooper-Hewitt team initially intended to have earners of badges to be able to design and develop their scholar badge. As the system developed, however, this practice was revised to an activity within the larger processes and practices of earning a badge rather than a fully realized motivating outcome.

Design Principles for Studying Learning

Like many of the badge systems involved in the Digital Media & Learning competition, Cooper-Hewitt did not intend any systematic practices for studying the activity in the badge system, though informally, they did take a strongly iterative approach to the design process that relied on feedback from participants and stakeholders. The team reports, “When we offer a badge we informally ask students for feedback and examine errors or mistakes in the upload process. We then adapt the structure and requirements in the next round of badges to account for the difficulties” (HASTAC Q&A).

Regardless of where you start, it’s more than likely you’ll end up somewhere other than your intended destination. That’s okay. Systems are living things, and your badge system needs to be flexible. You must embrace a bit of chaos in its design.
-Carla Casilli, Director of Design + Practice at the Badge Alliance