Changes in Student Attitudes Towards 6 Dimensions of Digital Engagement in a Program of Game Design Learning

By Dr Rebecca Reynolds

Introduction

This paper reports findings from the 2009/2010 pilot school year of Globaloria-WV. The project being piloted in West Virginia (WV) schools as a state-wide network of game design learning and social media engagement among youth. Middle, high school and community college students enroll in a blended learning game design elective course offered daily, for 1 or 2 semesters, credit and a grade. N=386 middle and high school students for Pilot Year 3 (2009/2010 school year). Overall, pre and post program survey results partially support our hypothesis that student attitudes towards the range of practices in several categories of participant engagement are positively influenced by their experience in the program. Especially for practices representing the more Constructionist engagement categories, our pre and post-program self-report survey analysis using t-test statistics indicates increases in student motivation towards, and understanding of these practices as a result of participation. That is, their post-program motivation was greater than their pre-program motivation, indicating a measure of success in the program at meeting the stated learning objectives. For three other less-Constructionist but active technology-use categories (e.g., information seeking), the results were more varied with regard to statistically significant increases. Ceiling effects may have played a role in this; pre-survey means were higher for such practices than the more Constructionist ones. The motivational findings on their own appear to indicate the appeal of the workshop style of learning among a large enough group of participants to see statistical effects of change.

Objectives

Approaches to teaching digital literacy in the school context have focused on the importance of imparting specific technology skills, and have been driven by association standards, for instance the NETS technology literacy standards promoted by the International Society for Technology in Education (ISTE, 2007), and the InfoPower information literacy standards promoted by the American Association of School Librarians and American Library Association (AASL, 2008). However, while earlier versions of the technology and information literacy standards tended to focus on more ‘Web 1.0’ forms of information-seeking activity, including searching, locating, evaluating and using informational resources online, the most recent updates to both sets of standards incorporate creative technology uses, and dispositions for productivity with technology tools.

. . . (the) standards exhort students to gain not just technology skills, but dispositions to use those skills . .

For instance, the ISTE NETS standards for students include learning objectives categories such as Standard 1, ‘Creativity and Innovation,’ which calls for students to be able to “demonstrate creative thinking, construct knowledge, and develop innovative products and processes using technology” (ISTE NETS Standards for Students, 2007). The AASL standards exhort students to gain not just technology skills, but dispositions to use those skills, and AASL Standard 4 is entirely focused on students’ pursuit of technology and information uses for personal and aesthetic growth.

Further, the importance of involving learners in programs of creative, project-based digital work is gaining more and more credence as digital literacy, participatory culture and digital divide concerns enter the national educational agenda (e.g., Jenkins, 2009; Hobbs, 2010; Knight Commission on Information Needs of Communities in a Democracy, 2009; Mossberger, Tolbert & McNeal, 2007; National Education Technology Plan, 2010). Achieving many of these standards’ objectives could be seen to require Constructionist interventions (e.g., Harel & Papert, 1991).

Unfortunately, while the updated standards and national priorities for instance those mapped out in the National Education Technology Plan of 2010 reflect significant advances in policy guidelines addressing technology integration for learning in schools, the reality is that actual implementation of substantive technology-based interventions in public schools nation-wide is still relatively rare.

This paper reports findings from the 2009/2010 pilot school year of a digital literacy project being conducted by a non-profit in NYC, with students and educators in West Virginia, funded through grants from the WV Governors Office and a private foundation. The project is a 7-year state-wide network of game design learning and social media engagement among youth, being piloted in WV schools. Middle, high school and community college students enroll in a blended learning game design elective course offered daily, for 1 or 2 semesters, credit and a grade. N=386 middle and high school students for Pilot Year 3 (2009/2010 school year).

Individual students interact with each other, teachers, and content resources on the wiki and in class . . .

ISTE and ALA standards and recommendations offer considerable synergy with the Constructionist approaches adopted in this state-wide network of game design learning. A non-profit provides students and teachers digital learning supports via a wiki-based social media platform called MyGLife.org, in-person twice-annual teacher training, and ongoing virtual webinars with students and teachers. Individual students interact with each other, teachers, and content resources on the wiki and in class, following a blended learning curriculum. They build games in Flash, constructing a portfolio of project-based work on wiki “Team Pages,” layering design documents, game assets, links, embedded game presentation videos, intra-group communications and final game files.

Here we investigate changes in student attitudes towards technology among participants as a result of engaging in the game design class in West Virginia. Results indicate important shifts in dispositions that we expect support our study research questions regarding students’ cultivation of 6 contemporary learning abilities through their participation.

Theoretical framework

In articulating the goals for the current initiative, we advance a framework of learning objectives that guides our applied program development and research, based on previous Constructionist literature, the organisation’s pilot research, and “digital literacy” scholarship. We propose that the 6 Contemporary Learning Abilities with Technology (6-CLAs) emerge through game design in the WV program, and these dimensions prepare students for effective practice in today’s knowledge economies and digital participatory cultures. The learning objectives of the project (Table 1) specify that students will develop a range of six ‘Contemporary Learning Abilities’ (CLAs), which are the six main dimensions of student practice and expertise that we use as learning objectives.

6 CLAs

Practices representing each CLA, and how they are articulated and integrated in Globaloria

1. Invention, progression, and completion of an original digital project idea (for an educational web-game or interactive simulation)

Brainstorming and developing game and simulation ideas and storylines (using Web2.0 tools such as wikis and blogs)
Choosing and researching a subject for a game design project
Developing an original approach to teaching the subject in an educational game
Writing an original game narrative and a proposal to explain it
Generating creative ideas for designs to express the subject of the game and the user experience
Planning game design execution using paper prototyping
Programming a game demo that illustrates the original game design and functionality
Programming and completing a final game
Developing knowledge of the game's domain or topic through game invention and creation

Coordinating the design, creation and programming of the game elements and managing the process of building it
Managing the project’s execution using a wiki (creating wiki pages, organizing and forM.C.ing the wiki, sharing project assets, and progress updates)
Managing the team work (defining and assigning team roles, coordinating tasks, and executing one's role within the team)
Project troubleshooting for self and others
Gaining leadership experience through the project management of all game production elements (e.g., design document, user flow, budget, schedule, introduction, overview, treatment, competitive analysis, teamwork, planning, managing implementation process)

Collaborating by using Web2.0 tools, such as posting to wikis, blogs, open source help forums, Instant messaging
Exchanging and sharing feedback and resources with others by posting information, links, source code questions and answers
Reading and commenting on blogs and wiki pages of others
Presenting final digital projects for others – virtually in game galleries and in person in live game demonstrations

5.Information-based learning, purposeful search, exploration

Searching the Web (using Google, wikipedia and other sources) for answers and help on specific issues related to programming games
Searching and finding resources on MyGLife.org network, website, and wiki
Searching the Web for new Flash design, animation and programming resources
Searching for information in support of the game’s educational subject and storyline

6. Surfing websites and experimenting with web applications and tools

Surfing to MyGLife.org starter kit site and other game sites and playing games online
Keeping track of and bookmarking surfing results that are relevant to projects
Browsing Web2.0 content sites such as Youtube, Flickr, Blogs, Google Tools

Reynolds & Harel (2009) and Harel Caperton (2010) establish the bases for this framework. CLAs 1, 2 and 3 are particularly reflective of Constructionist influence. Students learn introductory computer programming through Actionscript coding. The activities in some ways simulate those employed in workshop-based professional environments. Students collaborate on team game projects and learn time and project management through their project- and problem-based work. Some practices students experience in Globaloria, outlined above, simulate those engaged by active participants and communicators in today’s online digital cultures, such as creation of profile pages and use of the wiki to establish an online identity and to communicate with peers. Globaloria activities differ from more naturalistic social media uses by youth in that the communication and collaboration are centered for the most part around the productive game design activity.

A range of varying outcomes are expected. The primary one is that participants cultivate a new variation of ‘Constructionist Digital Literacy’ through their experience in successfully creating a relevant, meaningful digital artifact and engaging a range of technology practices in an integrated way, towards these ends. We expect that these experiences may prepare students with practices and habits of autonomous inquiry and problem-solving that can contribute towards success in participating in other technology-supported communities of practice and work environments of today. We expect that if successful, the program may lead some students to develop interest in and pursue future opportunities in computing, information management, and/or STEM disciplines. We hypothesise that students may cultivate dispositions towards using technology in more cultural- and social-capital enhancing ways, that can lead to their social mobility. We are also exploring students’ computational thinking outcomes. This study takes one small step in the direction of building the evidence base for these larger program claims. The Six CLAs serve as the learning objectives, outcome goals, and drivers for the continued program design and curriculum decisions made in iteratively developing the program. Given these objectives, we expect to see development across all categories among student participants. This paper addresses the following research question.

To what extent does student participation in Globaloria contribute to changes in technology habits, attitudes, and understanding, across a range of practices in which they engage?

Knowledge tests of the 6-CLAs are still in development. For instance, Reynolds (2011) has generated a content analysis approach to measure game design learning among individual and team game designers, through analysis of student final artifacts as a culminating knowledge product that is used as a dependent variable in several ongoing studies. Here, we focus on student attitudes and dispositions as they relate to each CLA category. Understanding student attitude shifts towards technology as a result of participation adds to the validity of the 6-CLAs model as a set of distinct factors, and supports the evidence base for the program’s achievement of its learning objectives. Further, understanding patterns of such shifts can help us identify new hypotheses regarding contribution of attitude shifts, towards measured learning outcomes, and such variables’ use in multi-level analysis also underway.

Methods

Intervention

In brief, a non-profit provided students and teachers digital learning supports via a wiki-based social media platform called MyGLife.org, in-person twice-annual teacher training, and ongoing virtual webinars with students and teachers. Individual students develop games, interacting with each other, teachers, and content resources on the wiki and in class, following a blended learning curriculum daily, for up to 90 minutes per session, across either a semester or a full year. The program applied many of the attributes of Constructionist design workshops described in Harel & Papert (1991) and Seely Brown (2005). Data sources for the attitudinal results are as follows:

Pre-program student survey data

Post-program student survey data

Educator progress reports bearing official student lists

Course wiki for verification

Pre and post-program surveys were conducted online in August of 2009, January of 2010, and May/June 2010, depending on student participation modality (first semester only, second semester only, or full year). Links were distributed to students via each pilot location wiki, with educator administration. Educators were strongly encouraged to introduce the voluntary surveys prior and subsequent to student engagement with the program, with follow-through by non-profit staff to monitor completion. Research was conducted with full parental consent and child assent, and IRB approval. Out of 386 middle and high school student participants, a total of 368 completed the pre-survey (95%), and 277 completed the post-survey (72%). Findings reported here reflect those who completed both pre and post. A total of 64 middle school and 322 high school students participated. The drop off from pre-survey to post-survey is due to a range of factors, including student voluntary opt-out, student absences at the end of the school year, and student discontinuations in the program, changing of schools, etc. Table 1. N of students by location, for the following variables: Gender, semester start, survey N, average participation months

School

No. of students

Female

Male

Semester 1 start

Semester 2 start

Pre-survey N

Post-Survey N

Average Partic. Months

Braxton Country High School

14

8

6

0

14

13

10

4

Bridgeport Middle School

16

7

9

16

16

15

4

Capital High School

33

13

20

33

0

31

22

9

Eastern Greenbrier Middle School

20

4

16

20

0

19

18

9

Greenbrier East High School

43

8

35

43

0

42

28

9

Greenbrier West High School

9

5

4

9

0

9

5

9

Liberty High School

15

2

13

15

0

15

12

9

Man High School

12

6

6

1

11

10

8

4.5

Oak Glen High School

20

10

10

20

20

13

4

Randolph Technical Center

28

8

20

18

9

25

19

4.2

Riverside High School

26

16

20

36

34

28

4

Sandy River Middle School

26

14

12

26

0

25

25

9

South Harrison High School

15

2

13

14

1

14

12

9

Spring Valley High School

76

8

68

76

0

73

48

8.5

Wheeling Park High School

10

2

8

10

0

10

9

9

Woodrow Wilson High School

13

4

9

13

12

6

4

386

117

269

265

120

368

277

Non-experimental pre/post design

Within the overall design-based research project we have undertaken, this study employed a non-experimental pre/post survey design to measure change in student attitudes towards a set of activities inherent to the intervention. The surveys included two types of self-report measure for each contemporary learning ability: enjoyment, and knowledge. See Appendix A for the specific items applied for each. Indices of enjoyment towards a range of activities were adapted from Ryan, Mims & Koestner (1983) for whom the interest/enjoyment subscale is considered the self-report measure of intrinsic motivation towards a given activity. We used a simple, single enjoyment question for a range of activity types. If students intrinsically enjoy their engagement in the activities, it indicates their value in the activities and a likelihood of their intention to engage again in the future.

For self-reported knowledge, we adapted Hargittai’s validated set of survey items for the concept of ‘digital literacy’ that she presents as a proxy for peoples’ actual technology skills (2005). Hargittai operationalises the ‘perceived knowledge’ proxy survey instrument as follows:

How familiar are you with the following Internet-related items? Please choose a number between 1 and 5 where 1 represents having ‘no understanding’ and 5 represents having ‘a full understanding’ of the item. (none, little, some, good, full).

She presents a range of then-relevant technology items. Among a random sample of U.S. adults the composite optimal index of internet-related items achieved a Cronbach’s alpha of .89, with a predictive power (adjusted R2) of .321 for actual internet competence, measured by a knowledge test of digital task completion -- the highest among all quantitative indices of digital literacy published. Here we adapt this ‘familiarity’ operationalisation, changing the technology items used to reflect those in which students engage in Globaloria. Given the adaptations, the measures do not carry the validity established in the earlier measure. We note the face validity of the adaptation, however.

Categories for enjoyment and knowledge in the CLA categories were created via factor analyses. We applied factor analysis to the pre-program survey items representing each CLA category. Results supported the relatedness of the individual items used to identify each of the CLA factors, with items specified in the Appendix hanging together for CLAs 1, 3, 4, 5 and 6 (with eigenvalues >1). For CLA 2, across the dimensions of self-reported frequency, motivation and knowledge, the factor analysis results indicated 2 sub-factors (creating with digital media and collaborating with team members online separately). Therefore we defined this as 2 separate categories. We performed additive combinations for the set of items in each CLA category identified (five single factors, and two sub-factors) for the measures for frequency, enjoyment and understanding. Pre/post program survey t-tests were then run in each of the 6 CLA categories.

Results

Motivation, prior to program

Middle and high school student group means for their motivation towards practices in the less-constructionist CLAs 4 – 6 (learning with social media, information-based learning and purposeful research, and surfing the internet) appear to be higher than the group means for the more constructionist CLAs 1 – 3 (invention progression & completion, project-based learning (creating and collaborating), and publishing/distribution digital media). This result was expected, since CLAs 1 – 3 reflect practices that are more complex, constructionist and project-based, representing activities, which most students have not experienced prior to participating. Also, these activities are more effortful, and thus may be perceived as less enjoyable.

Motivation, from pre to post

Our pre and post-program self-report survey analysis using t-test statistics indicates that middle school students report statistically significant increases in their enjoyment of program activities within all CLA dimensions 1 – 3, but not 4 - 6. It appears that participation positively influenced middle school students’ enjoyment of the more Constructionist dimensions of the CLAs in particular. It is also important to note that the N for middle school students was lower than that for high school, which can affect the significance level; the group means for CLAs 4-6 trend upwards.

High school students report statistically significant increases in their enjoyment of program activities within CLA dimensions, except CLAs 5 and 6. A ceiling effect for CLAs 5 and 6 again appears to be a factor.

Self-reported knowledge prior to program

The pattern of means for student self-reported knowledge prior to the program across the CLA categories parallel those for enjoyment in that means in the less-constructionist CLAs 4 – 6 (learning with social media, information-based learning and purposeful research, and surfing the internet) appear to be higher than the group means for the more constructionist CLAs 1 – 3 (invention progression & completion, project-based learning (creating and collaborating), and publishing/distribution digital media).

Self-reported knowledge from pre to post

Findings for middle school students indicate statistically significant increases in their self-reported knowledge of the program activities in CLAs 1-4. Results for high school students parallel those for middle school students.

Table 4. Middle school students’ pre- and post-program change in self-reported knowledge of practices in the 6 CLA categories

Discussion

Overall, the pre and post program survey results partially support our hypothesis that student attitudes towards the range of practices in each CLA category are positively influenced by their participation in the game design program. Especially for practices representing the more Constructionist CLAs 1-3, our pre and post-program self-report survey analysis using t-test statistics indicates increases in student motivation towards, and understanding of these practices as a result of participation. That is, their post-program engagement was greater than their pre-program engagement for the practices within CLA categories 1-3, indicating a measure of success in the program at meeting the stated learning objectives.

For CLAs 4-6, the results were more varied with regard to statistically significant increases. Ceiling effects may have played a role in this; pre-survey means were higher for practices in CLA categories 4-6 than 1-3.

The motivational findings on their own appear to indicate the appeal of the workshop style of learning . . .

The motivational findings on their own appear to indicate the appeal of the workshop style of learning among a large enough group of participants to see statistical effects of change. Future research should compare participant change to change in a matched-case set of control students to validate the causal linkage of participation. This program is unique in its focus on project-based game design learning among students, therefore increases for attitudes towards the practices representing CLA categories 1-3 are notable as they may indicate a likelihood for repeat engagement.

Reynolds & Harel Caperton (2011) discuss the discovery-based context for the learning occurring in the program, noting cases in qualitative mid-survey questionnaire, in which students convey a positive affect in particular towards autonomy afforded in the class, and express importance and value for the core program activities presented - whereas others express frustration towards the same program attributes. Reynolds (2011) follows up to report statistically significant findings in a regression model indicating that a student’s intrinsic motivational disposition is positively correlated to student knowledge outcomes in this context as measured by evaluation of student artifacts, whereas an extrinsic motivational disposition is negatively correlated to knowledge outcomes in this discovery-based context.

The motivational and self-reported knowledge results reported here reflect student attitudes/enjoyment towards specific domains of program activity. Given Reynolds (2011) results using generalised motivational inventories, it may be that intrinsic vs. extrinsic motivational disposition has a moderating effect on the activity-specific attitude shifts we see here. To what extent is generalised motivational orientation related to motivation towards differing types of Globaloria activities? Also, to what extent might Globaloria participation shift one from less to more of an intrinsic motivational disposition, or from more to less of an extrinsic motivational orientation? Such a result would be a finding of high significance. And, to what extent do the activity-specific shifts indicated in this study contribute to student knowledge outcomes? The interplay among generalised and activity-specific motivation, and student outcomes, needs to be investigated further.

Seely Brown (2005) states that:

today’s students want to create and learn at the same time. They want to pull content into use immediately. They want it situated and actionable - all aspects of learning-to-be, which is also an identity-forming activity. This path bridges the gap between knowledge and knowing (p. 6).

This program embodies a workshop model and the attitudinal findings indicate that for some students, this program is offering a new mode of engagement in the school setting that supports Seely Brown’s (2005) claims of appeal and resonance of such programs. The results here add to the evidence base for the program’s effectiveness. More research is needed to situate the motivation results here in the larger research agenda of the initiative, and continue advancing the validity of the research.

Reynolds, R & Harel Caperton, I. (2009) 'Development of High School and Community College Students' Contemporary Learning Abilities in [a game design program]', Presented at the annual conference of the American Educational Research Association,

San Diego, CA, April.

Reynolds, R & Harel Caperton, I. (2011) ‘Contrasts in student engagement, meaning-making, dislikes, and challenges in a discovery-based program of game design learning’ in Educational Technology Research and Development, 59 (2): pp. 267-289.

(Editor’s Note: This paper was originally presented at the annual conference of the American Education Research Association (AERA), April 2012, Vancouver, Canada and is printed here with the permission of the author.)

Dr Rebecca Reynolds is an Assistant Professor in the Department of Library and Information Science in the School of Communication & Information at Rutgers University, USA.