New Research Stresses Potential of Game-based Assessment

GlassLab’s research is one of the most complete studies yet around how to structure assessment within games.

GlassLab, a unique game development effort behind SimCityEDU, has published a new research paper aimed at outlining a framework for how to use games as a major part of educational assessment.

The paper, “Psychometric Considerations in Game-Based Assessment,” grew out of the lab’s work around the new SimCity game. The game sits on top of a massive new database that tracks the actions of players to gauge whether they were learning the lessons in the game.

“[W]e are tracking every time a student is hovering a mouse over a specific object, every time they are clicking on something, every choice, decision and action they make within the game to be able to solve that problem,” GlassLab’s general manager Jessica Lindl said about the new game.

Now, the researchers that worked with Lindl to develop the assessment engine behind SimCity have proposed using the the ideas more broadly.

They see emerging the emerging power of game-based data mixed with psychometics, the study of the mind that uses statistical tools to gauge the understanding of the student. It is this mix of gaming data and scientific tools that could reshape how teachers and administrators gauge student performance.

“Psychometrics allows designers of game-based assessments to create probability models that connect students’ performance in particular game situations to their skills, knowledge, identities, and values, both at a moment in time and as they change over time,” Anya Kamenetz wrote in the report’s executive summary.

The report is a dense piece of social science, running more than 150 pages, but its conclusion appears to be a powerful indicator of how games could affect student assessment in the years to come.

Game-based assessments may hold the promise of a richer, multi-dimensional portrait of student learning, but they also present a new frontier in assessment design, ripe with challenges and opportunities for psychometricians and game designers to explore collaboratively

co-author Robert Mislevy

“This paper provides a framework for the continued exploration of this new frontier and proposes a design approach for developing and testing new game-based assessments,” said co-author Robert Mislevy, a psychometrics consultant for GlassLab, in a statement.

To do this, though, there needs to be an integrated approach, mixing the psychometrics of assessment with the game design process. Only by mixing the two, the authors argue, can you build a game that actually assesses what students are truly learning. But if it can be done, and the authors point to SimCityEDU as an example, then the result can be a far more powerful tool for understanding learning than traditional tests could ever be.

Still, as Kamenetz herself admits in the executive summary, “The path that the authors have outlined is not easy. They call on experts in disparate fields to collaborate to build and test more GBAs (Game-Based Assessments) in order to bolster their empirical track record, which is necessary to see if they can fulfill their promise.

“GBAs offer a new flavor on the assessment platter; they address themselves to higher-order skills and concepts in a way that is open-ended, personalized and engaging, yet with the application of psychometrics to the rich data they generate, they have the potential to demonstrate high validity.”