The best of both worlds

Development of an evidence-based game concept

Author: Shengnan Chen

Introduction

Research is often conducted after serious game development has been finished. Although that type of research is necessary to prove the effectiveness of a serious game, it does not help to improve the game if it is found to be ineffective, which is unfortunately true in many cases. Introducing research early and making it an integral part of game development would reduce the risk of spending all the budget on a serious game that will prove to be ineffective in the end. Therefore, we have started to explore an approach for research-based game development (which I will briefly introduce in the next section).

There has been a lot of literature on research during serious game development (e.g., Jiang, McKanna, Calabrese, & El-Nasr, 2017; Katsikarelis, 2015; Olsen, Procci, & Bowers, 2011). However, the existing literature mostly comes from academia. I have not been able to find documentation of the integration of research into the whole cycle of game development done by serious games companies. Therefore, I would like to share what we learned at Grendel Games while applying academic knowledge to game development in a business context. The insights are not meant as instructional, but rather a springboard for critiques and improvement. We are trying to get the best of both worlds – using the power of research and the expertise of a game development company to make fun and effective serious games. These insights may be interesting for serious game companies, researchers, or clients who are interested in having a serious game developed.

We will start from this article, in which we look at the first stage of game development: creating a game concept. I will use a current project as an example. This project is still at the research and development phase, and the game concept we created is for further research to gain new insights, rather than a final game concept. I have omitted some details at the clients’ request. I will write about the steps we took from research to the game concept and the lessons learned. In this project, the clients are also the domain experts, so I will refer to them as clients/domain experts.

Research-based game development

First, I will briefly describe the research-based game development approach that we propose. We are currently testing this approach using our game development projects. This approach is based on a review of the literature about serious game development and interviews with staff who have been involved in research projects with games developed by Grendel Games.

Katsikarelis summarized the following four phases for serious games production: concept phase, pre-production phase, production phase, and post-production phase (Katsikarelis, 2015). In the concept phase, a game concept is devised based on requirements analyses; in the pre-production phase, a prototype is created and revised iteratively based on playtesting feedback; in the production phase, the game is created and improved iteratively; in the post-production phase, the production team provides support for the game.

The goals of research for each phase is different. Research at the concept phase aims to have an evidence-based game concept which is positioned to have a measurable impact (Kato, 2012). During the pre-production phase, the goal of research is to ensure that the core mechanics are effective in terms of both entertainment and education (Rassweiler, 2012). During the production phase, research aims to gather feedback about the usability, engagement, and learning goals of the game in order to continually make improvements (Procci, Chao, Bohnsack, Olsen, & Bowers, 2012). Finally, during the post-production phase, the final game is tested for its efficacy and effectiveness in terms of the learning goals (Kato, 2012). In addition to one-off research studies, in-game metrics can also be implemented to continually monitor the effectiveness (Serrano-Laguna, Manero, Freire, & Fernàndez-Manjòn, 2017).

Based on these different types of research needed for each production phase, we propose the following five steps:

Concept phase: making sure that we have a solid, evidence-based game concept (“How can the game make a measurable impact?”)

Pre-production phase: testing the core mechanics using prototypes (“Do the core mechanics work for the entertainment and education goals?”)

Production phase: testing the usability, engagement, and learning iteratively (“Is the game easy to understand, fun, and effective?”)

Post-production phase 1: doing efficacy and effectiveness research of the final game (“Is the game efficacious and effective in reaching the learning goal?”)

Post-production phase 2: tracking metrics after the game is released (“Is the game effective in the real life?” )

This article will focus on the first part – the concept phase. I will outline the steps that we took to develop the game concept in the next section.

Steps for creating an evidence-based game concept

Define the goal of the game

Defining the goal of the game is the most important step before creating a game concept. Ideally, we want a goal that is specific, time-bound, measurable, feasible, and relevant (Kato, 2012). We also want all stakeholders to agree on this goal. We started with: “Reduce pain intensity by at least XX% for Dutch adult chronic pain patients immediately, X months and X months after they play the game.”

We filled in the blanks using evidence in the literature. Using the decrease of pain intensity as an example, I will demonstrate the deliberation we went through to balance feasibility and relevance. Firstly, I chose to specify a percentage decrease instead of aiming for statistical significant changes because it is more relevant for patients (Dworkin et al., 2008). Statistical significance is still important and will be included in our effectiveness research but achieving statistical significance is a result of a combination of the size and variance of the treatment effects and the sample size (Dworkin et al., 2008). Therefore, it does not always mean clinical significance.

A 10-20% reduction in pain intensity is considered by patients to be minimally important, a reduction of 30% or more is considered to be moderately important, and a decrease of 50% or more is considered to be substantial (Dworkin et al., 2008; Farrar, Young Jr., LaMoreaux, Werth, & Poole, 2001). A previous study using a video game intervention to treat chronic pain patients found a 33% decrease in pain intensity from baseline to post-test (Jones, Moore, & Choo, 2016). Therefore, we decided to aim for a 30% decrease, which is both relevant for the patients and feasible according to previous research.

Do a literature review

To make sure that we have a scientific list of requirements for the game, we did literature research together with the clients/domain experts, which resulted in a document called “Problem Analyses”. Based on the structure recommended by a previous article (Kato, 2012), we defined the problems that the game tries to solve, the goal of the game, the theoretical background, and the existing treatment methods (including games and non-games). A second part of the “Problem Analyses” proposed candidate outcome measures and the preliminary planning of the efficacy and effectiveness research.

An example of the existing treatment methods that we found from the literature is Cognitive Behavioural Therapy. It focuses on people’s beliefs about pain and how their thoughts affect their behaviour (Eccleston, Williams, & Morley, 2009). We found that among all the psychological treatments, it has the most evidence and is the most effective for chronic pain (Eccleston et al., 2009). Hence, we decided to include Cognitive Behavioural Therapy as a treatment method in the game.

Extract requirements for game concept from the theoretical review

From the “Problem Analyses”, we extracted 20 requirements for the game concept. I will illustrate how we did that with one of the 20 requirements. We found that the variability of context can maximize the transfer of the new association learned in exposure therapy based on previous research (Craske, Treanor, Conway, Tbozinek, & Vervliet, 2014). From there, we extracted a requirement that the game should have a varied combination of exercises, scenarios, and environments.

It is important to note that we had limited budget for the first prototype, so it would not be feasible to incorporate all 20 requirements in this prototype. Furthermore, even though we understood from the literature that Cognitive Behavioural Therapy is the most effective treatment for chronic pain (Eccleston et al., 2009), we did not have the expertise to judge which techniques within Cognitive Behavioural Therapy were the most important.

Therefore, we sent the list of 20 requirements to the clients/domain experts and asked them to rate the priority of each requirement. Seven out of the 20 requirements were rated as “this must be done”, six were rated as “high priority”, two were rated as “low priority”, and four had uncertain priority due to insufficient evidence.

Create a game concept

Together with the game designer Remi, we had a brainstorming session and came up with 15 game concepts. We started from the requirement that we thought was the most important – the game should provoke movements that the patients usually avoid – and jotted down whatever concepts that came to mind. We did not filter the concepts using other requirements. Our ideas ranged from cheerleader management to a Last Guardian type of game.

After we had an initial list, we ran the concepts through the 20 requirements that we derived from the previous step and selected the game concept that met the most requirements (18 out of the 20 requirements).

Remi and I then met with Gerard, who is the programmer and the product owner for the game (i.e., product manager), to further discuss this concept. Gerard pointed out that the main gameplay (i.e., collecting) in the concept takes a lot of time to make since it requires a new environment for each session and suggested another gameplay (i.e., puzzle). He suggested that in a puzzle game, the gameplay loop can be repeated easily without making new environments. There can be more individual variations for patients and it is suitable for people of different ages. We did not go for an action-based gameplay, even though it is also easy to be made engaging, because we assumed that it would not be suitable for older patients.

We also revised the game concept so that all the requirements that were deemed mandatory and highly important by the clients/domain experts were reflected in the game. Some requirements, while not deemed important by them, were still included because we thought they had the greatest potential to distinguish the game from previous games for chronic pain.

Ask clients/domain experts to evaluate the game concept

The clients/domain experts looked at the first game concept and had a meeting with us. There was a large discrepancy between their ideas of the game and the initial concept. The clients had done multiple tests with a wide range of chronic pain patients (from 20 to 80 years old) using existing games, and they had obtained new insights about what the patients liked and what worked well. Because of the limited budget for the first prototype and the insights they gained so far, they suggested to narrow down the list of requirements to four requirements that they thought were the most important. The other requirements are still valid and can be added in future prototypes.

The main discrepancy was that from our literature review, we reached the conclusion that behavioral experiment and cognitive distraction were the most important treatment methods and we had a game concept based on that. However, the clients/domain experts thought that cognitive diffusion was the most important treatment method that should be included in the game. That had a big impact on the game concept.

Revise game concept

The setting, gameplay, and story were changed to more accurately reflect the new set of treatment methods. The clients/domain experts suggested an action-based gameplay and the game development team took the suggestion. This may seem like a hasty change compared to the time we spent on preparing the first game concept and making sure that it was based on the theoretical review. To address this issue, I will reflect on the reasons for the change:

The first reason was that the clients/domain experts came with new information that contradicted the assumption we had about the target audience. We previously assumed that an action-based gameplay would not be suitable for the target audience, but the clients/domain experts had tested an action-based game with the target audience and found out that they liked it.

The second reason for this radical change was that the clients/domain experts were much more specific about the requirements after they saw the first concept. They narrowed down the list of requirements to four requirements, and only two of these requirements came from the 13 requirements that they deemed highly important or mandatory before reading the game concept. The other two requirements were new.

The third reason was that we mis-judged which requirements were the most important because we did not have the domain knowledge and clinical experience to make the judgement.

Lessons learned

Nobody has made a game like this, so we cannot just use a magic formula. Going through all the steps above made sure that we have a game concept that is based on the best information and evidence we have so far. Nonetheless, there are many improvements we can make to the process:

In this case, changing the game concept radically based on clients/domain experts’ feedback was justified because the new concept adhered to new evidence, and the gameplay suggested by the clients/domain experts was considered by the game development team to be easier to be made engaging than the gameplay in the first concept.

However, this could go wrong sometimes. I interviewed Tristan, the Game Design Director at Grendel Games, about a game that the company made. In the beginning, the domain experts insisted that the educational elements should be separate from the gameplay elements because the target audience can be easily distracted and have difficulty completing the exercises. Although Tristan struggled with the dilemma that having purely educational components without gameplay would make the game boring, he went with the domain experts’ suggestion because he trusted their expertise. Later, when the game’s development was almost finished, he heard from another domain expert that it would not be problematic to incorporate gameplay into the educational components for this target group.

In the future, we should filter clients/domain experts’ feedback by separating the functional requirements from the creative suggestions. Specifically, we should follow the clients/domain experts’ requirements about the goals of the game but be more cautious when they suggest how the game should look, how the gameplay should be, or how the educational elements should be incorporated into the gameplay. If we are in doubt, we can get the opinions of other domain experts and the other members of the company or test the ideas with the target audience. If the clients/domain experts do not like the game concept, but it is effective and liked by the target audience, we should stand our ground.

Expect a communication curve and communicate early and often with the clients/domain experts

Figure 1. Communication curve.

From the initial ideas that the client came to us with to the current game concept, there have been multiple iterations. During every iteration, we discovered discrepancies between what we thought the clients/domain experts wanted and what they actually wanted. It feels like the discrepancy is getting smaller and smaller as we talk with each other more and more. Eventually we will reach the Bull’s eye – a game concept that we are both happy with (see Figure 1). Some parts of the curve are avoidable. For example, if we could get the requirements from the clients/domain experts and clarify them earlier, and if we could involve the clients more while making the first game concept, we could have made a game concept that was more on-point at the first time. Other parts of the curve are inevitable because the clients/domain experts may not be able to tell us what exactly they want until they see a concrete game concept. It indicates that when we plan for the development of a serious game, we should take the communication time into account, and communicate with the clients/domain experts earlier and more frequently.

Combine the domain experts’ opinions with the literature review

We spent a long time on doing the literature research, but all of the 20 requirements for the game concept came from the section about existing treatment methods (games and non-games). Out of the 20 requirements, only two ended up in the game concept for the first prototype. The academic in me had the tendency to do a thorough review, but the business world is much more result oriented than academia. The output : input ratio seemed really low using the thorough literature review method.

As mentioned before, although I could find a long list of effective treatment methods for chronic pain from the literature, I did not have enough domain knowledge in chronic pain to judge which ones were better and more important. A lot of time could have been saved by asking the domain expert to point us to the most effective treatment methods in the beginning of the literature research. Then we can focus on finding evidence to support those specific treatment methods and finding measures for the outcomes of those methods.

On the other hand, even though at the surface it seemed like a thorough literature review did not produce a lot of results, I still think it is necessary for a researcher to go through this thorough review. Although domain experts have a lot of practical experience, they may not be aware of all the research evidence. For example, while writing this article, I came across a Cochrane systematic review of psychological therapies for managing chronic pain (Williams, Eccleston, & Morley, 2012). I found out that Cognitive Behavioral Therapy (which we are using in the game) has small positive effects on disability and catastrophizing, but no impact on pain or mood compared to active control (Williams et al., 2012). Based on this evidence, I had a discussion with the clients/domain experts and decided to add disability and catastrophizing into the list of outcome measures. We kept pain as the main goal of the game because the systematic review did not focus on game-based interventions. It is possible that a serious game can have bigger treatment effects than traditional treatment methods because it is more motivating and engaging (Granic, Lobel, & Engels, 2014). Furthermore, a thorough literature review not only suggests what treatment methods to use, but also provides the rationale for why we use those methods.

In summary, we should ask the domain experts to guide us through the literature review, but at the same time we should back up their opinions with research evidence (whether we ask them for the evidence or look for it ourselves) because expert opinion is the lowest level of clinical evidence (Phillips et al., 2009).

Avoid bias when making judgement about what is fun

I learned in this process that there is not a set formula for fun. At first Remi and I came up with a collection-based game, then Gerard suggested a puzzle game based on his years of game development experience. After that, the clients/domain experts told us that they tested an action-based game with members of the target audience, and they enjoyed it. Each time we as a team quickly accepted the new idea and substituted the old game genre with the new. We made the right decision based on the best information we had within the time constraint. However, the process can still be improved.

While we have to make sure that the game genre fits well with the learning goal, there is a multitude of genres that can reach the same goal. As a game development team is likely to be consisted of core gamers, we can be biased when judging the preferences of the target audience for a serious game, who are usually not core gamers. We went in the right direction because the clients/domain experts tested an action-based game with the target audience. However, we did not know if they liked action games more than other types of games.

In small serious games companies, it is not feasible to always test different game genres with the target audience. Fortunately, researchers have investigated the preference of players from different groups such as age and gender (Breuer, Festi, & Quandt, 2014; Hartmann, Möller, & Krause, 2014). We can be informed partly by the experience of the game development team, and partly by evidence from previous research. We did the former but did not do the latter enough. If we had done a little more searching, we would have found out that older adults like intellectually challenging features in games and prefer puzzle games to action games (Blocker, Wright, & Boot, 2014). Even though not all action games are alike, that paper still raises some caution about developing an action game for our target audience. Fortunately, we have not yet entered the development phase, and we can still revise the game concept if the target audience do not like the gameplay in the prototype.

In summary, we should be more informed by evidence when making decisions about what is fun.

Participate in the testing with the target audience together with the clients/domain experts

Our clients/domain experts did a great thing – they did many tests with patients to evaluate the different treatment methods. From those tests, they narrowed down the treatment methods that worked the best. It would have been better if people from the game development team had also participated in the tests so that we could also observe what worked the best in terms of gameplay. We might have reached different conclusions based on our perspective.

Next step

With the background research and feedback from clients/domain experts, we have a solid game concept now. The next step is to bring the concept to life by making a prototype and testing the prototype with the target audience. Stay tuned!About the authorShengnan Chen comes from a psychological research background, and she became really fascinated by serious games because of their power to motivate people to learn and to change behaviors. After she finished her Research Master’s in Behavioural Science at Radboud University, she started to do an internship at Grendel Games. During her time here, she is learning to do game design and using research to make serious games more fun and effective.

Kato, P. M. (2012). The role of the researcher in making serious games for health. In S. Arnab, I. Dunwell, & K. Dabattista, Serious Games for Healthcare: Applications and Implications (pp. 213-231). Hershey, PA: IGI-Global.