Massively OP’s guide to understanding video game research

I noticed two strange things in the comments section of our lengthy Exploring the Video Game Debate series: People were angry at research that supported what they were trying to argue but hadn’t read well enough to see it, and people discredited the research because their personal social circles had vastly differently experiences (but also, often, didn’t read the research).

And you know what? That’s normal. While game research isn’t nearly as important as biology, a recent BBC article reminded me the internet has made it quite easy for people to plug into communities that reflect and validate their common opinion, sharing and replicating (mis)information again and again. While most of the core MOP community is between 25-44 and revolves around a shared fandom, it’s still possible to be affected by poor-quality input, as seen with Microsoft’s recent teen AI, Tay, and her venture into Twitter.

However, using some semi-recent research about a 14-year-old learning English through World of Warcraft, we can see how to approach research without being affected by bad data (or needing a research degree!).

Use your head, not your social identity

A three-part study by German researchers recently — and unsurprisingly — found that people judge scientific research based on their past experiences and social identity before trying to understand the actual studies. Though the research was about gamers and views of violent games, the researchers said that the same bias is true of most people and (sub)cultures.

While forming opinions based on past experience is essentially the basis of scientific research, basing beliefs on your social/cultural identity is quite dangerous. Remember, video game addiction still isn’t considered a “real” problem because psychologists argue it’s so tightly bound to culture. However, psychology is a “soft science.” There’s a difference between hard sciences and soft sciences; most people probably can’t explain the exact biology behind how their heart works, but they accept that doctors know enough to operate on it. However, if a psychologist says that games can make you desensitized to violence (which is true to a point), lay people who play video games and may not think they are violent will react (sometimes violently). Because soft sciences often use data gathered from personal experiences, people often feel their own experience or knowledge can disprove it.

The problem there is that science is based on past experience and generalizable results, meaning the findings have to apply in multiple and somewhat similar contexts. For example, I spent most of my teenage life never meeting anyone who played the Madden game series. When comparing my experience at E3 with videos of the event for the public, I saw that live Madden reveals got nearly no applause, while event recordings included canned cheers. However, the series clearly sells — but to sports fans, who aren’t well-represented in my personal social circles.

A simple way to check my bias was to google Madden and Sports Illustrated, a sports magazine popular enough that even this die-hard geek knows about it. Sure enough, there are tons of hits referencing the two, showing that, yes, there are some important institutes supporting the Madden series, even if they are barely on my radar. Checking with mainstream communities you don’t often associate with is a good way to mentally counter your bias and avoid overreacting.

In the research I mentioned earlier, the researchers suspected some gamer-participants may have rushed to finish an article they condemned for discussing games and violence, even though the findings did not support a link between the two. The researchers did another study that specifically included an attention check test to correct for that problem and weeded out those who failed it. Again, those who identified the most as gamers were biased against the research from the start and often at the end, some wholly distrusting science simply because the article potentially challenged their identity.

The problem is that people are pre-wired to rebel against information that may threaten their identity, and social media doesn’t help. However, if you know about that pitfall in advance, you can simply remind yourself of your bias and then actually consider what the research says.

Understand the research process

Now, don’t mistake me: Please don’t believe everything you read! It’s good to be critical. Just keep in mind that real research is “peer reviewed” — it has to make it through an obstacle course of other experts in the field — sometimes rivals — before it can be published in the more respectable and highly rated journals. Think of it like a peer review at work: You are both a potential ally and threat, so making it through that process isn’t always easy. Yes, Dan in accounting is a jerk that holds a grudge against you for spilling coffee on his favorite tie, but unless your work place is truly awful or corrupt, there are enough opinions within the company to explain what you do and how you work with others. Research, in theory, gets the same treatment.

Because of this, research also needs allies. By mentioning a long list of other studies (we’re talking pages of research titles!) that discuss the same topic as the conducted study, the researchers are essentially building a sturdy foundation to show that their study is based on other grounded studies that also went through the same peer-review process.

In addition to that, research also needs to be upfront about potential weaknesses and flaws and mount a defense of them so Dan can’t bring them up to delay publishing the findings for another year and holding up your funding. If you ever suspect something’s critically wrong with a study, looking at the actual article should reveal it.

At this point, most people will want to use personal experience. When this happens, ask yourself this: What will the opposition say? Think of my Madden example. It’s quite simple to reject if anyone says, “My experience is different!” Personal experience is just that: personal. When using personal experience, you want to find the parts of your experience that can be generalized since research does the same.

Use your experience

Let’s take a semi-recent example. Drs. Liss Kerstin Sylvén and Pia Sundqvist previously mentioned a 14-year-old boy who learned English through WoW. Dr. Sundqvist later wrote a short study on the boy. Without reading the article, many people will notice that it’s simply one person and question the results, and in some ways, that’s good. We want large group studies whenever possible. However, gaming still has a bit of a negative view in the world, and research on it is far from mainstream, which means smaller studies are still the norm out of necessity.

Sundqvist used a semi-structured interview, meaning the interviewee was encouraged to talk freely with little prompting, so researchers could later discuss the “stories” to highlight findings in relevant research, drawing conclusions to support or detract from other studies. The interviewee was a 14-year-old Bosnian boy who moved to Norway at age 4 and then Sweden at 6. He began learning English in third grade, and in sixth grade he went to a school that used English as the classroom language for many subjects (Swedish law permits only half of school time conducted in a language other than Swedish), so he was basically already multi-lingual and being primed for it, as even his school required him to learn another language (he’d chosen Spanish). He knows a little Croatian and Russian as they’re related to the Bosnian language. He wants to be a doctor like his parents, got to choose his vocational school, and actually approached the researchers on his own.

The boy had recently taken a test that indicated that he would have been able to pass a college-level English test. He’d specifically noted that he plays games for escapism and games allow him to do things he wouldn’t do in real life. The stories were what grabbed him initially, so he started off with single-player experiences before going online. Reading and writing in-game with others helped him correct some languages mistakes. For example, for some reason, he’d thought “s” was pronounced as “h” so “selling” became “helling,” something that his follow players helped him correct. This shows he has self awareness of learning, something that can give learners a very critical edge. He also thinks it’s “awesome to learn new things,” so he likes school.

So here’s what I need to consider when comparing the subject to my own experiences. First this, isn’t a normal kid. It’s difficult to compare his success story with the potential of a mono-lingual mid-western American trying to learn Swahili. Heck, reading this was personally a bit discouraging for me, as I’m an English teacher in Japan and have really struggled to improve my Japanese speaking and writing skills. Like the boy, I have some multi-lingual history, am aware of my learning habits, started off (in English and Japanese) playing games for their stories, and I even live in a foreign country that speaks my target language.

There are a few differences, though. As much as I love Japan, I generally haven’t imagined making this my permanent home, so my future job may not need me to know Japanese. The people I talk to rarely correct my Japanese (and the ones who do rarely get to see me). In fact, I felt like my gaming partners were more worried about trying to communicate with me than actually communicating, creating a stressful atmosphere that prevented any kind of escapism or ability to focus.

Generalize (or ungeneralize) your findings

This is where things get interesting. When you start reading more research, you can draw connections to past articles you’ve read. While personal experience is a decent starting point, simply being able to mention other studies that discuss your experience can help you better understand why your experience may or may not match what you’ve read.

For example, the same journal issue with Sundqvist’s study had another one on Japanese learners using radio programs to study English. They had a lot in common, but this study discussed motivation. Of particular interest was the breaking down of motivation types. The Bosnian boy had several types of motivation that seemed to lead to positive language acquisition: desire for international information exchange, willingness to communicate, confidence in communicating, and motivation to study alone. I, on the other hand, have low confidence in my Japanese ability, a key issue that saw other participants perform more poorly than those who were confident but had lower language skills.

Drs. Kowert and Bowman also wrote something around this time about the role of games in learning through creativity. They mentioned that creativity has been shown to help develop several psychological benefits, such as with overall mental health and problem solving, so gamers often have higher perceptional and cognitive abilities (which is supported by Naval Research). I might have higher perceptional and cognitive abilities as a gamer, but unfortunately my personal life in Japan negatively affected my mental health (mostly confidence), which is another large strike against the learning benefits I should have.

The most obvious way to see this is with a state of mind games can cause: “flow.” I’ve thought of it as “being in the zone” or “one with the game,” though it’s not limited to games. During “flow,” you ignore unimportant, non-related factors, home in on your required tasks, and suppress your inhibitions and self consciousness in a bid to perfectly respond to feedback. It’s not that there’s no spoon; there’s an idea of a spoon, and you’re creating, bending, and destroying spoons like some sort of god. It’s great when you can do this in a game since the perfectly balanced game primes you for this state, but it’s quite a feat to pull this off in real-world tasks. It’s even harder when you’re focusing more on personal problems and consciously trying relax among other people who are stressed out, which I experience with both gamers and non-gamers.

That brings us to identity construction. Games should help the player form an identity with their avatar. This is what helps the player distance him or herself from reality and consider problems from the perspective of the game world. My problem was that even in online games, my character’s issue was with communication and being an outsider. I had no gamer friends I brought into my online adventures with Japanese players, and there was more of a focus on communication than playing, to the point that I felt speaking to people in a PvP game and revealing my foreignness actually made me a larger ganking target.

As much as I wanted to be able to write-off the Bosnian boy’s success, I could see how it was useful: He had a good mix of skills that theoretically would help him learn a language, and it seemed to work. Although I possess some similar qualities, I was missing ones that would have made utilizing those qualities beneficial.

Read between the lines

Again, you don’t have to trust everything you read. It’s important to be critical, but remember that peer-reviewed research goes through a hell of a process. It’s tempting to use personal experience to try to debunk science, especially when the headlines threaten your social identity, but unless you can apply it outside your immediate social circle, it may mean your experience has limited generalizable value — just as mine did. Always ask yourself how your argument can be applied more broadly. For example, I’m often critical of modern game communication research that focuses only on text. My friend’s dad doesn’t really “get” her gaming habits, but the guy is serious about Skype, so how can modern research talk about communication in games without discussing voice chat?

By being able to check my bias, trust peer-reviewed research to be vetted against obvious fraud, and compare my thoughts/experiences with other research, I can make use of professional academic research to help me understand the world around me, digital and otherwise, without regurgitating some of the sludge that clogs up intelligent conversation on the internet.

Thanks for writing this one, Andrew! One thing that strikes me often is that we perhaps confuse “objective” with being “fair” – as another person posted down below, rarely is anyone “objective” … but that doesn’t mean that we can be fair and open to other opinions. As a researcher in video games and a frequent journal review (and as an editor of two scientific journals), I find myself having to remind colleagues to be open to data and ideas first, and then opinions of those data and ideas second. To be honest? The process works pretty well … too many people are involved for any one to “game the system” – but peer review of course means that other human scientists are reviewing work, and they can be flawed. Of course, the good news? Science is a dynamic rather than static process: we replicate and extend (and disprove) current studies in future work.

BalsBigBrother Yeah, trying to speak Japanese does help. However, it’s far from a cure all. I never found any Japanese or Koreans that didn’t KOS me when I tried using their language on the “Asian” Darkfall: Unholy Wars server. I did get some people to group with me on ArcheAge’s Japanese server, but it was never long term.
As you and Dr. Sundqvist and Dr. Sylvén noted, more “tedious” gameplay, like nearly everything I experienced in FF11 ;P, is conducive to making social connections online( http://www.engadget.com/2014/10/13/the-science-of-language-community-and-mmorpgs/ ), which probably helped your situation! I actually made a lot of in-game connections in the original DF, most of whom didn’t speak English natively. However, simply put, I feel there’s some big cultural differences between the way… well, the rest of the world plays and Japanese players play, especially since their PC gaming scene is probably about as vibrant as kendo clubs in American high schools.

Vexia I’m doing much, much better than before, which is why I’ve been speaking about it more freely. You are correct about the reluctance to correct mistakes, but also basic communication, and this actually applies to Tokyo too, though not nearly as often. For example, about ten years ago, I took my brother to a McDonald’s in Shinjuku. I tried to order him a milkshake in Japanese, and almost as soon as I opened my mouth, the cashier left me mid-sentence and got someone who spoke English to take my order. I still tried to speak in Japanese, but the cashier only replied in English. Unless you look “asian,” this will happen a lot. If you travel with an asian person, even if that person speaks no Japanese, the natives may continually attempt to communicate with the asian person, in Japanese, especially outside of Tokyo.

It’s a good experience though, and I’ve had a lot of great experiences out here too. My failures have just been more useful in terms of recent topics, especially as a counter point to some of the positives I often discuss.

This is a great idea for an article! Thanks for penning it, Andrew. I’m sure this kind of thing will be an ongoing issue, but hopefully this will make people more aware of how to approach a scholarly source.
And thanks for sharing your stories about Japan. I’m planning on studying there next year, for at least one semester, and I’m always interested to hear about other non-native people’s experiences, whether they’re good or bad ones. It sounds like you’re having a rough time right now, though. :( From what I understand, it seems there’s actually a reluctance among native Japanese speakers to correct learners’ mistakes? As in, it’s more of a cultural thing to keep those sorts of comments to yourself? Meanwhile, as in the example of the boy who used WoW to learn English, speakers of this language are often more than willing to correct mistakes, either with ill or good intent.

Nordavind This is good. I actually do something like this in my English classes but mostly because there’s “Japanese English” that’s used quite differently than in other parts of the world (“Left over” here is a baseball term meaning that the ball went over the left fielder, from what I’ve understood as a non-baseball fan).

It’s also why I mentioned to check biases (this whole article actually sprung from the “Gamers against science” article I linked, since from the title alone I didn’t think I’d agree but… well, here I am!).

crackfox This is true! That’s why there’s a note about checking research findings with sites outside your personal comfort zone (covering the search for peer reviewed counter points seemed a bit too heavy, so this was a compromise).

As for Japan, Tokyo is practically a different country from the rest of Japan. It’s like Los Angeles vs. Buford, Wyoming (no offense to anyone from there).

I think that even peer reviewed research can be subject to the same issues of social identity, if there is insufficient diversity among those conducting it: http://www.nytimes.com/2011/02/08/science/08tier.html?_r=2
Anyway, as interesting as Andrew’s articles are, I’m struck by how lonely and unhappy he seems in Japan. I have a brother in Tokyo and he seems very content there, but I have always imagined Japan to be a hard place to be a foreigner.

breetoplay Damonvile They were fun to collect on silly comments, and to see how many people you could make laugh…. but now that livefyre makes you actually go back to the thread to see who did it….they’re just extra work now!

Funny you should mention learning Japanese and English speakers.
One of the things I remember about my time playing FFXI with the Japanese players that were active in my timezone was that if you attempted to speak with them in their language even if you butchered the comment it did seem to make them more willing to try English with you. Otherwise they would 100% of the time stick to the stock phrases from the auto-translate function that game has or just down right refuse to speak. This did lead to to learn a few phrases from the Japanese players I grouped to help communication while in turn help them with some of the English they were struggling with and I have to say it was kind of nice :-)
That said the general slow pace of FFXI did help in this regard I imagine and I would have hated to try the same thing in any other game let alone one that was pvp orientated as you mention

It’s hard to be objective. Even when you think you are, you may bring to the table experience and knowledge – sometimes false! – that skews how you perceive what you read/see. This video from Veritasium explains why he starts out most of his videos with people telling how they THINK things work, often misconceptions, as a way to have them not just see the video and then still be misinformed (if they were).