More human than human: AI game bots pass Turing Test

For five years, the annual BotPrize competition has been using a variant of the Turing Test known as a "Computer game bot Turing Test" to challenge programmers, researchers and hobbyists to create a bot for Unreal Tournament 2004 (UT2004) that is indistinguishable from a human player. Fittingly, in the centenary year of Turing’s birth, not one but two teams have finally claimed the prize by achieving “humanness ratings” of over 50 percent. In comparison, human players received an average humanness rating of just over 40 percent.

For the uninitiated, UT2004 is a fast-paced, futuristic first person shooter (FPS) video game released, as the name suggests, in 2004. The game’s support for bots (computer-controlled opponents) and extensive modifications (mods) makes the game ideal fit for the BotPrize competition. The competition sees human players facing off against computer-controlled bots over multiple rounds. The human players also serve as judges and attempt to guess which opponents are human and which are the bots by tagging them with a “judging gun” that joins the player’s usual arsenal of weapons.

To take home the US$7,000 prize, entrants had to create a bot that achieved a humanness rating of at least 50 percent. The previous best was 37.5 percent achieved in 2011, but this year saw both the UT^2 bot from the University of Texas at Austin, and the MirrorBot created by Romanian computer scientist Mihai Polceanu, finally beat the 50 percent barrier with humanness ratings of 51.9 and 52.2 percent, respectively.

Of the four human players, only two achieved a humanness rating above 50 percent, with 53.3 percent the highest and 41.4 the average for human players. However, this did better the humanness rating for the bots, which came in at 34.2 percent.

“The idea is to evaluate how we can make game bots, which are nonplayer characters (NPCs) controlled by AI algorithms, appear as human as possible,” said Risto Miikkulainen, a professor of computer science in the College of Natural Sciences who created the UT^2 game bot with doctoral students Jacob Schrum and Igor Karpov. “When this ‘Turing test for game bots’ competition was started, the goal was 50 percent humanness. It took us five years to get there, but that level was finally reached last week, and it’s not a fluke.”

The UT^2 team took a two-pronged approach in getting their bot to most convincingly mimic as much of the range of human behavior as possible. While some behavior was modeled directly on previously observed human behavior, such as a human player’s propensity to pursue specific opponents as part of a grudge, the bot’s central battle behaviors were developed through a process modeled on biological evolution called neuroevolution.

“In the case of the BotPrize, a great deal of the challenge is in defining what 'human-like' is, and then setting constraints upon the neural networks so that they evolve toward that behavior,” said Schrum. “If we just set the goal as eliminating one’s enemies, a bot will evolve toward having perfect aim, which is not very human-like. So we impose constraints on the bot’s aim, such that rapid movements and long distances decrease accuracy. By evolving for good performance under such behavioral constraints, the bot’s skill is optimized within human limitations, resulting in behavior that is good but still human-like.”

Now that the initial goal of the BotPrize competition has been met, competition organizers plan to up the difficulty next year with a new challenge to get the competitors to take their bots to “the next level of human-like performance.”

The video below shows a human judge,"Miguel," as he faces off against "Ty," which is actually UT^2.