The Neuro-Evolution via Augmenting Topologies (NEAT)1 algorithm enables users to evolve neural networks without having to worry about esoteric details like hidden layers. Instead, NEAT is clever enough to incorporate all of that into the evolution process itself. You only have to worry about the inputs, outputs, and fitness evaluation.

This is a tutorial on how to use SharpNEAT 2, the second version of a popular C# implementation of the NEAT algorithm2 written by Colin Green. The source code for this tutorial is available here.

In this tutorial series, we'll be evolving neural networks to play Tic-Tac-Toe. I'm going to assume that you already know the basics of neural networks, evolutionary algorithms, and Tic-Tac-Toe. Part 1 will walk you through the basics of how to setup a new experiment, define a fitness evaluator, run the evolution, and use the resulting network in your application.3

Tic-Tac-Toe Game Engine

We first need to create a basic game engine that can simulate games between two players. The TicTacToeGame class defines a simple 3x3 game board:

Players are required to implement the IPlayer interface and define the GetMove method which takes a game board and returns the move to make:

public interface IPlayer
{
Move GetMove(SquareTypes[,] board);
}

Two hand-coded players are provided:

RandomPlayer - Randomly picks an available square at each move.

OptimalPlayer - Plays perfect strategy. The best you can do is draw against it.

The TicTacToeGame class provides a helper method to compete two players against each other:

/// <summary>
/// A helper method for competing two players against each other.
/// </summary>
/// <param name="xPlayer">The player to act first.</param>
/// <param name="oPlayer">The player to act second.</param>
/// <returns>
/// The square type of the winner, or SquareTypes.N if the game
/// was a draw.
/// </returns>
public static SquareTypes PlayGameToEnd(IPlayer xPlayer, IPlayer oPlayer)

Designing the Experiment

Our networks will have nine inputs and nine outputs-- one for each square on the board. The NeatPlayer class takes an IBlackBox object and the player's square type:

The IBlackBox interface is an abstraction for a general model that takes an array of real-valued inputs and outputs an array of real-valued inputs. In our case, we're evolving neural networks, but the code is reusable if you ever want to use a different model type.

Each square on the board will be converted into an integer value and fed into the input nodes. If the square belongs to the opponent, it's -1; if it belongs to us, it's 1; and if it's empty, it's 0:

/// <summary>
/// Gets the next move as dictated by the neural network.
/// </summary>
public Move GetMove(SquareTypes[,] board)
{
...
// Convert the game board into an input array for the network
setInputSignalArray(Brain.InputSignalArray, board);

Once the inputs are set, the model is activated. This takes the inputs and runs them through the model, populating the OutputSignalArray of the model. The available square with the highest output node activation is then chosen as the move:

Creating an Experiment

The convention in SharpNEAT is to define a class implementing INeatExperiment. An experiment encapsulates all the basic functionality needed to create, setup, and run the evolutionary algorithm. However, for the vast majority of projects, almost all of the code will be identical. To simplify the tutorial, I've created an abstract class, SimpleNeatExperiment, which hides all the common setup code so we can focus on the experiment-specific components.4 All you have to define in your TicTacToeExperiment class are four simple properties:

/// <summary>
/// Defines the setup for the Tic-Tac-Toe evolution experiment.
/// </summary>
public class TicTacToeExperiment : SimpleNeatExperiment
{
/// <summary>
/// Gets the Tic-Tac-Toe evaluator that scores individuals.
/// </summary>
public override IPhenomeEvaluator<IBlackBox> PhenomeEvaluator
{
get { return new TicTacToeEvaluator(); }
}
/// <summary>
/// Defines the number of input nodes in the neural network.
/// The network has one input for each square on the board,
/// so it has 9 inputs total.
/// </summary>
public override int InputCount
{
get { return 9; }
}
/// <summary>
/// Defines the number of output nodes in the neural network.
/// The network has one output for each square on the board,
/// so it has 9 outputs total.
/// </summary>
public override int OutputCount
{
get { return 9; }
}
/// <summary>
/// Defines whether all networks should be evaluated every
/// generation, or only new (child) networks. For Tic-Tac-Toe,
/// we're evolving against a random player, so it's a good
/// idea to evaluate individuals again every generation,
/// to make sure it wasn't just luck.
/// </summary>
public override bool EvaluateParents
{
get { return true; }
}
}

Setting XML Parameters

The rest of the parameters are configured via an XML configuration file, "tictactoe.config.xml":

ActivationScheme - How the neural network will be activated. Since NEAT networks can have recurrent connections, iterating through the network multiple times may give you different answers each time. Here's we're fixing the number of iterations to 2. Alternatively, you could set Scheme to Relax which iterate the network until the output between iterations was very small. Typically though, 2 iterations is fine for most problems.

ComplexityRegulationStrategy and ComplexityThreshold - SharpNEAT uses these parameters to determine when to switch from complexification (i.e., adding nodes and connections during mutation) to decomplexification (i.e., subtracting nodes and connections). A good rule of thumb is about 3 * (inputs + outputs).

Description - A text description of our experiment. This is mostly used by the SharpNEAT GUI, which we don't use for this tutorial.

It will probably take some time and tweaking to get a feeling for these parameters. Typically, the tougher the domain, the bigger the population, species count, and complexity threshold.

Creating an IPhenomeEvaluator

The phenome evaluator is where the bulk of the work is done. This is where you put the code that evaluates each neural network based on how it performs in your domain. For our Tic-Tac-Toe evaluator, we're going to play each network for 100 games against a random opponent, half as X and half as O, then play one game as each side against the optimal player. For each game, a win is worth ten points, a tie is worth one point, and a loss is worth zero points.5

/// <summary>
/// Evaluate the provided IBlackBox against the random tic-tac-toe player
/// and return its fitness score. Each network plays 10 games against the
/// random player and two games against the expert player. Half of the
/// games are played as circle and half are played as x.
///
/// A win is worth 10 points, a draw is worth 1 point, and a loss is worth 0
/// points.
/// </summary>
public FitnessInfo Evaluate(IBlackBox box)
{
double fitness = 0;
SquareTypes winner;
OptimalPlayer optimalPlayer = new OptimalPlayer(SquareTypes.O);
RandomPlayer randomPlayer = new RandomPlayer();
NeatPlayer neatPlayer = new NeatPlayer(box, SquareTypes.X);
// Play 50 games as X against a random player
for (int i = 0; i < 50; i++)
{
// Compete the two players against each other.
winner = TicTacToeGame.PlayGameToEnd(neatPlayer, randomPlayer);
// Update the fitness score of the network
fitness += getScore(winner, neatPlayer.SquareType);
}
// Play 50 games as O against a random player
neatPlayer.SquareType = SquareTypes.O;
for (int i = 0; i < 50; i++)
{
// Compete the two players against each other.
winner = TicTacToeGame.PlayGameToEnd(randomPlayer, neatPlayer);
// Update the fitness score of the network
fitness += getScore(winner, neatPlayer.SquareType);
}
// Play 1 game as X against an optimal player
neatPlayer.SquareType = SquareTypes.X;
optimalPlayer.SquareType = SquareTypes.O;
// Compete the two players against each other.
winner = TicTacToeGame.PlayGameToEnd(neatPlayer, optimalPlayer);
// Update the fitness score of the network
fitness += getScore(winner, neatPlayer.SquareType);
// Play 1 game as O against an optimal player
neatPlayer.SquareType = SquareTypes.O;
optimalPlayer.SquareType = SquareTypes.X;
// Compete the two players against each other.
winner = TicTacToeGame.PlayGameToEnd(optimalPlayer, neatPlayer);
// Update the fitness score of the network
fitness += getScore(winner, neatPlayer.SquareType);
// Update the evaluation counter.
_evalCount++;
// If the network plays perfectly, it will beat the random player
// and draw the optimal player.
if (fitness >= 1002)
_stopConditionSatisfied = true;
// Return the fitness score
return new FitnessInfo(fitness, fitness);
}

Now that we've defined our parameters and our evaluator, the only thing left is to run the evolution.

Running the Evolutionary Algorithm

The main execution loop for SharpNEAT 2 is pretty simple. We create our experiment, load the XML parameters, create an instance of NeatEvolutionAlgorithm, and tell it to start. We also add an UpdateEvent handler that saves the best network to file periodically.

When you run the program, it will periodically print out the current fitness score of the best network. The way we have the algorithm setup, a perfect score of all wins would be 1020 (100 games vs. random + 2 games vs. optimal, with 10 points for a win). However, the optimal player will never lose and the random players will sometimes play optimally as well, just by chance. You can go ahead and stop the evolution whenever you're satisfied that fitness has stagnated.

Playing Against Our Neural Network

To play against the neural network we just evolved, run the NeatTicTacToe program. The only code worth showing here is the code to load the network from file:

SharpNEAT saves networks as genomes, which have to be decoded back into their phenotypic neural network state. Fortunately, SimpleNeatExperiment provides a handy helper function to get a genome decoder. The rest of the GUI works similar to how it did in the evolution program.

Conclusion

That's it! In this tutorial, we saw how to create a basic experiment in SharpNEAT 2, a powerful framework for evolving artificial neural networks. We created an experiment to evolve a Tic-Tac-Toe AI by playing it against hand-coded opponent strategies. By now you know how to setup your experiment, set parameters, evolve the networks, and handle file I/O.

If you played against our AI, you probably noticed it really sucks. That's okay! The next part of this tutorial will explore some of the details hidden by SimpleNeatExperiment and introduce co-evolution. The result will (hopefully) be a stronger AI opponent.

Hi
interesting tut i haven’t read all yet, but i made some improvements on the OptimalPlayer, cause i won the first game against it. Now you can’t win just stop the other from winning.

Here the knew code:

/* ***************************************************************************
* This file is part of the NashCoding tutorial on SharpNEAT 2.
*
* Copyright 2010, Wesley Tansey (wes@nashcoding.com)
*
* Both SharpNEAT and this tutorial are free software: you can redistribute
* it and/or modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* SharpNEAT is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with SharpNEAT. If not, see .
*/
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

//if we can't win, check if there are any moves that we have to make
//to prevent ourselves from losing
for (int i = 0; i < 3; i++)
for (int j = 0; j < 3; j++)
{
if (board[i, j] != SquareTypes.N)
continue;

//if we're here, that means we have made at least 1 move already and can't win
//nor lose in 1 move, so just make the optimal play which would be to a free
//corner that isn't blocked
Move move = null;
int max = -1;
for (int i = 0; i < 3; i++)
for (int j = 0; j < 3; j++)
{
if (board[i, j] != SquareTypes.N)
continue;