Test run

So, the holidays got busy, and then I got sick. Overall, I hadn’t done much on this, but that change now.

Today, I’m trying a test run, with several rounds of a mid-sized Neural Network (3 layers, 500 nodes each) against a completely random opponent. This was mainly to lay out the framework in my code for loading models, evaluating data, generating batches of training data, etc.

I generated a set of 100,000 training items by playing the random number generator against itself, and trained the network against this set for 10 epochs. I then began playing games between the random player and the AI, alternating first move, until I had another 100,000 training lines generated.

After the first cycle:

Total games played: 16,643
When Random went first, Random won 2,652 out of 8,320 games, or 31.88%
When AI went first, AI won 6,473 out of 8,320 games, or 77.80%

Hmm, not bad for one training cycle. Let’s run another cycle and see if things improve:

Total games played: 16,770
When Random went first, Random won 2,565 out of 8,384 games, or 30.59%
When AI went first, AI won 6,668 out of 8,384 games, or 79.53%

A slight improvement… Let’s try a few more cycles and see how much my NN improves!

Total games played: 16,745
When Random went first, Random won 2,556 out of 8,372 games, or 30.53%
When AI went first, AI won 6,706 out of 8,372 games, or 80.10%
...
Total games played: 16,947
When Random went first, Random won 2,500 out of 8,474 games, or 29.50%
When AI went first, AI won 6,865 out of 8,473 games, or 81.02%
...
Total games played: 17,002
When Random went first, Random won 2,422 out of 8,501 games, or 28.49%
When AI went first, AI won 6,869 out of 8,501 games, or 80.80%

What we’re seeing here is clear: a small, but measurable, difference after each cycle. Just for kicks, I’m going to increase the number of training epochs at each stage from 10 to 50, then run a few cycles. This should see a larger increase…

Total games played: 17,046
When Random went first, Random won 2,221 out of 8,522 games, or 26.06%
When AI went first, AI won 7,070 out of 8,522 games, or 82.96%

That was something like 8 more cycles. Definitely an improvement, but not a dramatic one. As a framework for my larger test & round-robin tournament, it works.

Before I begin the tournament proper, however, I’m going to run some more tests with different sizes of training sets, different numbers of epochs, and different learning rates.