Results by Depth and Width and Depth (Rounds 11-15)

I’m still tracking number of games won during the tournament, but as those results are relative to each other it’s hard to see if any absolute progress is being made. So, as previously reported, I started playing each model against the Smart Random player for 100 games and seeing how often the NN won. This will allow me to see absolute improvements, whereas the tournament only lets me know relative rankings.

1 x 26 26 34 26 29
2 x 25 26 29 27 27
3 x 23 26 27 22 23
4 x 22 26 26 24 23
5 x 19 25 23 21 23

(Numbers above are the averages; for instance, the group of single layered networks won, on average, 26 games out of 100 on the 11th round, and then 29 out of 100 on the 15th)

The previous trend of favoring deep networks is completely reversed here. At every step, the shallower networks were the clear favorites. Also, we see an interesting peak in quality after the 13th round, and the 14th and 15th actually regressed.

x 100 9 10 13 10 13
x 250 16 23 22 19 20
x 500 21 26 22 25 25
x 1000 24 25 24 21 24
x 2500 22 23 30 22 22
x 5000 23 22 29 22 21

Here, again, we see a different trend from before. Where before the wider networks did less poorly, here the wider networks developed a clear advantage. There does seem to be a sweet spot between 500 and 1000 nodes wide, but it doesn’t seem terribly pronounced.

Now, let’s see how this compares to games won, not against the Smart player, but against other AI:

Depth Round 11 Round 12 Round 13 Round 14 Round 15
1 47 50 48 49 48
2 47 49 47 50 48
3 45 47 47 50 49
4 49 47 45 49 49
5 48 46 46 49 52
Width Round 11 Round 12 Round 13 Round 14 Round 15
100 57 53 55 57 54
250 52 53 48 55 51
500 45 53 47 44 47
1000 46 44 47 46 46
2500 47 42 45 44 44
5000 43 43 46 43 47

Again, this is number of games won out of 100. What we see here is interesting… Again, the bias towards deeper,  narrower models.

Now this is perplexing. The networks that do well against the Smart Random player do worse against other networks. That’s… fascinating.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s