-
Notifications
You must be signed in to change notification settings - Fork 1
FAP vs Sparcraft v2
I create to separate versions of my bot from master, and switch using configuration what combat estimation method bot should use. Each variant play 24 games against opponent, which was not pre-trained on my bot. All games should be on the same map (Benzene).
WARNING: I have too much serious mistakes to take this comparison seriously. I promise create new comparison will at least less errors, but it will take more time right now, since I have to play more games
Mistake in testing setup was to not control race of AKBot, and I was lazy, so I run 24 games, to somehow account for that. Number 24 was taken deliberately as 2 * Count of bot races * 2 * Count of starting positions on map.
| Bot |
|---|
| PurpleWave |
| 100382319 |
| Hao Pan |
| OpprimoBot |
| Marine Hell |
| Goliat |
| Bjorn P Mattsson |
| DaleeTYC |
| Oleg Ostroumov |
| UC3ManoloBot |
| Roman Danielis |
| Korean |
| Bryan Weber |
| HOLD Z |
| Kruecke |
| MadMixZ |
| AyyyLmao |
| Travis Shelton |
| Lukas Moravec |
| Jakub Trancik |
| MadMixT |
| UPStarCraftAI 2016 |
| KaonBot |
| Niels Justesen |
| Opponent | Spracraft wins | FAP wins | Sparcraft Avg. Duration | FAP Avg. Duration |
|---|---|---|---|---|
| PurpleWave | 12 | 9 | 529.892884816667 | 274.7247885 |
| 100382319 | 21 | 22 | 73.5540207291667 | 65.7766821541667 |
| Hao Pan | 4 | 11 | 296.3858326625 | 110.2506482 |
| OpprimoBot | 4 | 13 | 498.589050529167 | 150.649752958333 |
| Marine Hell | 5 | 10 | 289.315761258333 | 89.11732025 |
| Goliat | 23 | 14 | 245.098135937 | 276.060866566667 |
| Bjorn P Mattsson | 24 | 24 | 55.3131592791667 | 53.8102806041667 |
| DaleeTYC | 12 | 14 | 202.190096645833 | 95.0314442291667 |
| Oleg Ostroumov | 23 | 24 | 55.6143562125 | 53.64181405 |
| UC3ManoloBot | 23 | 24 | 83.6172546041667 | 60.9130681125 |
| Roman Danielis | 11 | 18 | 248.078314720833 | 99.488444775 |
| Korean | 19 | 12 | 110.759966304167 | 71.7638120541667 |
| Bryan Weber | 14 | 12 | 201.618030045833 | 144.045309404167 |
| HOLD Z | 9 | 9 | 75.6426302958333 | 65.6706770083333 |
| Kruecke | 10 | 10 | 342.182368104167 | 88.5675233666667 |
| MadMixZ | 16 | 12 | 273.914519583333 | 88.5682084083333 |
| AyyyLmao | 24 | 24 | 56.8274248958333 | 53.26245915 |
| Travis Shelton | 18 | 14 | 313.979046970833 | 99.0185034875 |
| Lukas Moravec | 16 | 14 | 141.445919995833 | 166.56462935 |
| Jakub Trancik | 24 | 24 | 99.784521125 | 97.7009552791666 |
| MadMixT | 18 | 14 | 134.352378466667 | 97.1388986083333 |
| UPStarCraftAI 2016 | 11 | 7 | 76.4071761791667 | 59.5341572708333 |
| KaonBot | 17 | 14 | 340.026204616667 | 137.444390891667 |
| Niels Justesen | 6 | 5 | 309.8448227375 | 121.397800595833 |
Data was collected from observations from here #15
Game duration here is measured in seconds, and likely correlated with how much frames game takes. Even if FAP claimed to be faster, then that should not affect overall performance of bot in significant way. If that not true, let me know.
| Opponent | Spracraft wins | FAP wins | Sparcraft Avg. Duration | FAP Avg. Duration |
|---|---|---|---|---|
| Bjorn P Mattsson | 24 | 24 | 55.3131592791667 | 53.8102806041667 |
| HOLD Z | 9 | 9 | 75.6426302958333 | 65.6706770083333 |
| Kruecke | 10 | 10 | 342.182368104167 | 88.5675233666667 |
| AyyyLmao | 24 | 24 | 56.8274248958333 | 53.26245915 |
| Jakub Trancik | 24 | 24 | 99.784521125 | 97.7009552791666 |
Not much difference in term of performance. Time of play looks more or like the same, except Kruecke.
| Opponent | Spracraft wins | FAP wins | Sparcraft Avg. Duration | FAP Avg. Duration |
|---|---|---|---|---|
| 100382319 | 21 | 22 | 73.5540207291667 | 65.7766821541667 |
| Hao Pan | 4 | 11 | 296.3858326625 | 110.2506482 |
| OpprimoBot | 4 | 13 | 498.589050529167 | 150.649752958333 |
| Marine Hell | 5 | 10 | 289.315761258333 | 89.11732025 |
| DaleeTYC | 12 | 14 | 202.190096645833 | 95.0314442291667 |
| Oleg Ostroumov | 23 | 24 | 55.6143562125 | 53.64181405 |
| UC3ManoloBot | 23 | 24 | 83.6172546041667 | 60.9130681125 |
| Roman Danielis | 11 | 18 | 248.078314720833 | 99.488444775 |
Total amount of win increase: 1 + 7 + 9 + 5 + 2 + 1 + 1 + 7 = 33
Not much interesting increase, I want more bots jump to 5 win games. Looks like drop in game duration corresponding to the loss of games mostly.
| Opponent | Spracraft wins | FAP wins | Sparcraft Avg. Duration | FAP Avg. Duration |
|---|---|---|---|---|
| PurpleWave | 12 | 9 | 529.892884816667 | 274.7247885 |
| Korean | 19 | 12 | 110.759966304167 | 71.7638120541667 |
| Bryan Weber | 14 | 12 | 201.618030045833 | 144.045309404167 |
| MadMixZ | 16 | 12 | 273.914519583333 | 88.5682084083333 |
| Travis Shelton | 18 | 14 | 313.979046970833 | 99.0185034875 |
| Lukas Moravec | 16 | 14 | 141.445919995833 | 166.56462935 |
| MadMixT | 18 | 14 | 134.352378466667 | 97.1388986083333 |
| UPStarCraftAI 2016 | 11 | 7 | 76.4071761791667 | 59.5341572708333 |
| KaonBot | 17 | 14 | 340.026204616667 | 137.444390891667 |
| Niels Justesen | 6 | 5 | 309.8448227375 | 121.397800595833 |
Total amount of win decrease: 3 + 7 + 2 + 4 + 4 + 2 + 4 + 4 + 3 + 1 = 34
Looks like drop in performance here is worse then increase in other areas where FAP works better.