Request for Feedback: Allow for a small amount of timeout retries per game

[bvanvugt]

It's frustrating for any player when their Battlesnake times out for reasons beyond their control, especially if that results in them losing a game. Network hiccups happen -- that's a reality of web development, and we worry that the game engine is a bit too punishing in these cases that are clearly outside of player control.

So we'd like to propose adding some timeout leniency into the Battlesnake API.

One option is to allow up to N timeout retries per Battlesnake per game, with timeouts being immediately retried until N is reached (at which point the current move forward behavior kicks in). An example of this would be a "3-Strike" system.

Another option is to allow N timeout retries per M turns, ie: you get 1 retries every 100 turns, and it's wasted if not used. This is slightly more forgiving, and doesn't punish longer games like. A variation could be everyone has 1 timeout allowed, and once used, it recharges after N turns.

It may also make sense to increase the timeout on retries, in an effort to give servers that have gone idle a chance to wake up. If the game timeout is set to 500ms, maybe your first retry is 1s, and your second retry is 2s.

We're also open to other ideas here, and would love to hear what the community thinks.

Some identified concerns:

• Timeout allowances can be abused by top tier Battlesnakes for additional compute. This is technically true, but also top players would understand the risk of not leaving timeout allowances for actual unexpected situations. So it's a risk.
• The API would likely need to be updated to indicate that a timeout occurred, so that Battlesnakes would be aware of what allowances are available to them. Ideally this would be a backwards compatible change, and we'd want to be thoughtful here.
• Ideally the game board UI also reflects when these retries occur, and what the current retry state is for each Battlesnake.

[aubreyyan]

for these constants N/M, do you have the numbers on hand for the average number of outlier latency events that might reasonably be attributed to network latency (as opposed to a server fault)? It's probably important that they're not arbitrary.

What if the problem is avoidable? As in, maybe the problem lies with your hosting strategy and if you're using an unreliable provider instead of AWS EC2 (as a reliable counterpoint) then you might even argue the snake owner is responsible and should accept that risk? This might be a testable hypothesis, if you ran some controlled tests on different hosting platforms to determine whether one is more/less reliable than the other.

[xtagon]

Hot take, but IMO timeouts are better solved on the snake server's end than the engine side. There are a few things that cause timeouts:

1. The snake is simply offline/unreachable. If this is because of issues like Heroku's auto-sleep, the solution should be to either comply with Heroku's free tier or circumvent it with regular pings, but introducing retry in Battlesnake does not seem like an appropriate solution. If instead it is because the snake is having server problems, a retry is not likely to solve it within a turn or two.

2. The snake exceeds the allotted time because of network delays. IMO this is also more appropriate to solve as the snake developer. Retries will work around temporary network blips, but in the cases where this is a regular problem, the issue is usually that the server is too far away and again retries are probably only a bandaid, whereas it is relatively easy for snake developers to learn about network locality, and there are plenty of cheap hosting options in-region.

3. The snake exceeds the allotted time because their algorithm. Not much needs to be said here :) Retries in this case would just be unfair.

4. The snake exceeds the allotted time because of concurrency of games. This is definitely a challenge, because time-limiting a snake's algorithm is much easier when there's only a concurrency of 1, but when you're in multiple games simultaneously, it's harder to scale. But firstly, this is part of the Battlesnake experience: you learn about web servers, you learn about performance, and you learn about scaling and concurrency, out of necessity. And secondly, the engine already takes some measures to account for this and limit how many games a snake is in at any given time.

[aubreyyan]

piggybacking off of that, it would be nice to have some kind of console or dashboard that visualizes how often our snake is timing out. especially if it's due to network issues, it's hard to track timeouts from the snake server side. that feels like a better solution to me.

[robbles]

I agree with @xtagon here, with the caveat that there might be odd edge cases that pop up occasionally that feel "unfair" even if the player knows about them.
Maybe a combination of features like this would be a good compromise:

• a very minimal retry policy intended to catch rare timeouts (like the 1 retry that recharges every N turns)
• a way of surfacing timeout statistics to the player that allows them to dig into some of the possible reasons above. Like a histogram of response times, or a breakdown of where the timeouts tend to occur for their snake (connect, response timeout, etc.)

I think long term a "performance dashboard" for your snake would be incredible for learning about optimizing code 😄

[bvanvugt]

Yeah, I want to clarify that I also agree with every point made by @xtagon! Server reliability is "part of the game" and we should work hard to keep it that way. 💯

I want to refer back to the original post:

It's frustrating for any player when their Battlesnake times out for reasons beyond their control, especially if that results in them losing a game. Network hiccups happen -- that's a reality of web development, and we worry that the game engine is a bit too punishing in these cases

and emphasize the phrase "beyond their control". That's the heart of this proposal :)

(Xtagon, you mention this in #2: "Retries will work around temporary network blips")

Even the most reliable providers aren't 100% uptime, so it seems unfair to expect that of Battlesnakes. I'm more searching for a way of lowering that to something like... 99.9%?

I think what @robbles suggests as a compromise is super aligned with the original idea here, I like it.

[xtagon]

Yeah I like the idea of a compromise or at least limiting the scope of this