run_trial crashes when gz sim can't reach fuel

[M1chaelM]

Most of our models are stored remotely in gz fuel. By default, they are cached during the first run. However, for our docker image, every run is the first run, so we are not getting any caching benefit. This means that the sever is slow to run in the best case scenario.

Making this worse, the gazebo fuel service is highly unreliable, and frequently cannot be reached. Without local caching, this problem becomes more pronounced. When the service can't be reached, gz produces Error in REST request messages when the models we need to load the simulation can't be found, and eventually it causes the whole process to crash, which makes it look like our server doesn't work.

The simplest workaround is to run a command that caches the models we need in the image itself when we build it. This will mean the build will still fail if fuel is down, but after the image is built we don't have to worry about it.

A second solution would be to keep local copies of the models in our repository like we used to, and hold off on pushing things to fuel until the service is fixed. An advantage of this is that users can still build the image successfully even if fuel is down. Disadvantages are that it's more work and our repository gets bigger.

[M1chaelM]

The first workaround is complicated by this bug in gz fuel gazebosim/gz-fuel-tools#349 , which causes it to report success even when it has failed. This means that if the user tries to build the server when fuel is down, it will build without the cache and there is no way of knowing this other than carefully watching the console output.