You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When the osquery runner cannot launch an osquery instance, we currently return an error, which will shut down launcher entirely.
Looking over the logs and past issues we've investigated, I see two primary errors: 1) timeout waiting for osqueryd to create socket, indicating the osquery process did not start up, and 2) could not create an extension client where the socket file does not exist or the connection is refused.
In both of these cases, restarting launcher is overkill, and even detrimental to solving the issue. In some cases, we can see these errors happen when the current osquery version is old and not compatible with the current database; restarting launcher in this case is actively harmful because it resets the autoupdate delay, preventing a newer osquery version from being downloaded.
So! We want to change the runner behavior to repeatedly retry starting osquery instances and not exit from the runner.
If osquery instance launch fails, retry launching the instance -- potentially with backoff
If osquery instance launch fails, also consider triggering an autoupdate check for osquery
Runner should still be responsive to calls to Shutdown
The text was updated successfully, but these errors were encountered:
RebeccaMahany
changed the title
When the osquery runner cannot launch an osquery instance, do not return an error. Instead, retry launching osquery instances (maybe with a bit of backoff?). We should tackle this issue first and earlier, to handle the edge case noted in 1 above. We could also consider terminating the autoupdate delay in this case, to download a new osquery version quicker. Could also potentially tackle the item below ("Make the osquery instance status available in knapsack") at the same time, if easier.
Retry launching osquery issues on failure
Nov 5, 2024
RebeccaMahany
changed the title
Retry launching osquery issues on failure
Retry launching osquery instance on failure
Nov 5, 2024
When the osquery runner cannot launch an osquery instance, we currently return an error, which will shut down launcher entirely.
Looking over the logs and past issues we've investigated, I see two primary errors: 1)
timeout waiting for osqueryd to create socket
, indicating the osquery process did not start up, and 2)could not create an extension client
where the socket file does not exist or the connection is refused.In both of these cases, restarting launcher is overkill, and even detrimental to solving the issue. In some cases, we can see these errors happen when the current osquery version is old and not compatible with the current database; restarting launcher in this case is actively harmful because it resets the autoupdate delay, preventing a newer osquery version from being downloaded.
So! We want to change the runner behavior to repeatedly retry starting osquery instances and not exit from the runner.
Shutdown
The text was updated successfully, but these errors were encountered: