[Response Ops][Task Manager] Propagate msearch error status code so backpressure mechanism responds correctly

[ymao1]

Resolves https://github.com/elastic/response-ops-team/issues/240

Summary

Creating an MsearchError class that preserves the status code from any msearch errors. These errors are already piped to the managed configuration observable that watches for and responds to ES errors from the update by query claim strategy so I updated that filter to filter for msearch 429 and 503 errors as well.

To Verify

1. Make sure you're using the mget claim strategy (xpack.task_manager.claim_strategy: 'mget') and start ES and Kibana.
2. Inject a 429 error into an msearch response.

--- a/x-pack/plugins/task_manager/server/task_store.ts
+++ b/x-pack/plugins/task_manager/server/task_store.ts
@@ -571,6 +571,8 @@ export class TaskStore {
     });
     const { responses } = result;

+    responses[0].status = 429;
+
     const versionMap = this.createVersionMap([]);

3. See task manager log the msearch errors and eventually reduce polling capacity

[2024-10-23T15:35:59.255-04:00][ERROR][plugins.taskManager] Failed to poll for work: Unexpected status code from taskStore::msearch: 429
[2024-10-23T15:35:59.756-04:00][ERROR][plugins.taskManager] Failed to poll for work: Unexpected status code from taskStore::msearch: 429
[2024-10-23T15:36:00.257-04:00][ERROR][plugins.taskManager] Failed to poll for work: Unexpected status code from taskStore::msearch: 429
[2024-10-23T15:36:00.757-04:00][ERROR][plugins.taskManager] Failed to poll for work: Unexpected status code from taskStore::msearch: 429
...

[2024-10-23T15:36:06.267-04:00][WARN ][plugins.taskManager] Poll interval configuration is temporarily increased after Elasticsearch returned 19 "too many request" and/or "execute [inline] script" error(s).
[2024-10-23T15:36:06.268-04:00][WARN ][plugins.taskManager] Capacity configuration is temporarily reduced after Elasticsearch returned 19 "too many request" and/or "execute [inline] script" error(s).

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[ymao1]

@elasticmachine merge upstream

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: df910b7

Metrics [docs]

✅ unchanged

History

• 💛 Build #245243 was flaky 3b0d716

cc @ymao1

[mikecote]

Changes LGTM! Left a few nits but testing locally worked as expected!

[mikecote]

optional nit: not sure if we need this to ensure the catch path goes through.

Suggested change

	await store.msearch([{}]);
	await store.msearch([{}]);
	throw new Error('should have thrown');

[mikecote]

nit: It's interesting that response.status could be null. If ever that occurs, should we consider it an unexpected status code as well?

[mikecote]

I added the v8.16.0 label so we can make this part of that release as well when we have clusters opting in.