Cleanup opensearch output

[Pablu23]

Description

Cleanup and optimize Opensearch async output

Assignee

• The changes adhere to the contribution guidelines
• I have performed a self-review of my code
• My changes generate no new warnings (e.g. flake8/mypy/pytest/...) other than deprecations
• Change base branch from poc-mainloop to main, after merge from mainloop

Documentation

• README.md is up-to-date
• CHANGELOG.md is up-to-date
• Documentation (doc/source) is up-to-date
• Preview for docs looks fine
• Notebooks are up-to-date and functional
• Examples are up-to-date and functional (including compose & k8s)

Code Quality

• Patch test coverage > 95% and does not decrease
• New code uses correct & specific type hints

How did you verify that the changes work in practice?

• List of (preferably easy reproducible) tests including OS

Reviewer

• The code is readable/maintainable and follows the contribution guidelines
• Documentation is up-to-date
• Tests (unit & acceptance) are meaningful and not over-mocked

---

The rendered docs for this PR can be found here.

[Pablu23]

bulk_id = uuid.uuid4()
actions = {**event.data, "_id": f"{bulk_id}_{index}"} for index, event in enumerate(events)
index = 0
async for success, item in helpers.async_streaming_bulk(client, actions, **kwargs):

  # This should not be possible!
  assert index < len(events)
  
  assert index == int(item["create"]["_id"][37:])

This proofs that helpers.async_streaming_bulk keeps the order for actions and yield iteration the same

[Pablu23]

This could possibly stay in the code, but I dont like generating a uuid here and setting it as an id, if opensearch probably does that more performantly

[mhoff]

Many thanks for your work. Here the few comments we already discussed

[mhoff]

Please add a follow-up ticket for us that we might want send the chunks concurrently in the future, depending on where we identify actual performance bottlenecks

[Pablu23]

https://github.com/fkie-cad/Logprep/tree/wip-async-output-no-helper here is a wip implementation, but this is only around 50 eps faster, should look into this