feat: add alertmanager client

[mpenet]

Adds simple alertmanager client with a default implementation using aleph for http transport.

[arnaudgeiser]

This approach works for me as long as you take care of doing the PRs on all the artifacts that would need to replace the Riemann client with the Alert client.

Down the line, we should aim at having those concerns directly available in Mania which should basically provide a base system and configuration to which we'll be able to augment with a subsystem from the artifact we are creating. But we are not there yet.

[mpenet]

I am not super happy with this being part of reporter personally (it was my undestanding it had to be). I would rather have it separate and depended upon in mania down the line as you suggest. Otherwise, yes, the ultimate goal is to get out of riemann on all artifacts.

[arnaudgeiser]

I am not super happy with this being part of reporter personally (it was my undestanding it had to be). I would rather have it separate and depended upon in mania down the line as you suggest. Otherwise, yes, the ultimate goal is to get out of riemann on all artifacts.

This is a bit more nuanced than this and I would say it's two efforts:

• A short-term one, where the ultimate goal is to move out from Riemann without effort (ideally zero) on the artifacts
• A medium-term one, where we are shaping what we expect in terms of alerting from the orchestrators. Our stance (which still needs discussion) is that not everything can be metric-shaped and we should have alerts with enough context. The following write-up [1] is a good starting point

Regarding the short-term measure, it doesn't have to be implemented specially on reporter, but it has to be implemented somewhere centrally so it would be almost a no-op on the client side. An alternative path that could have been considered :

• Vendor the reporter namespaces inside mania (a private, internal repository)
• Alter the riemann-events! function to send alert to the Alertmanager API directly (and deprecate the function right away)
• Hardcode as much as possible inside mania (URL and so on..)
• Create a new alert! function (or send!) in a dedicated mania namespace that would eventually replace reporter

  reporter/src/spootnik/reporter/impl.clj

  Line 234 in 8cd24d5

  (riemann-send client)

Then, on the consuming artifacts it would be just a matter of bumping mania and removing the reporter dependency. Maybe a bit more will be needed but you get the idea.

As you may I guessed, we don't have a strong appetite to have a special library dedicated just to this.

[1] : https://app.shortcut.com/exoscale/story/115506/orchestrators-alerting-strategy#activity-117365

[mpenet]

closing in favor of https://github.com/exoscale/mania/pull/26 [wip]