output/cloudv2: Binary-based payload

[codebien]

It adds the HTTP client for flushing the metric to the remote cloud service and the new Protobuf-based required payload.

Updates #2954

[codecov-commenter]

Codecov Report

Merging #2963 (92c6694) into master (ddecdfc) will decrease coverage by 1.70%.
The diff coverage is 13.00%.

❗ Current head 92c6694 differs from pull request most recent head b368bd5. Consider uploading reports for the commit b368bd5 to get more accurate results

@@            Coverage Diff             @@
##           master    #2963      +/-   ##
==========================================
- Coverage   75.21%   73.51%   -1.70%     
==========================================
  Files         236      238       +2     
  Lines       17722    18220     +498     
==========================================
+ Hits        13330    13395      +65     
- Misses       3533     3955     +422     
- Partials      859      870      +11     

Flag	Coverage Δ
ubuntu	73.51% <13.00%> (-1.70%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
output/cloud/expv2/pbcloud/metric.pb.go	7.58% <7.58%> (ø)
output/cloud/expv2/metrics_client.go	58.00% <58.00%> (ø)
cloudapi/client.go	72.91% <100.00%> (ø)
cmd/run.go	72.76% <100.00%> (ø)

... and 1 file with indirect coverage changes

[codebien]

@na-- @olegbespalov I added the aggregation proposal also, it shouldn't be optimal but it should be a good first step to achieve a reasonable rate. We can iterate on it after we will get some benchmarks (I don't plan them for this PR).

I would get the reviews for it before I do the refactor asked in #2963 (comment) so I can address it all at once.

[olegbespalov]

I've done a first round, will do the other one a bit later

[codebien]

Rebased after the aggregation merge

[mstoykov]

Shouldn't we just break them in parts then? 100kb doesn't soudn that big. Especially before compression.

[codebien]

How would you do that? Here the limit is related to the protobuf, how do you chunk it without breaking the metric set message? I'm not aware of Protobuf supporting a streaming/chunked version. Did I miss it?

This should be an extreme check, if we need to chunk, then I think we should do that before calling the metric's client, so in its caller. WDYT?

[mstoykov]

You can use https://pkg.go.dev/google.golang.org/protobuf/proto#Size to find out how big it will be before serializing and then splitting if necessary.

And I guess a method like Push can know that protobuf will be used and cut it in pieces when necessary.

We can probably leave that for after we merge everything. But I would like to know how many metrics are 100kb before we actually make this the default. As dropping your metrics because they were 101kb sounds like terrible experience.

[mstoykov]

LGTM! But we should test how big is 100kb encoded metrics before we actually make this the default way to push metrics to the cloud.

[codebien]

But we should test how big is 100kb encoded metrics

Yes, I already have it on my list. I add it also to the worklog in #2954 as a reminder.