Cloud output v2

[codebien]

Context

#2954 introduces the new experimental Coud output with a Protobuf-based protocol.

Memory usage

After the first iteration, the memory usage is higher than required. Especially for the Trend metrics is very easy to saturate the bandwidth in a range from tons of KiloBytes up to the remote limit (1 MB).

We also decided to denormalize some fields to reduce the workload and keep the implementation simple on the remote server but the load generated on the client is high, we should revisit this decision.

Fault tolerance

The current flush process could be more fault tolerant, it doesn't retry on failures.

Validation

__name__ and test_run_id are reserved labels for the remote service and if a test also sets them then there are conflicts generating unexpected behavior for the user. A more dev-friendly UX should be implemented.

Proposal

We identified some actions that should drive us to the goal:

• A more compact Protobuf representation for Histogram.
• Split in multiple requests when the flush process gets a number of time series higher than the MaxMetricSamplesPerPackage variable.
• Normalize as MetricSet's fields the common fields across time series.
• Fault-tolerant flush operation.
• Exclude __name__ and test_run_id from the allowed tag names.

Acceptance criteria

Change the Cloud output default version to 2.

Worklog

Must have

Beta Give feedback

1. output/cloudv2: Flush chunks #3108
   +0
   
2. output/cloudv2: Retry and flushers pool #3104
   +0
   
3. cloudv2: Not replicate common fields #3144
   +0
   
4. output/cloudv2: Compact histogram #3169
   cloud +1
   
5. Warn and discard any label equal to the reserved #3155
   cloud +1
   
6. Use a dedicated option instead of MaxMetricSamplesPerPackage #3156
   cloud +1
   
7. chore: add MaxTimeSeriesInBatch to the printable config #3172
   +0
   
8. Concurrent flush across batches #3192
   +0
   
9. cloud: Set v2 as the default output #3258 (postponed)
10. changelog: Cloud output v2 #3161
    +0
    

Nice to have (in case we need to reduce the scope)

Beta Give feedback

1. output/cloudv2: Unlimited size for body payload #3120
   +0
   
2. output/cloudv2: Use a static remote service url #3125
   +0
   
3. TestOutputFlush always enabled #3146
   cloud lower prio tests +3
   
4. cloudv2: s/ReferenceID/TestRunID/ #3137
   +0
   
5. Snappy-Framing support #3122
   cloud enhancement performance +3
6. Revaluate the current periodic and abort signal architecture/interaction (output/cloudv2: Error handling for flush #3082 (comment), output/cloudv2: Retry and flushers pool #3104 (comment))
7. Unexport all the strucs/methods/fields not required as exported

[codebien]

Most of the critical work expected here has been merged and will be released on v0.46.0. Remaining work will continue on #3258