Agent performance improvements

[krdln]

Tackle low-hanging fruits in terms of agent performance improvements and remove major bottlenecks. Also try to reduce GC pressure.

The largest CPU utilization (10%) is RequestToInputWithServices which is creating the input for Rego. We can avoid this for flows that don't have any matching classifiers by running selector matching stage before creating this input.
Classify function that invokes Rego is 6.5%
CheckRequest is 3.6%
Log processing is 10%
Batch processor (for sending cloud metrics) is 5%
Grpc stream handling is 13%
go runtime schedule is 10
gc is about 10%

• Skip RequestToInput convertion with no classifier applied #2171
• reduce mallocs
  • actually profile mallocs @krdln
    • write it down somewhere
  • Use vtprotobuf for faster unmarshalling and less mallocs #2212
  • try pooling in other places
• skip rego for simple classifiers (extractors)
• try to shrink check response (check response is marshaled in Check and unmarshaled in logs processing). (or even cache check responses as they're agent-to-agent?)
• other bottlenecks
  • deep copies of entities: Avoid deep copy in entity lookup #2178
  • speed up sort in rollupprocessor.key: perf: Misc speedups in otelcollector processors #2186
  • try to reduce Gets from attribute map
  • StartTracesOp is > 2%, what's that? This is caused by obsreport tracing, which we don't use
  • AddCheckResponseBasedLabels: perf: Misc speedups in otelcollector processors #2186
  • label convertions
• investigate why we spend in 12% in runtime.mcall – is it normal or do we do sth weird with goroutines?

In italics are marked things that require some bigger changes