[receiver/awsfirehose] Implement encoding extension framework

[axw]

Component(s)

receiver/awsfirehose

Is your feature request related to a problem? Please describe.

awsfirehosereceiver has a limited set of baked-in unmarshallers: cwlogs (CloudWatch logs), cwmetrics (CloudWatch metric streams JSON format), otlp_v1 (CloudWatch metric streams OTLP 1.0 format).

Firehose itself does not know or care about the format of data records. In theory you could write OTLP logs to a delivery stream using https://docs.aws.amazon.com/firehose/latest/APIReference/API_PutRecord.html, but there is no way for the receiver to unmarshal this. OTLP is just an example: one might produce arbitrary observability data (or other types of data, but out of scope here) to Firehose, which also cannot be decoded without additional unmarshallers.

The unmarshallers within the awsfirehosereceiver module may also be useful for other receivers, e.g. in the S3 receiver for decoding data sent by Firehose to an S3 bucket. Since they are internal to the module, they cannot be reused.

Describe the solution you'd like

I would like to enhance the receiver to implement the encoding extension framework, so that users may inject additional encodings. I have created an untested PoC of this at axw@1904aa5. In that PoC I have left the existing unmarshallers in the receiver, but I would envisage them eventually moving to dedicated extension components.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/awsfirehose: @Aneurysm9

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[axw]

If this direction is acceptable, I would be happy to polish and contribute the branch I referenced above.

[VihasMakwana]

This sounds like a valid request to me.

FWIW, there are many components that use custom unmarshalers/marshalers. Now that we have encoding extension, we should do this for all such components.

[axw]

/label waiting-for-code-owners

[MovieStoreGuy]

Not sure for Firehose strictly, however the extension paths makes sense to me and embracing that adoption.

[Aneurysm9]

This seems reasonable to me. I might have some concerns about the implementation, but I'll defer those until it's out of draft.

[axw]

Thanks for the feedback, I'll clean up the branch and open a PR soon.

[axw]

/label -waiting-for-code-owners

[axw]

@Aneurysm9 seeing as the receiver has alpha stability, there's technically an option to break config straight away. Would you prefer:

1. Wait for Add cloudwatch encoding #37222 and then immediately remove the built-in unmarshallers and record_type config, adding an encoding config that supports only extensions
2. Provide backwards compatibility and remove the built-in unmarshallers later on once 37222 has baked: [receiver/awsfirehose] Add support for encoding extensions #37262

[Aneurysm9]

Regardless of stability level my preference is to not make unnecessary breaking changes and to do as much as possible to both ease the transition and provide as much notice and time for users to adapt as possible. I don't think there's any reason not to follow the standard breaking changes process here.

[axw]

@Aneurysm9 sounds good, thanks - that's my preference too. #37262 maintains backwards compatibility but deprecates the old config, so it's ready for review.