Use environment variables in data prepper configs

[sandervandegeijn]

Is your feature request related to a problem? Please describe.
Other logforwarders support the use of environment variables in configs. Logstash and FLuentbit both support the ${ENV_VAR} notation in their config files. This is useful in docker/kubernetes implementations where you might have several instances using the same config data. The data prepper docs don't reference this so I presume it's not supported yet.

Describe the solution you'd like
Use environment variables in the configs like Fluent-bit and Logstash do

Describe alternatives you've considered (Optional)
None

Additional context
None

[dlvenable]

I think we'd be supporting this with a syntax like ${{env.ENV_VAR}}. We laid the groundwork for this in the AWS Secrets Manager work: #3382

[sandervandegeijn]

Would be very useful, especially in a container environment 👍

[batpiste]

very useful indeed :)

[anand3493]

@dlvenable Would really like to use ${{env.ENV_VAR}} in our EKS data prepper deployment. Do we have any plan on doing this ?