helm-chart(v4): Use env var for setting CONTAINER_LOG_PATH, new default for containerRuntime

[joshuabaird]

Fixes #1670 (in combination with PR #1773).

Because this PR contains breaking changes, it bumps the helm chart to v4.0.0-RC.1. There are additional changes we wish to make with v4.0 of the helm chart -- this is the first.

What?

This PR modernizes and simplifies the chart's process for configuring containerRuntime which influences how Fluentbit collects logs from nodes:

• Changes the default for containerRuntime from docker to containerd -- support for the docker runtime was removed in Kubernetes v1.24 (2022) and modern distributions use containerd
• Adds a CONTAINER_LOG_PATH environment variable to the fluent-operator Deployment which is used by Support setting CONTAINER_LOG_PATH via an environment variable #1773
• Removes initContainers on the fluent-operator Deployment that were used to dynamically configure the the log location for the docker containerRuntime
• Provides backwards compatibility with fluent-operator <=3.5 by providing a fluent-operator-env ConfigMap that exposes CONTAINER_ROOT_DIR as an environment variable which is mounted to the FluentBit container(s) as a file
• Simplifies the configuration for container log paths
  • Deprecated: operator.logPath.containerd and operator.logPath.crio
  • Added: operator.containerLogPath for direct path specification
  • New helper function automatically sets sensible defaults:
    • containerd → /var/log/containers
    • crio → /var/log/containers
    • docker → /var/lib/docker/containers
• Updated documentation
• Started a MIGRATION-v4 document which detailed migration instructions

Why?

The initContainers used a legacy and outdated vendored image which was difficult to maintain. Using files to provide the environment variable, while functional, was overly complex. Most of the complexity was because we were supporting custom log paths for the docker runtime. This runtime is no longer in use as of Kubernetes.

If users want to continue using the docker runtime and provide a custom log path, they can do so via the operator.containerLogPath value -- which is documented in the migration guide.

[joshuabaird]

@cw-Guo @marcofranssen This is the first v4 helm-chart feature. I think the CRD re-factor being discussed in #1765 is another candidate. The changes here are compatible with fluent-operator <=3.5 as well.

[marcofranssen]

1. Some Nits regarding docs clarity.
2. Suggestion on the values.
3. question on one of the macro functions.

Furthermore we also have to run helm-docs to update the README with updated values.

[marcofranssen]

The trim suffix would only work if people don't customize the paths.

This makes me wonder if we should even allow people to configure the log paths. Why don't we just always default based on the containerRuntime? I guess it is very unlikely people will use customized log paths with any containerRuntime.

[joshuabaird]

Allowing custom log paths was mainly for backwards compatibility for those users who may still be running the docker runtime with non standard log paths.

Do you think we should remove the option to configure custom paths?

[marcofranssen]

Yeah since we are doing a major version bump I don't think we need to support previous/deprecate things.

In case we want to do a more smooth migration we probably even want to do a 3.x release that only deprecates and falls back to old settings.

[joshuabaird]

See 8239036