fix(helm): Add startup probe read

[someStrangerFromTheAbyss]

What this PR does / why we need it:

Which issue(s) this PR fixes:
Fixes [#15191]
Special notes for your reviewer:
Check the default value in the value file for the helm chart
Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[someStrangerFromTheAbyss]

Default value of values.yaml file can be changed. Just want to make sure the issue no longer appears

[jkroepke]

please see my comments.

[jkroepke]

Tests failing because of the startup probe:

Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    10m                  default-scheduler  Successfully assigned loki-5wqncvl3p4/loki-read-685f48b986-pk44v to chart-testing-control-plane
  Warning  FailedMount  10m                  kubelet            MountVolume.SetUp failed for volume "kube-api-access-77mhk" : failed to sync configmap cache: timed out waiting for the condition
  Normal   Pulled       2m30s (x4 over 10m)  kubelet            Container image "docker.io/grafana/loki:3.5.7" already present on machine
  Normal   Created      2m30s (x4 over 10m)  kubelet            Created container: loki
  Normal   Started      2m30s (x4 over 10m)  kubelet            Started container loki
  Warning  Unhealthy    1s (x12 over 8m31s)  kubelet            Startup probe failed: HTTP probe failed with statuscode: 401
  Normal   Killing      1s (x4 over 7m31s)   kubelet            Container loki failed startup probe, will be restarted

[someStrangerFromTheAbyss]

Tests failing because of the startup probe:

Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    10m                  default-scheduler  Successfully assigned loki-5wqncvl3p4/loki-read-685f48b986-pk44v to chart-testing-control-plane
  Warning  FailedMount  10m                  kubelet            MountVolume.SetUp failed for volume "kube-api-access-77mhk" : failed to sync configmap cache: timed out waiting for the condition
  Normal   Pulled       2m30s (x4 over 10m)  kubelet            Container image "docker.io/grafana/loki:3.5.7" already present on machine
  Normal   Created      2m30s (x4 over 10m)  kubelet            Created container: loki
  Normal   Started      2m30s (x4 over 10m)  kubelet            Started container loki
  Warning  Unhealthy    1s (x12 over 8m31s)  kubelet            Startup probe failed: HTTP probe failed with statuscode: 401
  Normal   Killing      1s (x4 over 7m31s)   kubelet            Container loki failed startup probe, will be restarted

Hummm 401. Probably some authentication....It was bold of me to assume that there would be no authentication on read pods.

Ill remove it from the default values, but comment the startupProbe code with a link to the issue the default values.yaml. That way, if futur user have the issue, they can just uncomment the comment block.

Will do the commit in a couple of minutes.

[jkroepke]

@someStrangerFromTheAbyss The problem is different.

I guess the proposed solution works only, if

https://grafana.com/docs/loki/latest/operations/multi-tenancy/

is not enabled.

In that case, the header X-Scope-OrgID is missing. Maybe the canary tenant could be re-used.

Just document the solution wont help either. Maybe another loki Rest call would fit better.

And is this needed on the querier component as well? read exists only on SSD.

[someStrangerFromTheAbyss]

Just document the solution wont help either. Maybe another loki Rest call would fit better.

Hummm... i'll see what i can do.

And is this needed on the querier component as well? read exists only on SSD.

I never deployed in Microservices mode, so i don't know. From the linked issue, this has been reported only in Simple scalable mode.

[jkroepke]

httpProbe supports header, maybe X-Scope-OrgID: fake solves the issues. fake is the default value, if auth is disabled.

[jkroepke]

Hi,

I can confirm, it works on my test setup. However, I would not set this as default. having a 60 second delay feels unnecessary for most users. Instead I would recommend to document that, similar how istio is documented /cc @JStickler