Loki on goat is missing is loki-gateway ingress

[QuantumEnigmaa]

I got paged by several alerts for goat and goatproxy clusters because the log agents couldn't send their logs to loki. This was due to the fact that Loki on goat MC was missing its loki-gateway ingress so I created a user-values configmap to have it deployed which solved the incident.

However I shouldn't have had to do that as the loki-gateway ingress is defined in the shared-configs repo so we should investigate why goat's loki-chart-values configmap wasn't matching the shared-configs' config.

I guess that goat being a private cluster might again be at the root of this issue.

[QuentinBisson]

Maybe the gitrepo of flux is matching a custom config branch that does not define the ingress?

[QuantumEnigmaa]

No, all flux gitrepos are aimed at the main (or master) branch :

NAMESPACE       NAME                            REVISION                                                SUSPENDED       READY   MESSAGE                                                                             
flux-giantswarm collection                      master@sha1:48fa209b4b2f1710b4c84d805576e40d0c1bf353    False           True    stored artifact for revision 'master@sha1:48fa209b4b2f1710b4c84d805576e40d0c1bf353'
flux-giantswarm giantswarm-config               main@sha1:0a718686fd16f0546743592c7c39ff2a5b631a98      False           True    stored artifact for revision 'main@sha1:0a718686fd16f0546743592c7c39ff2a5b631a98'  
flux-giantswarm management-clusters-fleet       main@sha1:29ec6ee4b4b25d2b41964325a842f08978689a6c      False           True    stored artifact for revision 'main@sha1:29ec6ee4b4b25d2b41964325a842f08978689a6c'  
flux-giantswarm releases                        master@sha1:479b8f482181742bef3fc32d1bf86dd37007334a    False           True    stored artifact for revision 'master@sha1:479b8f482181742bef3fc32d1bf86dd37007334a'
flux-giantswarm shared-configs                  main@sha1:0cc2c9dccce5342ccf4d8307c48b6cc86ee98126      False           True    stored artifact for revision 'main@sha1:0cc2c9dccce5342ccf4d8307c48b6cc86ee98126'  
flux-giantswarm vsphere-aws-addons-collection   main@sha1:6053804f4d04ea69b8b0951a8e0fb737221c5ba5      False           True    stored artifact for revision 'main@sha1:6053804f4d04ea69b8b0951a8e0fb737221c5ba5'

[QuentinBisson]

I think the ingress was deleted by hand because I removed your user-values CM and it is still there 🤔

[QuantumEnigmaa]

Oh could explain the situation, but why would someone delete the ingress ?

[QuentinBisson]

By accident I Hope 😂

[QuantumEnigmaa]

Ok so the ingress part is back in the loki-chart-values configmap and after manually forcing the chart's redeployment, the ingress is still here so we can consider that this is settled :)