Memory usage never goes down, even after a scale out of the service.

[sambuc]

Memory usage seems to only grow until we get to an OOMkill event.

We lowered the autoscale percentage memory limit as it is computed on the average memory usage of all the currently pods, but this does not help the memory allocation spread out between all the service pods.

In case of a scale out event, I would expect a new pod to start (it does) and then the memory load to eventually even out between all the running pods.

[leafty]

In case of a scale out event, I would expect a new pod to start (it does) and then the memory load to eventually even out between all the running pods.

Software is not magic, the memory load cannot "spread out" like a fluid.

If go is not triggering memory garbage collection before it goes over the limit, this indicates that we do not properly set the go memory limit environment variable, so we should start there.