Unbounded RAM usage?

[guillemc23]

Hey! I'm trying to deploy a minimal Titiler instance on a docker container. I'm running the latest version, pulling my image from ghcr.io/developmentseed/titiler:0.18.9. I'm not adding any custom endpoints, just the extra cogValidateExtension, cogViewerExtension and ColorMapFactory.

I have noticed that RAM usage seems unbounded (the maximum I've seen before killing the process was around 20GB). I have tried setting resource limitation on my docker-compose.yml and GDAL environment variables as follows:

    environment:
      - GDAL_CACHEMAX=200 # Recommended value
    deploy:
      resources:
        limits:
          cpus: "4.00"
          memory: "4g"

I am not sure what effects do these variables have, since RAM usage keeps growing all the time, looking like a memory leak. To me, it looks as if GDAL is not being able to properly close the datasets after reading.

This is what docker ps returns for this container, with the RAM usage limited at 4GB. I haven't seen it go down for hours now.

I'm not sure if this is expected or if some of my configuration parameters are wrong, so any kind of input would be appreciated! If you need any extra information, I will be happy to provide it to you.

Thank you!

EDIT: This was happening to me as well when running Titiler v0.15.6 outside of Docker

[vincentsarago]

@guillemc23 can you try using ghcr.io/developmentseed/titiler-uvicorn instead ?

the memory usage has been quite a weird thing and not limited to titiler

• is there a memory leak?  cogeotiff/rio-tiler#713
• How does the titiler do memory release #481
• Memory Usage #465
• Memory caching while running multiple raster read operation rasterio/rasterio#3181

[vincentsarago]

I would also make sure to run only 1 process, if you have 200mb for 100 subprocess/threads then you could get 20Gb maybe 🤷‍♂️

[guillemc23]

I'm currently spawning uvicorn with 4 workers as in:

uv run --frozen uvicorn src.app.main:app --host 0.0.0.0 --port 8000 --proxy-headers --workers 4

Will test your suggestions and report back!

[guillemc23]

Dear @vincentsarago ,
I've tested reducing the amount of workers to 1 and also pulling our docker from: ghcr.io/developmentseed/titiler-uvicorn. I have tested as well the docker-compose.yml files in https://github.com/developmentseed/titiler/tree/main/dockerfiles.

Still no luck, our instance never frees up memory even if it's not being used. If it's being used, it looks like it frees up a small amount just enough to not overload.

Not sure if related, but we are reading our COGs from a remote filesystem, in this case, through a signed http request.
Also, when generating our cogs, we're keeping them unit16, I don't think it's related either, but just in case.