EMFILE: too many open files, watch '/logs/traefik.log error on backend

[arcoast]

Hi, not a Pangolin user, but trying to get this stood up with my own longstanding Traefik stack. I'm clearly doing something wrong and would really appreciate it if someone could take a look and see if they can spot where I'm failing!

My traefik setup

networks:
    traefik:
        external: false
        name: traefik
        ipam:
            driver: default
            config:
                - subnet: 10.87.210.0/24
                #   gateway: 10.87.210.254

services:
    traefik:
        image: traefik
        container_name: traefik
        networks:
            traefik:
                ipv4_address: 10.87.210.253
        command:
            - --log.level=info
            - --global.checknewversion=true
            - --global.sendanonymoususage=false
            - --serversTransport.insecureSkipVerify=true
            # API
            - --api.dashboard=true
            - --api.insecure=false
            # Entrypoint - websecure
            - --entryPoints.websecure.address=:443
            - --entryPoints.websecure.forwardedHeaders.trustedIPs=127.0.0.1/32, 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8
            - --entryPoints.websecure.transport.respondingTimeouts.readTimeout=0s
            - --entryPoints.websecure.http.tls.certResolver=letsencrypt-cloudflare-dns
            - --entryPoints.websecure.http.tls.domains[0].main=${DOMAIN}
            - --entryPoints.websecure.http.tls.domains[0].sans=*.${DOMAIN} 
            # Certs - Cloudflare
            - --certificatesresolvers.letsencrypt-cloudflare-dns.acme.email=${TRAEFIK_EMAIL}
            - --certificatesresolvers.letsencrypt-cloudflare-dns.acme.storage=acme.json
            # - --certificatesresolvers.letsencrypt-cloudflare-dns.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory #Uncomment for staging
            - --certificatesresolvers.letsencrypt-cloudflare-dns.acme.dnsChallenge.provider=cloudflare
            - --certificatesResolvers.letsencrypt-cloudflare-dns.acme.dnsChallenge.resolvers=1.1.1.1:53, 1.0.0.1:53
            - --certificatesresolvers.letsencrypt-cloudflare-dns.acme.dnsChallenge.propagation.delayBeforeChecks=20s
            - --certificatesResolvers.letsencrypt-cloudflare-dns.acme.dnsChallenge.propagation.disableChecks=true
            # Providers - docker
            - --providers.docker=true
            # - --providers.docker.endpoint=unix:///var/run/docker.sock # Disable for Socket Proxy. Enable otherwise.
            - --providers.docker.endpoint=tcp://docker-socket-proxy:2375 # Enable for Socket Proxy. Disable otherwise.
            - --providers.docker.exposedByDefault=false
            - --providers.docker.defaultRule=Host(`{{ index .Labels "com.docker.compose.service"}}.${DOMAIN}`)
            - --providers.docker.network=traefik
            # Providers - file
            - --providers.file.directory=/configs
            - --providers.file.watch=true
            # Providers - redis
            - --providers.redis.endpoints=traefik-redis:6379
            - --providers.redis.password=${TRAEFIK_REDIS_PASS}
            # Access Log
            - --accessLog=true
            - --accessLog.filePath=/logs/traefik.log
            - --accessLog.format=json
            - --accessLog.fields.defaultMode=keep
            - --accessLog.fields.headers.defaultMode=keep
        ports:
            - 443:443
        environment:
            - TZ=$TZ
            - DOMAIN=${DOMAIN}
            - HOSTIP=${HOSTIP}
            - CF_API_EMAIL=${TRAEFIK_CF_API_EMAIL}
            - CF_DNS_API_TOKEN=${TRAEFIK_CF_DNS_API_TOKEN}
        volumes:
            - ${CONFIG}/traefik/traefik/configs:/configs:ro
            - ${CONFIG}/traefik/traefik/acme.json:/acme.json # Must create file and chmod 600
            - ${CONFIG}/traefik/traefik/logs/:/logs
        depends_on:
            - docker-socket-proxy
        restart: unless-stopped
        labels:
            - "traefik.enable=true"
            - "traefik.docker.network=traefik"
            - "traefik.http.routers.traefik.service=api@internal"
            - "traefik.http.routers.traefik.entrypoints=websecure"
            - "traefik.http.routers.traefik.rule=Host(`traefik.${DOMAIN}`)"
            - "traefik.http.routers.traefik.middlewares=authelia@file, securityHeaders@file"

The relevant part I think from the above is this:

            # Access Log
            - --accessLog=true
            - --accessLog.filePath=/logs/traefik.log
            - --accessLog.format=json
            - --accessLog.fields.defaultMode=keep
            - --accessLog.fields.headers.defaultMode=keep

and

 - ${CONFIG}/traefik/traefik/logs/:/logs

My Traefik Log Dashboard setup

networks:
    traefik-log-dashboard:
        external: false
        name: traefik-log-dashboard

services:
    backend:
        image: ghcr.io/hhftechnology/traefik-log-dashboard-backend:1.0
        container_name: traefik-dashboard-backend
        networks:
            - traefik-log-dashboard
        environment:
            - NODE_ENV=production
            - PORT=3001
            - TRAEFIK_LOG_PATH=/logs/traefik.log
            - TRAEFIK_LOG_FILE=/logs/traefik.log
        volumes:
            - ${CONFIG}/traefik/traefik/logs/:/logs:ro
        ports:
            - 3001:3001
        restart: unless-stopped
        healthcheck:
            test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:3001/health"]
            interval: 30s
            timeout: 10s
            retries: 3

    frontend:
        image: ghcr.io/hhftechnology/traefik-log-dashboard-frontend:1.0
        container_name: traefik-dashboard-frontend
        networks:
            - traefik
            - traefik-log-dashboard
        # ports:
        #     - 80:80
        depends_on:
            - backend
        restart: unless-stopped
        healthcheck:
            test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost/"]
            interval: 30s
            timeout: 10s
            retries: 3
        labels:
            - "traefik.enable=true"
            - "traefik.docker.network=traefik"
            - "traefik.http.services.traefik-log-dashboard.loadbalancer.server.port=80"
            - "traefik.http.routers.traefik-log-dashboard.entrypoints=websecure"
            - "traefik.http.routers.traefik-log-dashboard.rule=Host(`traefik-log-dashboard.${DOMAIN}`)"           
            - "traefik.http.routers.traefik-log-dashboard.middlewares=authelia@file, securityHeaders@file"

On starting up I get this error:

2025-07-30T16:39:54.270856314Z Server running on port 3001
2025-07-30T16:39:54.271318062Z Starting geolocation pre-caching...
2025-07-30T16:39:54.276420872Z Found 1 unique public IP addresses to geolocate.
2025-07-30T16:39:54.336618599Z Pre-cached 1 of 1 IPs
2025-07-30T16:39:54.336639517Z Finished geolocation pre-caching.
2025-07-30T16:39:54.336825612Z Tail starting...
2025-07-30T16:39:54.336834932Z filename: /logs/traefik.log
2025-07-30T16:39:54.336856278Z encoding: utf-8
2025-07-30T16:39:54.337022085Z fromBeginning: true
2025-07-30T16:39:54.337175772Z filesystem.watch present? true
2025-07-30T16:39:54.337182030Z useWatchFile: false
2025-07-30T16:39:54.337934796Z watch strategy: watch
2025-07-30T16:39:54.338527237Z watch for /logs/traefik.log failed: Error: EMFILE: too many open files, watch '/logs/traefik.log'
2025-07-30T16:39:54.341015641Z Error [ERR_UNHANDLED_ERROR]: Unhandled error. ("watch for /logs/traefik.log failed: Error: EMFILE: too many open files, watch '/logs/traefik.log'")
2025-07-30T16:39:54.341024858Z     at Tail.emit (node:events:513:17)
2025-07-30T16:39:54.341026850Z     at new Tail (/app/node_modules/tail/lib/tail.js:71:18)
2025-07-30T16:39:54.341028350Z     at LogParser.setLogFile (file:///app/src/logParser.js:42:17)
2025-07-30T16:39:54.341029767Z     at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
2025-07-30T16:39:54.341031237Z   code: 'ERR_UNHANDLED_ERROR',
2025-07-30T16:39:54.341032651Z   context: "watch for /logs/traefik.log failed: Error: EMFILE: too many open files, watch '/logs/traefik.log'"
2025-07-30T16:39:54.341034224Z }

I can see the log file by opening a shell in the backend container and it's definitely in json.

/app # ls -la /logs/
total 18532
drwxr-xr-x    2 99       users           43 Jul 30 16:38 .
drwxr-xr-x    1 root     root            18 Jul 30 16:39 ..
-rw-r--r--    1 root     root       5652767 Jul 30 16:51 traefik.log

Thanks for taking a look.

[hhftechnology]

why two mounts??

[arcoast]

Well I saw both referenced in the docs/yaml files so was just trying anything at that point!

EDIT: Tried with both of them individually with the same error message

[arcoast]

I'd like to update this with details of a resolution, I looked at inodes etc on host thinking that might be the issue, but nothing apparent, in the end I shutdown all my other docker containers (other than Traefik) and then tried and it worked.

I can only conclude it was a host/docker engine issue with regards to the number of files being utilised.