High TIME_WAIT Connections and Growing Cached Memory in NGINX with OpenResty

[hydde7]

I'm experiencing an issue with my NGINX configuration where:

• A large number of connections are stuck in the TIME_WAIT state.
• The system's cached memory continuously increases, eventually consuming most of the available RAM.

I’ve already tried multiple optimizations without success.

What I’ve Tried

NGINX Configuration

• Adjusted keep-alive settings:

  proxy_set_header Connection "keep-alive";

• Increased buffer settings:

  proxy_buffer_size 16k;
  proxy_buffers 4 16k;
  proxy_busy_buffers_size 32k;

Linux Kernel Parameters

• Enabled tcp_tw_reuse:

  net.ipv4.tcp_tw_reuse = 1
  

• Reduced tcp_fin_timeout:

  net.ipv4.tcp_fin_timeout = 30
  

• Verified these settings with:

  sysctl -p

Clearing Cached Memory

• Manually cleared cache using:

  echo 3 > /proc/sys/vm/drop_caches

  However, the cached memory growth resumes almost immediately.

Guidance Needed

• How to reduce the high number of TIME_WAIT connections.
• How to identify and manage the growing cached memory usage.

Any advice or suggestions would be greatly appreciated.

Here's my nginx.conf:

user www-data;
worker_processes auto;
worker_rlimit_nofile 65000;
pid /run/nginx.pid;

events {
    worker_connections 4096;
    use epoll;
    multi_accept on;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    lingering_timeout 20s;
    lingering_close always;

    access_log off;

    error_log /usr/local/openresty/nginx/logs/error.log error;

    lua_package_path "/usr/local/openresty/lualib/resty/?.lua;;";

    more_set_headers 'X-Proxy-Me: $hostname';
    more_set_headers 'X-Proxy-Session: $request_id';

    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 15s;
    keepalive_requests 20;
    client_header_timeout 20;
    client_body_timeout 20;
    reset_timedout_connection on;
    send_timeout 20;
    sendfile on;

    map $http_origin $cors_origin {
        ''      '*';
        default $http_origin;
    }

    server {
        listen 80;
        server_name _;

        gzip  on;
        gzip_types application/vnd.apple.mpegurl video/f4m application/dash+xml text/xml;

        vod_mode remote;
        vod_upstream_location /s3;
        vod_max_metadata_size 32m;
        vod_ignore_edit_list on;
        vod_last_modified_types *;
        vod_expires 1h;
        vod_segment_duration 3000;
        vod_hls_absolute_index_urls off;
        vod_performance_counters off;

        location ~ ^/my_location/([0-9a-f\-]+)/([^/]+)/index\.m3u8$ {
            resolver 127.0.0.11;

            if ($request_method = 'OPTIONS') {
                add_header 'Access-Control-Allow-Origin' '*' always;
                add_header 'Access-Control-Allow-Headers' 'Origin,Range,Accept-Encoding,Referer,Cache-Control,Authorization' always;
                add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' always;
                add_header 'Access-Control-Max-Age' 1728000;
                add_header 'Content-Type' 'text/plain; charset=UTF-8';
                add_header 'Content-Length' 0;
                return 204;
            }

            set $file $1;
            content_by_lua_block {
                local utils = require "utils"
                local bucket, filename, region = utils.get_file_info(ngx.var.file)

                if not bucket or not filename or not region then
                    ngx.status = ngx.HTTP_BAD_REQUEST
                    return
                end

                ngx.req.set_header("bucket", bucket)
                ngx.req.set_header("region", region)
                ngx.exec("/hls/" .. ngx.var.file .. "/" .. filename .. "/" .. "index.m3u8")
            }
        }

        location ^~ /s3/hls {
            resolver 8.8.8.8;
            rewrite ^/s3/hls/(.*) /$1 break;

            set $bucket '';
            set $region '';
            set $s3_host '';

            rewrite_by_lua_block {
                local bucket = ngx.req.get_headers()["bucket"]
                local region = ngx.req.get_headers()["region"]

                if not bucket then
                    ngx.exit(ngx.HTTP_BAD_REQUEST)
                end

                if not region then
                    ngx.exit(ngx.HTTP_BAD_REQUEST)
                end

                local s3_host
                if region == "us-east-1" then
                    s3_host = bucket .. ".s3.amazonaws.com"
                else
                    s3_host = bucket .. ".s3." .. region .. ".amazonaws.com"
                end

                ngx.var.s3_host = s3_host
                require("resty.aws-signature").s3_set_headers(ngx.var.s3_host, ngx.var.uri, region)
            }

            proxy_hide_header       x-amz-id-2;
            proxy_hide_header       x-amz-request-id;
            proxy_hide_header       Set-Cookie;
            proxy_ignore_headers    Set-Cookie;
            proxy_http_version      1.1;
            proxy_set_header        Connection "keep-alive";
            proxy_set_header        Host $s3_host;
            proxy_pass              http://$s3_host;
        }

        location /hls/ {
            if ($uri ~ ^/hls/(.*)/(.*)/(.*)$) {
                set $file $1;
            }

            rewrite ^/hls/(.*)/(.*)/(.*) /hls/$2/$3 break;

            set $drm_server '';

            set_by_lua_block $drm_server{
                return os.getenv("DRM_SERVER")
            }

            vod hls;

            vod_hls_encryption_method aes-128;

            vod_hls_encryption_key_uri "$drm_server/drm/$file";

            vod_drm_enabled on;
            vod_drm_request_uri "/kaltura/$file";
            vod_drm_upstream_location /keystore;

            add_header Cache-Control "public, max-age=3600";
            add_header Access-Control-Allow-Headers "Origin,Range,Accept-Encoding,Referer,Cache-Control,Authorization";
            add_header Access-Control-Expose-Headers "Server,Content-Length,Content-Range,Date";
            add_header Access-Control-Allow-Methods "GET, HEAD, OPTIONS";
            add_header Access-Control-Allow-Origin "*";
        }

        location ~ ^/my_location/([0-9a-f\-]+)/([^/]+)/seg-.+\.ts$ {
            resolver 127.0.0.11;

            if ($request_method = 'OPTIONS') {
                add_header 'Access-Control-Allow-Origin' '*' always;
                add_header 'Access-Control-Allow-Headers' 'Origin,Range,Accept-Encoding,Referer,Cache-Control,Authorization' always;
                add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' always;
                add_header 'Access-Control-Max-Age' 3600;
                add_header 'Content-Type' 'text/plain; charset=UTF-8';
                add_header 'Content-Length' 0;
                return 204;
            }

            set $file $1;

            access_by_lua_block {
                local utils = require "utils"
                local bucket, filename, region = utils.get_file_info(ngx.var.file)

                if not bucket or not filename or not region then
                    ngx.status = ngx.HTTP_BAD_REQUEST
                    return
                end

                local uri = ngx.var.uri
                local seg_part = string.match(uri, "seg%-.+%.ts")

                local new_uri = "/hls/" .. ngx.var.file .. "/" .. filename .. "/" .. seg_part

                ngx.req.set_header("bucket", bucket)
                ngx.req.set_header("region", region)
                ngx.req.set_uri(new_uri)
            }

            proxy_set_header Origin $http_origin;
            proxy_pass http://127.0.0.1/;
        }

        location ^~ /keystore/kaltura {
            internal;
            resolver 8.8.8.8;

            set $file '';
            set $drm_server '';
            set $drm_token '';

            set_by_lua_block $file{
                local regex = "/keystore/kaltura/(?<file>[a-zA-Z0-9-_]+)"

                local m, err = ngx.re.match(ngx.var.uri, regex)
                if err then
                    ngx.log(ngx.ERR, "error: ", err)
                    return
                end

                return m["file"]
            }

            set_by_lua_block $drm_server{
                return os.getenv("DRM_SERVER")
            }
            set_by_lua_block $drm_token{
                return os.getenv("DRM_TOKEN")
            }

            set_by_lua_block $drm_server_without_protocol{
                local regex = "https?://(?<server>[a-zA-Z0-9-_.]+)"

                local m, err = ngx.re.match(ngx.var.drm_server, regex)
                if err then
                    ngx.log(ngx.ERR, "error: ", err)
                    return
                end
                return m["server"]
            }

            set $proxy_url "$drm_server/drm/kaltura/$file";

            proxy_ssl_verify        off;
            proxy_ssl_server_name   on;

            proxy_set_header Origin $http_origin;
            proxy_set_header x-vlab-drm-token $drm_token;
            proxy_pass $proxy_url;
        }

        location /appstatus {
            default_type text/plain;
            return 200;
        }
    }
}

[HenriqueOtsuka]

I'm with the same issue.

The Cached Memory just increase and I have uncountable TIME_WAIT status in my netstat -an

I'm trying to fix one case per time to focuses on each problem.

About the TIME_WAIT, my sysctl.conf also has the configuration below.

Note that I'm running Nginx on Docker Environment and it was applied in host config and in docker run parameters.

net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_time = 120
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 5

For the Cache issue, even when i enable some configs in nginx.conf like openfile_, proxy_buffers and vod_metadata / vod_performance, the usage increase a lot.

I don't know what else is possible here...

[ySp-chld]

TIME_WAIT is no longer an issue in modern linux distro as they will be freed and reuse if ephemeral port number are exhausted.
Also high memory cache usage is not an issue on linux.

However if you still start to see performance issue with nginx you should configure properly proxy pass to reuse open connections with keep_alive : https://www.f5.com/company/blog/nginx/avoiding-top-10-nginx-configuration-mistakes#no-keepalives

You should not need any sysctl configuration to fix this.

keepalive requieres to define an upstream server or it will not be apply

[erankor]

Regarding the cache memory, looking at your configuration, I see you are using "remote" mode against S3 storage, so it doesn't make much sense - nginx doesn't read from the file system, it reads from a remote server, so how can it affect the OS cache?
Btw, speaking about "cache", I don't see any vod_xxx_cache directive here, which means the MP4 metadata is read over-and-over again for each segment request. I would highly recommended using these directives (vod_metadata_cache etc.)