gRPC service keepalive configuration not working

[manishkulkarni]

We have a gRPC dotnet core service running on Kubernetes behind Nginx Ingress. The service has bidirectional streaming endpoint.

The ingress is configured with grpc_read_timeout, grpc_send_timeout & client_body_timeout as per documentation to allow the connections to be kept open.

The gRPC service is consumed by a dotnet client app and it uses SocketsHttpHandler so that it can send KeepAlive pings during inactivity period.

From the Nginx debug logs we can see that keepalive pings are being sent from client but Nginx server resets stream when grpc_read_timeout / grpc_send_timeout is reached during period of inactivity. We tried the client with keepalive pings disabled but, the behaviour is still the same, connection remains open until read/send timeout is reached. Therefore keepalive pings seem to be adding no value.

My assumption was that if we keep sending keepalive pings, Nginx server should not reset stream. Not sure if we have some configuration or implementation issue or if we are misinterpreting how keepalive works & this is by design?

Any pointers or help is much appreciated. Thanks.

[brianehlert]

Can you please tell us what version of the NGINX Ingress Controller you are using?

[brianehlert]

first - exec into the NIC container and try to get a response from the backend service pods.
That is just to ensure connectivity from NIC to the backed service pods.
Next is to verify the deployment and nic service definition and ports.
After that is to share the NIC and service configurations and logs.

[chris-brown]

Hey @brianehlert - We managed to get our gRPC services running behind the NIC. However we still see the behaviour of the timeouts not respecting the keep-alive pings. We installed th NIC using the helm chart. Chart nginx-ingress-0.12.0 2.1.0 app version, so latest as of today.

Nginx logs below. This is a filtered snapshot of the logs as there is alot of [notice] log noise.

I0124 23:09:40.026707       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"my-service", Name:"my-service-ingress", UID:"8f438679-bdcc-43fb-9afb-9011017784e3", APIVersion:"networking.k8s.io/v1", ResourceVersion:"377484", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for my-service/my-service-ingress was added or updated 
I0124 23:09:40.026878       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"my-service", Name:"my-service-ingress", UID:"8f438679-bdcc-43fb-9afb-9011017784e3", APIVersion:"networking.k8s.io/v1", ResourceVersion:"377484", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for my-service/my-service-ingress was added or updated 
I0124 23:09:40.027254       1 event.go:285] Event(v1.ObjectReference{Kind:"Secret", Namespace:"ingress-nginx", Name:"nginx-ingress-nginx-ingress-default-server-tls", UID:"1e396129-4610-4b59-8db4-2b1de4ae87d2", APIVersion:"v1", ResourceVersion:"347512", FieldPath:""}): type: 'Normal' reason: 'Updated' the special Secret ingress-nginx/nginx-ingress-nginx-ingress-default-server-tls was updated
I0124 23:09:40.028285       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"my-service", Name:"my-service-ingress", UID:"8f438679-bdcc-43fb-9afb-9011017784e3", APIVersion:"networking.k8s.io/v1", ResourceVersion:"377484", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for my-service/my-service-ingress was added or updated 
I0124 23:09:40.161419       1 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-ingress-nginx-ingress", UID:"8d0f04a7-cf4c-4640-a5aa-c54b8f5137f0", APIVersion:"v1", ResourceVersion:"351618", FieldPath:""}): type: 'Normal' reason: 'Updated' Configuration from ingress-nginx/nginx-ingress-nginx-ingress was updated 
I0124 23:09:40.161445       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"my-service", Name:"my-service-ingress", UID:"8f438679-bdcc-43fb-9afb-9011017784e3", APIVersion:"networking.k8s.io/v1", ResourceVersion:"377484", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for my-service/my-service-ingress was added or updated 
192.168.65.3 - - [24/Jan/2022:23:11:12 +0000] "POST /public.payment.v3.Payment/FromMessage HTTP/2.0" 408 0 "-" "grpc-dotnet/2.41.0 (.NET 6.0.1; CLR 6.0.1; net6.0; windows; x64)" "-"

NIC configuration

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:

worker_processes  auto;
daemon off;

error_log  stderr notice;
pid        /var/lib/nginx/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    map $upstream_trailer_grpc_status $grpc_status {
        default $upstream_trailer_grpc_status;
        '' $sent_http_grpc_status;
    }


    access_log  /dev/stdout  main;




    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout 65s;
    keepalive_requests 100;

    #gzip  on;

    server_names_hash_max_size 1024;
    server_names_hash_bucket_size 256;

    variables_hash_bucket_size 256;
    variables_hash_max_size 1024;

    map $request_uri $request_uri_no_args {
        "~^(?P<path>[^?]*)(\?.*)?$" $path;
    }

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }
    map $http_upgrade $vs_connection_header {
        default upgrade;
        ''      $default_connection_header;
    }








    server {
        # required to support the Websocket protocol in VirtualServer/VirtualServerRoutes
        set $default_connection_header "";
        set $resource_type "";
        set $resource_name "";
        set $resource_namespace "";
        set $service "";

        listen 80 default_server;


        listen 443 ssl default_server http2;



        ssl_certificate /etc/nginx/secrets/default;
        ssl_certificate_key /etc/nginx/secrets/default;






        server_name _;
        server_tokens "on";






        location / {
            return 404;
        }
    }
    # stub_status
    server {
        listen 8080;

        allow 127.0.0.1;
        deny all;

        location /stub_status {
            stub_status;
        }
    }
    server {
        listen unix:/var/lib/nginx/nginx-status.sock;
        access_log off;



        location /stub_status {
            stub_status;
        }
    }

    include /etc/nginx/config-version.conf;
    include /etc/nginx/conf.d/*.conf;

    server {
        listen unix:/var/lib/nginx/nginx-502-server.sock;
        access_log off;



        return 502;
    }

    server {
        listen unix:/var/lib/nginx/nginx-418-server.sock;
        access_log off;



        return 418;
    }
}

stream {
    log_format  stream-main  '$remote_addr [$time_local] '
                      '$protocol $status $bytes_sent $bytes_received '
                      '$session_time "$ssl_preread_server_name"';

    access_log  /dev/stdout  stream-main;





    include /etc/nginx/stream-conf.d/*.conf;
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/avif                                       avif;
    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/wasm                                 wasm;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/config-version.conf:
server {
    listen unix:/var/lib/nginx/nginx-config-version.sock;
        access_log off;



    location /configVersion {
        return 200 1;
    }
}
map $http_x_expected_config_version $config_version_mismatch {
        "1" "";
        default "mismatch";
}
# configuration file /etc/nginx/conf.d/my-service-my-service-ingress.conf:
# configuration for my-service/my-service-ingress

upstream my-service-my-service-ingress--my-service-service-http {
        zone my-service-my-service-ingress--my-service-service-http 256k;
        random two least_conn;

        server 10.1.0.122:5001 max_fails=1 fail_timeout=10s max_conns=0;
        server 10.1.0.123:5001 max_fails=1 fail_timeout=10s max_conns=0;
        server 10.1.0.124:5001 max_fails=1 fail_timeout=10s max_conns=0;

}
upstream my-service-my-service-ingress-example.com-my-service-service-grpc {
        zone my-service-my-service-ingress-example.com-my-service-service-grpc 256k;
        random two least_conn;

        server 10.1.0.122:5002 max_fails=1 fail_timeout=10s max_conns=0;
        server 10.1.0.123:5002 max_fails=1 fail_timeout=10s max_conns=0;
        server 10.1.0.124:5002 max_fails=1 fail_timeout=10s max_conns=0;

}
upstream my-service-my-service-ingress-example.com-my-service-service-http {
        zone my-service-my-service-ingress-example.com-my-service-service-http 256k;
        random two least_conn;

        server 10.1.0.122:5001 max_fails=1 fail_timeout=10s max_conns=0;
        server 10.1.0.123:5001 max_fails=1 fail_timeout=10s max_conns=0;
        server 10.1.0.124:5001 max_fails=1 fail_timeout=10s max_conns=0;

}


server {




        listen 443 ssl http2;


        ssl_certificate /etc/nginx/secrets/my-service-external-ssl;
        ssl_certificate_key /etc/nginx/secrets/my-service-external-ssl;







        server_tokens on;

        server_name example.com;

        set $resource_type "ingress";
        set $resource_name "my-service-ingress";
        set $resource_namespace "my-service";








        location /healthz {
                set $service "my-service-service";




                grpc_connect_timeout 60s;
                grpc_read_timeout 60s;
                grpc_send_timeout 60s;
                grpc_set_header Host $host;
                grpc_set_header X-Real-IP $remote_addr;
                grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                grpc_set_header X-Forwarded-Host $host;
                grpc_set_header X-Forwarded-Port $server_port;
                grpc_set_header X-Forwarded-Proto $scheme;

                grpc_pass grpc://my-service-my-service-ingress-example.com-my-service-service-http;


        }
        location /bank.payment.v1.Payment {
                set $service "my-service-service";




                grpc_connect_timeout 60s;
                grpc_read_timeout 60s;
                grpc_send_timeout 60s;
                grpc_set_header Host $host;
                grpc_set_header X-Real-IP $remote_addr;
                grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                grpc_set_header X-Forwarded-Host $host;
                grpc_set_header X-Forwarded-Port $server_port;
                grpc_set_header X-Forwarded-Proto $scheme;

                grpc_pass grpc://my-service-my-service-ingress-example.com-my-service-service-grpc;


        }
        location /public.payment.v3.Payment {
                set $service "my-service-service";




                grpc_connect_timeout 60s;
                grpc_read_timeout 60s;
                grpc_send_timeout 60s;
                grpc_set_header Host $host;
                grpc_set_header X-Real-IP $remote_addr;
                grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                grpc_set_header X-Forwarded-Host $host;
                grpc_set_header X-Forwarded-Port $server_port;
                grpc_set_header X-Forwarded-Proto $scheme;

                grpc_pass grpc://my-service-my-service-ingress-example.com-my-service-service-grpc;


        }
        location / {
                set $service "my-service-service";




                grpc_connect_timeout 60s;
                grpc_read_timeout 60s;
                grpc_send_timeout 60s;
                grpc_set_header Host $host;
                grpc_set_header X-Real-IP $remote_addr;
                grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                grpc_set_header X-Forwarded-Host $host;
                grpc_set_header X-Forwarded-Port $server_port;
                grpc_set_header X-Forwarded-Proto $scheme;

                grpc_pass grpc://my-service-my-service-ingress--my-service-service-http;


        }

        error_page 400 @grpcerror400;
        error_page 401 @grpcerror401;
        error_page 403 @grpcerror403;
        error_page 404 @grpcerror404;
        error_page 405 @grpcerror405;
        error_page 408 @grpcerror408;
        error_page 414 @grpcerror414;
        error_page 426 @grpcerror426;
        error_page 500 @grpcerror500;
        error_page 501 @grpcerror501;
        error_page 502 @grpcerror502;
        error_page 503 @grpcerror503;
        error_page 504 @grpcerror504;


        location @grpcerror400 { default_type application/grpc; return 400 "\n"; }
        location @grpcerror401 { default_type application/grpc; return 401 "\n"; }
        location @grpcerror403 { default_type application/grpc; return 403 "\n"; }
        location @grpcerror404 { default_type application/grpc; return 404 "\n"; }
        location @grpcerror405 { default_type application/grpc; return 405 "\n"; }
        location @grpcerror408 { default_type application/grpc; return 408 "\n"; }
        location @grpcerror414 { default_type application/grpc; return 414 "\n"; }
        location @grpcerror426 { default_type application/grpc; return 426 "\n"; }
        location @grpcerror500 { default_type application/grpc; return 500 "\n"; }
        location @grpcerror501 { default_type application/grpc; return 501 "\n"; }
        location @grpcerror502 { default_type application/grpc; return 502 "\n"; }
        location @grpcerror503 { default_type application/grpc; return 503 "\n"; }
        location @grpcerror504 { default_type application/grpc; return 504 "\n"; }

}

Please note we never see any keep alive pings in the NIC logs. Can you confirm the expected behaviour with bi-directional streaming and gRPC. Would you expect nginx to respect the keep-alive pings and not timeout after 60s as configured above?

[pleshakov]

Hi @manishkulkarni @chris-brown

NGINX expected behavior is to not support keepalive pings. Please see this https://trac.nginx.org/nginx/ticket/1887 for further explanation.

If you'd like to use TCP keepalives, as mentioned in https://trac.nginx.org/nginx/ticket/1887 , this can be added to the Ingress Controller by adding so_keepalive parameter to the listen directive https://github.com/nginxinc/kubernetes-ingress/blob/v2.1.0/internal/configs/version1/nginx.tmpl#L116 and propagating that modification via the custom templates feature https://github.com/nginxinc/kubernetes-ingress/tree/v2.1.0/examples/custom-templates

[tstraley]

Would grpc_socket_keepalive on as a snippet be easier than making a custom template to modify the listen directive?

I may be mis-interpreting what each of these settings do (grpc_socket_keepalive might just be for upstream while listen so_keepalive is only for downstream?)

[tstraley]

I've also opened #4043 as an idea to add non-snippet configuration option for client_body_timeout directive, which I found to be necessary for gRPC streams through NIC. Maybe some of the references in there would be helpful to others having issues with gRPC services behind this ingress controller.