NewRelic Agent 10.19.0 breaks gRPC calls

[jeffpapp]

Description

With the NewRelic Agent 10.19.0, we started seeing our gRPC calls fail with a Failed to deserialize response message error.

After doing some research it seems like this can happen if something is intercepting the request/response. I found a similar issue at dotnet/aspnetcore#39317 which made me think maybe the NewRelic agent might be causing an issue since it was just updated.

To check I rebuilt my docker images and specified version 10.18.0 directly and redeployed and tested the same exact gRPC call and it worked correctly.

Your Environment

I am running via docker container on AKS.

For Maintainers Only or Hero Triaging this bug
Suggested Priority (P1):
Suggested T-Shirt size (S, M, L, XL, Unknown):

[workato-integration]

https://new-relic.atlassian.net/browse/NR-178762

[tippmar-nr]

@jeffpapp thanks for submitting this issue. We just released v10.19.1 with a fix that we think will help.

If you can, please give the new release a try and confirm back here whether it did or didn’t resolve your issue. Thanks!

[jeffpapp]

Same error. Here are the gRPC debug level logs from the client side of the call. If there are other logs that would be helpful let me know and I can try to provide them

dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[4]
      Channel picker updated.
dbug: Grpc.Net.Client.Balancer.Subchannel[11]
      Subchannel id '3-1' state changed to Connecting. Detail: 'Connection requested.'.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[3]
      Channel state updated to Connecting.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[8]
      Picked queued.
dbug: Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport[2]
      Subchannel id '3-1' connected to socket xxx:5000.
dbug: Grpc.Net.Client.Balancer.Subchannel[11]
      Subchannel id '3-1' state changed to Ready. Detail: 'Successfully connected to socket.'.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[3]
      Channel state updated to Ready.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[4]
      Channel picker updated.
dbug: Grpc.Net.Client.Balancer.Internal.ConnectionManager[6]
      Successfully picked subchannel id '3-1' with address xxx:5000. Transport status: InitialSocket
info: System.Net.Http.HttpClient.XXXServiceClient.LogicalHandler[100]
      Start processing HTTP request POST http://cxxx:5000/xxx/xxx
info: System.Net.Http.HttpClient.XXXServiceClient.ClientHandler[100]
      Sending HTTP request POST http://xxx:5000/xxx/xxx
dbug: Grpc.Net.Client.Internal.GrpcCall[18]
      Sending message.
info: System.Net.Http.HttpClient.XXXServiceClient.ClientHandler[101]
      Received HTTP response headers after 1673.5675ms - 200
info: System.Net.Http.HttpClient.XXXServiceClient.LogicalHandler[101]
      End processing HTTP request after 1692.3809ms - 200
dbug: Grpc.Net.Client.Internal.GrpcCall[13]
      Reading message.
dbug: Grpc.Net.Client.Internal.GrpcCall[9]
      Message not returned from unary or client streaming call.
info: Grpc.Net.Client.Internal.GrpcCall[3]
      Call failed with gRPC error status. Status code: 'Internal', Message: 'Failed to deserialize response message.'.
dbug: Grpc.Net.Client.Internal.GrpcCall[4]
      Finished gRPC call.

[tippmar-nr]

Thanks for the feedback, @jeffpapp. We've been able to reproduce the issue using a gRPC server and are continuing to investigate.

[floge07]

I cannot describe in words of what a nightmare it has been for us to trace this problem back to the newrelic agent...
Always use static version constraints on your packages folks.