Access to sockets forbidden by its access policy

[tihomir-kit]

Hi!

First - thanks for the library, really nice work so far. The library made my life easier. :)

Well, I stumbled upon a problem. We have a .net webapp hosted on Azure (currently Basic B1 tier) and the InfluxDB is running in an Azure Linux VM instance. Recently we started noticing the following error:

» 2 Sep 2015 09:44:01.733 Wed Sep 02 08:44:02.3041 +00:00 2015 <Namespace>.Business.Services.ReadingService : Error, System.Net.Http.HttpRequestException: An error occurred while sending the request. 
---> System.Net.WebException: Unable to connect to the remote server 
---> System.Net.Sockets.SocketException: An attempt was made to access a socket in a way forbidden by its access permissions at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress) at System.Net.Sockets.Socket.InternalBind(EndPoint localEP) at System.Net.Sockets.Socket.BeginConnectEx(EndPoint remoteEP, Boolean flowContext, AsyncCallback callback, Object state) at System.Net.Sockets.Socket.UnsafeBeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state) at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception) 
--- End of inner exception stack trace 
--- at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar) 
--- End of inner exception stack trace 
--- at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task) at Microsoft.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Microsoft.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at InfluxDB.Net.InfluxDb.<QueryAsync>d__f.MoveNext() 
--- End of stack trace from previous location where exception was thrown 
--- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at <Namespace>.Business.Services.InfluxDbService.<QueryReading>d__20.MoveNext() in c:\BuildAgent\work\1b725b0f890f9b97\<Namespace>.Business\Services\InfluxDbService.cs:line 227 
--- End of stack trace from previous location where exception was thrown 
--- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at <Namespace>.Business.Services.InfluxDbService.<GetLastReadings>d__f.MoveNext() in c:\BuildAgent\work\1b725b0f890f9b97\<Namespace>.Business\Services\InfluxDbService.cs:line 129 
--- End of stack trace from previous location where exception was thrown 
--- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at <Namespace>.Business.Services.InfluxDbService.<GetLastReading>d__a.MoveNext() in c:\BuildAgent\work\1b725b0f890f9b97\<Namespace>.Business\Services\InfluxDbService.cs:line 100 
--- End of stack trace from previous location where exception was thrown 
--- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at <Namespace>.Business.Services.ReadingService.<ResolveSensorReading>d__13.MoveNext() in c:\BuildAgent\work\1b725b0f890f9b97\<Namespace>.Business\Services\ReadingService.cs:line 95 Context

Which boils down to the following message: "An attempt was made to access a socket in a way forbidden by its access permissions at System.Net.Sockets.Socket.DoBind"

I googled it up and at first it seemed that the connections are being refused by the InfluxDB VM, but after I tried to reproduce the thing from my dev machine, I couldn't. I made a lot more requests from my dev machine than from what I needed to do with the production, and I didn't see the same symptoms. That led me to believe that the problem was not in the InfluxDB instance but in some kind of Azure website limitation.

I used these two netstat commands on InfluxDB to monitor connections on the InfluxDB VM..

This one gives connection counts per IP:

watch "netstat -plan|grep :80|awk {'print \$5'} | cut -d: -f 1 | sort | uniq -c | sort -nk 1"

So, something like:

30 one_ip
3138 another_ip

And this one gives counts by tcp connection status:

netstat -tn | awk 'NR>2 {print $6}' | sort | uniq -c | sort -rn

And something like:

285 TIME_WAIT
1802 FIN_WAIT2
82 ESTABLISHED
3 FIN_WAIT1
2 SYN_RECV

In production, after I got at about ~1500 FIN_WAIT2 (which basically means that the server is waiting for the connection to be closed (and the client acknowledged that it know it can close the connection), I would start seeing the access to socket being forbidden message. In essence - lingering connections remain, and it takes them up to 3 minutes to clean themselves up.

On my dev machine FIN_WAIT2 went all the way up to ~10000 and everything still worked fine and I didn't want to wait any more so I stopped the test.

Then I decided to take a look into your code and noticed that you're not disposing the HttpClient after you get the response from the InfluxDB server (at RequestInnerAsync()). I checked on SO whether HttpClient should be disposed after usage and it seems that unless it's being reused for later. In the case of InfluxDB.Net, I guess it should be disposed then. I tried disposing the HttpClient, the request and the response objects and it didn't help. Then I decided to try and set the requestTimeout from RequestAsync (I set it to 5 seconds), but that didn't change anything either (FIN_WAIT2 didn't go away earlier because of that). The last thing I tried was to make a singleton wrapper around HttpClient like this:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Text;

namespace InfluxDB.Net
{
    public class HttpClientSingletonWrapper : HttpClient
    {
        private static readonly Lazy<HttpClientSingletonWrapper> Lazy = new Lazy<HttpClientSingletonWrapper>(() => new HttpClientSingletonWrapper());

        public static HttpClientSingletonWrapper Instance { get { return Lazy.Value; } }

        private HttpClientSingletonWrapper()
        {

        }
    }
}

But this didn't help either. So now I'm kind of out of options and was hoping you might have an idea or suggestion on how to solve this? If you need any additional info, I'll gladly provide it. I would like to get to the bottom of this because someone else might have the same problem in the future, so it would be good if we were able to fix this. :)

Additional links:

• SocketException preventing use of C# TCPListener in Windows Service
• How to close (kill, release?) a socket, which is in FIN_WAIT_2 state?
• Best practice for using HttpClient within a REST service?

Thanks!