Automagic socket sharing for net/tls connections via proxy server

[natevw]

This integrates a client for https://github.com/tessel/proxy into the runtime net/tls modules. In this way, some of the built-in CC3000 WiFi module's limitations and quirks may be avoided. Socket number limitation is raised, and perhaps even IPv6 endpoints via proxy [untested].

Overview

The proxy is enabled and optionally configured via the following process.env keys. These can be set manually, before any require statements that may cause the net module to be evaluated, or via the tessel CLI interface once tessel/t1-cli#167 is released.

In short, you will need to at least provide PROXY_TOKEN/TM_API_KEY to enable proxied net connections, and PROXY_TRUSTED=1 once you understand its implications.

Sample usage

Once the proxy settings are configured, they are applied automatically to all relevant outgoing network connections:

process.env.TM_API_KEY = "FIXME";    // get via https://oauth.tessel.io
require('http').get("http://ipcalf.com/?format=text", function (res) {
  res.on('data', function (d) {
    console.log("proxy ip is:", d.toString())
  });
});

Configurable environment variables

• PROXY_HOST (default: "proxy.tessel.io") — hostname or IP address of the proxy server you wish to use.
• PROXY_PORT (default: 443) — the proxy server will be contacted on this port. Even though the client-proxy connection is not HTTPS, we use port 443 by default to help circumvent overly strict firewalls 🙈
• PROXY_CERT — If you are tunneling through your own proxy server and it uses a self-signed certificate, you will need to provide its public key in PEM format. The default value is empty, in which case the default certificate trust chain is used.
• PROXY_TRUSTED (default: 0) — By default, only net sockets will be proxied, and tls will use their own CC3K sockets directly. Set this to 1 to trust the proxy server with secure socket data; this disables end-to-end encryption! See important TLS information below for details.
• PROXY_IDLE (default: 90e3) — The tunnel to the proxy will be closed if no new sockets are opened within this timeout. Set to 0 to leave the tunnel connected only while actively used, or set to Infinity to keep the tunnel open indefinitely once it is first used.
• PROXY_LOCAL — space separated list of hosts for which the net/tls libraries should still use direct (native CC3K) connections. Your list may include individual IP addresses (192.168.1.1), IP address ranges in CIDR notation (192.168.0.255/4), exact domain matches (www.example.com) or a domain including subdomains (.example.net). Default setting includes common LAN ranges and localhost` [xref: https://github.com/CC3K can't connect to localhost/127.0.0.1 [or even own network IP address?!] t1-firmware#60].
• PROXY_TOKEN or TM_API_KEY (required) — to enable an proxy usage, you must provide a valid token used to authenticate with the proxy server. For the hosted server, this should be the API key displayed in your profile at https://oauth.tessel.io. For a self-hosted proxy, this will likely be your AUTH_HARDCODED password or whatever custom authentication you have configured. (PROXY_TOKEN takes precedence over the more general TM_API_KEY.)

Per-socket options

When using the options form of the net.connect or tls.connect helpers, there is now an additional setting:

• {proxy:false} — you can force a net or tls connection to bypass the proxy (i.e. use CC3K directly) via this extension to the normal socket options.

The HTTP/HTTPS modules tend to forward their options objects into the respective network modules, so you may also be able to control proxying at that level as well. This has not been thoroughly reviewed, and connection-reuse via the default global Agent may interfere in unintended ways.

Important TLS information

Although the client-proxy "tunnel" connection is always done over a tls socket, it was impractical to connect the Tessel runtime's built-in TLS library to a proxied (virtual) socket. Therefore, there is no end-to-end encryption for proxied sockets: the proxy server has access to all data sent/received.

By default, if you create an outgoing connection via the tls (or https) module, the proxy tunnel will not be used — it will use a native CC3K socket to make a direct connection. If you trust the proxy server and would like it to handle tls connections on your behalf, you will need to set PROXY_TRUSTED=1.

Implementation details

There is a new quasi-"abstract base class" Socket which now underlies the original TCPSocket and the new ProxiedSocket classes. Should you create a socket via socket = new net.Socket(), all expected method calls are buffered until you call socket.connect(). At this point, the target host is examined and used (along with all other relevant settings) to determine which type of socket should be used. You socket instance is converted in-place to the appropriate "concrete" subclass and completes its initialization via Don't Try This At Home™ means.

Now, if the connection is via a TCPSocket it pretty much [mis-]behaves as it did before. If it is a ProxiedSocket, however, then a whole different mechanism rolls into action. First, we check to see if there is already a tunnel open to the proxy server. If so, our job is relatively easy: just create a new "substream" (passing a tls/net flag to the server based on the local socket type) and forward the provided host/port information (and a few other methods/events) over to the proxy server.

When no tunnel is active, one is set up by connecting to the proxy server, sending the authorization token, and waiting for its signal to proceed. Most of this is managed by way of the streamplex module, the proxy server and client just add back some of the socket-specific handling. Once all sockets using a tunnel are closed, the tunnel itself is shut down.

Any error on the underlying tunnel will cause it to be closed. The event will propagate to all subsockets, however, so a new tunnel can be re-established when a new proxied socket is opened.

An oracle concerning the CC3000

While tunneling all end-user sockets through a single proxy connection can avoid some pitfalls, any issue that would plague a single network socket will now plague all of your (proxied) sockets together. Beware. Beware!

Like such as in the this issue: tessel/t1-firmware#128

[natevw]

Original TODO:

• explain/document this patch
• convert user-configurable stuff to use process.env (xref Adds Global and Local Configuration Variables t1-cli#167)
• implement tls options needed to support secure tunnel (separate patch, xref TLS: cannot provide custom ca list on connection #701)
• test on device

[natevw]

The floor is now open.

[tcr]

Comment the first: the entire PR message is a thing of beauty. 🙈

[tcr]

A comment here to explain what this cert is

[natevw]

Added in f49d968

[tcr]

This PR is of good quality and most of my complaints are minor. I don't want to land this without at least some additional proxy-specific tests, which we should talk about over Zulip; otherwise this generally improves code quality and doesn't have any standout issues.

[jiahuang]

just curious, why object.create instead of new?

[natevw]

streamProto becomes the prototype for an already-existing socket object (see @tcr's "I hate you" comment above ;-) and this bypasses calling the actual constructors again, only preparing the [think of it like a] "mix-in" we now need.

If it weren't for needing a link between socket instances and their associated tunnel (next two lines) the logic that uses this would just ProxiedSocket.prototype directly instead.

[jiahuang]

lgtm, will do more testing of this over the weekend.