SNOW-1042878: Axios ERR_FR_TOO_MANY_REDIRECTS (proxy error)

[a-tortevois]

Please answer these questions before submitting your issue.
In order to accurately debug the issue this information is required. Thanks!

1. What version of NodeJS driver are you using?
   snowflake-sdk@1.9.3

2. What operating system and processor architecture are you using?
   W10 AMD64

3. What version of NodeJS are you using?
   (node --version and npm --version)
   18.18.2

4. What are the component versions in the environment (npm list)?

$ npm list
snowflake@1.0.0 C:\dev\tests\snowflake
└── snowflake-sdk@1.9.3

5.Server version:
8.4.1
6. What did you do?

Behind a corporate proxy I get this error when I try to connect to snowflake

{"level":"TRACE","message":"[2:49:47.001 PM]: CALL POST with timeout 90000: https://***.snowflakecomputing.com/session/v1/login-request?requestId=***&warehouse=***&databaseName=***&roleName=***"}
{"level":"DEBUG","message":"[2:49:50.240 PM]: Encountered an error when sending the request. Details: {\"message\":\"Maximum number of redirects exceeded\",\"name\":\"Error [ERR_FR_TOO_MANY_REDIRECTS]\",\"stack\":\"Error [ERR_FR_TOO_MANY_REDIRECTS]: Maximum number of redirects exceeded\\n    at AxiosError.from (C:\\\\dev\\\\tests\\\\snowflake\\\\node_modules\\\\axios\\\\dist\\\\node\\\\axios.cjs:837:14)\\n    at RedirectableRequest.handleRequestError (C:\\\\dev\\\\tests\\\\snowflake\\\\node_modules\\\\axios\\\\dist\\\\node\\\\axios.cjs:3087:25)\\n    at RedirectableRequest.emit (node:e"}

In another context, it helps me get through the corporate proxy : https://antoinevastel.com/nodejs/2022/02/26/nodejs-optimized-https-proxy-no-dependencies.html

7. What did you expect to see?

   Successful connection :)

[sfc-gh-dszmolka]

Hi - generally, the driver works using a proxy, so I'm wondering what is the difference in your case. Let me confirm how it behaves using Windows and a proxy.

[sfc-gh-dszmolka]

just tested using the following simple script

var snowflake = require('snowflake-sdk');
//require('log-timestamp');

console.log(`\nNode.js process version is: ${process.version}\n`);

//snowflake.configure({ logLevel: 'trace' });

var connection = snowflake.createConnection({
    validateDefaultParameters: true,
  // please set up environment variables accordingly
    account: process.env.SFACCOUNT,
    username: process.env.SFUSER,
    password: process.env.SFPASS,
	proxyHost: '172.20.XXX.XXX', // please replace me
	proxyPort: 8080 // me too
});
connection.connect(function(err, conn) {
    if (err) {
        console.error('Unable to connect: ' + err.message);
    } else {
        console.log('Successfully connected as id: ' + connection.getId());
    }
});

var statement = connection.execute({
    sqlText: 'select \'Query run successfully on Snowflake\' as result;',
    complete: function(err, stmt, rows) {
        if (err) {
            console.error('Failed to execute statement due to the following error: ' + err.message);
        } else {
            console.log('[queryID ' + statement.getStatementId() + ', requestId ' + statement.getRequestId() + '] Number of rows produced: ' + rows.length);
        }
    }
});

on a Windows 10 machine:

>node test_conn.js
[2024-02-05T15:14:00.700Z]
Node.js process version is: v18.18.2

..
[2024-02-05T15:14:04.779Z] Successfully connected as id: d35ccb9d-b7c4-47e6-8dc4-7f5b915610e3
[2024-02-05T15:14:06.031Z] [queryID 01b22652-0102-5aeb-0002-477600157092, requestId 28bf84c8-cf54-45da-af99-3cdaf51ebf82] Number of rows produced: 1
>ver

Microsoft Windows [Version 10.0.19044.3086]

looks like Snowflake Node.js driver generally works with proxy on Windows 10.

Can you please provide more details on your issue? For example, run the above test script with tracing enabled, and provide me with the full logs (all the logs generated from start to end of execution) ? Please sanitize the logs if you need, or you can work directly 1 on 1 with a Snowflake Support engineer if you don't want to share the logs here.

Alternatively if you could please provide a reproduction snippet, which when run, leads to the issue - that might be also good but for now I would like to be able to determine if your issue stems from a driver bug, or an environmental issue which is specific for your runtime. Thank you in advance !

[a-tortevois]

var connection = snowflake.createConnection({
    validateDefaultParameters: true,
    account: `${SNOWFLAKE_ORGANIZATION_NAME}-${SNOWFLAKE_ACCOUNT_NAME}`,
    username: SNOWFLAKE_USERNAME,
    role: SNOWFLAKE_ROLE,
    warehouse: SNOWFLAKE_WAREHOUSE,
    database: SNOWFLAKE_DATABASE,
    authenticator: 'SNOWFLAKE_JWT',
    privateKey: SNOWFLAKE_PRIVATE_KEY,
    proxyHost: 'http://proxy.***.com',
    proxyPort: 8080,
});

Node.js process version is: v18.18.2

{"level":"DEBUG","message":"[8:43:27.236 AM]: 300"}
{"level":"DEBUG","message":"[8:43:27.250 AM]: Contacting SF: /session/v1/login-request?requestId=***&warehouse=***&databaseName=***&roleName=***, (1/7)"}
{"level":"TRACE","message":"[8:43:27.253 AM]: Create and add to cache new agent https://***.snowflakecomputing.com:443-keepAlive"}
{"level":"DEBUG","message":"[8:43:27.254 AM]: Proxy settings used in requests: // PROXY environment variables: HTTP_PROXY: http://proxy.***.com:8080 HTTPS_PROXY: http://proxy.***.com:8080 NO_PROXY: localhost,127.0.0.1,localaddress,.localdomain.com. // Proxy configured in Connection: protocol=http proxy=http://proxy.***.com:8080"}
{"level":"TRACE","message":"[8:43:27.255 AM]: CALL POST with timeout 90000: https://***.snowflakecomputing.com/session/v1/login-request?requestId=***&warehouse=***&databaseName=***&roleName=***"}
{"level":"DEBUG","message":"[8:43:27.255 AM]: --createStatementPreExec"}
{"level":"DEBUG","message":"[8:43:27.256 AM]: numBinds = 0"}
{"level":"DEBUG","message":"[8:43:27.256 AM]: threshold = 100000"}
{"level":"DEBUG","message":"[8:43:27.256 AM]: RowStatementPreExec"}
{"level":"DEBUG","message":"[8:43:27.257 AM]: context.bindStage=undefined"}
{"level":"DEBUG","message":"[8:43:40.137 AM]: Encountered an error when sending the request. Details: {\"message\":\"Maximum number of redirects exceeded\",\"name\":\"Error [ERR_FR_TOO_MANY_REDIRECTS]\",\"stack\":\"Error [ERR_FR_TOO_MANY_REDIRECTS]: Maximum number of redirects exceeded\\n    at AxiosError.from (C:\\\\dev\\\\tests\\\\snowflake\\\\node_modules\\\\axios\\\\dist\\\\node\\\\axios.cjs:837:14)\\n    at RedirectableRequest.handleRequestError (C:\\\\dev\\\\tests\\\\snowflake\\\\node_modules\\\\axios\\\\dist\\\\node\\\\axios.cjs:3087:25)\\n    at RedirectableRequest.emit 
(node:e"}

$ systeminfo
OS Name:                   Microsoft Windows 10 Enterprise
OS Version:                10.0.19044

[sfc-gh-dszmolka]

thank you !

the logs (and the repro script :) ) indicate that both

• envvars HTTP_PROXY / HTTPS_PROXY
• and proxyHost / proxyPort

are configured. This is incorrect and may lead to problems, as you already see. As a test, would it be please possible to:

1. remove proxyHost and proxyPort from Connection, and re-test, only using the envvars HTTP_PROXY / HTTPS_PROXY (this is not documented, but works most of the time)
2. add back proxyHost and proxyPort to Connection, remove envvars HTTP_PROXY / HTTPS_PROXY (this is documented, and should always work)

and re-test?

[a-tortevois]

Hello,
I'm sorry but it's not working.
I modified the sdk and used the native agent as describe here: https://antoinevastel.com/nodejs/2022/02/26/nodejs-optimized-https-proxy-no-dependencies.html
... and it's work well

It use something like a tunnel with a first CONNECT http.request: https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/CONNECT

Edit: I also remove proxy on requestOptions on http/base.js (line 233 => proxy: false)

How can you add support for this proxy ?

[sfc-gh-dszmolka]

so i tested the following scenarios, still on Windows 10, now using keypair auth to match your situation.

1. use only proxyHost / proxyPort in the documented way (proxyHost: '10.20.30.40', proxyPort: 8080) - works
2. use only HTTP_PROXY / HTTPS_PROXY as '10.20.30.40:8080' - works
3. use both proxyHost: '10.20.30.40', proxyPort: 8080 and HTTP_PROXY / HTTPS_PROXY as '10.20.30.40:8080' - works
4. use only proxyHost / proxyPort but prepend the http scheme (proxyHost: 'http://10.20.30.40', proxyPort: 8080) - works
5. use only HTTP_PROXY / HTTPS_PROXY as 'http://10.20.30.40:8080' - still works
6. use both proxyHost: 'http://10.20.30.40', proxyPort: 8080 and HTTP_PROXY / HTTPS_PROXY as 'http://10.20.30.40:8080' - and this still works

edit: further tests indicate that as soon as HTTPS_PROXY is set, the contents of proxyHost/proxyPort is ignored/overridden when actually sending the request.

so I'm starting to think maybe as next step it would be worth looking at the proxy service you're using - is this perhaps something which is available as a community version or a trial so I could try it ? I'm currently using mitmproxy to test and as you see, the issue does not reproduce in the above permutations.

Also if you could please share more details on the 'it's not working' part - which of the above scenarios you're attempting to use ? Is it still not working as recommended initially: ditching the environment variables and using proxyHost: 'your.pro.xy' (without http://) and proxyPort ?

I also understand you made it work for your environment by modifying the SDK and it's great news to hear you're unblocked here, but for the sake of the rest of the users I would like to understand and issue better how it is not working in your environment if that's possible.

Anyways, I'll keep trying to reproduce the issue.

[sfc-gh-dszmolka]

a random question, but do you by any chance use ZScaler ?

[a-tortevois]

Yes We use ZScaler :(

[sfc-gh-dszmolka]

Thank you for confirming - asking because we had another customer having very similar problem to yours. The solution in their case was to disable the ZScaler client connector on the host, since the connection was already sent through the proxy once due to the envvars - and using the 'client connector' again somehow directed the traffic to the proxy.

The workaround you found (dropping the whole proxy configuration from requestOptions by setting int to false) maybe did something similar, and enabled the requests to be only proxied once (instead of more than once, somehow getting into a redirect loop)

[a-tortevois]

Thank you for your reply. In my case I can't disable ZScaler :(

[sfc-gh-dszmolka]

I finally managed to reproduce the same issue even without ZScaler, using tinyproxy
Started it up with a minimal config, then confirmed it works in another client which is not axios - logged in with the browser through the proxy, to the Snowflake account, it was successful.

Then run HTTPS_PROXY=http://my.pro.xy:8888 node test.js and observed the too many redirects error as the proxy gets into a redirect loop between http and https requests to the Snowflake login-requests endpoint.

Another scenario was working well, when I did not use the environment variable HTTPS_PROXY, but configured

proxyHost: my.pro.xy, // no http:// is prepended!!
proxyPort: 8888

and this went through the proxy as well, but was successfully connecting and logging in.

We'll take a look how to address this in snowflake-sdk; in the meantime maybe using the Connection proxy options can help. (proxyHost/proxyPort instead of the envvar HTTPS_PROXY)

[a-tortevois]

Good to hear!
It will work with a HttpAgent/HttpsAgent if you add in the axios configuration proxy: false, httpAgent: xxx (or httpsAgent)
Both should be configurable
This agent could be the one I gave you in the link before

[a-tortevois]

Another idea, for proxyHost: my.pro.xy, // no http:// is prepended!!
Could you manage in the lib if the host is given with a protocol or not and also with a port or not ?
The configuration could be proxyHost: 'http://my.pro.xy:8888'
Or proxyHost: 'my.pro.xy', proxyPort: '8888', proxyProtocol: 'http'

[sfc-gh-dszmolka]

that might be indeed help some issues if the axios config could be exposed with a pre-existing default. Thank you for the idea, we'll take a look into it and I'll keep this thread posted.

[sfc-gh-dszmolka]

forgot to update this one - Snowflake node.js driver latest released 1.15.0 introduced support for *_PROXY envvars, and with that, this issue should be fixed. (Had a test case for this specific scenario too, failed with 1.14.0 with the exact error mentioned here, and worked with the new fix branch, which is now released since couple days)