Unauthorised after redirect (Authorization header is lost).

[italo-turing]

Hi. I'm trying to move my program that uses requests to httpx, but I'm having difficulties with redirects. I'm using the Quora Ads API, and the way its pagination works is that it gives you the URL to the next bit in the response. These URLs come as HTTP, but Quora doesn't accept HTTP; the requests are redirected to HTTPS. If you force the HTTP request, you get a 301. However, it seems that httpx is not sending the Authorization header with the redirected requests, which gives me an Unauthorized error.

This is my function:

async def get_json_response(
    session: AsyncClient, url: str, params: Dict[str, str]
) -> dict:
    try:
        print()
        opt = dict(url=url, params=params, headers=get_headers())
        request = session.build_request("GET", **opt)
        print("before url:", request.url)
        print("before headers:", request.headers)
        # response = await session.get(**opt)
        response = await session.send(request)
        response.raise_for_status()
        res = response.json()
        if res is None:
            raise ValueError(response)
        return res
    except (ValueError, HTTPError) as e:
        # logging.error(e)
        print()
        print('error:', e)
        print("after url:", e.request.url)
        print("after headers:", e.request.headers)

This is the output.

before url: http://api.quora.com/ads/v0/accounts/XXXXXXX?fields=accountId%2CaccountName%2CcampaignId%2CcampaignName&level=CAMPAIGN&sort=campaignId&offset=25   

before headers: Headers({'host': 'api.quora.com', 'accept': '*/*', 'accept-encoding': 'gzip, deflate, br', 'connection': 'keep-alive', 'user-agent': 'python-httpx/0.19.0', 'authorization': '[secure]'})                                                                                                                                                                           

error: 401 Client Error: Unauthorized for url: https://api.quora.com/ads/v0/accounts/XXXXXXX?fields=accountId%2CaccountName%2CcampaignId%2CcampaignName&level=CAMPAIGN&sort=campaignId&offset=25                                                                               For more information check: https://httpstatuses.com/401                                     

after url: https://api.quora.com/ads/v0/accounts/XXXXXXX?fields=accountId%2CaccountName%2CcampaignId%2CcampaignName&level=CAMPAIGN&sort=campaignId&offset=25                      

after headers: Headers({'host': 'api.quora.com', 'accept': '*/*', 'accept-encoding': 'gzip, deflate, br', 'connection': 'keep-alive', 'user-agent': 'python-httpx/0.19.0'})   

This is httpx 0.19.0 on WSL2:

❯ uname -a                                                                                   
Linux malacandra 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux 

I don't know what request's behaviour is. But this works with the default options there.

[tomchristie]

Ah interesting.

Here's how requests deals with removing Authorization headers on redirects...

https://github.com/psf/requests/blob/1e5fad7433772b648fcbc921e2a79de5c4c6be8b/requests/sessions.py#L119-L142

Here's how httpx deals with removing Authorization headers on redirects...

httpx/httpx/_client.py

Lines 524 to 530 in 0b4a832

	if not same_origin(url, request.url):
	# Strip Authorization headers when responses are redirected away from
	# the origin.
	headers.pop("Authorization", None)
	# Update the Host header.
	headers["Host"] = url.netloc.decode("ascii")

The difference is that requests has a special casing to allow preserving the Authorization header on http->https redirects, which "isn't specified by RFC 7235, but is kept to avoid breaking backwards compatibility with older versions of requests".

I'm not 100% sure what the best resolution is here, but I think we might well want to allow an exclusion for the http->https case, because it kinda makes sense, regardless of specs.

(Ideally a pagination API like the one you've described ought to return links with the correct scheme & host in the first place, but the real world is messy, eh...)

[tomchristie]

Promoted this to #1850

[italo-turing]

In the meantime, what would be the solution? Disable automatic redirects and do them manually, passing the header again?

[tomchristie]

For your specific case, I'd probably directly fix up the URLs that you get back from the pagination API, using...

url = url.copy_with(scheme="https")