Bearer token refresh after HTTP 401 and/or 30 minutes

[Jaharmi]

What are strategies for implementing refresh of a bearer token? I'm looking for ways that are both quick and pragmatic. If I can do it at the session level, that would be great.

I'm working with an HTTP API that uses basic authentication to request a bearer token. The bearer token remains valid for 30 minutes or invalidated on demand before it expires. The token can be refreshed using a currently-valid token. Once it expires, the API endpoint returns HTTP 401.

I've looked into event hooks and custom authentication. Event hooks didn't look like they would help. Custom authentication appears a bit above my current level of understanding, at least from the docs. I could possibly insert some comparison of wall clock time against the token expiration time, but that seems both cumbersome and would affect a lot of code.

Background

I've got a script using httpx successfully without dealing with the bearer token expiration scenario. It is my first script to use httpx and so I was learning about the library along the way. Under almost all circumstances, it completes all necessary API calls in less than 30 minutes.

I was also trying to learn asyncio and Python 3, coming mostly from running Python 2.x scripts at the desktop level. I ended up switching from async to sync methods because of issues with the API endpoint. I'd eventually like to try async again, as the API endpoint issues may be resolved, but for now, sync calls are enough.

The script sets up a (synchronous, not async) session after obtaining the bearer token. I'm unsure how to add in support for refreshing the bearer token when it has expired (HTTP 401) or is possibly near expiration (almost 30 minutes). I want to make the script more robust and handle the possibility of token expiration. I'd also like to make changes in the fewest places possible, so I'm hoping doing something at the session level can apply across all of the functions that make and process API calls.

[tomchristie]

A custom authentication class is probably what you want here.

I would do something like the example here - store the bearer token as state on the auth class. Any time you get a 401, attempt to refresh the bearer token.

Is that enough info to get you started on this? It'd be fab if you could share how you progress on it - no doubt it'll end up being useful to someone else further down the line.

[Jaharmi]

My head is still spinning trying to understand how to fit this together and adapt it with the code I have.

I have a function that creates a session, client. (Everything I'm doing is in functions.) I pass this function the user and password, which then is added to the session as client.auth. This is used in a request, and from the JSON response to that, I get the bearer token. I then pass client in to other functions so they use the session.

If I understand the examples correctly, I would need to use the requires_response_body example. With that, I can get the JSON response containing the bearer token along with the expiration date/time. However, this example seems to be the most complex so I'm still trying to figure it out.

Since I'd be overriding Auth with the class, would I use the class to get the bearer token in all situations — including the initial connection?

Instead of passing the username and password two-tuple into client.auth, I would need to pass MyCustomAuth to it?

Hopefully I'm even bringing up sensible questions. For reference, here's the authentication information. Thanks!

[pssolanki111]

not sure how far along you are with the implementation, but I'd personally suggest authlib + httpx for oauth applications.

see the package tda-api for an implementation. authlib handles the access_token refresh internally.
@tomchristie @Jaharmi

[Jaharmi]

@pssolanki111 Thanks, I'll take a look!

The API isn't officially Oauth. I've read about Oauth and the flow does seem similar. I don't know if it's close enough for Oauth to work.

[lordmauve]

I also have this problem. I solved it by writing a time-based cache of tokens implemented within the slightly weird generator-based async framework of httpx.Auth.auth_flow.

But the API for that is not really up to the job; there are two problems:

1. I don't have a concurrency primitive that lets me avoid cache stampede. Currently I'm allowing cache stampede, just extending token expiry times while speculatively revalidating so that cache stampede only happens if a token has definitely expired.

2. I need to set requires_response_body to True but this modifies all requests, even the request that I'm trying to add authentication to. If this is a huge download it is problematic to do this. So auth flow needs to say if it wants the response body for each request. In the original PR for requires_response_body there's an observation

   I can't figure out a way to ensure these tests all pass with requires_response_body=True. It's difficult because the response needs to be read before passing it back to auth_flow, but we need to pass it back in order to get the StopIteration to determine when to return.

   ...which is exactly this issue. Unfortunately it looks like the thought got sidestepped:

   The message I'm getting here is that it does not, in fact, matter if an auth class has already called read() on the response or not.

At least I can reimplement the Auth class with a complete new sync_auth_flow and async_auth_flow, but this doesn't make the existing Auth class more correct.