Allow aws-vault to safely be run in parallel

[timvisher-dd]

I run aws-vault as a credentials_provider for SSO backed AWS profiles. This is often in parallel contexts of dozens to hundreds of invocations at once.

In that context when credentials needed to be refreshed, aws-vault would open an unconstrained amount of browser tabs in parallel, usually triggering HTTP 500 responses on the AWS side and failing to acquire creds.

To mitigate this I developed a small wrapper around aws-vault that would use a Bash dir lock (Wooledge BashFAQ 45) when there was a possibility that the credentials would need to be refreshed. This worked but it was also quite slow as it would lock the entire aws-vault execution rather than just the step that might launch a browser tab. The dir locking strategy was also sensitive to being killed by process managers like terraform and so had to do things like die voluntarily after magic amounts of seconds to avoid being SIGKILLed and leaving a stale lock around.

This changeset introduces that locking mechanism directly into aws-vault in a cross-platform manner using a library that is tolerant of all forms of process death (even SIGKILL). I've been testing it extensively and not only does it out perform my bash wrapper by at least an order of magnitude or two but it so far has not had any stale locking problems.

Because of how much more performant the locking mechanism is I also found that OIDC cred retrieval can 429 so this changeset also handles the Retry-After headers with jitter. In a stress test across 646 AWS profiles in 9 SSO Directories I'm able to retrieve creds successfully in 36 seconds on my box. This fails irrevocably on HEAD because the browser storm overwhelms the IAM/SSO APIs.

Testing

Unit tests have been added for this. I also used some local integration test scripts that clear out the creds and run aws-vault export --format=json across different sets of my profiles and assert that it succeeds. Finally I've converted my local tooling to use this fork of aws-vault and have been exercising it there without issue.

Colofon

I did not write any of this code. Codex did. That said I have read through it in some detail and it looks reasonable to me.

Co-authored-by: Codex noreply@openai.com

[mbevc1]

No need to open a PR here, since it has been abandoned. Thanks for opening ByteNess#291

[timvisher-dd]

No need to open a PR here, since it has been abandoned. Thanks for opening ByteNess#291

Yep! Didn't notice that somehow until after I created this PR. It'll close once the other PR merges. :)