AWS temporary credentials from the ASF endpoint do not work for NISAR granules

[ajelenak]

This code:

import earthaccess as ea  # version 0.15.1
auth = ea.login(strategy="netrc", persist=False)
creds = auth.get_s3_credentials(daac="ASF")

seems to generate credentials that produce 403 HTTP errors for the NISAR granules currently available. The credentials I manually get from https://nisar.asf.earthdatacloud.nasa.gov/s3credentials work.

[chuckwondo]

Hi @ajelenak, thanks for raising this.

When specifying daac="ASF", earthaccess ends up using the endpoint https://sentinel1.asf.alaska.edu/s3credentials, which explains why you had to explicitly specify the correct endpoint.

Unfortunately, our daac/provider -> endpoint mapping is hardcoded, so this type of problem is a known issue, and we have had some discussions around how to properly address this.

Some initial experimentation was started around a "crawler" that we might possibly run on a periodic basis to embed such a mapping, or somehow make it easy to get an updated mapping when something changes, such as in this case of NISAR, which introduced a new endpoint.

[ajelenak]

I reported this to ASF as well. Maybe they will be motivated to settle on a single endpoint for all their collections.

[jhkennedy]

Ah @ajelenak I just replied to your email. If you specify the endpoint directly, it will work:

import earthaccess

auth = earthaccess.login()
creds = auth.get_s3_credentials(endpoint='https://nisar.asf.earthdatacloud.nasa.gov/s3credentials')

Maybe they will be motivated to settle on a single endpoint for all their collections.

Unfortunately, the Sentinel-1 distribution endpoint was set up before even Earthdata Cloud was a thing -- we were the first DAAC to distribute data from AWS. Right now, ASF has these cloud distribution endpoints:

• nisar.asf.earthdatacloud.nasa.gov has NISAR data
• sentinel1.asf.alaska.edu has Sentinel-1 data
• cumulus.asf.earthdatacloud.nasa.gov has everything else from ASF in the cloud

The Sentinel-1 endpoint will likely go away at some point in favor of the general cumulus one, but the NISAR one is here for the long term.

And really, the DAAC -> single cloud endpoint isn't universal across multiple DAACs (#1135 (comment)), and with ESDIS's intention to shift from DAACs to centralized enterprise solutions and Science Enabling Teams, this will break down further over time.

I think we should probably eliminate daac.py entirely and instead focus on returning the access keys and metadata for a particular collection/dataset or granule(s) based on what's reported in CMR.

That said, I like having the helper method above that lets you specify an endpoint, but I don't think it should include DAAC or provider options.

[chuckwondo]

And really, the DAAC -> single cloud endpoint isn't universal across multiple DAACs (#1135 (comment)), and with ESDIS's intention to shift from DAACs to centralized enterprise solutions and Science Enabling Teams, this will break down further over time.

I think we should probably eliminate daac.py entirely and instead focus on returning the access keys and metadata for a particular collection/dataset or granule(s) based on what's reported in CMR.

That said, I like having the helper method above that lets you specify an endpoint, but I don't think it should include DAAC or provider options.

I have to look again, but I think the "crawler" was intended to create a mapping from bucket to s3 endpoint in order to support a DAAC having multiple endpoints. With such a mapping, we wouldn't care about DAAC/provider. We would simply be able to yield the appropriate s3 credentials endpoint for any s3 URL pointing to an object within the Earthdata holdings.

[ajelenak]

With such a mapping, we wouldn't care about DAAC/provider. We would simply be able to yield the appropriate s3 credentials endpoint for any s3 URL pointing to an object within the Earthdata holdings.

Yes, this is probably the most enduring approach.

@jhkennedy I appreciate you answering my issue not once, but twice! 😁

I am closing this issue since a specific S3 credentials endpoint can be supplied to earthaccess already.