No access to S3

[QuentinMaz]

Hi,

I have recently joined the GFTS project and I can't access S3.
While I don't understand exactly why (other than the configuration profile ''gfts''is not be found), I can provide you with the error raised.

The code to access to S3:

import s3fs

storage_options = {
    "anon": False,
    "profile": "gfts",
    "client_kwargs": {
        "endpoint_url": "https://s3.gra.perf.cloud.ovh.net/",
        "region_name": "gra",
    },
}

s3 = s3fs.S3FileSystem(**storage_options)
filepath = "s3://destine-gfts-data-lake/"

s3.ls(filepath)

Error raised:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/srv/conda/envs/notebook/lib/python3.12/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/conda/envs/notebook/lib/python3.12/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "/srv/conda/envs/notebook/lib/python3.12/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
                ^^^^^^^^^^
  File "/srv/conda/envs/notebook/lib/python3.12/site-packages/s3fs/core.py", line 1006, in _ls
    files = await self._lsbuckets(refresh)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/conda/envs/notebook/lib/python3.12/site-packages/s3fs/core.py", line 973, in _lsbuckets
    files = (await self._call_s3("list_buckets"))["Buckets"]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/conda/envs/notebook/lib/python3.12/site-packages/s3fs/core.py", line 358, in _call_s3
    await self.set_session()
  File "/srv/conda/envs/notebook/lib/python3.12/site-packages/s3fs/core.py", line 544, in set_session
    self._s3 = await s3creator.__aenter__()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/conda/envs/notebook/lib/python3.12/site-packages/aiobotocore/session.py", line 25, in __aenter__
    self._client = await self._coro
                   ^^^^^^^^^^^^^^^^
  File "/srv/conda/envs/notebook/lib/python3.12/site-packages/aiobotocore/session.py", line 149, in _create_client
    verify = self.get_config_variable('ca_bundle')
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/conda/envs/notebook/lib/python3.12/site-packages/botocore/session.py", line 323, in get_config_variable
    return self.get_component('config_store').get_config_variable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/conda/envs/notebook/lib/python3.12/site-packages/botocore/configprovider.py", line 465, in get_config_variable
    return provider.provide()
           ^^^^^^^^^^^^^^^^^^
  File "/srv/conda/envs/notebook/lib/python3.12/site-packages/botocore/configprovider.py", line 671, in provide
    value = provider.provide()
            ^^^^^^^^^^^^^^^^^^
  File "/srv/conda/envs/notebook/lib/python3.12/site-packages/botocore/configprovider.py", line 761, in provide
    scoped_config = self._session.get_scoped_config()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/conda/envs/notebook/lib/python3.12/site-packages/botocore/session.py", line 422, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (gfts) could not be found

[annefou]

1. Could you check if you have a file name $HOME/.aws/credentials ?
2. If this credential file exists, could you try removing "profile": "gfts",?

[QuentinMaz]

Surprisingly, there isn't .aws folder at all...

...Yet, even though I haven't created an issue as indicated in the user guide (here), it seems like my username is included in the current list of authorized GFTS users (linked there).

[QuentinMaz]

@tinaok helped me fix the issue yesterday. We configured the access to the AWS S3 bucket with the following:

1. aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID.
2. aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY.
3. aws configure set aws_endpoint_url $AWS_ENDPOINT_URL_S3.
4. Change the value in brackets [...] of .aws/credentials (line 1) to [gfts]. There might be a way to avoid doing this manually (with vim) with another aws command.

As a last note, I don't know if I missed a step during the onboarding or if this configuration step is currently missing in the onboarding procedure.

Let me know if you need further information or if I can close the issue!

[annefou]

The config profile (gfts) should now be found. do you know have another error?

[QuentinMaz]

No, we checked that the gfts configuration is found and consequently the bucket is successfully accessed.

Should I close the issue?

[annefou]

Yes you can close it. Thanks!

…

On Thu, 21 Nov 2024 at 12:51, Mazouni Quentin ***@***.***> wrote: No, we checked that the gfts configuration is found and consequently the bucket is successfully accessed. Should I close the issue? — Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB6KIPDSIFJLUP2PIER3NPT2BXJO7AVCNFSM6AAAAABSBYZA3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJRGA2TSOJWGE> . You are receiving this because you commented.Message ID: ***@***.***>

[capetienne]

I am new to the project and i do not find.awsfolder

[tinaok]

@annefou @minrk , if we can't create .aws automatically for new users,

• we can add documentation, or
• create script which 'prepare' enviroment and it can be executed in the initialisation process?