Skip to content

Commit

Permalink
Filter logging to hide massive stacktrace for access issue
Browse files Browse the repository at this point in the history
With this change, we will return the ClientError without any extra error
or logs in the terminal. As a result, if we capture the ClientError in
the script, we will have minimum output.

Example query:
```python
from datachain.error import ClientError
from datachain.lib.dc import C, DataChain

try:
    wds = (
        DataChain.from_storage("az://amrit-datachain-test/shards")
        .filter(C.name.glob("00000000.tar"))
        .settings(parallel=8, cache=True)
    )

    wds.show()
except ClientError as e:
    print(str(e))

print("Exception caught")
```

Closes #600
  • Loading branch information
amritghimire committed Jan 24, 2025
1 parent 2674392 commit 38e2c70
Showing 1 changed file with 5 additions and 0 deletions.
5 changes: 5 additions & 0 deletions src/datachain/lib/listing.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import logging
import posixpath
from collections.abc import Iterator
from typing import TYPE_CHECKING, Callable, Optional, TypeVar
Expand All @@ -23,6 +24,10 @@

D = TypeVar("D", bound="DataChain")

# Disable warnings for remote errors in clients
logging.getLogger("aiobotocore.credentials").setLevel(logging.CRITICAL)
logging.getLogger("gcsfs").setLevel(logging.CRITICAL)


def list_bucket(uri: str, cache, client_config=None) -> Callable:
"""
Expand Down

0 comments on commit 38e2c70

Please sign in to comment.