Unexpected error occurred when providing invalid credentials #24
Description
Bug Report
Description
When using the webdavs remote with DVC, if you don't provide both a username and password for your WebDAV server, DVC will will fail with an internal error message that may not be clear to users.
Reproduce
- Set up an SFTPGo server that supports WebDAV. This server will listen on a specific port number (10080) on your local machine.
#!/usr/bin/env bash
CONTAINER_NAME="webdav"
SECRETS_DIR="/usr/local/src/secrets"
graceful_shutdown() {
docker stop $CONTAINER_NAME && docker rm $CONTAINER_NAME
exit
}
trap graceful_shutdown INT TERM
docker create --name $CONTAINER_NAME \
-a STDOUT \
-p 8080:8080 \
-p 2022:2022 \
-p 10080:10080 \
-e SFTPGO_WEBDAVD__BINDINGS__0__PORT=10080 \
-t "drakkan/sftpgo:v2.5.5" || exit
docker cp -q ./secrets/ $CONTAINER_NAME:$SECRETS_DIR || exit
docker start --attach $CONTAINER_NAME || exit- Initialize DVC.
dvc init --no-scm- Track some changes
touch example.xml && dvc add example.xml- Add the remote
dvc remote add test webdavs://localhost:10080- Specify a username for the remote
dvc remote modify test user myuser- Try to push changes to the remote.
$ dvc push --remote test
ERROR: unexpected error - sequence item 1: expected a bytes-like object, NoneType found
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!Verbose output
$ dvc push
ERROR: unexpected error - sequence item 1: expected a bytes-like object, NoneType found
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
(.venv) ➜ .local git:(3.30.1) dvc push --verbose
2023-11-22 12:21:44,644 DEBUG: v3.30.1, CPython 3.11.2 on <REDACTED>
2023-11-22 12:21:44,644 DEBUG: command: push --verbose
2023-11-22 12:21:44,862 DEBUG: Preparing to transfer data from '/home/sid/workspace/dvc/.local/.dvc/cache/files/md5' to 'https://localhost:10080/files/md5'
2023-11-22 12:21:44,863 DEBUG: Preparing to collect status from 'files/md5'
2023-11-22 12:21:44,863 DEBUG: Collecting status from 'files/md5'
2023-11-22 12:21:44,864 DEBUG: Querying 1 oids via object_exists
2023-11-22 12:21:44,951 ERROR: unexpected error - sequence item 1: expected a bytes-like object, NoneType found
Traceback (most recent call last):
File "/home/sid/workspace/dvc/dvc/repo/push.py", line 117, in push
push_transferred, push_failed = ipush(
^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_data/index/push.py", line 68, in push
result = transfer(
^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_data/hashfile/transfer.py", line 204, in transfer
status = compare_status(
^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_data/hashfile/status.py", line 176, in compare_status
dest_exists, dest_missing = status(
^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_data/hashfile/status.py", line 148, in status
exists.update(odb.oids_exist(hashes, jobs=jobs, progress=pbar.callback))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_objects/db.py", line 411, in oids_exist
return list(wrap_iter(remote_oids, callback))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_objects/db.py", line 36, in wrap_iter
for index, item in enumerate(iterable, start=1):
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_objects/db.py", line 359, in list_oids_exists
in_remote = self.fs.exists(paths, batch_size=jobs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_objects/fs/base.py", line 358, in exists
if self.fs.async_impl:
^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py", line 47, in __get__
return prop.__get__(instance, type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py", line 25, in __get__
res = instance.__dict__[self.fget.__name__] = self.fget(instance)
^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_webdav/__init__.py", line 86, in fs
return WebdavFileSystem(**self.fs_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/fsspec/spec.py", line 79, in __call__
obj = super().__call__(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/webdav4/fsspec.py", line 103, in __init__
self.client = client or Client(base_url, auth=auth, **client_opts)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/webdav4/client.py", line 265, in __init__
self.http: HTTPClient = http_client or HTTPClient(**client_opts)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_client.py", line 646, in __init__
super().__init__(
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_client.py", line 180, in __init__
self._auth = self._build_auth(auth)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_client.py", line 433, in _build_auth
return BasicAuth(username=auth[0], password=auth[1])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_auth.py", line 130, in __init__
self._auth_header = self._build_auth_header(username, password)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_auth.py", line 139, in _build_auth_header
userpass = b":".join((to_bytes(username), to_bytes(password)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: sequence item 1: expected a bytes-like object, NoneType found
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/sid/workspace/dvc/dvc/cli/__init__.py", line 211, in main
ret = cmd.do_run()
^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/cli/command.py", line 27, in do_run
return self.run()
^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/commands/data_sync.py", line 64, in run
processed_files_count = self.repo.push(
^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/repo/__init__.py", line 60, in wrapper
return f(repo, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/repo/push.py", line 125, in push
_update_meta(
File "/home/sid/workspace/dvc/dvc/repo/push.py", line 18, in _update_meta
new = build_data_index(
^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/dvc/repo/index.py", line 783, in build_data_index
if not fs.exists(out_path):
^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_objects/fs/base.py", line 355, in exists
return self.fs.exists(path)
^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py", line 47, in __get__
return prop.__get__(instance, type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/funcy/objects.py", line 25, in __get__
res = instance.__dict__[self.fget.__name__] = self.fget(instance)
^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/dvc_webdav/__init__.py", line 86, in fs
return WebdavFileSystem(**self.fs_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/fsspec/spec.py", line 79, in __call__
obj = super().__call__(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/webdav4/fsspec.py", line 103, in __init__
self.client = client or Client(base_url, auth=auth, **client_opts)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/webdav4/client.py", line 265, in __init__
self.http: HTTPClient = http_client or HTTPClient(**client_opts)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_client.py", line 646, in __init__
super().__init__(
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_client.py", line 180, in __init__
self._auth = self._build_auth(auth)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_client.py", line 433, in _build_auth
return BasicAuth(username=auth[0], password=auth[1])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_auth.py", line 130, in __init__
self._auth_header = self._build_auth_header(username, password)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sid/workspace/dvc/.venv/lib/python3.11/site-packages/httpx/_auth.py", line 139, in _build_auth_header
userpass = b":".join((to_bytes(username), to_bytes(password)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: sequence item 1: expected a bytes-like object, NoneType found
2023-11-22 12:21:44,999 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-11-22 12:21:44,999 DEBUG: Removing '/home/sid/workspace/dvc/.bQmqxAzbbLKem5ydCE5cQL.tmp'
2023-11-22 12:21:44,999 DEBUG: Removing '/home/sid/workspace/dvc/.bQmqxAzbbLKem5ydCE5cQL.tmp'
2023-11-22 12:21:44,999 DEBUG: Removing '/home/sid/workspace/dvc/.bQmqxAzbbLKem5ydCE5cQL.tmp'
2023-11-22 12:21:45,000 DEBUG: Removing '/home/sid/workspace/dvc/.local/.dvc/cache/files/md5/.87YBr5G5csNYhnrtPGBXpo.tmp'
2023-11-22 12:21:45,001 DEBUG: Version info for developers:
DVC version: 3.30.1
-------------------
Platform: Python 3.11.2 on <REDACTED>
Subprojects:
dvc_data = 2.22.0
dvc_objects = 1.1.0
dvc_render = 0.6.0
dvc_task = 0.3.0
scmrepo = 1.4.1
Supports:
azure (adlfs = 2023.10.0, knack = 0.11.0, azure-identity = 1.15.0),
gdrive (pydrive2 = 1.17.0),
gs (gcsfs = 2023.9.2),
hdfs (fsspec = 2023.9.2, pyarrow = 14.0.1),
http (aiohttp = 3.9.0, aiohttp-retry = 2.8.3),
https (aiohttp = 3.9.0, aiohttp-retry = 2.8.3),
oss (ossfs = 2021.8.0),
s3 (s3fs = 2023.9.2, boto3 = 1.28.17),
ssh (sshfs = 2023.10.0),
webdav (webdav4 = 0.9.8),
webdavs (webdav4 = 0.9.8),
webhdfs (fsspec = 2023.9.2)
Config:
Global: /home/sid/.config/dvc
System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sda9
Caches: local
Remotes: webdavs
Workspace directory: ext4 on /dev/sda9
Repo: dvc (no_scm)
Repo.site_cache_dir: /var/tmp/dvc/repo/f3aaa8fd4f4e85f507ddd8996bd9016b
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-11-22 12:21:45,001 DEBUG: Analytics is disabled.Expected
DVC to fail with an error message stating that the password field is not specified in .dvc/config.
Environment information
Output of dvc doctor:
$ dvc doctor
DVC version: 3.30.1
-------------------
Platform: Python 3.11.2 on <REDACTED>
Subprojects:
dvc_data = 2.22.0
dvc_objects = 1.1.0
dvc_render = 0.6.0
dvc_task = 0.3.0
scmrepo = 1.4.1
Supports:
azure (adlfs = 2023.10.0, knack = 0.11.0, azure-identity = 1.15.0),
gdrive (pydrive2 = 1.17.0),
gs (gcsfs = 2023.9.2),
hdfs (fsspec = 2023.9.2, pyarrow = 14.0.1),
http (aiohttp = 3.9.0, aiohttp-retry = 2.8.3),
https (aiohttp = 3.9.0, aiohttp-retry = 2.8.3),
oss (ossfs = 2021.8.0),
s3 (s3fs = 2023.9.2, boto3 = 1.28.17),
ssh (sshfs = 2023.10.0),
webdav (webdav4 = 0.9.8),
webdavs (webdav4 = 0.9.8),
webhdfs (fsspec = 2023.9.2)
Config:
Global: /home/sid/.config/dvc
System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sda9
Caches: local
Remotes: webdavs
Workspace directory: ext4 on /dev/sda9
Repo: dvc (no_scm)
Repo.site_cache_dir: /var/tmp/dvc/repo/f3aaa8fd4f4e85f507ddd8996bd9016b