hf-transfer progress bar

[cbensimon]

PR content

• Turn chunk_size and max_files into shared constants
• Use a context manager for tqdm instead of manually creating / closing
• Use the same tqdm object for both hf-transfer and regular downloader
• Log progress through tqdm if hf-transfer supports it
• Still log progress in the end anyway (getting total size, speed, etc. even on old hf-transfer versions)
• Only use hf-transfer for file sizes > 10 x CHUNK_SIZE (~100MB)
  • Now that we anyway open a first request to get the content-length header event on hf-transfer

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[Wauplin]

Thanks for integrating this @cbensimon! Clean PR and logic looks good to me :)

Just to be sure about the 100MB threshold before using hf_transfer. Is it that you expect hf_transfer to be slower than requests on a 50MB file for example?

In addition to the code part, we should tweak a few parts in the documentation now that progress bars are supported. Would you mind updating them in the same PR?

• download.md It is tested and production-ready, but it lacks user-friendly features such as advanced error handling, resumable downloads and proxies.
• environment_variables.md Additionally, hf_transfer lacks several user-friendly features such as advanced error handling, progress bars during uploads, resumable downloads and proxies.

Last thing, when is it planned to make a new release of hf_transfer? It would be nice to have it soon so that huggingface_hub can use progress bars as expected.

Thanks in advance!

[cbensimon]

Just to be sure about the 100MB threshold before using hf_transfer. Is it that you expect hf_transfer to be slower than requests on a 50MB file for example?

As we now already open a request to get the Content-Length header, file size has to be larger for hf_transfer to be faster (iter_content on an already opened socket will be faster than re-opening the request and download it, even with multiple workers). But sure, this 100MB threshold is a bit arbitrary. Anything between CHUNK_SIZE (10MB) and 100MB seems ok to me. I let you chose the value !

[cbensimon]

In addition to the code part, we should tweak a few parts in the documentation now that progress bars are supported. Would you mind updating them in the same PR?

Done. Let me know what you think

[cbensimon]

Last thing, when is it planned to make a new release of hf_transfer? It would be nice to have it soon so that huggingface_hub can use progress bars as expected.

cc @Narsil (I made some tests on real-world environments like Spaces and 0.1.4rc1 seem to work without issues)

[Wauplin]

Thanks for making the changes @cbensimon!

[Wauplin]

Top! Approved!