Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Xet download workflow #2875

Draft
wants to merge 13 commits into
base: xet-integration
Choose a base branch
from
Draft

Conversation

hanouticelina
Copy link
Contributor

@hanouticelina hanouticelina commented Feb 18, 2025

🏗️ This PR is a work in progress, don't review yet.

Partially resolves #2713.

This PR adds the Xet download workflow implemented in xetpoc_huggingface_hub (internal).Tthe upload one will be integrated in a separate PR

The main branch for xet storage integration into huggingface_hub is xet-integration.

Main changes:

  • Make hf_xet available as an optional dependency via pip install huggingface_hub[hf_xet]
    Note: since it's a common part for download and upload, this has been pushed directly into xet-integration branch.
  • Integrate changes from the xet poc for the download workflow only.
  • Add tests.
  • Add documentation.

to try it in from this branch:

pip install -e ".[dev,hf_xet]"
export HF_DEBUG=1 #  if you want to set huggingface_hub logger to debug level
huggingface-cli download huggingface/distilbert-base-uncased-xet

cc @bpronan @assafvayner @rajatarya

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants