Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/hub/_toctree.yml
Original file line number Diff line number Diff line change
Expand Up @@ -421,6 +421,8 @@
new: true
isExpanded: true
sections:
- local: storage-buckets-access
title: Access Patterns
- local: storage-buckets-security
title: Bucket Security

Expand Down
68 changes: 68 additions & 0 deletions docs/hub/storage-buckets-access.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Access Patterns

Beyond the [CLI and Python SDK](./storage-buckets#managing-files), there are several ways to access bucket data from your existing tools and workflows.

## Choosing an Access Method

| Method | Best for | Details |
|--------|----------|---------|
| **hf-mount** | Mount as local filesystem — any tool works | [See below](#mount-as-a-local-filesystem) |
| **Volume mounts** | HF Jobs & Spaces (same idea, managed for you) | [See below](#volume-mounts-in-jobs-and-spaces) |
| **hf:// paths** (fsspec) | Python data tools (pandas, DuckDB) | [See below](#python-data-tools) |
| **CLI sync** | Batch transfers, backups | [Sync docs](./storage-buckets#syncing-directories) |

## Mount as a Local Filesystem

[hf-mount](https://github.com/huggingface/hf-mount) lets you mount buckets (and repos) as local filesystems via NFS (recommended) or FUSE. Files are fetched lazily — only the bytes your code reads hit the network.

Install:

```bash
curl -fsSL https://raw.githubusercontent.com/huggingface/hf-mount/main/install.sh | sh
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

really need to submit it to brew

```

Mount a bucket:

```bash
hf-mount start bucket username/my-bucket /mnt/data
```

Once mounted, any tool that reads or writes files works with your bucket — pandas, DuckDB, vLLM, training scripts, shell commands, etc.

> [!TIP]
> Buckets are mounted read-write; repos are read-only. See the [hf-mount repository](https://github.com/huggingface/hf-mount) for full documentation including backend options, caching, and write modes.

## Volume Mounts in Jobs and Spaces

Volume mounts in [Jobs](./jobs) and [Spaces](./spaces) are the same idea as `hf-mount`, managed for you by the platform — no extra setup needed. Buckets are mounted read-write by default.

```bash
hf jobs run -v hf://buckets/username/my-bucket:/data python:3.12 python script.py
```

For the full volume mount syntax and Python API, see the [Jobs configuration docs](./jobs-configuration#volumes) and the [Spaces volume mount guide](/docs/huggingface_hub/guides/manage-spaces#mount-volumes-in-your-space).

## Python Data Tools

The [`HfFileSystem`](/docs/huggingface_hub/guides/hf_file_system) provides [fsspec](https://filesystem-spec.readthedocs.io)-compatible access to buckets using `hf://buckets/` paths. Any Python library that supports fsspec can read and write bucket data directly.

**pandas:**

```python
import pandas as pd

df = pd.read_parquet("hf://buckets/username/my-bucket/data.parquet")
df.to_parquet("hf://buckets/username/my-bucket/output.parquet")
```

**DuckDB** (Python client):

```python
import duckdb
from huggingface_hub import HfFileSystem

duckdb.register_filesystem(HfFileSystem())
duckdb.sql("SELECT * FROM 'hf://buckets/username/my-bucket/data.parquet' LIMIT 10")
```

For more on `hf://` paths and supported operations, see the [`HfFileSystem` guide](/docs/huggingface_hub/guides/hf_file_system) and the [Buckets Python guide](/docs/huggingface_hub/guides/buckets).
3 changes: 3 additions & 0 deletions docs/hub/storage-buckets.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ You can interact with buckets using the Hub web interface, the [`hf` CLI](https:
> [!TIP]
> Buckets are available to all users and organizations. See [hf.co/storage](https://huggingface.co/storage) for pricing details.

> [!TIP]
> Want to use bucket data with pandas, DuckDB, or mount buckets as a local filesystem? See [Access Patterns](./storage-buckets-access).

## Buckets vs Repositories

The Hub offers two types of storage: Git-based **repositories** for versioned, collaborative work and **buckets** for fast, mutable object storage.
Expand Down
Loading