ct2hf

ct2hf is a user-friendly CLI tool designed to simplify the process of converting supported Hugging Face transformer models into a CTranslate2-compatible format. Additionally, it seamlessly uploads the converted model to your Hugging Face repository. No interaction required!

Highlights

• Convert, quantise and upload your models completely hands-free
• Automatically cleans up all dependencies, even if the program has terminated unexpectedly
• Utilises the least amount of memory possible to handle large model conversions
• Avoids common permission and storage pitfalls with sensible defaults
• Only uses Git LFS when necessary, avoiding unnecessary issues that arise from using LFS

Setup

You have to be logged in to Hugging Face to upload the converted model onto your account.

uvx --no-cache --from huggingface-hub huggingface-cli login

Or without uv.

huggingface-cli login

Usage

You may use pip to install ct2hf but we recommended using uv to avoid polluting your Python environment.

uvx --no-cache --python 3.14 --from git+https://github.com/winstxnhdw/ct2hf ct2hf --help

By default, ct2hf avoids leaving behind any unnecessary files. If you would like to preserve the downloaded models, you can use the --preserve-models flag. Additionally, if --quantisation is not specified, the quantisation type defaults to int8.

Tip

ct2hf has some memory usage optimisations that may not be compatible with some models. If you encounter any issues, try using the --compatibility flag.

usage: ct2hf

convert and upload a transformer model to huggingface

positional arguments:
  model-id              transformer model to convert

options:
  --output-name         name of the output model
  --files-to-copy       files to copy to the output model
  --preserve-models     do not delete the downloaded models
  --revision            revision of the model to convert
  --quantisation        none, int8_float32, int8_float16, int8_bfloat16, int16, float16, bfloat16, float32
  --compatibility       use compatibility mode at the cost of higher memory usage

Example

The minimal usage of ct2hf involves specifying the model ID of the transformer model you would like to convert.

ct2hf openchat/openchat-3.6-8b-20240522

You can also specify the name of the output model.

ct2hf openchat/openchat-3.6-8b-20240522 --output-name openchat-3.6-ct2-int8

You can also specify the files to copy to the output model.

ct2hf openchat/openchat-3.6-8b-20240522 --files-to-copy tokenizer.json