Skip to content

Commit

Permalink
docs: Add docs explaining how to install pre-built wheels.
Browse files Browse the repository at this point in the history
  • Loading branch information
abetlen committed Apr 4, 2024
1 parent c50309e commit 1db3b58
Showing 1 changed file with 45 additions and 0 deletions.
45 changes: 45 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,15 @@ This will also build `llama.cpp` from source and install it alongside this pytho

If this fails, add `--verbose` to the `pip install` see the full cmake build log.

**Pre-built Wheel (New)**

It is also possible to install a pre-built wheel with basic CPU support.

```bash
pip install llama-cpp-python \
--extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
```

### Installation Configuration

`llama.cpp` supports a number of hardware acceleration backends to speed up inference as well as backend specific options. See the [llama.cpp README](https://github.com/ggerganov/llama.cpp#build) for a full list.
Expand Down Expand Up @@ -108,6 +117,30 @@ To install with cuBLAS, set the `LLAMA_CUDA=on` environment variable before inst
CMAKE_ARGS="-DLLAMA_CUDA=on" pip install llama-cpp-python
```

**Pre-built Wheel (New)**

It is also possible to install a pre-built wheel with CUDA support. As long as your system meets some requirements:

- CUDA Version is 12.1, 12.2 or 12.3
- Python Version is 3.10, 3.11 or 3.12

```bash
pip install llama-cpp-python \
--extra-index-url https://abetlen.github.io/llama-cpp-python/whl/<cuda-version>
```

Where `<cuda-version>` is one of the following:
- `cu121`: CUDA 12.1
- `cu122`: CUDA 12.2
- `cu123`: CUDA 12.3

For example, to install the CUDA 12.1 wheel:

```bash
pip install llama-cpp-python \
--extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu121
```

</details>

<details>
Expand All @@ -119,6 +152,18 @@ To install with Metal (MPS), set the `LLAMA_METAL=on` environment variable befor
CMAKE_ARGS="-DLLAMA_METAL=on" pip install llama-cpp-python
```

**Pre-built Wheel (New)**

It is also possible to install a pre-built wheel with Metal support. As long as your system meets some requirements:

- MacOS Version is 11.0 or later
- Python Version is 3.10, 3.11 or 3.12

```bash
pip install llama-cpp-python \
--extra-index-url https://abetlen.github.io/llama-cpp-python/whl/metal
```

</details>
<details>

Expand Down

0 comments on commit 1db3b58

Please sign in to comment.