llm-test

Description

Sample docker container for running LLM (Large Language Model) locally.

Requirement

Software

• Visual Studio Code
• Docker Desktop 4.44.3 (202357)
• Docker version 28.3.2, build 578ccf6
• Docker Compose version v2.39.1-desktop.1

Hardware

• CPU: Intel Core i7-10700
• RAM: 32 GB
• GPU: NVIDIA GeForce RTX 4070 Ti

Install

Fork and clone this repository.

$ git clone git@github.com:yourname/llm-test.git

Usage

Start container.

$ docker-compose up -d

Pull LLM model.

$ curl http://localhost:11434/api/pull -d '{"name": "llama3"}'

From API,

$ curl http://localhost:11434/api/chat -d '{"model": "llama3","messages": [{"role": "user", "content": "Hello!"}], "stream": false}'

From CLI,

$ docker exec -it ollama_gpu bash
root@ddaeda92a635:/# ollama run llama3
>>> 

Press Ctrl + d to exit.

Contribution

1. Fork this repository
2. Create your feature branch
3. Commit your changes
4. Push to the branch
5. Create new Pull Request

License

MIT License

Author

minato