Microsoft.Extensions.AI for Local LLMs

This provides sample codes that uses Microsoft.Extensions.AI for locally running LLMs through Docker Model Runner, Foundry Local, Hugging Face and Ollama.

This is a trim-down version of OpenChat Playground, specific to dealing with locally running LLMs. If you want to see more language models supported including MaaS and other vendors, try out OpenChat Playground instead.

Prerequisites

• .NET SDK 9
• Visual Studio Code + C# DevKit or Visual Studio 2022 v17.14+
• GitHub CLI
• PowerShell 7.5+ 👉 Windows only
• Docker Desktop
• Foundry Local
• Ollama

Getting Started

Get the repository ready

1. Login to GitHub.

   gh auth login

2. Check login status.

   gh auth status

3. Fork this repository to your account and clone the forked repository to your local machine.

   gh repo fork devkimchi/meai-for-local-llms --clone --default-branch-only

4. Navigate to the cloned repository.

   cd meai-for-local-llms

5. Get the repository root.

   # bash/zsh
   REPOSITORY_ROOT=$(git rev-parse --show-toplevel)

   # PowerShell
   $REPOSITORY_ROOT = git rev-parse --show-toplevel

Use GitHub Models

As a default, this app uses GitHub Models.

1. Make sure you are at the repository root.

   cd $REPOSITORY_ROOT

2. Add GitHub Personal Access Token (PAT) for GitHub Models connection. Make sure you should replace {{YOUR_TOKEN}} with your GitHub PAT.

   # bash/zsh
   dotnet user-secrets --project $REPOSITORY_ROOT/src/MEAIForLocalLLMs.WebApp \
       set GitHubModels:Token "{{YOUR_TOKEN}}"

   # PowerShell
   dotnet user-secrets --project $REPOSITORY_ROOT/src/MEAIForLocalLLMs.WebApp `
       set GitHubModels:Token "{{YOUR_TOKEN}}"

   For more details about GitHub PAT, refer to the doc, Managing your personal access tokens.

3. Run the app. The default language model is openai/gpt-4o-mini.

   dotnet run --project $REPOSITORY_ROOT/src/MEAIForLocalLLMs.WebApp

   If you want to change the language model, add the --model option with a preferred model name. You can find the language model from the GitHub Models catalog page.

   dotnet run --project $REPOSITORY_ROOT/src/MEAIForLocalLLMs.WebApp -- --connector-type GitHubModels --model <model-name>

4. Open your web browser, navigate to http://localhost:5160, and enter prompts.

Use Docker Model Runner

1. Make sure Docker Desktop is up and running.

   docker info

2. Download language model, ai/gpt-oss, to your local machine.

   docker model pull ai/gpt-oss

3. Make sure you are at the repository root.

   cd $REPOSITORY_ROOT

4. Run the app. The default language model is ai/gpt-oss.

   dotnet run --project $REPOSITORY_ROOT/src/MEAIForLocalLLMs.WebApp -- --connector-type DockerModelRunner

   If you want to change the language model, add the --model option with a preferred model name.

   dotnet run --project $REPOSITORY_ROOT/src/MEAIForLocalLLMs.WebApp -- --connector-type DockerModelRunner --model <model-name>

5. Open your web browser, navigate to http://localhost:5160, and enter prompts.

Use Foundry Local

1. Make sure Foundry Local is up and running.

   foundry service status

   If Foundry Local service is not up and running, run the following command:

   foundry service start

2. Download language model, gpt-oss-20b, to your local machine.

   foundry model download gpt-oss-20b

3. Make sure you are at the repository root.

   cd $REPOSITORY_ROOT

4. Run the app. The default language model is gpt-oss-20b.

   dotnet run --project $REPOSITORY_ROOT/src/MEAIForLocalLLMs.WebApp -- --connector-type FoundryLocal

   If you want to change the language model, add the --model option with a preferred model name.

   dotnet run --project $REPOSITORY_ROOT/src/MEAIForLocalLLMs.WebApp -- --connector-type FoundryLocal --model <model-name>

5. Open your web browser, navigate to http://localhost:5160, and enter prompts.

Use Hugging Face

Models from Hugging Face can be run through Ollama server.

1. Make sure Ollama is up and running.

   ollama start

2. In a separate terminal, download language model, LGAI-EXAONE/EXAONE-4.0-1.2B-GGUF, to your local machine.

   ollama pull hf.co/LGAI-EXAONE/EXAONE-4.0-1.2B-GGUF

3. Make sure you are at the repository root.

   cd $REPOSITORY_ROOT

4. Run the app. The default language model is hf.co/LGAI-EXAONE/EXAONE-4.0-1.2B-GGUF.

   dotnet run --project $REPOSITORY_ROOT/src/MEAIForLocalLLMs.WebApp -- --connector-type HuggingFace

   If you want to change the language model, add the --model option with a preferred model name.

   dotnet run --project $REPOSITORY_ROOT/src/MEAIForLocalLLMs.WebApp -- --connector-type HuggingFace --model <model-name>

5. Open your web browser, navigate to http://localhost:5160, and enter prompts.

Use Ollama

1. Make sure Ollama is up and running.

   ollama start

2. In a separate terminal, download language model, gpt-oss, to your local machine.

   ollama pull gpt-oss

3. Make sure you are at the repository root.

   cd $REPOSITORY_ROOT

4. Run the app. The default language model is gpt-oss.

   dotnet run --project $REPOSITORY_ROOT/src/MEAIForLocalLLMs.WebApp -- --connector-type Ollama

   If you want to change the language model, add the --model option with a preferred model name.

   dotnet run --project $REPOSITORY_ROOT/src/MEAIForLocalLLMs.WebApp -- --connector-type Ollama --model <model-name>

5. Open your web browser, navigate to http://localhost:5160, and enter prompts.