Enabling API-keyless usage & locally-run Ollama integration

[pierreolivierbonin]

Hello,

Fantastic repo, really appreciate the YT video as well. Just wanna check in to see if you'd be interested in allowing the users to run this entire app without API keys. As a user and developer, and for privacy purposes, some organizations may want to avoid querying external APIs and instead favour locally-run open source models.

Would you be interested, and if so, how much work do you think this kind of integration would represent?

[madtank]

Yes, Josh has put together something very useful. I added the option for AWS bedrock, that doesn't require an api key. It'll use your local credentials file used for AWS cli or boto3. Or if you run the container in AWS, it'll use a role.
I did implement ollama for local models, but at the time it did not stream, so I didn't offer to contribute that. The way Josh has setup, it was pretty easy and straightforward to add ollama.

[JoshuaC215]

Thanks Pierre! I’d be very happy to have local Ollama support added, the use case makes total sense. I think the lack of streaming support for that model is OK too if it doesn’t totally break things. This also supports my long term goal with the project to let people fully run their own agent stack without relying on a third party.

The recent PR for adding deepseek shows the places I think you’d need to update for this:

https://github.com/JoshuaC215/agent-service-toolkit/pull/134/files

Sounds like @madtank may already have it in a fork, if someone wanted to post a working PR I’m happy to help as needed!

[JoshuaC215]

Forgot to mention, I think you’d just do something like env variable to enable it / point to the ollama path instead of an api key

[pierreolivierbonin]

@JoshuaC215 @madtank fantastic, thank you so much for your quick responses. I've just cloned @madtank's OllamaAssist repo and so far, it looks like it's working well.

More specifically, I'm looking for an Ollama integration + RAG capacity. I was able to develop something on my side using Langchain, ollama, and chromadb that works but the UI component with Streamlit was missing.

In short, I'd love to contribute the missing parts so that we can easily run something like a locally-run chatbot powered by an Ollama-served open source model, leveraging RAG capacity from ChromaDB (or any open source vector database). Will keep tinkering and keep you posted :)

[JoshuaC215]

I just added experimental ollama support with #160. I would love feedback on how it works for you. Instructions are here.

[pierreolivierbonin]

so far, I'm only getting errors while trying to run the app:

[pierreolivierbonin]

For clarity and replicability, I've followed the instructions but there's a bit of a lack of clarity around the .env file, I believe.

I had already installed Ollama on my machine, so for the rest of my use case, what I am doing right now is:

• copy the environment variables set in .env.example and paste those into a .env file, and then:
• uncomment line 35: OLLAMA_MODEL=llama3.2
• uncomment line 40: AGENT_URL=http://0.0.0.0:8080
• run python src/run_service.py in the first terminal
• run streamlit run src/streamlit_app.py in the second terminal

Another thing I found unclear is: where do we ensure ollama is running in the instructions? Does it presuppose that the user has it running in the standalone app? Or is the user meant to run ollama serve in a third terminal?

With my fork of @madtank's OllamaAssist repo, for instance, I only have to run ollama serve in a first terminal, and then streamlit run <appname.py> in a second terminal, which quite convenient. Maybe we can achieve a similar result here?

[madtank]

Deployment Guide: Docker and Local Setup

Docker Deployment

Running the service using Docker Compose:

docker compose watch

Required Environment Variables (.env)

# Ollama Configuration
OLLAMA_MODEL=MFDoom/deepseek-r1-tool-calling:14b
# Note: Set this if running service in docker and ollama on bare metal
OLLAMA_BASE_URL=http://host.docker.internal:11434
# Agent URL for Streamlit app (defaults to http://{HOST}:{PORT} if not set)
AGENT_URL=http://0.0.0.0:8080

Local Deployment

To run the service locally:

python src/run_service.py
streamlit run src/streamlit_app.py

Required Environment Variables (Local)

# Ollama Configuration
OLLAMA_MODEL=MFDoom/deepseek-r1-tool-calling:14b
# Agent URL for Streamlit app (defaults to http://{HOST}:{PORT} if not set)
AGENT_URL=http://0.0.0.0:8080

Important Note

When selecting a model, ensure it supports tool calling functionality. Using models without tool support will result in errors like:

ollama._types.ResponseError: registry.ollama.ai/library/deepseek-r1:14b does not support tools (status code: 400)

For optimal performance, use an agent or LLM that explicitly supports tool calling capabilities.

[JoshuaC215]

Thanks @pierreolivierbonin and @madtank ! Sorry it was causing trouble for you Pierre-Olivier.

I think I realized that you're right - I was running the desktop app (e.g. I have the Ollama icon in my system tray) which meant the server was running without me explicitly setting it.

I just did some further testing and found that when I killed the desktop app, it started giving errors similar to what you shared. Then when I ran ollama serve in a separate terminal and tried again, things worked. So, I'm guessing that was the missing step for you?

Let me know if that resolves it for you, and we can add an instruction to call out that step in the README. I have NOT tested on Windows, so it's possible there are other problems - might need your help debugging. The logs from the service could be another clue.

[JoshuaC215]

Also @pierreolivierbonin re:

If you would allow me to be a contributor to this repo, I'd love to provide feedback but I want to make sure I can push some code to this repo not just comments in a discussion. Would that be okay with you?

Yep, the recommended path is just to submit a PR, a bunch of folks besides me have already made contributions including some sizable ones and new features! I would suggest if you're spending more than an hour or two on something, it may be worth a quick chat ahead of time since I haven't accepted every contribution and/or sometimes have design feedback - trying to keep it generally applicable and things I can reasonably maintain. Happy to do in an issue or privately. For small things, just submit a PR! And please follow the contributing instructions / use pre-commit to reduce back and forth.