Getting 404s from locally running Ollama

[DVG]

I'm attempting to build a simple proof of concept for sending a message to deepseek-r1 running under ollama (I'm not intent on this model specifically, it's just the one I pulled because it supports tools and I just want to play around with it)

I have this initializer code:

RubyLLM.configure do |config|
  config.ollama_api_base = "http://127.0.0.1:11434"
end

Ollama is running and responds to curls locally:

curl -s http://localhost:11434/api/tags
{"models":[{"name":"deepseek-r1:latest","model":"deepseek-r1:latest","modified_at":"2025-07-27T10:13:18.115821618-04:00","size":5225376047,"digest":"6995872bfe4c521a67b32da386cd21d5c6e819b6e0d62f79f64ec83be99f5763","details":{"parent_model":"","format":"gguf","family":"qwen3","families":["qwen3"],"parameter_size":"8.2B","quantization_level":"Q4_K
_M"}},{"name":"deepseek-r1:1.5b","model":"deepseek-r1:1.5b","modified_at":"2025-07-26T23:09:52.215113449-04:00","siz
e":1117322768,"digest":"e0979632db5a88d1a53884cb2a941772d10ff5d055aabaa6801c4e36f3a6c2d7","details":{"parent_model":
"","format":"gguf","family":"qwen2","families":["qwen2"],"parameter_size":"1.8B","quantization_level":"Q4_K_M"}},{"n
ame":"deepseek-r1:70b","model":"deepseek-r1:70b","modified_at":"2025-07-26T22:32:25.403239188-04:00","size":42520397
873,"digest":"d37b54d01a76512a5adbaa9da4ee1c00833bf2dfefb0d4cc54351ba529e0f4da","details":{"parent_model":"","format
":"gguf","family":"llama","families":["llama"],"parameter_size":"70.6B","quantization_level":"Q4_K_M"}}]}%

But when I attempt to hit it with ruby llm, I don't get anything:

chat = RubyLLM.chat(model: 'deepseek-r1', provider: 'ollama')
# WARN -- RubyLLM: Assuming model 'deepseek-r1:1.5b' exists for provider 'RubyLLM::Providers::Ollama'. Capabilities may not be accurately reflected.
chat.ask "Hello"
# `<main>': 404 page not found (RubyLLM::Error)`

I've tried various combinations of specifying the model differently such as deepseek/deepseek-r1 and deepseek-r1:latest and deepseek/deepseek-r1:latest but never get anything back. The warning makes me think I'm doing something wrong but not sure where to turn next

[DVG]

Okay, progress. After looking at the ollama server api logs I saw the 404 errors, so it was getting to the server but the endpoint is incorrect int he gem:

llmtest(dev)* module RubyLLM
llmtest(dev)*   module Providers
llmtest(dev)*     module Ollama
llmtest(dev)*       # Chat methods of the Ollama API integration
llmtest(dev)*       module Chat
llmtest(dev)*                 def completion_url
llmtest(dev)*                   'chat'
llmtest(dev)*                 end
llmtest(dev)*       end
llmtest(dev)*     end
llmtest(dev)*   end
llmtest(dev)> end

Adding this monkey patch let it hit the correct endpoint, which I can see it returned. aresponse.

The new problem is when it's parsing the response

llmtest(dev)> chat.ask "hello"
(llmtest):24:in `<main>': undefined method `fetch' for nil (NoMethodError)

      @role = options.fetch(:role).to_sym
                     ^^^^^^

[DVG]

The only place this code appears is in Message

def initialize(options = {})
      @role = options.fetch(:role).to_sym

It's defaulted so I don't know why it would possibly be nil ¯\_(ツ)_/¯

[tpaulshippy]

I suggest adding /v1 to your configured base URL as specified here -- https://rubyllm.com/configuration#ollama-api-base-ollama_api_base

[DVG]

That was part of the issue! But the bigger problem seems to be some api / response breaking changes from whenever ollama was implemented to now. I made a PR: https://github.com/crmne/ruby_llm/pull/300/files

[tpaulshippy]

That was part of the issue! But the bigger problem seems to be some api / response breaking changes from whenever ollama was implemented to now. I made a PR: https://github.com/crmne/ruby_llm/pull/300/files

I don't think you need the changes in the PR. Ollama has an openai compatible endpoint that RubyLLM uses and it works. See https://ollama.readthedocs.io/en/openai/#v1chatcompletions for more details.