Accessing local LM's in WLJS?

[t-Pai]

I see that it is possible to access LLM's through their API's. How about accessing local LM's in WLJS?

[JerryI]

I haven't tried by myself. As far as I know, WL does not support local LLMs (@KirillBelovTest might correct me on that).

But the handcrafting an adapter can be something... https://community.wolfram.com/groups/-/m/t/3235028

[JerryI]

Or do you mean to use local LLM as a notebook assistant?

[t-Pai]

Thank you for your help, Jerryl.
In Windows Powershell I can talk to my local LM (lLM) like this:

Invoke-RestMethod -Uri http://localhost:11434/api/generate -Method Post -ContentType 'application/json' -InFile payload.json

My request to the lLM is in the payload file:

{
  "model": "qwen3:8b",
  "prompt": "Why is the sky blue?",
  "stream": false
}

Response:

model                : qwen3:8b
created_at           : 2025-08-29T01:38:08.7546099Z
response             : <think>
                       Okay, the user is asking why the sky is blue. ...  (long, but relevant explanation)   

I tried to make the connection to the lLM from WLJS according to your suggestion:

chat = GPTChatObject["Hi There", 
   "Endpoint" -> "https://localhost:11434/api/generate", (* replace with yours *)
   "Model" -> "qwen3:8b" (* replace with yours *)
];

GPTChatComplete[chat];

chat["Messages"][[-1]]

The instruction produces a pop up window saying something about "join string" and "expected" but it disappears so fast that I can't read it completely. I don't know where to look for the logs of the session. Sorry I'm such a noob.

Anyway, your suggested instruction generates a response:

<|"role"->"system","date"->(*VB[*)(DateObject[{2025,9,1,7,50,44.1446768`9.397453210680194},"Instant","Gregorian",12.`])(*,*)(*"1:eJxTTMoPSmNiYGAo5gUSYZmp5S6pyflFiSX5RcEsQBGXxJLUYEkgQ8k3P0/BUCE4tUDByMDIVMHA3MrUwMrERAkAEdwPZA=="*)(*]VB*),"content"->"Hi There"|>

But I don't think the response originates from the lLM.

[JerryI]

Thank you @t-Pai !
It gives a lot of info. The problem is in our library.

The instruction produces a pop up window saying something about "join string" and "expected" but it disappears so fast that I can't read it completely. I don't know where to look for the logs of the session. Sorry I'm such a noob.

No worries, this is our bad to have such unfrendly design ;)
It could help us a bit more, if you click on

Then it will show accumulated log messages. And by the way, are you using ollama with some facebook models? Then we can try on our local machines to fix this issue.

Thx!

[t-Pai]

I'm downloading the Gemma3 model. Takes ages.

[t-Pai]

I've got the Meta gemma3 model installed now and the response is the same as from the qwen3 model.
Input:

chat = GPTChatObject["Hi There", 
   "Endpoint" -> "https://localhost:11434/api/generate", (* replace with yours *)
   "Model" -> "gemma3:12b" (* replace with yours *)
];

GPTChatComplete[chat];

chat["Messages"][[-1]]

Response:

<|"role"->"system","date"->(*VB[*)(DateObject[{2025,9,3,7,29,32.5077788`9.264562197802105},"Instant","Gregorian",12.`])(*,*)(*"1:eJxTTMoPSmNiYGAo5gUSYZmp5S6pyflFiSX5RcEsQBGXxJLUYEkgQyk8NUXBWCE4tUDByMDIVMHA3MrI0srYSAkAEUYPXw=="*)(*]VB*),"content"->"Hi There"|>

It produces the same warning (not from the AI model obviously). I found it under "Show messages" as you indicated:

Warning

StringJoin::string: String expected at position 2 in Bearer <>Missing[NotAvailable].

Gemma3 runs fine when used from the Power Shell command line in the same way as qwen3. I'm actually quite impressed how well these models perform at a fraction (5%?) of the size of the LLM's online. They could make very good notebook assistants is what I think. Surely better than the Wolfram notebook assistant.

[JerryI]

I've got the Meta gemma3 model installed now and the response is the same as from the qwen3 model.

Input:

chat = GPTChatObject["Hi There", 

   "Endpoint" -> "https://localhost:11434/api/generate", (* replace with yours *)

   "Model" -> "gemma3:12b" (* replace with yours *)

];



GPTChatComplete[chat];



chat["Messages"][[-1]]

Response:

<|"role"->"system","date"->(*VB[*)(DateObject[{2025,9,3,7,29,32.5077788`9.264562197802105},"Instant","Gregorian",12.`])(*,*)(*"1:eJxTTMoPSmNiYGAo5gUSYZmp5S6pyflFiSX5RcEsQBGXxJLUYEkgQyk8NUXBWCE4tUDByMDIVMHA3MrI0srYSAkAEUYPXw=="*)(*]VB*),"content"->"Hi There"|>

It produces the same warning (not from the AI model obviously). I found it under "Show messages" as you indicated:

Warning



StringJoin::string: String expected at position 2 in Bearer <>Missing[NotAvailable].

Gemma3 runs fine when used from the Power Shell command line in the same way as qwen3. I'm actually quite impressed how well these models perform at a fraction (5%?) of the size of the LLM's online. They could make very good notebook assistants is what I think. Surely better than the Wolfram notebook assistant.

Thanks! We more or less located issues within GPTLink library. The rest is on our side :)

[JerryI]

Fixed and tested in WLJS Notebook 2.8.3 with qwen3:8b!

[JerryI]

PS: Ok. I shall reopen it, so that the discussion is visible