[FR]: Extend compatibility to OpenAI compatible services

[mustafamohsen]

What would you like to have?

There are plenty of LLM providers that are compatible with OpenAI APIs (e.g. Deepseek). It would add more versatility to have a generic OpenAI Compatible provider that accepts base URL, API key, and model as inputs, and calls accordingly. This approach is implemented by VS Code's extension Cline, as well as others

Code of Conduct

• I agree to follow this project's Code of Conduct

[calderonsamuel]

Thanks for the idea, we are thinking on changing our internal logic to rely on the new {ellmer} . That should allow us to support (hopefullly) everything that package can handle. I see that we can extend the chat_openai() method with a custom URL and API KEY very easily. The model might be a bit more trickier. We'll be sharing any updates on this thread.

Btw, if you have Ollama + Deepseek running locally, it should be already supported

[mustafamohsen]

@calderonsamuel
That's interesting. Thanks for the hints.
Yes, I have Ollama, but Deepseek is kinda slow to run locally, even on an M2 Ultra, so I primarily rely on the APIs

[calderonsamuel]

Hi @mustafamohsen . #239 intends to solve this request. However, the Deepseek API currently does not allow me to add any billing information to give it a try.

I can't merge anything without confirming it works as expected. But you can start using it when you please!

Install from the PR's branch with:

pak::pak("MichelNivard/gptstudio#239")

Set OPENAI_API_URL to the Deepseek API ("https://api.deepseek.com"), and OPENAI_API_KEY. This should be done at the user level, usethis::edit_r_environ() will open that file for you.

Let me know if it works!

[mustafamohsen]

Hi @calderonsamuel
Great! Thank you
Yes, DeepSeek's been suffering overload recently so they stopped selling more credits until further notice
Luckily, I still have some little credit which I can use to test. Also, if you want I can email you an API key for your testing. Just let me know

[mustafamohsen]

@calderonsamuel I'm away for a while so I can't test it maybe until next weekend. However, I took a peak at the diff, and I didn't find anything related to model selection, which I doubt will work like that

[stevegbrooks]

In my company we have a LLM service that uses the openai protocol, so I can test it with that. I'll try it out on Monday. But to echo @mustafamohsen , we would definitely need the params model_name, api_key, and api_url at minimum, and kwargs for good measure.

[calderonsamuel]

Thanks for the comments everyone. Each service, including OpenAI, has already an implementation of a call to the "models" endpoint, which populates the "models" dropdown in the settings panel. As I understand, Deepseek has the exact same endpoint. So, #239 is not trying to add a new service, but to use whatever exists for OpenAI and re-route to use Deepseek's API. @mustafamohsen I appreciate the API key offer, but I don't think I should have that power 😿. I'm not actually the official maintainer of the package, just a trusted collaborator (with merge privileges)

[stevegbrooks]

I tried this out yesterday, and wasn't able to get it to work. It seems to be something to do with the SSL certificate, which I'm not sure how to specify in the gptstudio API.

 # load tangbao package with reticulate
> library(reticulate)
> tangbao <- import("tangbao")
> 
> apollo <- tangbao$apollo
> apollo <- apollo$Apollo()
> 
> Sys.setenv(OPENAI_API_URL="https://api-mgmt.boehringer-ingelheim.com/llm-api")
> Sys.setenv(OPENAI_API_KEY = apollo$token)
> Sys.setenv(CURL_CA_BUNDLE = apollo$`_cert_path`)
>
> # test
> curl::curl_fetch_memory(
+     Sys.getenv("OPENAI_API_URL"),
+     handle = curl::new_handle()
+ )$status_code
[1] 200
> response <- request(apollo$`_base_url`) |>
+     req_url_path_append("application/iam") |>
+     req_headers(.headers = apollo$`_api_headers`) |>
+     req_auth_bearer_token(apollo$token) |>
+     req_options(ssl_verifypeer = TRUE,
+                 ssl_verifyhost = 2,
+                 cainfo = apollo$`_cert_path`) |>
+     req_perform()
> response
<httr2_response>
GET https://api-mgmt.boehringer-ingelheim.com:8065/llm-api/application/iam
Status: 200 OK
Content-Type: application/json
Body: In memory (78 bytes)
>
> # gptstudio
> gptstudio::get_available_models(service = "openai")
Error in `httr2::req_perform()`:
! HTTP 401 Unauthorized.
• OAuth error
• realm: OpenAI API
Run `rlang::last_trace()` to see where the error occurred.

So here its working with curl::curl_fetch_memory and directly calling the API with httr2, but not working in gptstudio.

Another thing I found odd was when I tried to build the skeleton request:

> skeleton <- gptstudio:::gptstudio_skeleton_build.gptstudio_request_openai()
> skeleton$api_key == apollo$token
[1] TRUE
> skeleton$url == Sys.getenv("OPENAI_API_URL")
[1] FALSE
> skeleton$url
https://api.openai.com/v1/chat/completions

So its respecting the env var for OPENAI_API_KEY, but it seems the OPENAI_API_URL is not changing the url in this skeleton builder.

So my theory is its either the URL specification not being updated to respect changes in the OPENAI_API_URL env var, or something in the CA specification.

[calderonsamuel]

Thanks for this report @stevegbrooks . Please, confirm that you are using the branch from #239 (pak::pak("MichelNivard/gptstudio#239")). From my side, it looks like OPENAI_API_URL works as expected.

skeleton <- gptstudio:::gptstudio_skeleton_build.gptstudio_request_openai()

# If TRUE, the skeleton working as expected
skeleton$api_key == Sys.getenv("OPENAI_API_KEY")
#> [1] TRUE

# This is expected to be false
skeleton$url == Sys.getenv("OPENAI_API_URL")
#> [1] FALSE

# Because the endpoint is appended to the original URL
skeleton$url
#> https://api.deepseek.com/chat/completions

# You should get the same commit reference
sessioninfo::package_info("gptstudio", dependencies = FALSE)
#>  package   * version    date (UTC) lib source
#>  gptstudio   0.4.0.9009 2025-02-19 [1] Github (MichelNivard/gptstudio@718b18c)

^{Created on 2025-02-26 with reprex v2.1.1}

[stevegbrooks]

Hi,

Thanks for the clarification. I wasn't on the right commit ref.

After getting onto the right version, it still wasn't working. After some troubleshooting I realized that if I set Sys.setenv(OPENAI_API_URL = url) in the session, it won't work, because of how zzz.R is defining the gptstudio.openai_url option.

Basically, the skeleton for the api call requires the OPENAI_API_URL to be set in the .Renviron so that its present for the .onLoad() call. Once I defined OPENAI_API_URL in my .Renviron, it worked as expected.

When I then look at the available models, however, I get this result:

> gptstudio::get_available_models(service = "openai")
[1] "gpt-4o-mini"       "gpt-3.5-turbo"     "gpt-4-turbo"       "gpt-4o"            "gpt-4o-2024-11-20" "gpt-4o-eu"         "gpt-4o-mini-eu"   
[8] "gpt-4o-mini-us"    "gpt-4o-us"

As the user, I want to be able to filter for the models that I think the chat app should use, especially including our claude 3.5 sonnet deployment. I shouldn't be forced to choose only from models that start with "gpt".

> response <- request(Sys.getenv("OPENAI_API_URL")) |>
+     req_url_path_append("models") |>
+     req_headers(.headers = apollo$`_api_headers`) |>
+     req_auth_bearer_token(apollo$token) |>
+     req_options(ssl_verifypeer = TRUE,
+                 ssl_verifyhost = 2,
+                 cainfo = Sys.getenv("SSL_CERT_FILE")) |>
+     req_perform()
> 
> response
<httr2_response>
GET https://api-mgmt.boehringer-ingelheim.com:8065/llm-api/models
Status: 200 OK
Content-Type: application/json
Body: In memory (3938 bytes)
> models <- httr2::resp_body_json(response)
> 
> models$data[[22]]
$id
[1] "claude_3_5_sonnet"

$object
[1] "model"

$created
[1] 1677610602

$owned_by
[1] "openai"

I can think of two strategies here:

1. We simply add a condition in list_available_models.openai. If getOption("gptstudio.openai_url") isn't equal to "https://api.openai.com/v1" (i.e., the user is using an openai-compatible API service), then we have a series of these stringr::str_subset(..., negate = TRUE) calls to filter out (as best we can) the non-chat models.
2. We have a way for the user to configure which model names they want for the service. So, they might specify something like CHAT_MODELS = ['gpt-4o', 'claude_3_5_sonnet', ...]