Misleading HTTP 200 Status Code on Network Connection Lost Error via Portkey

[Nathanpamart]

What Happened?

Issue Summary:

When making a request to OpenAI's API through Portkey, I received an HTTP 200 status code despite the response body indicating a "Network connection lost." error. This behavior is misleading and prevents automatic retry mechanisms from triggering, as they typically rely on non-200 status codes to indicate failure.

{ "status": 200, "headers": { "content-type": "application/json" }, "body": { "html-message": "Network connection lost." }, "responseTime": 108, "lastUsedOptionJsonPath": "config" }

Steps to Reproduce:
Send a POST request to https://api.openai.com/v1/chat/completions via Portkey with the following headers and body:

Headers: (List only the relevant headers, especially those related to Portkey and retry logic)
Content-Type: application/json
X-Portkey-Retry: {"attempts": 5}
(Include any other headers that are crucial to replicating the issue)
Body: (Simplify to illustrate the request without exposing sensitive information)

see the request json below

{ "url": "https://api.openai.com/v1/chat/completions", "method": "POST", "headers": { "accept": "application/json", "accept-encoding": "gzip, br", "cf-visitor": "{\"scheme\":\"https\"}", "connection": "Keep-Alive", "content-length": "7325", "content-type": "application/json", "host": "api.portkey.ai", "user-agent": "OpenAI/Python 1.14.2", "x-forwarded-proto": "https", "x-portkey-retry": "{\"attempts\": 5}", "x-real-ip": "213.27.249.134", "x-stainless-arch": "x64", "x-stainless-async": "false", "x-stainless-lang": "python", "x-stainless-os": "MacOS", "x-stainless-package-version": "1.14.2", "x-stainless-runtime": "CPython", "x-stainless-runtime-version": "3.10.0" }, "body": { "messages": [ { "role": "system", "content": "<my_system_prompt>" }, { "role": "user", "content": "<my content>" } ], "model": "gpt-4-1106-preview", "n": 1, "stream": false, "temperature": 0 }, "portkeyHeaders": { "x-portkey-api-key": "rD********Hg=", "x-portkey-config": "{\"cache\":{\"mode\":\"simple\",\"maxAge\":864000}}", "x-portkey-provider": "openai", "x-portkey-trace-id": "f2fc3222-e3e5-4081-9d3f-f92cf2d312b0" } }

What Should Have Happened?

In the event of a network connection issue or similar error, I expect to receive an HTTP status code indicative of an error (e.g., 4xx or 5xx). This would allow client-side logic, such as automatic retries, to function correctly based on the status code.

Actual Behavior:
Received an HTTP 200 status code, suggesting a successful response, despite the body containing an error message indicating a lost network connection.

Suggested Fix:
Adjust the API or Portkey's error handling logic to return a more appropriate HTTP status code when an error occurs, especially for network-related issues.

Relevant Code Snippet

I trigger this request from a python script using langchain, as below

from langchain_community.chat_models import ChatOpenAI from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

headers = createHeaders( provider="openai", api_key=PORTKEY_API_KEY, # Cache Options config={ "cache": {"mode": "simple", "maxAge": 864000} }, # in seconds (10 days) retry={"attempts": 5}, # Metadata environment="test", user="<user>", organisation="<org>", prompt="test", )

after initializing an LLM object

llm = ChatOpenAI( model_name=model_name, temperature=0, base_url=PORTKEY_GATEWAY_URL, default_headers=headers, )

Your Twitter/LinkedIn

https://www.linkedin.com/in/nathanpamart/

[rootAvish]

I faced this bug too. An additional side effect of this bug is Portkey's cache caches the error response (assume because of the 200 response code). You don't get a valid response till you force cache refresh.

[vrushankportkey]

Hi @Nathanpamart @rootAvish sorry for missing out on this! Tagging @narengogi who can take a look at this

[narengogi]

@rootAvish @Nathanpamart This issue has since been fixed, the gateway now handles http failures correctly

[rootAvish]

@narengogi I encountered this again today. Have sent trace ids on an email thread @VisargD is marked in. Kindly refer to the same, thanks.