Litellm Add Callback failing with SSL:Certificate_Verify_Failed

[skuchipu]

Seeing below error when adding lanfguse as call in Litellm .

deployed bothe litellm and langfuse using helm charts and both the deployments are on the same self hosted k8s cluster .

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1028)
Traceback (most recent call last):
File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
yield
File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 233, in handle_request
resp = self._pool.handle_request(req)
File "/usr/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
raise exc from None
File "/usr/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
response = connection.handle_request(
pool_request.request
)
File "/usr/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
raise exc
File "/usr/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 78, in handle_request
stream = self._connect(request)
File "/usr/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 156, in _connect
stream = stream.start_tls(**kwargs)
File "/usr/lib/python3.13/site-packages/httpcore/_backends/sync.py", line 154, in start_tls
with map_exceptions(exc_map):
~~~~~~~~~~~~~~^^^^^^^^^
File "/usr/lib/python3.13/contextlib.py", line 162, in exit
self.gen.throw(value)
~~~~~~~~~~~~~~^^^^^^^
File "/usr/lib/python3.13/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1028)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/lib/python3.13/site-packages/langfuse/client.py", line 392, in auth_check
projects = self.client.projects.get()
File "/usr/lib/python3.13/site-packages/langfuse/api/resources/projects/client.py", line 46, in get
_response = self._client_wrapper.httpx_client.request(
"GET",
...<27 lines>...
else 0, # type: ignore
)
File "/usr/lib/python3.13/site-packages/langfuse/api/core/http_client.py", line 100, in request
response = self.httpx_client.request(*args, **kwargs)
File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 827, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 914, in send
response = self._send_handling_auth(
request,
...<2 lines>...
history=[],
)
File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 942, in _send_handling_auth
response = self._send_handling_redirects(
request,
follow_redirects=follow_redirects,
history=history,
)
File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
response = self._send_single_request(request)
File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1015, in _send_single_request
response = transport.handle_request(request)
File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 232, in handle_request
with map_httpcore_exceptions():
~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/lib/python3.13/contextlib.py", line 162, in exit
self.gen.throw(value)
~~~~~~~~~~~~~~^^^^^^^
File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1028)
13:40:46 - LiteLLM Proxy:ERROR: _health_endpoints.py:282 - litellm.proxy.proxy_server.health_services_endpoint(): Exception occured - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1028)
INFO: 100.80.28.182:52446 - "GET /health/services?service=langfuse HTTP/1.1" 500 Internal Server Error

[skuchipu]

I've added the following setting and ENV to the deployment , still getting the same error
SSL_CERTIFICATE: "/etc/ssl/certs/cabundle.pem"

Any inputs ?