feat(integrations): add litellm integration #4864
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| litellm.success_callback = litellm.success_callback or [] | ||
| if _success_callback not in litellm.success_callback: | ||
| litellm.success_callback.append(_success_callback) | ||
|
|
||
| litellm.failure_callback = litellm.failure_callback or [] | ||
| if _failure_callback not in litellm.failure_callback: | ||
| litellm.failure_callback.append(_failure_callback) |
There was a problem hiding this comment.
It seems as if both success_callback and failure_callback are run in a thread, which might finish after completion returns. As the span is closed in either callback, it may occur that the span is finished after the surrounding transaction does, resulting it being absent completely. This should definitely be pointed out somewhere.
There was a problem hiding this comment.
There is definitely the potential for a timing issue but I don't see a way around it at the moment since the LiteLLM integration might not be in control of the overarching transaction.
From your testing when developing this, was this a real issue when something like a web framework was managing the transaction?
There was a problem hiding this comment.
It was only an issue when writing code like this:
with sentry_sdk.start_transaction(...):
result = completion(...)When using it in a framework (tried with FastAPI) I could not reproduce this error.
| params = { | ||
| "model": SPANDATA.GEN_AI_REQUEST_MODEL, | ||
| "stream": SPANDATA.GEN_AI_RESPONSE_STREAMING, | ||
| "max_tokens": SPANDATA.GEN_AI_REQUEST_MAX_TOKENS, | ||
| "presence_penalty": SPANDATA.GEN_AI_REQUEST_PRESENCE_PENALTY, | ||
| "frequency_penalty": SPANDATA.GEN_AI_REQUEST_FREQUENCY_PENALTY, | ||
| "temperature": SPANDATA.GEN_AI_REQUEST_TEMPERATURE, | ||
| "top_p": SPANDATA.GEN_AI_REQUEST_TOP_P, | ||
| } |
There was a problem hiding this comment.
It is not clear where to actually put these parameters in the arguments to completion
There was a problem hiding this comment.
Don't understand this comment, can you elaborate? What do the params have to do with completion?
There was a problem hiding this comment.
completion takes quite generic kwargs that are then passed on to the model provider API. These above are used for openai (at least I suspect that this is where and how we retrieve them)
![seer-by-sentry[bot]](https://avatars.githubusercontent.com/in/801464?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| params = { | ||
| "model": SPANDATA.GEN_AI_REQUEST_MODEL, | ||
| "stream": SPANDATA.GEN_AI_RESPONSE_STREAMING, | ||
| "max_tokens": SPANDATA.GEN_AI_REQUEST_MAX_TOKENS, | ||
| "presence_penalty": SPANDATA.GEN_AI_REQUEST_PRESENCE_PENALTY, | ||
| "frequency_penalty": SPANDATA.GEN_AI_REQUEST_FREQUENCY_PENALTY, | ||
| "temperature": SPANDATA.GEN_AI_REQUEST_TEMPERATURE, | ||
| "top_p": SPANDATA.GEN_AI_REQUEST_TOP_P, | ||
| } |
There was a problem hiding this comment.
Don't understand this comment, can you elaborate? What do the params have to do with completion?
| litellm.success_callback = litellm.success_callback or [] | ||
| if _success_callback not in litellm.success_callback: | ||
| litellm.success_callback.append(_success_callback) | ||
|
|
||
| litellm.failure_callback = litellm.failure_callback or [] | ||
| if _failure_callback not in litellm.failure_callback: | ||
| litellm.failure_callback.append(_failure_callback) |
There was a problem hiding this comment.
There is definitely the potential for a timing issue but I don't see a way around it at the moment since the LiteLLM integration might not be in control of the overarching transaction.
From your testing when developing this, was this a real issue when something like a web framework was managing the transaction?
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
TY for incorporating my feedback, LGTM. From my POV there are 2 outstanding points:
LMK if you need support with any of these or if you want my feedback on anything else. |
I've added tests and added the minimum versions to the list. I hope that works out.
Done |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
<!-- Use this checklist to make sure your PR is ready for merge. You may delete any sections you don't need. --> ## DESCRIBE YOUR PR Add docs for the LiteLLM Python SDK integration. Requires getsentry/sentry-python#4864 Closes https://linear.app/getsentry/issue/TET-1217/litellm-docs ## IS YOUR CHANGE URGENT? Help us prioritize incoming PRs by letting us know when the change needs to go live. - [ ] Urgent deadline (GA date, etc.): <!-- ENTER DATE HERE --> - [ ] Other deadline: <!-- ENTER DATE HERE --> - [x] None: Not urgent, can wait up to 1 week+ ## SLA - Teamwork makes the dream work, so please add a reviewer to your PRs. - Please give the docs team up to 1 week to review your PR unless you've added an urgent due date to it. Thanks in advance for your help! ## PRE-MERGE CHECKLIST *Make sure you've checked the following before merging your changes:* - [ ] Checked Vercel preview for correctness, including links - [ ] PR was reviewed and approved by any necessary SMEs (subject matter experts) - [ ] PR was reviewed and approved by a member of the [Sentry docs team](https://github.com/orgs/getsentry/teams/docs) ## LEGAL BOILERPLATE <!-- Sentry employees and contractors can delete or ignore this section. --> Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms. ## EXTRA RESOURCES - [Sentry Docs contributor guide](https://docs.sentry.io/contributing/) --------- Co-authored-by: Ivana Kellyer <ivana.kellyer@sentry.io>
Add a first implementation of the litellm integration, supporting completion and embeddings
Closes https://linear.app/getsentry/issue/PY-1828/add-agent-monitoring-support-for-litellm
Closes https://linear.app/getsentry/issue/TET-1218/litellm-testing
Note
Introduce
LiteLLMIntegrationthat instruments LiteLLM chat/embeddings calls with spans, token usage, optional prompt logging, and exception capture.sentry_sdk/integrations/litellm.pywithLiteLLMIntegrationregistering LiteLLMinput/success/failurecallbacks.chat/embeddings, setgen_ai.*metadata (provider/system, operation, model, params likemax_tokens,temperature,top_p,stream).api_base,api_version,custom_llm_provider.include_promptsand PII are enabled.usageand capture exceptions; always finish spans.Written by Cursor Bugbot for commit 1ecd559. This will update automatically on new commits. Configure here.