feat: Added instrumentation for chat completion streams #1884
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1884 +/- ##
=======================================
Coverage 96.87% 96.87%
=======================================
Files 209 209
Lines 39699 39759 +60
=======================================
+ Hits 38458 38518 +60
Misses 1241 1241
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
bizob2828
force-pushed
the
instrument-completion-stream
branch
2 times, most recently
from
0b3fdb1 to
963dc88
Compare
bizob2828
changed the title
bizob2828
force-pushed
the
instrument-completion-stream
branch
from
963dc88 to
4878711
Compare
bizob2828
force-pushed
the
instrument-completion-stream
branch
from
4878711 to
55be332
Compare
I am also convinced of this. It seems to silently swallow any errors thrown by the stream. Not knowing TS, it's really difficult to figure out all of their nested wrappers around their stream handling. |
jsumners-nr
previously approved these changes
Nov 27, 2023
…sdoc on an optional param
jsumners-nr
approved these changes
Nov 27, 2023
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds instrumentation for chat completion streams. It intercepts the stream generator and proxies the iterations through chunks. It will then update the segment time and record LLM events when it finishes. Errors are funky and thanks to @jsumners-nr for helping create a test for it. I'm convinced the openai library has a bug but didn't spend time completely tracking down. It's worth noting that we only instrument streams in 4.12.2+. The interface was different in past versions and not worth supporting both. In that case the completion span + llm events will not occur. We still will get child spans for the external http call.
I also refactored the versioned tests as 1 file was getting unwieldy. Lastly, I fixed the span duration and manually end segments. We typically end segments implicitly and at transaction finalization we will calculate the durations. However since we're using the segment duration in the LLM events they must be the same so we end the segment right before recording events. This will be fine but a little different from what we typically do.
Related Issues
Closes #1844