Skip to content

Conversation

AkisAya
Copy link
Contributor

@AkisAya AkisAya commented Sep 29, 2025

What type of PR is this?

fix:
add a workaround to preserve stream filed in user raw request, because openai-go's openai.ChatCompletionNewParams doesn't have this field, see issue #209

What this PR does / why we need it:

stream mode is not supported well, when user use an auto model, stream filed in of user raw input is dropped, so a non-sse response is returned , this pr reuse the ExpectStreamingResponse field introduced by this pr https://github.com/vllm-project/semantic-router/pull/203/files, to identify whether this request expect a stream response

  1. detect if the request header Accept: text/event-stream
  2. extract stream field from raw request body

and finally return a mutated request body to envoy with stream field

Which issue(s) this PR fixes:

Fixes #209

Release Notes: No

Copy link

netlify bot commented Sep 29, 2025

Deploy Preview for vllm-semantic-router ready!

Name Link
🔨 Latest commit f54e71d
🔍 Latest deploy log https://app.netlify.com/projects/vllm-semantic-router/deploys/68dbbe735722f800083ef644
😎 Deploy Preview https://deploy-preview-282--vllm-semantic-router.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Copy link

github-actions bot commented Sep 29, 2025

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned:

📁 src

Owners: @rootfs, @Xunzhuo, @wangchen615
Files changed:

  • src/semantic-router/pkg/extproc/request_handler.go

vLLM

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

@rootfs
Copy link
Collaborator

rootfs commented Sep 29, 2025

@tao12345666333 can you review this? thanks

@rootfs
Copy link
Collaborator

rootfs commented Sep 29, 2025

@AkisAya can you sign DCO?

Ensure you have a local copy of your branch by [checking out the pull request locally via command line](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/checking-out-pull-requests-locally).
In your local branch, run: git rebase HEAD~2 --signoff
Force push your changes to overwrite the branch: git push --force-with-lease origin fix/stream-mode

@tao12345666333
Copy link
Contributor

@tao12345666333 can you review this? thanks

Yes, please assign to me, I will review this PR tomorrow

Signed-off-by: akisaya <akikevinsama@gmail.com>
@AkisAya AkisAya closed this Sep 30, 2025
@AkisAya AkisAya reopened this Sep 30, 2025
@AkisAya
Copy link
Contributor Author

AkisAya commented Sep 30, 2025

@AkisAya can you sign DCO?

Ensure you have a local copy of your branch by [checking out the pull request locally via command line](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/checking-out-pull-requests-locally).
In your local branch, run: git rebase HEAD~2 --signoff
Force push your changes to overwrite the branch: git push --force-with-lease origin fix/stream-mode

signed

@Xunzhuo
Copy link
Member

Xunzhuo commented Sep 30, 2025

have u tested when u set stream=true and got stream response?

@AkisAya
Copy link
Contributor Author

AkisAya commented Sep 30, 2025

have u tested when u set stream=true and got stream response?

sure.

you can get a stream response if your request body have stream: true or set a header Accept: text/event-stream

  1. with stream field or with a header
image
  1. without stream field or the header
image

@Xunzhuo Xunzhuo changed the title fix stream mode feat: add stream mode support Sep 30, 2025
@Xunzhuo Xunzhuo merged commit 53bb830 into vllm-project:main Sep 30, 2025
8 checks passed
@Xunzhuo
Copy link
Member

Xunzhuo commented Sep 30, 2025

tested it and working great, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

stream mode is not supported
5 participants