Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Triton RuntimeStatus.MethodInfos is missing ModelStreamInfer #80

Open
Legion2 opened this issue Jan 20, 2024 · 1 comment · May be fixed by #81
Open

Triton RuntimeStatus.MethodInfos is missing ModelStreamInfer #80

Legion2 opened this issue Jan 20, 2024 · 1 comment · May be fixed by #81
Labels
enhancement New feature or request

Comments

@Legion2
Copy link

Legion2 commented Jan 20, 2024

Triton provides an extension to the standard gRPC inference api for streaming (inference.GRPCInferenceService/ModelStreamInfer), this extension is required to use vLLM backend with triton.
However currently the triton runtime adapter does not advertise the existence of this gRPC method and trying to call it results in an error (inference.GRPCInferenceService/ModelStreamInfer: UNIMPLEMENTED: Method not found or not permitted: inference.GRPCInferenceService/ModelStreamInfer)

To resolve this issue, I think the ModelStreamInfer method must be added here:

modelmesh-runtime-adapter/model-mesh-triton-adapter/server/server.go

Lines 267 to 269 in f9781d2

mis := make(map[string]*mmesh.RuntimeStatusResponse_MethodInfo)
mis[tritonServiceName+"/ModelInfer"] = &mmesh.RuntimeStatusResponse_MethodInfo{IdInjectionPath: path1}
mis[tritonServiceName+"/ModelMetadata"] = &mmesh.RuntimeStatusResponse_MethodInfo{IdInjectionPath: path1}

Legion2 added a commit to Legion2/modelmesh-runtime-adapter that referenced this issue Jan 21, 2024
@Legion2
fix: Add ModelStreamInfer to triton MethodInfos (kserve#80)
81f6d8b
@Legion2 Legion2 linked a pull request Jan 21, 2024 that will close this issue
Open
@Legion2
Copy link
Author

Legion2 commented Jan 21, 2024

I have created a PR #81 and tested in our environment that the ModelStreamInfer requests work with the patch.

Legion2 added a commit to Legion2/modelmesh-runtime-adapter that referenced this issue Jan 21, 2024
@Legion2
fix: Add ModelStreamInfer to triton MethodInfos (kserve#80)
9ce4393 
Signed-off-by: Leon Kiefer <leon.k97@gmx.de>
@rafvasq rafvasq added the enhancement New feature or request label Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: Add ModelStreamInfer to Triton MethodInfos Legion2/modelmesh-runtime-adapter
2 participants
@rafvasq @Legion2