Skip to content

Conversation

rgerganov
Copy link
Collaborator

Allow rpc-server to expose multiple devices from a single endpoint. Change RPC protocol to include device identifier where needed. Add new API to get the device count from an RPC endpoint.

closes: #15210

Allow rpc-server to expose multiple devices from a single endpoint.
Change RPC protocol to include device identifier where needed.
Add new API to get the device count from an RPC endpoint.

closes: ggml-org#15210
@rgerganov
Copy link
Collaborator Author

@slaren I am still working on this but I'd appreciate some early feedback on the API changes in ggml-rpc.h

@github-actions github-actions bot added examples ggml changes relating to the ggml tensor library for machine learning labels Sep 26, 2025
@rgerganov rgerganov marked this pull request as ready for review September 30, 2025 08:51
@rgerganov
Copy link
Collaborator Author

I am also considering changing the naming scheme for RPC devices. Now I am using RPC<X>[<host>:<port>] for both name and description which doesn't look nice in various logs.

I am thinking of switching to RPC<X> for device name and [<host>:<port>] for device description. The X number should be global monotonic counter making RPC devices appear like this:

RPC0 ([localhost:50052])
RPC1 ([localhost:50052])
RPC2 ([localhost:18053])
RPC3 ([localhost:18054])
...

Thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
examples ggml changes relating to the ggml tensor library for machine learning
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature Request: Support multiple devices on a single rpc-server
2 participants