[Feature]: LiteLLM MCP Roadmap #9891
Replies: 7 comments 23 replies
-
Beta Was this translation helpful? Give feedback.
-
|
Can this be made into a discussion? @ishaan-jaff it'll be easier to track |
Beta Was this translation helpful? Give feedback.
-
|
done |
Beta Was this translation helpful? Give feedback.
-
|
I think a very common problem is that very quickly the number of MCP servers you have installed explodes. I think that makes prompt engineering much more difficult because you have to carefully tune your prompt that it's not choosing the wrong tools for the task you plan to do. So I think it would be a great feature if the proxy would have several endpoints to offer different subsets of MCP servers that are installed so that you can have one endpoint for your IDE focused on software development tools and another for a general chat client. Just a mechanism to have different profiles with different sets of MCP servers available. |
Beta Was this translation helpful? Give feedback.
-
|
Could we leverage something like tags here? https://docs.litellm.ai/docs/proxy/tag_routing |
Beta Was this translation helpful? Give feedback.
-
|
I wonder if you should leave room for the possibility of doing this, but not invest in it right away. LLMs are pretty smart and I don't know that picking the wrong tools would happen that often unless you have a lot of very similar tools (maybe some people have that use case; I don't see that being an issue for me). Maybe try it and then solve if that pops up a lot? But make sure whatever you do API-wise leaves room for that possibility. Maybe you can throw |
Beta Was this translation helpful? Give feedback.
-
|
Another thought off the top of my head - what if there was a way to specify which tools are automatically used based on prompt language and which aren't (and have to be explicitly requested)? That lets you specify that say the "check deployment status" tool can be invoked automatically but the "do deployment tool" has to be explicitly requested so there are no accidental deploys. Could allow that on the tool level or if you end up doing tool collections, it could be specified on the tool collection level - i.e.: idempotent_tools and non_idempotent_tools. |
Beta Was this translation helpful? Give feedback.
-
|
@ishaan-jaff Yes, now I get it. That would probably do the trick. At least on a client level. I think many users would like to remove the necessity to manage multiple sets MCPs for different clients. But have one central place to manage and maintain the MCP installations while still be able to have different sets of MCPs for different applications. |
Beta Was this translation helpful? Give feedback.
-
|
I like the "segmentation" idea too. If you can narrow down the types of tools you want to expose to the LLM, that can only help.
This is also a good one. It would definitely be good to be able to limit access to certain MCPs by org/team membership. |
Beta Was this translation helpful? Give feedback.
-
|
Would love to tackle this one for the initial proxy features:
One thing to consider is restricting litellm to only use HTTP streamable transport since SSE requires a constant connection. If we wanted to support SSE, we would need to either stay connected consistently or only connect on tool runs and periodically listen for new changes. I think the authentication is an important discussion and would be curious to see where you guys are going with that requirement. It makes sense to include a pass through of a token from the litellm proxy to the MCP servers for specific authn/authz at the server. Do we also want litellm proxy to be able store it's own cert to generate jwt like the new feature? |
Beta Was this translation helpful? Give feedback.
-
|
I made a bunch of progress on the backend parts for this. Essentially the db and config for hosting the servers will look like model LiteLLM_MCPServerTable {
server_id String @id @default(uuid())
alias String?
description String?
url String
transport String @default("http")
spec_version String @default("2025-03-26")
auth_type String?
created_at DateTime? @default(now()) @map("created_at")
created_by String?
updated_at DateTime? @default(now()) @updatedAt @map("updated_at")
updated_by String?
}and there will be the following defaults: transport: "http", # one of (http|sse) - defaults to sse
spec_version: "2025-03-26", # one of (2024-11-05|2025-03-26) - defaults to 2025-03-26
auth_type: API_KEY, # one of (API_KEY|BASIC|BEARER_TOKEN) defaults to no authNow just waiting for the streamable http to come out this week to implement the last parts of the backend changes here. For context why I am pushing for streamable http support for litellm's proxy instead of SSE is the constant connection needed and other gunicorn and workers issues when running as explained here. In the meantime, I will start working towards the frontend and tests this week. If @ishaan-jaff or @krrishdholakia has time to connect, that would be great since there will be breaking changes to the proxy with this PR and I want to validate that is okay :)! As for the duplicate tools defined in #10404, I am hoping the team + virtual key additions are enough, but if not we can look towards prefixing the tools with the server's alias i.e. |
Beta Was this translation helpful? Give feedback.
-
Is there any intention to extend this to be user-specific per upstream? Specifically looking at using LiteLLM as a proxy layer for multi-tenant applications, and would want to pass through user specific auth credentials per request. |
Beta Was this translation helpful? Give feedback.
-
I am in the same boat. There is authentication to the LiteLLM proxy to limit which MCP servers the requestor has access to. There still needs some way to authenticate to the MCP server (if auth is required) when requesting for tools. I have a few ideas, but there is no design yet if you have any suggestions |
Beta Was this translation helpful? Give feedback.
-
@wasaga is taking a look over on the Pomerium side, but we realized we might be converging on a similar solution. He might have some input on design here |
Beta Was this translation helpful? Give feedback.
-
|
Can anyone help me in understanding how LiteLLM MCP related to the MCP registry and MCP Proxy discussion here in below links : modelcontextprotocol/registry#11 https://github.com/orgs/modelcontextprotocol/discussions/73 Is LiteLLM MCP a registry implementation or a proxy ? What are the plans to adopt the official registry implementation. |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Additionally, does Prometheus have monitoring for the MCP service? |
Beta Was this translation helpful? Give feedback.
-
|
Heya @harvardfly -- litellm proxy now supports the basic adding, editing, and deleting MCP servers in a least privileged principle approach supporting SSE servers at the moment. Since the functionality from the UI is not there for configuring MCP servers on keys and teams just yet, only admins will have access to these configured servers. For your question about Prometheus, are you looking to get observations from litellm proxy connecting to the server and calling tools? Or does the MCP server itself connect |
Beta Was this translation helpful? Give feedback.
-
|
@wagnerjt get observations from litellm proxy connecting to the server and calling tools，Similar to how model services are called through the LiteLLM proxy, it should be possible to view monitoring metrics such as the number of calls through Prometheus. |
Beta Was this translation helpful? Give feedback.
-
|
@wagnerjt We look forward to the prompt implementation of the following features:
|
Beta Was this translation helpful? Give feedback.
-
|
Hi team, One point I’d like to raise for consideration is the importance of forwarding user credentials (or some representation of the user identity, such as a JWT or access token) to the MCP, in addition to the LiteLLM API key. This would allow the MCP to enforce fine-grained authentication and authorization based on the actual end user’s identity. This is especially relevant when: Access to specific tools or actions in the MCP is restricted per user or role. Audit logs or usage policies need to reflect the individual user, not just the API key. Multi-tenant scenarios require isolation at the user level. Ensuring that user-level context is passed through LiteLLM to the MCP would enable better security, traceability, and policy enforcement. Is this something that’s already on the roadmap or being considered? |
Beta Was this translation helpful? Give feedback.
-
|
I'm with you! I know I would need this so I made the following ways litellm could proxy by supporting the auth types Just not sure how we want to pass in the extra auth when making the request since I personally use the Authorization header of |
Beta Was this translation helpful? Give feedback.
-
|
OpenAI mcp tools + anthropic allow passing this in the tool definition - https://platform.openai.com/docs/api-reference/responses/create#responses-create-tools from openai import OpenAI
client = OpenAI()
resp = client.responses.create(
model="gpt-4.1",
tools=[
{
"type": "mcp",
"server_label": "deepwiki",
"server_url": "https://mcp.deepwiki.com/mcp",
"require_approval": "never",
"headers": {..} # Note: Anthropic only supports a bearer token
},
],
input="What transport protocols are supported in the 2025-03-26 version of the MCP spec?",
)
print(resp.output_text) |
Beta Was this translation helpful? Give feedback.
-
|
This is great! I'll read up more on it |
Beta Was this translation helpful? Give feedback.
-
|
What is the roadmap for other mcp resource types like resources and prompts? It seems like in the docs the only supported type right now are tools? Is it possible to build in a way in the future that each new resource type doesn't need additional dev work from litellm to support the new mcp protocol option? Using litellm for a prompt library and controlling access across teams would be really cool since you get the litellm access controls for free without having to build it into the mcp server |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
The Feature
Starting this to discuss LiteLLM MCP Roadmap
Demo Video
Existing MCP walkthrough here
Are you a ML Ops Team?
No
Twitter / LinkedIn details
No response
Beta Was this translation helpful? Give feedback.
All reactions