Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support for Ascend NPU backend #3853

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

botieking98
Copy link

@botieking98 botieking98 commented Feb 25, 2025

Motivation

Support for Ascend NPU backend

Modifications

inspired by #3782
and also support run on multiple NPU devices.

Checklist

Test result

python -m sglang.launch_server --model-path Qwen/Qwen2.5-72B-Instruct --device npu --grammar-backend outlines --attention-backend torch_native --tp 16 --port 40000 --host 0.0.0.0 --trust-remote-code

image

22dimensions and others added 2 commits February 25, 2025 20:39
Signed-off-by: Song Zhang <gepin.zs@antgroup.com>
@22dimensions
Copy link

We are very happy to see that someone is also interested in sglang on Ascend.

Thanks for moving this forward, would you mind linking #3781 to make more people know background in details? I'd like to work with you together to support Ascend NPU.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants