Dev/use vllm #1053
Merged
Dev/use vllm #1053
+1,145
−6
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- 新增基于队列和线程的异步推理机制 - 优化同步推理接口，使用新机制实现
- 删除了 LLM 类中的 async_llm_inference 方法 - 该方法尚未使用，且再在loop_thread之外运行后会导致 vllm 崩溃，因此将其移除
- 新增 speed_test.ipynb 文件，用于测试 CosyVoice2模型的性能 - 包含测试环境配置、默认情况下的使用示例、使用 vllm 加速 LLM 推理的步骤
- 移除任务队列和单任务处理限制 - 使用 asyncio.run_coroutine_threadsafe() 在后台线程中运行推理任务
|
@qi-hua 你好，感谢你的分享，我尝试你这个分支，想问下需要多少显存可以呢。gpu_memory_utilization在代码的哪块加呢 |
vllm需要大概3-4G的显存就可以了，gpu_memory_utilization的设置目前放在了cosyvoice/llm/llm_vllm.py:39 ENGINE_ARGS中，还没有暴露设置的位置，需要手动修改。 |
- 在 Frontend 中，恢复原本逐个生成文本令牌 - 在 Model 类中，移除了不必要的日志信息和断言，简化了文本令牌的处理流程
哦哦，感谢感谢，另外，我发现一个问题是，我按下面的测试脚本测试出现了下面的错误， |
|
我不了解这两种方式的区别，但默认会启动很多的进程。 |
|
@lyblsgo 麻烦帮忙看一下这个代码 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
目前使用 vllm 的 AsyncLLMEngine 加速推理
增加了 cosyvoice.llm.llm_vllm.VllmQwen2LM ，其他文件主要是简单修改；
VllmQwen2LM目前支持多任务推理，并发需对原接口适当修改；
使用trt的情况下，加速后效果 rtf 能够达到 0.1-0.15。