load local visual language models

[Adorablepet]

Currently, the visual model is downloaded locally, and when the service starts, it is deployed and invoked through vLLM. Will it later be supported to call a model that has already been deployed on another server via an HTTP API?thanks.

[github-actions]

👋 Welcome! Thanks for opening your first issue. If you'd like to take a crack at fixing it, feel free to open a pull request — otherwise, we'll take a look as soon as we can!

[mandalsouvik3333]

We already support this. You can start the docext app with these args

--vlm_server_host: Host where the vLLM/OLLAMA server running
--vlm_server_port: Port for the vLLM/OLLAMA server
--model_name: Name of the model to use. Use 'ollama/' prefix for OLLAMA models and 'hosted_vllm/' prefix for hosted vLLM models.

export API_KEY=<your_api_key> # incase you have used a API key to host the model
python -m docext.app.app --model_name hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct-AWQ --vlm_server_host <your_ip> --vlm_server_port <your_port>

Let me know if you face any issues

[Adorablepet]

We already support this. You can start the docext app with these args

--vlm_server_host: Host where the vLLM/OLLAMA server running
--vlm_server_port: Port for the vLLM/OLLAMA server
--model_name: Name of the model to use. Use 'ollama/' prefix for OLLAMA models and 'hosted_vllm/' prefix for hosted vLLM models.

export API_KEY=<your_api_key> # incase you have used a API key to host the model
python -m docext.app.app --model_name hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct-AWQ --vlm_server_host <your_ip> --vlm_server_port <your_port>

Let me know if you face any issues

ok, i will try it,thanks.