Dify with OIlama keeps restarting

[dougpiv]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Dify version

8.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I am experiencing a strange behavior when I run the workflow in Dify. It makes the call to Ollama, but it is restarting the model at each iteration. It didn’t do this in the previous version 7.3. The model is llama3.1.

✔️ Expected Behavior

I expect that the memory on gpu keeps full and dont restart de model every iteration

❌ Actual Behavior

Every iteration the model restarts on ollama

[dosubot]

Hey @dougpiv! 👋 I'm here to help you with any bugs, questions, or contributions you have for Dify with OIlama! I'm an expert bot and I'll assist you while we wait for a human maintainer to jump in.

I found a similar discussion regarding the Ollama model restarting at each iteration in Dify version 8.2. The issue is due to a connection problem between Docker containers and the Ollama service. The Docker container cannot access the Ollama service using localhost. This can be resolved by configuring the service to use host.docker.internal instead of localhost:

http://host.docker.internal:11434

This change ensures that the Docker container can correctly access the Ollama service, preventing the model from restarting at each iteration. Additionally, if you are deploying Dify and Ollama using Docker, consider using the local network IP address, e.g., http://192.168.1.100:11434 or the Docker host machine IP address, e.g., http://172.17.0.1:11434 ^[1][2].

For more details, you can refer to the discussion on this issue ^[3].

To continue talking to Dosu, mention @dosu.

[dougpiv]

I am already using host.docker.internal and the connection is working. What is happening is that it seems to be restarting the model on the GPU with every request in a cycle in Dify.

[dosubot]

The model on the GPU is likely restarting with every request in a cycle because the Ollama service is not properly exposed to the network. Ensure that the Ollama service is accessible from the Docker container by setting the OLLAMA_HOST environment variable to 0.0.0.0 to bind the service to all network interfaces.

For macOS:

launchctl setenv OLLAMA_HOST "0.0.0.0"

For Linux:

[Service]
Environment="OLLAMA_HOST=0.0.0.0"

For Windows:

1. Quit Ollama by clicking on it in the taskbar.
2. Edit system environment variables from the control panel.
3. Edit or create a new variable for your user account: OLLAMA_HOST=0.0.0.0.
4. Click OK/Apply to save.
5. Run ollama from a new terminal window.

Additionally, if you are deploying Dify using Docker, use the local network IP address or the Docker host machine's IP address instead of localhost:

http://192.168.1.100:11434

or

http://172.17.0.1:11434

Ensure that the Dify container can access the endpoint of Xinference if you are using it for local model deployment. Replace localhost with your machine's IP address in the Dify settings:

http://<Machine_IP>:9997

Start Xinference with the -H 0.0.0.0 option to allow access from any non-local client:

xinference-local -H 0.0.0.0

Finally, ensure that the DIFY_BIND_ADDRESS environment variable in your docker-compose.yaml is set to 0.0.0.0:

DIFY_BIND_ADDRESS: ${DIFY_BIND_ADDRESS:-0.0.0.0}

This configuration should prevent the model from restarting with every request ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.