Unable to Connect to Triton Server at localhost:8000

[osuzdalev]

Description
I am experiencing a connection error when trying to access Triton's ready endpoint to verify that the server and the models are ready for inference. When using the command curl -v localhost:8000/v2/health/ready, I receive the error message
"Failed to connect to localhost port 8000 after 4 ms: Connection refused."

Triton Information
I am using Triton version 23.01-py3 running in a docker container. I pulled the image from nvcr.io/nvidia/tritonserver:23.01-py3.

To Reproduce
Run the Triton container using the command docker run --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:23.01-py3 tritonserver --model-repository=/models.

Verify Triton Is Running Correctly with curl -v localhost:8000/v2/health/ready

Expected behavior
I expected the command curl -v localhost:8000/v2/health/ready to return a status code of 200, indicating that Triton is ready and the models are ready for inference.

Environement
I am on a MacBook Pro with an M1 chip.

[krishung5]

Hi @osuzdalev, we don't officially support Mac OS, but some things that's just off the top of my head:

1. Running Trion with logging enabled should provide more context. Adding this flag to the docker run command --log-verbose=1
2. Is the port 8000 set correctly? Is there other process running on the same port?

[osuzdalev]

Hi @krishung5, thank you for answering!

I added the flag and attached the logs in text files below.

When I run:
$ curl -v localhost:8000/v2/health/ready

I still get the error:

*   Trying 127.0.0.1:8000...
* connect to 127.0.0.1 port 8000 failed: Connection refused
*   Trying ::1:8000...
* connect to ::1 port 8000 failed: Connection refused
* Failed to connect to localhost port 8000 after 6 ms: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 8000 after 6 ms: Connection refused

However I have found that sending an inference request from the Triton SDK container works. (logs attached)

When I try to call the python exemples like "image_client.py" it doesn't, I get the following:

$ python image_client.py -v -m inception_graphdef -s INCEPTION qa/images/mug.jpg                                                                             

GET /v2/models/inception_graphdef, headers None
Traceback (most recent call last):
  File "/Users/osuz/PycharmProjects/ZEISS/venv/lib/python3.9/site-packages/geventhttpclient/connectionpool.py", line 163, in get_socket
    return self._socket_queue.get(block=False)
  File "src/gevent/queue.py", line 335, in gevent._gevent_cqueue.Queue.get
  File "src/gevent/queue.py", line 350, in gevent._gevent_cqueue.Queue.get
  File "src/gevent/queue.py", line 319, in gevent._gevent_cqueue.Queue._Queue__get_or_peek
    '''Return the approximate size of the queue (not reliable!).'''
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/osuz/PycharmProjects/ZEISS/image_client.py", line 324, in <module>
    model_metadata = triton_client.get_model_metadata(
  File "/Users/osuz/PycharmProjects/ZEISS/venv/lib/python3.9/site-packages/tritonclient/http/__init__.py", line 534, in get_model_metadata
    response = self._get(request_uri=request_uri,
  File "/Users/osuz/PycharmProjects/ZEISS/venv/lib/python3.9/site-packages/tritonclient/http/__init__.py", line 269, in _get
    response = self._client_stub.get(request_uri)
  File "/Users/osuz/PycharmProjects/ZEISS/venv/lib/python3.9/site-packages/geventhttpclient/client.py", line 266, in get
    return self.request(METHOD_GET, request_uri, headers=headers)
  File "/Users/osuz/PycharmProjects/ZEISS/venv/lib/python3.9/site-packages/geventhttpclient/client.py", line 226, in request
    sock = self._connection_pool.get_socket()
  File "/Users/osuz/PycharmProjects/ZEISS/venv/lib/python3.9/site-packages/geventhttpclient/connectionpool.py", line 166, in get_socket
    return self._create_socket()
  File "/Users/osuz/PycharmProjects/ZEISS/venv/lib/python3.9/site-packages/geventhttpclient/connectionpool.py", line 127, in _create_socket
    raise first_error
  File "/Users/osuz/PycharmProjects/ZEISS/venv/lib/python3.9/site-packages/geventhttpclient/connectionpool.py", line 114, in _create_socket
    sock = self._connect_socket(sock, sock_info[-1])
  File "/Users/osuz/PycharmProjects/ZEISS/venv/lib/python3.9/site-packages/geventhttpclient/connectionpool.py", line 136, in _connect_socket
    sock.connect(address)
  File "/Users/osuz/PycharmProjects/ZEISS/venv/lib/python3.9/site-packages/gevent/_socketcommon.py", line 607, in connect
    raise _SocketError(err, strerror(err))
ConnectionRefusedError: [Errno 61] Connection refused

sdk_log.txt
server_log.txt

The port 8000 is not used by any other process.

[dyastremsky]

This isn't an issue with Triton. This is a Docker networking issue specific to Macs.

According to the Docker documentation, The host networking driver only works on Linux hosts, and is not supported on Docker Desktop for Mac, Docker Desktop for Windows, or Docker EE for Windows Server. I believe this is because Docker in Mac is run in a Linux virtual machine so the host shares the virtual machine's network instead of the host's.

You may have to do more research to see how to properly expose/forward the port to have it accessible between containers on a Mac. The above should work once the networking is properly set up.