[Bug] Unable to spin up chatQnA on Xeon #972

skkama1h · 2024-10-16T23:07:46Z

Priority

P1-Stopper

OS type

Ubuntu

Hardware type

Xeon-SPR

Installation method

Pull docker images from hub.docker.com
Build docker images from source

Deploy method

Docker compose
Docker
Kubernetes
Helm

Running nodes

Single Node

What's the version?

c930bea

Description

Unable to spin up chatQnA containers using the latest commit - c930bea
It was working until last week. Last working commit on a Xeon: 2de1bfc

Error: When a query is presented, an "Internal Server Error" message is returned.

Docker logs points to "chatqna-xeon-backend-server" service referencing "embedding-tei" service which used to previously run on port 6000. This service seems to have been removed from the YAML file in the previous two commits.

Do not see any reference to embedding service running on port 6000 in the latest compose.yaml file c930bea

After reverting to 2de1bfc, containers work as expected and I am able to send queries and receive responses with no errors.

Reproduce steps

Clone chatQnA repo
Switch to xeon dir
export host_ip=<>, export no_proxy=<>
source set_env.sh
docker compose up -d
Upload sample PDF to chat (use nike.pdf as example)
Send chat query using curl command or UI. Receive "Internal Server Error"

Raw log

chatqna-xeon-backend-server  | [2024-10-16 23:01:37,171] [    INFO] - comps-core-orchestrator - {'text': 'What is the revenue of Nike in 2023?'}
chatqna-xeon-backend-server  | [2024-10-16 23:01:37,172] [    INFO] - comps-core-orchestrator - {'text': 'What is the revenue of Nike in 2023?'}
chatqna-xeon-backend-server  | INFO:     192.168.20.122:49540 - "POST /v1/chatqna HTTP/1.1" 500 Internal Server Error
chatqna-xeon-backend-server  | ERROR:    Exception in ASGI application
chatqna-xeon-backend-server  | Traceback (most recent call last):
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1073, in _wrap_create_connection
chatqna-xeon-backend-server  |     sock = await aiohappyeyeballs.start_connection(
chatqna-xeon-backend-server  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/aiohappyeyeballs/impl.py", line 104, in start_connection
chatqna-xeon-backend-server  |     raise first_exception
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/aiohappyeyeballs/impl.py", line 81, in start_connection
chatqna-xeon-backend-server  |     sock = await _connect_sock(
chatqna-xeon-backend-server  |            ^^^^^^^^^^^^^^^^^^^^
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/aiohappyeyeballs/impl.py", line 166, in _connect_sock
chatqna-xeon-backend-server  |     await loop.sock_connect(sock, address)
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 638, in sock_connect
chatqna-xeon-backend-server  |     return await fut
chatqna-xeon-backend-server  |            ^^^^^^^^^
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 678, in _sock_connect_cb
chatqna-xeon-backend-server  |     raise OSError(err, f'Connect call failed {address}')
chatqna-xeon-backend-server  | ConnectionRefusedError: [Errno 111] Connect call failed ('0.0.0.0', 6000)
chatqna-xeon-backend-server  | 
chatqna-xeon-backend-server  | 
chatqna-xeon-backend-server  | The above exception was the direct cause of the following exception:
chatqna-xeon-backend-server  | 
chatqna-xeon-backend-server  | 
chatqna-xeon-backend-server  | Traceback (most recent call last):
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
chatqna-xeon-backend-server  |     result = await app(  # type: ignore[func-returns-value]
chatqna-xeon-backend-server  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
chatqna-xeon-backend-server  |     return await self.app(scope, receive, send)
chatqna-xeon-backend-server  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
chatqna-xeon-backend-server  |     await super().__call__(scope, receive, send)
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__
chatqna-xeon-backend-server  |     await self.middleware_stack(scope, receive, send)
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
chatqna-xeon-backend-server  |     raise exc
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
chatqna-xeon-backend-server  |     await self.app(scope, receive, _send)
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 174, in __call__
chatqna-xeon-backend-server  |     raise exc
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 172, in __call__
chatqna-xeon-backend-server  |     await self.app(scope, receive, send_wrapper)
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
chatqna-xeon-backend-server  |     await self.app(scope, receive, send)
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
chatqna-xeon-backend-server  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
chatqna-xeon-backend-server  |     raise exc
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
chatqna-xeon-backend-server  |     await app(scope, receive, sender)
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
chatqna-xeon-backend-server  |     await self.middleware_stack(scope, receive, send)
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
chatqna-xeon-backend-server  |     await route.handle(scope, receive, send)
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
chatqna-xeon-backend-server  |     await self.app(scope, receive, send)
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
chatqna-xeon-backend-server  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
chatqna-xeon-backend-server  |     raise exc
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
chatqna-xeon-backend-server  |     await app(scope, receive, sender)
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
chatqna-xeon-backend-server  |     response = await f(request)
chatqna-xeon-backend-server  |                ^^^^^^^^^^^^^^^^
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
chatqna-xeon-backend-server  |     raw_response = await run_endpoint_function(
chatqna-xeon-backend-server  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
chatqna-xeon-backend-server  |     return await dependant.call(**values)
chatqna-xeon-backend-server  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chatqna-xeon-backend-server  |   File "/home/user/GenAIComps/comps/cores/mega/gateway.py", line 184, in handle_request
chatqna-xeon-backend-server  |     result_dict, runtime_graph = await self.megaservice.schedule(
chatqna-xeon-backend-server  |                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chatqna-xeon-backend-server  |   File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 67, in schedule
chatqna-xeon-backend-server  |     response, node = await done_task
chatqna-xeon-backend-server  |                      ^^^^^^^^^^^^^^^
chatqna-xeon-backend-server  |   File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 204, in execute
chatqna-xeon-backend-server  |     async with session.post(endpoint, json=inputs) as response:
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 1353, in __aenter__
chatqna-xeon-backend-server  |     self._resp = await self._coro
chatqna-xeon-backend-server  |                  ^^^^^^^^^^^^^^^^
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 657, in _request
chatqna-xeon-backend-server  |     conn = await self._connector.connect(
chatqna-xeon-backend-server  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 564, in connect
chatqna-xeon-backend-server  |     proto = await self._create_connection(req, traces, timeout)
chatqna-xeon-backend-server  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 975, in _create_connection
chatqna-xeon-backend-server  |     _, proto = await self._create_direct_connection(req, traces, timeout)
chatqna-xeon-backend-server  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1350, in _create_direct_connection
chatqna-xeon-backend-server  |     raise last_exc
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1319, in _create_direct_connection
chatqna-xeon-backend-server  |     transp, proto = await self._wrap_create_connection(
chatqna-xeon-backend-server  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chatqna-xeon-backend-server  |   File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1088, in _wrap_create_connection
chatqna-xeon-backend-server  |     raise client_error(req.connection_key, exc) from exc
chatqna-xeon-backend-server  | aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host 0.0.0.0:6000 ssl:default [Connect call failed ('0.0.0.0', 6000)]

The text was updated successfully, but these errors were encountered:

arun-gupta · 2024-10-17T01:18:58Z

Seems like this might be related #949

letonghan · 2024-10-17T05:52:32Z

Hi @skkama1h , this is caused by the update of this PR.
The embedding and reranking microservice wrappers are removed and set as default mode in ChatQnA, so the published version of docker images do not match the latest source code.
You need to you build the docker image from source following this instruction. And modify the docker image version in compose.yaml.
Also, the docker image of latest version will be soon published to docker hub. Once the latest images are updated, we will let you know.

skkama1h · 2024-10-17T15:35:10Z

Okay, thanks @letonghan.

letonghan · 2024-10-18T02:24:39Z

Hi @skkama1h , The latest version docker images which matched the main source code have been released and validated.
You can use the docker image opea/xxxx:latest to start up ChatQnA services and try it again. Thanks : )

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] Unable to spin up chatQnA on Xeon #972

[Bug] Unable to spin up chatQnA on Xeon #972

skkama1h commented Oct 16, 2024

arun-gupta commented Oct 17, 2024

letonghan commented Oct 17, 2024 •

edited

Loading

skkama1h commented Oct 17, 2024

letonghan commented Oct 18, 2024

[Bug] Unable to spin up chatQnA on Xeon #972

[Bug] Unable to spin up chatQnA on Xeon #972

Comments

skkama1h commented Oct 16, 2024

Priority

OS type

Hardware type

Installation method

Deploy method

Running nodes

What's the version?

Description

Reproduce steps

Raw log

arun-gupta commented Oct 17, 2024

letonghan commented Oct 17, 2024 • edited Loading

skkama1h commented Oct 17, 2024

letonghan commented Oct 18, 2024

letonghan commented Oct 17, 2024 •

edited

Loading