bentoml.exceptions.BentoMLRpcError: Failed to upload MyBentoService to the remote yatai server: Deadline Exceeded

[BartlomiejSkwira]

I have a problem uploading my BentoService to a local Yatai repository. I have 2 separate containers running on my local laptop: one for saving my service with artefacts (models) and second as a Yatai repository server:

1. A Yatai repository server is running under Docker via command bentoml yatai-service-start --grpc-port=${YATAI_GRPC_PORT} --ui-port=${YATAI_UI_PORT} --verbose \. YATAI_GRPC_PORT is set to50050 and exposed outside this container
2. The other container is responsible for packing and saving MyBentoService. The code is a bit like so:

my_service = MyBentoService()

# Pack trained model artifacts
for model_package in models: # this is a list of 20 models with name, model and metadata
    name = model_package["name"]
    model = model_package["model"]
    metadata = model_package["metadata"]

    my_service.pack(name, model, metadata=metadata)

# Save
yatai_url = "http://host.docker.internal:50050" #docker communication between separate containers 
saved_path = my_service.save(yatai_url=yatai_url, labels=labels)

When I run this pack/save I get this:

[2021-08-21 17:46:46,368] WARNING - pip package requirement pytorch-lightning already exist
[2021-08-21 17:46:46,368] INFO - PytorchLightningArtifact is packing a pytorch lightning model instance as torchscript module
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/bentoml/yatai/client/bento_repository_api.py", line 598, in _upload_bento
    timeout=DEFAULT_GRPC_REQUEST_TIMEOUT,
  File "/opt/conda/lib/python3.7/site-packages/grpc/_interceptor.py", line 360, in __call__
    compression=compression)
  File "/opt/conda/lib/python3.7/site-packages/grpc/_interceptor.py", line 396, in _with_call
    return call.result(), call
  File "/opt/conda/lib/python3.7/site-packages/grpc/_channel.py", line 343, in result
    raise self
  File "/opt/conda/lib/python3.7/site-packages/grpc/_interceptor.py", line 386, in continuation
    compression=new_compression)
  File "/opt/conda/lib/python3.7/site-packages/grpc/_channel.py", line 1142, in with_call
    return _end_unary_response_blocking(state, call, True, None)
  File "/opt/conda/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "{"created":"@1629568020.965692100","description":"Error received from peer ipv4:192.168.65.2:50050","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scripts/upload_models.py", line 95, in <module>
    main()
  File "scripts/upload_models.py", line 90, in main
    yatai_url=args.yatai_url,
  File "scripts/upload_models.py", line 78, in upload_models_to_yatai
    pack_and_save_models(models=models_to_upload, labels=labels, yatai_url=yatai_url)
  File "/code/src/model_management/bento_packer.py", line 53, in pack_and_save_models
    saved_path = lstm_classifier_service.save(yatai_url=yatai_url, labels=labels)
  File "/opt/conda/lib/python3.7/site-packages/bentoml/service/__init__.py", line 739, in save
    return yc.repository.upload(self, version, labels)
  File "/opt/conda/lib/python3.7/site-packages/bentoml/yatai/client/bento_repository_api.py", line 162, in upload
    return self.upload_from_dir(tmpdir, labels)
  File "/opt/conda/lib/python3.7/site-packages/bentoml/yatai/client/bento_repository_api.py", line 216, in upload_from_dir
    saved_bento_path,
  File "/opt/conda/lib/python3.7/site-packages/bentoml/yatai/client/bento_repository_api.py", line 605, in _upload_bento
    f'Failed to upload {bento_name}:{bento_version} to '
bentoml.exceptions.BentoMLRpcError: Failed to upload LstmBentoService:1.1.20210821174651_1C04CB to the remote yatai server: Deadline Exceeded

The connection is established, Yatai server in debug prints this:

bentoml_yatai     | [2021-08-21 17:46:53,320] DEBUG - Session acquired
bentoml_yatai     | [2021-08-21 17:46:53,320] DEBUG - 	READ on LstmBentoService_1.1.20210821174651_1C04CB
bentoml_yatai     | [2021-08-21 17:46:53,332] DEBUG - Session released after 0.011773824691772461s
bentoml_yatai     | [2021-08-21 17:46:54,143] DEBUG - Session acquired
bentoml_yatai     | [2021-08-21 17:46:54,143] DEBUG - 	WRITE on LstmBentoService_1.1.20210821174651_1C04CB
bentoml_yatai     | [2021-08-21 17:46:54,156] DEBUG - Session released after 0.0133819580078125s
bentoml_yatai     | [2021-08-21 17:46:54,971] DEBUG - Created temporary directory: /tmp/bentoml-temp-zx95v8od
bentoml_yatai     | [2021-08-21 17:47:00,967] DEBUG - BentoML in debug mode, keeping temp directory "/tmp/bentoml-temp-zx95v8od"
bentoml_yatai     | [2021-08-21 17:47:00,969] ERROR - RPC ERROR UploadBento:

Not sure how to tackle such an issue. Is this a docker/container issue or maybe something is not right with my pack/save sequence? Any advice would be very appreciated.

[BartlomiejSkwira]

So I tried to save my service to an online Yatai instance and it worked. Seems that this is a local Docker connection/configuration issue

[cateto]

hello @BartlomiejSkwira
what is online yatai instance? more detail plz.
I have same problem.

[BartlomiejSkwira]

I was using BentoML with Yatai, you can read about it more here https://github.com/bentoml/Yatai in short it's a repository + UI to manage models/deployments and more.

So my problem and the exception above was on local (my laptop) docker-compose setup. When I run a production Yatai instance the models were uploaded correctly. @cateto debug where is your deployment destination and if Bento is able to connect to it.

[aarnphm]

Hi, https://github.com/bentoml/Yatai is only compatible for 1.0 and above. For 0.13 we have a different architecture for yatai. I believe that the RpcError is saying that the bento could not be uploaded. So I guess check for your yatai connection and settings.