How to deploy Triton on Vertex AI?

[btseitlinatwork]

Hello

I am excited to use Triton with Google Cloud Vertex AI, but I can't seem to find a way.

I followed these tutorials:

• https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/vertex_endpoints/nvidia-triton/get_started_with_triton_ensemble.ipynb
• https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/vertex_endpoints/nvidia-triton/nvidia-triton-custom-container-prediction.ipynb
• https://github.com/NVIDIA/nvidia-gcp-samples/blob/master/vertex-ai-samples/prediction/triton_inference.ipynb

None of them produced a working deployment. I am able to build the container (or rather rename the NGC container and upload it to google artifact registry). But deployment of the model to a Vertex endpoint fails with a non-descriptive error:

exec /opt/tritonserver/nvidia_entrypoint.sh: exec format error

It seems like the container is receiving some unexpected command. But I don't understand what it was: there are no logs.

Does anyone have an up-to-date guide on how to deploy to Vertex?

Also, I see that all questions and discussions of Triton on Vertex are from 2022 and older. Should I look into deploying on GKE or another solution altogether?