Best practice for multiple service serving

[tguner-arteriaai]

Hi everyone,

I am just wondering what are your thoughts on the best practice of serving multiple bentoml service with their own endpoints.

I mean, let's say you have 3 bentoml.service you want to serve, one of them uses the other two using bentoml.depends to call them async and merge their outputs. But, I also need to serve those two independently as well.

You can only serve one service when you call bentoml serve service:(one of those three) but you can run them separately at different ports as well.

What I am trying to achieve is, I was thinking to write a quick nginx conf to forward all three individual ports to one port with their corresponding endpoint, so that when this app is being deployed to Cloud, even though they run on their own ports while container is running, calls from outside host/endpoint1, host/endpoint2, host/endpoint3 can be forward to local:port1/endpoint1, local:port2/endpoint2, etc.

Long story short, All these three services can be called individually and can have their own traffic load. What do you think?

[tguner-arteriaai]

I have a follow up question:

We can also use one of the services as a router as well like:

import asyncio
import bentoml
import numpy as np
from transformers import pipeline

@bentoml.service(resources={"gpu": 1, "memory": "4Gi"})
class ModelAService:
    model_a_ref = bentoml.models.get("model_a:latest")

    def __init__(self) -> None:
        self.pipeline_a = pipeline('task_a', model=self.model_a_ref.path)

    @bentoml.api
    def model_a_predict(self, input_data: np.ndarray) -> np.ndarray:
        data_a = self.pipeline_a(input_data)
        return data_a

@bentoml.service(resources={"gpu": 1, "memory": "4Gi"})
class ModelBService:
    model_b_ref = bentoml.models.get("model_b:latest")

    def __init__(self) -> None:
        self.pipeline_b = pipeline('task_b', model=self.model_b_ref.path)

    @bentoml.api
    def model_b_predict(self, input_data: np.ndarray) -> np.ndarray:
        # Dummy preprocessing steps
        data_b = self.pipeline_b(input_data)
        return data_b

@bentoml.service(resources={"cpu": "4", "memory": "8Gi"})
class EnsembleService:
    service_a = bentoml.depends(ModelAService)
    service_b = bentoml.depends(ModelBService)

    @bentoml.api
    async def ensemble_predict(self, input_data: np.ndarray) -> np.ndarray:
        result_a, result_b = await asyncio.gather(
            self.service_a.to_async.predict(input_data),
            self.service_b.to_async.predict(input_data)
        )
        # Dummy aggregation
        return (result_a + result_b) / 2

    @bentoml.api
    async def model_a_predict(self, input_data: np.ndarray) -> np.ndarray:
        result = await self.service_a.to_async.predict(input_data)
        return result

    @bentoml.api
    async def model_b_predict(self, input_data: np.ndarray) -> np.ndarray:
        result = await self.service_b.to_async.predict(input_data)
        return result

and when we run bentoml serve service:EnsembleService, I can have these endpoints that can be called to route ServiceA and ServiceB services.

However, my question is, are we using the services with THEIR full capabilities this way through bentoml.depends? I mean like we are defining services like individual pods with their assigned cpu, gpu, memory, and timeout values. We should keep it this way, so I want to know using ServiceA and ServiceB like this calls them like endpoint hit or it is just using them like calling functions?

What is the best practice here?

[ssheng]

Whether to expose an endpoint for each individual service depends on the use case. The router service you have above is a perfectly valid pattern if all models endpoints should be exposed. Alternatively, if a service should be used as internal functions that assist on another function exposed externally, it can just be invoked as an internal function without an external endpoint.

I think you are absolutely correct services were designed to run and scale independently with dedicate CPU and GPU resources. That is why they can be defined individually for each service. When a bento with multiple services is deployed to BentoCloud, this is exactly the topology it is deployed where each service is a replica set.