Add new health check configuration fields

[spal1]

🚀 What

Adds new configuration fields to truss config for models and chains. Naming and defaults can be seen in examples below.

for models, we add a new health_checks section to runtime:

runtime:
  health_checks:
    restart_check_delay_seconds: 0
    restart_failure_threshold_seconds: 1800 # 18 consecutive failures over 30 minutes
    stop_traffic_failure_threshold_seconds: 1800 # 18 consecutive failures over 30 minutes

for chains, we add a new runtime field to RemoteConfig that includes health_checks:

remote_config = chains.RemoteConfig(
  runtime=chains.Runtime(
    health_checks=truss_config.HealthCheck(
      restart_check_delay_seconds=0,
      max_time_before_restart_seconds=1800,
      max_time_before_stopping_traffic_seconds=1800
    )
  )
)		

🔬 Testing

Added a couple of integration test cases

[linear]

BT-13066 Add configuration parameters to truss config

[squidarth]

I think we want to do these validations on the backend. That gives us more flexibility to not be stuck with these checks for old clients.

Can you look into how that might work?

[spal1]

Discussed offline--I'll use this as an opportunity to create a first version of a validate_truss_config GQL mutation that we can call from here: https://github.com/basetenlabs/truss/blob/main/truss/remote/baseten/remote.py#L124

[marius-baseten]

I don't understand why we write so much boilerplate code for this here when pydantic can do most of this. - more a comment on pre-existing code though...

I think we should refactor this whole file soon. It's also super error prone that you have to add key literals to to_dict methods each time you add a field.

[marius-baseten]

Chains doesn't map 1:1 to the structure of truss config. I wonder if it would make sense to add this field to ChainletOptions instead of creating a new class.

If you want to proceed with a new class, then there should be some differentiator, explaining which future fields go into Runtime and which ones go into ChainletOptions.

[marius-baseten]

I don't understand why we write so much boilerplate code for this here when pydantic can do most of this. - more a comment on pre-existing code though...

I think we should refactor this whole file soon. It's also super error prone that you have to add key literals to to_dict methods each time you add a field.