Improve the autoscaling benchmark scripts

[Jeffwan]

🚀 Feature Description and Motivation

Copy some TODO items from issue #650

WIP items:

• Improving aibrix/benchmarks/generator/client.py with async and streaming mode
  • TTFT
  • TPOT
  • Goodput
• Adjust the QPS scale
• Improving plot script
  • Tokens/s times series plot
  • TTFT times series plot
  • TPOT times series plot
  • Goodput times series plot

Use Case

This is to improve the qualify of the autoscaling experiments

Proposed Solution

No response

[happyandslow]

Addressing in #676

[Jeffwan]

@happyandslow what's the status of this story?

1. All above metrics in the issue description are all collected?
2. Did you make changes to the plot diagram?
3. is Ensure that requests are evenly distributed across each second, instead of being sent in a burst #667 resolved?

[Jeffwan]

@happyandslow Any updates?

[happyandslow]

@happyandslow Any updates?

The modification is applied to #676
Improving aibrix/benchmarks/generator/client.py with async and streaming mode

• TTFT
• TPOT
• Goodput
• Adjust the QPS scale

Improving plot script

• Tokens/s times series plot
• TTFT times series plot
• TPOT times series plot
• Goodput times series plot

I'm leaving out Goodput since it is a one time measurement throughout the entire experiment.

I'm waiting for some free cycles to test these out before PR can be merged.

[Jeffwan]

I'm waiting for some free cycles to test these out before PR can be merged.

@happyandslow

Do you mean the changes has been included in #676 or in coming PR?

Can you confirm whether #667 is resolved or not?