Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to sweep GPUs using Model Analyzer #906

Open
lionsheep24 opened this issue Jun 27, 2024 · 1 comment
Open

How to sweep GPUs using Model Analyzer #906

lionsheep24 opened this issue Jun 27, 2024 · 1 comment

Comments

@lionsheep24
Copy link

lionsheep24 commented Jun 27, 2024 •
edited
Loading

I am using Model Analyzer to analyze metrics such as TPS and latency while incrementally increasing the number of GPUs. However, I could not find how to conduct experiments by increasing the number of GPUs. Could you provide guidance on how to set up and perform these experiments using Model Analyzer? Please refer to below.
1.config.yaml

model_repository: /workspace/models

run_config_search_disable: True
profile_models:
  pipeline_model:
    parameters:
      concurrency: [1, 2]
    model_config_parameters:
      max_batch_size: [1, 2]
      dynamic_batching:
        max_queue_delay_microseconds: [200, 300]
      instance_group:
        - kind: KIND_GPU
          count: [1, 2]
          gpus: [0]
        - kind: KIND_GPU
          count: [1, 2]
          gpus: [0, 1]
        - kind: KIND_GPU
          count: [1, 2]
          gpus: [0, 1, 2]
        - kind: KIND_GPU
          count: [1, 2]
          gpus: [0, 1, 2, 3]
    perf_analyzer_flags:
      shape:
        - pipeline_input:19174

report_model_configs:
  model_config_default:
    plots:
      throughput_v_latency:
        title: Title
        x_axis: perf_latency_p99
        y_axis: perf_throughput

2.Model analyzer log

[Model Analyzer] Profiling server only metrics...
[Model Analyzer]
[Model Analyzer] Starting brute mode search
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_default
[Model Analyzer]
[Model Analyzer] Profiling pipeline_model_config_default: client batch size=1, concurrency=1
[Model Analyzer] Saved checkpoint to /workspace/checkpoints/0.ckpt
[Model Analyzer] Profiling pipeline_model_config_default: client batch size=1, concurrency=2
[Model Analyzer] Saved checkpoint to /workspace/checkpoints/0.ckpt
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_0
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_1
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_2
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_3
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_4
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_5
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_6
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_7
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_8
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_9
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_10
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_11
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_12
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_13
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_14
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_15
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_16
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_17
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_18
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_19
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_20
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_21
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_22
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_23
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_24
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_25
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_26
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_27
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_28
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_29
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_30
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_31
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 200}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_32
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_33
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_34
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_35
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_36
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_37
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_38
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_39
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_40
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_41
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_42
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_43
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_44
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_45
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_46
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_47
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 1, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_48
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_49
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_50
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_51
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_52
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_53
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_54
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_55
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_56
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_57
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_58
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_59
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_60
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_61
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 1, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_62
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]
[Model Analyzer]
[Model Analyzer] Creating model config: pipeline_model_config_63
[Model Analyzer]   Setting dynamic_batching to {'max_queue_delay_microseconds': 300}
[Model Analyzer]   Setting instance_group to [{'kind': 'KIND_GPU', 'count': 2, 'gpus': [0]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2]}, {'kind': 'KIND_GPU', 'count': 2, 'gpus': [0, 1, 2, 3]}]
[Model Analyzer]   Setting max_batch_size to 2
[Model Analyzer]
[Model Analyzer] Skipping illegal run configuration
[Model Analyzer] No changes made to analyzer data, no checkpoint saved.
@lionsheep24 lionsheep24 changed the title How to incrementally increase GPUs for TPS and latency analysis using Model Analyzer How to sweep GPUs using Model Analyzer Jul 2, 2024
@nv-braf
Copy link
Contributor

nv-braf commented Jul 8, 2024

Unfortunately, MA does not support varying the number of GPUs used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lionsheep24 @nv-braf