Add warmup-ms argument to benchmark_requests

[avbokovoy]

This PR aims to stabilize nbit_device test for small L and D sizes. Recommended safe warmup-ms value is 100

[liligwu]

This is my solution:
just disabled the iteration numbers. warmup and profiling runs are time-based.
diff_inf_warmup_run_time.txt

[avbokovoy]

This is my solution: just disabled the iteration numbers. warmup and profiling runs are time-based. diff_inf_warmup_run_time.txt

My goal was to give user an option to choose iteration based and/or time based benchmarking. And I'm not sure that omitting this functionality is needed. Also, I'm not sure that after a proper warmup we need a time based profiling.

Anyway, I don't insist on my approach. @liligwu wdyt?

[liligwu]

This is my solution: just disabled the iteration numbers. warmup and profiling runs are time-based. diff_inf_warmup_run_time.txt

My goal was to give user an option to choose iteration based and/or time based benchmarking. And I'm not sure that omitting this functionality is needed. Also, I'm not sure that after a proper warmup we need a time based profiling.

Anyway, I don't insist on my approach. @liligwu wdyt?

Yeah, I understand. I'm also wondering how we should use these changes. Will we finally upstream it?

[avbokovoy]

This is my solution: just disabled the iteration numbers. warmup and profiling runs are time-based. diff_inf_warmup_run_time.txt

My goal was to give user an option to choose iteration based and/or time based benchmarking. And I'm not sure that omitting this functionality is needed. Also, I'm not sure that after a proper warmup we need a time based profiling.
Anyway, I don't insist on my approach. @liligwu wdyt?

Yeah, I understand. I'm also wondering how we should use these changes. Will we finally upstream it?

From my point of view we need to do the following:

1. Test it internally
2. Describe the problem to the customer and suggest this PR as a solution (with default value of warmup-ms=100)
3. Upstream it
4. Make according changes in our internal benchmark repository