Add maxIterations option to arrival-rate executors

[na--]

Feature Description

It will be nice if users can specify exactly how many iterations at most a constant-arrival-rate or ramping-arrival-rate scenario should make before it stops. Especially when there is a set of data items we can only use once, as is the case of #2305 (comment)

For constant-arrival-rate that can somewhat be accomplished by some math carefully choosing the duration, but it's still a PITA. For ramping-arrival-rate it's technically also possible, but so difficult not to be worth it.

Suggested Solution (optional)

Add a new maxIterations option. Arrival-rate executors should end whenever they hit their maximum duration OR after they start the maxIterations iteration globally. This is going to be similar to what shared-iterations and per-vu-iterations already do.

This should be fairly straightforward to implement. For example, ramping-arrival-rate already has gets the iterations it's supposed to execute over a channel:

k6/lib/executor/ramping_arrival_rate.go

Line 460 in 804203b

for nextTime := range ch {

So it should just be a matter of closing that channel whenever the desired number of iterations was reached, I think...

Already existing or connected issues / PRs (optional)

#1386 explores time bucketing of iterations, another potential improvement for arrival-rate executors

And before we implement either this issue or #1386, we should probably refactor the arrival-rate executors in a similar manner to #2155 🤔

[mstoykov]

So it should just be a matter of closing that channel whenever the desired number of iterations was reached, I think...

Does that mean that we will count dropped iterations, due to no VU available to execute them, toward the maxIterations?

If yes, I would argue that this can also be done through some kind of js helper which does the math, which isn't really all that much of a PITA IMO ;).

I have no idea what the API for that will be for ramping-arrival-rate, but I also find

  scenarios: {
    contacts: {
      executor: 'ramping-arrival-rate',
      startRate: 50,
      timeUnit: '1s',
      preAllocatedVUs: 50,
      maxVUs: 100,
      maxIterations: 250,
      stages: [
        { target: 200, duration: '30s' },
        { target: 0, duration: '30s' },
      ],
    },
  },

To be confusing. Just looking at it, I can't decide if maxIterations will end at 15th second or if it's too much (and the numbers here are pretty small) at which Point I don't know what the point is.

Do we intend on having maxIterations and no duration in these cases, like:

  scenarios: {
    contacts: {
      executor: 'ramping-arrival-rate',
      startRate: 50,
      timeUnit: '1s',
      preAllocatedVUs: 50,
      maxVUs: 100,
      stages: [
        { target: 200, maxIterations: 150 },
        { target: 0, maxIterations: 50 },
      ],
    },
  },

And to get the duration out of this?
Or the other way around, have maxIteration and duration but have no target :

  scenarios: {
    contacts: {
      executor: 'ramping-arrival-rate',
      startRate: 50,
      timeUnit: '1s',
      preAllocatedVUs: 50,
      maxVUs: 100,
      stages: [
        { duration: "30s", maxIterations: 150 },
        { duration: "30s", maxIterations: 50 },
      ],
    },
  },

I feel like the latter one will have problems with startRate as it's not what is being directly configured.

All of this to me seems like a good place for some helper JS functions which explore those areas and generate the currently supported config instead of starting with more options being added.

p.s. For completeness’ sake, just the first stage of starts 50*30 + (200-50)*30/2=3750 iterations

[na--]

Does that mean that we will count dropped iterations, due to no VU available to execute them, toward the maxIterations?

If yes, I would argue that this can also be done through some kind of js helper which does the math, which isn't really all that much of a PITA IMO ;).

Good question 🤔 I am not sure which would be better, though I think "no" might be the right answer here. I think we should probably try to keep this in sync with scenario.iterationInTest from k6/execution, which doesn't count dropped iterations. See this script for example:

import exec from 'k6/execution';
import { sleep } from 'k6';

export const options = {
    scenarios: {
        sc1: {
            executor: 'constant-arrival-rate',
            preAllocatedVUs: 1,
            rate: 1,
            duration: '30s',
        },
    },
}

export default function () {
    console.log(`[t=${(new Date()) - exec.scenario.startTime}ms] VU{${exec.vu.idInInstance}/${exec.vu.idInTest}} ran scenario iteration ${exec.scenario.iterationInTest} (${exec.vu.iterationInScenario} for VU) in ${exec.scenario.name}`);
    sleep(1.9);
}

I will output something like this:

INFO[0000] [t=0ms] VU{1/1} ran scenario iteration 0 (0 for VU) in sc1  source=console
WARN[0001] Insufficient VUs, reached 1 active VUs and cannot initialize more  executor=constant-arrival-rate scenario=sc1
INFO[0002] [t=2000ms] VU{1/1} ran scenario iteration 1 (1 for VU) in sc1  source=console
INFO[0004] [t=4000ms] VU{1/1} ran scenario iteration 2 (2 for VU) in sc1  source=console
INFO[0006] [t=6000ms] VU{1/1} ran scenario iteration 3 (3 for VU) in sc1  source=console
INFO[0008] [t=8000ms] VU{1/1} ran scenario iteration 4 (4 for VU) in sc1  source=console
INFO[0010] [t=10001ms] VU{1/1} ran scenario iteration 5 (5 for VU) in sc1  source=console
INFO[0012] [t=12001ms] VU{1/1} ran scenario iteration 6 (6 for VU) in sc1  source=console
INFO[0014] [t=14000ms] VU{1/1} ran scenario iteration 7 (7 for VU) in sc1  source=console
INFO[0016] [t=16001ms] VU{1/1} ran scenario iteration 8 (8 for VU) in sc1  source=console
INFO[0018] [t=18000ms] VU{1/1} ran scenario iteration 9 (9 for VU) in sc1  source=console
INFO[0020] [t=20001ms] VU{1/1} ran scenario iteration 10 (10 for VU) in sc1  source=console
INFO[0022] [t=22001ms] VU{1/1} ran scenario iteration 11 (11 for VU) in sc1  source=console
INFO[0024] [t=24000ms] VU{1/1} ran scenario iteration 12 (12 for VU) in sc1  source=console
INFO[0026] [t=26001ms] VU{1/1} ran scenario iteration 13 (13 for VU) in sc1  source=console
INFO[0028] [t=28001ms] VU{1/1} ran scenario iteration 14 (14 for VU) in sc1  source=console
INFO[0030] [t=30001ms] VU{1/1} ran scenario iteration 15 (15 for VU) in sc1  source=console

And you are right - if we wanted to count dropped iterations, we can probably somewhat easily make a JS helper that calculates the duration of the scenario/stages to get us exactly the desired number of iterations. So this way we cover both use cases. The implementation won't be as clean as what I suggested above, but it probably won't be much more complicated than it 🤞

I have no idea what the API for that will be for ramping-arrival-rate, but I also find

...
To be confusing. Just looking at it, I can't decide if maxIterations will end at 15th second or if it's too much (and the numbers here are pretty small) at which Point I don't know what the point is.

The point is to be able to easily define a scenario for this use case, which we currently can't do: "Execute exactly the specified number of iterations at the given constant/variable iteration rate"

And yes, I think we should still have duration, for the same reason we require maxDuration in shared-iterations and per-vu-iterations.

[joanlopez]

All of this to me seems like a good place for some helper JS functions which explore those areas and generate the currently supported config instead of starting with more options being added.

Gonna close this issue as it has no reactions (does it really worth the effort?) and we don't foresee capacity to focus on this. Plus, there's no clear idea about how to progress (see the discussion above, it would require more thinking before starting to write an implementation), and as Mihail mentioned, if somebody definitely wants to give it a try, it's probably achievable in the form of a JS helper that anyone could write.

If the latter is the case, and you, future reader, have a more concrete implementation suggestion, ideally proofed, feel free to leave it here or open a new issue with the specific details.

Thanks!