Getting mean/stdev in test results

[indeediansbrett]

I was wondering if there was a way to get mean and standard deviation in test results. It provides median, p95, p99, etc which are great. But if I have two test results, I don't have a good way to know if the differences are significant and I need to run more scenarios.

[hassy]

hi @indeediansbrett 👋 Artillery does not report mean or average of latency measurements as a design choice. Curious to learn more about how you'd like to be able to compare multiple test results (it's a feature we'd like to support better in Artillery).

[indeediansbrett]

Basically need a way to know if the difference in metrics are statistically significant. My thought was that at the very least average and standard deviation are supplied to do an unpaired t-test. Also some way to calculate higher degree moments like a standard deviation of standard deviation for multiple scenario sets. Then I could answer a question like "How many scenarios do I need to run to be able to detect a significant difference if it exists?"

[indeediansbrett]

Hi @hassy what are your thoughts on providing avg/stdev in addition to percentile metrics?