Bad VMs leading to Inaccurate results

[exterkamp]

Running this a few times with only 1 run on my personal site gh repo I got inconsistent results. This seemed to be due to the underlying VMs.

100 score & 1026 benchmark idx
98 score & 347 benchmark idx
88 score & 106 benchmark idx

Ideas:

• Less than a 500 benchmark index, the run should be killed. These results won't be accurate and can't be asserted against.

• We should run a pre-flight check with the benchmarker Lighthouse uses (it's some pretty simple js to run, seen here: https://benchmark.exterkamp.codes) and check if the VM we got for our action was DOA. If it was, then exit code 1 and print out something about re-running the action because of a bad VM.

[alekseykulikov]

Thank you @exterkamp for a great research!
I hope it will go away as the Github Actions platform matures. Otherwise, we have to request a new VM to provide consistent performance testing results. Any ideas on how to force a new VM and not fail the workflow?

[paulirish]

Reading https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops it doesn't provide many expectations around CPU load.. or any options for it.

But it does say "(Linux only) Run steps in a cgroup that offers 6 GB of physical memory and 13 GB of total memory" so that's something.

cc @connorjclark

[paulirish]

Oh and much like azure pipelines... github actions apparently also allows self-hosted runners: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-self-hosted-runners

[alekseykulikov]

As a solution, we may collect Lighthouse reports using PageSpeed Insights. It's more consistent (https://treo.sh/demo/6) and will allow running audits in parallel to speed up a build.

- name: Audit URLs using Lighthouse
  uses: treosh/lighthouse-ci-action@v2
  with:
   urls: |
      https://example.com/
      https://example.com/demo
      https://example.com/dashboard
   psiToken: page-speed-insights-token
   runs: 3 # use median for performance runs