Distributed load testing on AWS Fargate. Full support for Playwright.

[hassy]

Artillery v2.0.0-33 is hot off the press and includes support for running distributed load tests on AWS Fargate, with full support for using Playwright.

Run thousands of real headless browsers with no infrastructure to set up or manage, from your own AWS account.

Read full release notes here: https://github.com/artilleryio/artillery/releases/tag/artillery-2.0.0-33

[travisdieckmann]

Is anyone able to execute using run:fargate?
npx artillery run:fargate --region us-west-2 tests.yaml exits non-zero with no error messages

npx artillery run:fargate tests.yaml results in:

Checking AWS connectivity... [10:33:04]
Checking cluster... [10:33:04]
Could not check cluster "artilleryio-cluster" in us-east-1
Error: Could not find cluster artilleryio-cluster in us-east-1

[hassy]

Hi @travisdieckmann 👋 This could be due to AWS permissions you're running Artillery with. Did you set up a custom IAM role (docs) or are you running under an admin role?

[travisdieckmann]

I am running under an admin role in the AWS account.

[sambordo1]

@travisdieckmann If you find it's not an AWS permissions issue, I got a similar error message until I manually created an " artilleryio-cluster " cluster in my AWS account. After manually creating that cluster the tests went through just fine using that cluster. Not sure if this is the right way of doing it, but it worked for me, hope that helps.

[hassy]

@travisdieckmann creating and setting a cluster explicitly as @sambordo1 suggests should work as a workaround. Will ship a fix asap.

[hassy]

@travisdieckmann Just published a canary release of Artillery which should fix the issue. You can update to artillery@2.0.0-34-f96bae8 to try it. Let me know if you still run into any issues.

[giuliodali]

Hi,

I've found an issue with playwright and fargate. If you have this file structure, then it won't find the test in the folder to run.

• TheTestFolder
  • TheTest.js
• TheTest.yml (contents has this line processor: './TheTestFolder/TheTest.js')

This works fine locally, just not on fargate where you get:

worker error, id: 1 Error: Cannot find module '/artillery-pro/test_data/TheTestFolder/TheTest.js'

[bernardobridge]

Hi @TestItEver ! Thanks for reporting the issue. I haven't been able to replicate it with a similar folder structure to yours. A couple of questions:

• In the folder structure you shared above, what is the processor path that TheTest.yml points to? And does process.js get required by TheTest.js or vice-versa?
• Could you share what command you ran, and from which folder you ran it?

Thank you!

[TestItEver]

Hi @bernardobridge! Out folder structure is like this:
1. level: folder Performance
2. level: folder tests
3. level: folder artillery
- TheTest.js
- TheTest.yml with
- payload: -path: ./TheUser.csv
- processor: ./TheTest.js
3. level: folder playwright
- process.js

Yes, the process.js (and some other js files from playwright folder) is required from TheTest.js: require("../playwright/process.js"). Process.js itself doesn't require TheTest.js but required several other js-files from playwright folder.

I run the command from the folder Performance:
npx artillery run-fargate --region xxxx tests/artillery/TheTest.yml

.. and my test bundle looks like this:
artillery\TheTest.yml
artillery\TheUser.csv
artillery\TheTest.js
playwright\process.js
...
and this doesn't work.

If I have everything, that is required in bundle, under folder artillery, the test bundle looks like this:
TheTest.yml
TheUser.csv
TheTest.js
process.js
...
and this works.

[bernardobridge]

Hi @TestItEver and others 👋 . Thank you for your help!

This is likely to be the same bug reported here #2092, originating from having the same file name on the .yml and .js files.

We have a fix for it now, and a new CLI version will be available with it soon.

In the meantime, you should be able to work around the issue by naming the files with different names.

[TestItEver]

Hi @bernardobridge! In wich version is the fix included? I updated my artillery version to 2.0.0-37 but I still have the same error with folders.
I see, that the folder structure is build up:

|-- artillery\config.yaml
|-- artillery\login-logout.yaml
|-- artillery\processor.js
|-- artillery\users.csv
|-- helpers\date.js

But at this point

WARNING - config.processor is now resolved relative to the config file

Expected to find file at: /artillery-pro/test_data/processor.js
Error: ENOENT: no such file or directory, open
'/artillery-pro/test_data/users.csv'
Code: ENOENT
cleanup called, signal:
EXIT

the program looks after users.csv simply under test_data... BUT the file seems to be under test_data/artillery/users.csv?

[AtefBellaaj]

Hi @bernardobridge
I am having the same issue unfortunately! v 2.0.0-37 is this tracked in an issue? Thank you for your time 😄

[mtulioalcantara]

Hello, I am getting:

Workers are running, waiting for reports... [18:47:12]

Summary report @ 18:47:24(-0300)

No measurements recorded during this period

Why I do not have any measurements recorded?

Any help? Thanks a loto!

[mtulioalcantara]

I mean, where can I see the results of the test?

[hassy]

hi @mtulioalcantara 👋 This can happen if the Artillery workers on Fargate crashed for some reason. You'll need to check workers logs in AWS CloudWatch. To find the logs:

Go to AWS Console -> CloudWatch -> Logs -> Log Groups -> search for "artilleryio-log-group/artilleryio-cluster". All the logs from all workers should be there.

[mtulioalcantara]

I went to the log page on CloudWatch and I found that, somehow, Playwright together with Artillery try to access the test script as "test.js" filename. As soon as I changed my JS test script name (airport-gap-test.js) to that, It just worked! I still doesn't know if that's a fact (poor documentation in this part), but it's working fine and I've been launching several tests using Fargate since then, using always my JS script test file named "test.js".

Does it make any sense?

[hassy]

thanks! it looks like the same issue as the one reported by @giuliodali earlier in the discussion. We're looking into it!

[Malvika-sys]

Can anyone help me how do we authenticate from local machine to connect automation framework to AWS FARGATE : provide key id and secrets?

[hassy]

Please see the section on AWS authentication in the docs: https://www.artillery.io/docs/load-testing-at-scale/aws-fargate#:~:text=The%20SDK%20requires,chap%2Dconfigure.html

[mtulioalcantara]

I'm using Artillery with Playwright and Fargate for my tests. I'm having this kind of results:

vusers.completed: .............................................................. 361
vusers.created: ................................................................ 405
vusers.created_by_name.0: ...................................................... 405
vusers.failed: ................................................................. 44

What does it really mean to have 44 failed VU?

And in this same test I get:

browser.page.FCP.https://mywebsite.com/:
min: ......................................................................... 911.3
max: ......................................................................... 2181.3
median: ...................................................................... 1274.3
p95: ......................................................................... 1620
p99: ......................................................................... 1826.6

The timing results above refers only to the 361 completed VUs or are influenced somehow by the 44 failed?

I sent those 405 VUs to a real webpage in AWS with 8 powerful EC2 instances (t3.2xlarge) with load balancing, during a 5 minutes walking thru a 6 pages test. Can I attest I am ready to receive 405 users to my website?

Also, I tried different configurations for ~400 VUs total, fot the test, considering de --count flag and the arrivalCount in the YML config file:

count 12 x arrivalCount 45 for a total 405 VUs (right?)
count 10 x arrivalCount 40 for a total 400 VUs (right?)
count 20 x arrivalCount 20 for a total 400 VUs (right?)

All of them give me around 10 - 16% of failed VUs.

How can I improve my tests (get no failed VUs)? Should I change something in my infira considering I have good timing already and a powerful line of EC2 instances?

Thank you very much!

[mtulioalcantara]

The main question here is: are those failed VUs due to the capacity of my infrastructure of due to the Artllery/Playwright/Fargate capacity?

[mtulioalcantara]

@hassy could you kindly take a look at my scenario please? I apreciate very much your help!

[yogi-sagar]

Hi I am trying to run the tests in aws:fargate on us-west-2 region but it is saying this region is not supported. What could be the reason us-west-2 not being supported. Any plans in future to support it? We don't have service in east-1 region so this is the only option for us to set it up.

artillery run-fargate --count 1 --region ap-east-1 --task-role-name artilleryio-ecs-worker-role --cpu 1 --memory 1024 accounts-test.yml
- Unsupported region (us-west-2) provided. Please specify one of: us-east-1, us-west-1, eu-west-1, eu-central-1, ap-south-1, ap-northeast-1

[bernardobridge]

Hello @yogi-sagar! Support for us-west-2 has been added here, but a new version isn't released yet. You can use the latest canary release (artillery@2.0.3-1091257) if you need this now, and it will also be available in the next CLI release, which should happen this week.

[yogi-sagar]

Great! Thank you so much.

[yogi-sagar]

@bernardobridge I don't see new release being out after our discussion. Can you please confirm? I see the last one 2.0.3 was on Dec 6 https://github.com/artilleryio/artillery/releases.

[bernardobridge]

Apologies @yogi-sagar, it took us a little while longer.

Artillery 2.0.4 is now out: https://github.com/artilleryio/artillery/releases/tag/artillery-2.0.4

[mustafa89]

Is there a way to tag the resources in AWS for proper cost allocation? Also the docs say there is a cleanup : https://www.artillery.io/docs/load-testing-at-scale#clean-up-aws-resources

There isn't all resources are still there. We would like if there was a command to do a full cleanup of artillery-created resources. Is that possible?

[hassy]

@mustafa89 AWS tagging is not supported at the moment but it's something we're considering.

The only long-lived resources are the IAM roles and an empty Fargate cluster. Everything else is created on demand (SQS queues, Fargate tasks), or has a lifecycle policy associated with it (S3 objects, CloudWatch logs)

[mustafa89]

Ideally, the hope would be that everything is cleaned up (SSM as well although they don't cost anything), but tagging for us would be very nice. Thankyou for the quick help on this.