Test NodeJS App throughput (iops) with artillery?

[ghost]

Hello,
I would like to test the max throughput of my NodeJS app. So 100% CPU = how many res/sec?

[hassy]

That's exactly the use case for Artillery. Bearing in mind that throughput will depend on the environment and exact nature of the workload of course, the process is: run some load on your app, see what CPU usage is like under some level of load, increase load to understand the relationship between workload increase and increase in CPU usage.

[B3ns44d]

Getting started

Artillery can be installed like any other Node package, using NPM:

npm install -g artillery 

Doing a quick test

Artillery can be used to do one-off tests to a specific URL in a similar way of tools like Apache Benchmark.

That is done using the quick command.

artillery quick -c 10 -n 20 https://httpbin.org/get

This will launch 10 virtual users that will do 20 requests each, to the specified URL.

The quick command supports some other flags to control how the load will be distributed, like rate, which can be used to define the number of new users per second, or the duration, which defines a fixed time for the test.

For example, the following command will run the test during 10s, with 2 new users arriving each second, resulting in around 20 requests in the total.

artillery quick https://httpbin.org/get -r 2 -d 10

You can combine all these flags, to achieve the desired pattern of load for your test.

In terms of other options, I found this command to be pretty limited as you can´t configure almost anything more of the request, like defining request headers. It also only allows GET requests.

So, if you just want to do some one-off test directly from the command line and need a little more flexibility configuring the request, there are probably better tools like Apache Benchmark.

But the power of Artillery lies in being able to simulate realistic user behavior with scenarios and flows, using declarative YAML files.

Scnearios and YAML configuration

The most basic test case can be defined like this:

config:
  target: 'https://httpbin.org/'
  phases:
    - duration: 60
      arrivalRate: 20
scenarios:
  - flow:
    - get:
        url: "/get"

The config section allows defining some generic configuration about the test, like the target URL, the duration of the test and the way users will generate load.

In this example, we define one phase, which will last 60 seconds with 20 new virtual users (arriving every second (on average)).

The scenarios section defines the scenarios this test will cover. Each scenario has a flow associated with it, which is a set of steps that each user will do in your application.

We will see this in more detail, later in this article.

In this particular case, we have defined one single scenario, with a single step, which is a GET request to the https://httpbin.org/get endpoint.

You can run this test using the run command:

artillery run <path_to_file>.yml

This is very similar to the "quick" example, just a request to a specific endpoint.
But unlike the "quick" command, using a configuration allows you to define a lot more params of the request, like HTTP method, headers, cookies, request payload, etc. You can even load your payload dynamically from CSV files.

Here is a more advanced example demonstrating how to do a POST request with payload data loaded from a CSV file.

  config:
    payload:
      # path is relative to the location of the test script
      path: "users.csv"
      fields:
        - "username"
        - "password"
  scenarios:
    - flow:
        - post:
            url: "/auth"
            json:
              username: "{{ username }}"
              password: "{{ password }}"

This test will do a "POST" request to the "/auth" endpoint, with a JSON body containing the fields "username" and "password", which values are loaded from a CSV file "users.csv", from the specified fields.

You can find out about all the possibilities in the Documentation.

more complete user flow

We have seen how to define a scenario with a simple flow.
Next, we will see how you can define an entire user journey.

Considering a typical E-commerce site. You might want to simulate a user flow, starting from the catalog page, selecting a product, and add it to the cart.

This can be achieved with the following configuration:

config:
  target: "https://staging1.local"
  phases:
    - duration: 60
      arrivalRate: 5
  payload:
    path: "keywords.csv"
    fields:
      - "keywords"
scenarios:
  - name: "Search and buy"
    flow:
      - post:
          url: "/search"
          body: "kw={{ keywords }}"
          capture:
            json: "$.results[0].id"
            as: "id"
      - get:
          url: "/details/{{ id }}"
      - post:
          url: "/cart"
          json:
            productId: "{{ id }}"

Some things we haven´t seen before.

A flow can consist of multiple steps, representing the user journey in the application. Each user spawned by Artillery will execute all the defined steps sequentially.

In this particular example, The first step is doing a POST request to the "/search" endpoint with a body, containing keywords, loaded from a CSV file.

- post:
          url: "/search"
          body: "kw={{ keywords }}"
          capture:
            json: "$.results[0].id"
            as: "id"

The "capture" block allows us to get a value from the response and save it as a variable for later use. You can use "JSONPath", "XPath", "Regex" and more to parse the response and get the value you want.

For example, to get a URL form an anchor tag in HTML response you could do as following:

- get:
    url: "/some/page"
    capture:
      - selector: "a[class^=productLink]"
        attr: "href"
        as: "productUrl"

Back to our E-commerce flow, We will use the captured "id" variable from the "search" step, to go to the "product details" page and to add the product to cart:

- get:
     url: "/details/{{ id }}"
- post:
    url: "/cart"
    json:
       productId: "{{ id }}"

As you can see with a few lines of YAML we can define an entire user flow to test.

Artillery supports many more functions to work with HTTP requests and responses. You can find then in the http reference section of the documentation.

Multiple Scenarios

In the previous example, we saw how to define a single scenario with multiple steps.
But you can also define multiple scenarios to simulate users doing different things at your application at the same time.

Let´s say you want to simulate users searching and buying products at the same time. You can create two different scenarios to test both flows. Note that each user will only execute one of the scenarios.

The scenario supports a "weight" property which you can use to indicate how likely it is that scenario to be picked by a particular user.

By default, each scenario will have the same probability. If you define a scenario with weight 2, that scenario is twice as likely to be picked as the one with 1 (which is the default value).

Setting success conditions

It´s possible to configure Artillery to return a non-zero exit code if the test run doesn't comply with specified conditions based on a set of parameters like error rate, minimum, maximum and percentile based latency.

This is really useful in a CI environment as you can make the test fail if it doesn´t meet your performance requirements.

The following configuration will ensure that at least 95% of requests are executed below 200ms, otherwise, the command will exit with an error.

config:
  ensure:
    p95: 200

Extend Artillery

Hooks

Artillery has support for "hooks", which allow for custom JS functions to be executed at certain points during the execution of a scenario.

The following extension points are available:

• beforeScenario and afterScenario - called before/after a virtual user executes a scenario
• beforeRequest - called before a request is sent. request parameters can be customized here.
• afterResponse - called after a response has been received. The response can be inspected and custom variables can be set here.
• function - which can be inserted as a step at any point in a scenario.

You can use these hooks, to log extra information you need or to modify the request of some sort before doing a request.

this article shows a very good example of this, by leveraging Faker.js to generate random data for the tests.

Plugins

Artillery supports custom plugins that you can write using Javascript.

Artillery has some official plugins like artillery-plugin-expect that can be used to add expectations/assertions to your HTTP scenarios, allowing you to do some basic functional tests together with your load tests or the artillery-publish-metrics, which allows sending metrics about your test runs to external services like Datadog, InfluxDB or StatsD.

Here is an example of the artillery-plugin-expect plugin in use. See The "expect" section:

scenarios:
  - name: Get pets
    flow:
      - get:
          url: "/pets"
          capture:
            - json: "$.name"
              as: name
          expect:
            - statusCode: 200
            - contentType: json
            - hasProperty: results
            - equals:
              - "Tiki"
              - "{{ name }}"

You can build your own to add extra functionality.

[hassy]

Great answer @B3ns44d! You should publish that as a blog post. 😄

[B3ns44d]

thank you 😄, will do hahaha

[ghost]

Yeah, I expected a small explanation and he posted a whole tutorial 😅 here you deserve it 🍕👉👈

[hassy]

@B3ns44d how's that blog post coming along? :D we're putting together a page in the official docs to include and give shout outs to blog posts covering Artillery, lmk if/when you publish yours so that I can include it there.

[B3ns44d]

@hassy Actually, I am not planning to publish it, you can repost it on your docs you have all right to do so