k6 pm.expect behaviour vs postman

[datimms]

Imagine I have a postman request with an expected response of:-

{
  "data": {
    "someProperty": true
  },
  "errors": []
}

and I have a test as follows:-

const jsonData = pm.response.json();

pm.test("Some Property is returned", function () {
    //ignore the fact this is a bad idea
    pm.expect(jsonData.data.someProperty).to.exist;
});

pm.test("Shouldn't contain another property", function () {
    pm.expect(jsonData.data.anotherProperty).to.not.exist;
});

When I run this in Postman, both tests pass. Postman doesn't throw any js errors.
When I convert and run this in k6, the first test passes and the second test is not reported and k6 itself stops the iteration, logging level=error msg="TypeError: Cannot read property 'notThere' of undefined. It then starts another iteration if available.

The main issue really is that k6 fails silently.

Imagine now my system under test starts to fail under load and begins to respond with:-

{
  "data": null,
  "errors": [{...},{...}]
}

In postman now both tests would fail.
In k6 both tests would not be reported, k6 would have failed silently then continue another iteration and at the end of the run report 100% pass all green. If you were diligent you would notice that the number of http_reqs didn't marry up to the number of iterations but that would be the only way to tell.

Is this expected behaviour of postman requests running in k6, compared to the way Postman handles it?

[thim81]

Hi @datimms,

The pm.response.json() is a very sensitive function, meaning that if the API response is anything but JSON, it will throw an error (like Unexpected token '<') in Postman.
So depending on your API configuration, your webservers might return a HTML error page or something that is not JSON, which makes the pm.response.json() fail.

I'm not 100% sure but my guess is that Postman-to-K6 wrapper, catches the failed iteration silently.

A tip could be to make your Postman tests more fault tolerant, which would make the K6 tests also more stable.
Typically I advice to try/catch the pm.response.json()

// Set response object as internal variable
let jsonData = {};
try {jsonData = pm.response.json();}catch(e){console.log(e);}

pm.test("Some Property is returned", function () {
    //ignore the fact this is a bad idea
    pm.expect(jsonData.data.someProperty).to.exist;
});

Would this be a idea that could work for you?