Can RestTestGen detect wrong status codes?

[henning410]

I have some test API with endpoint GET /testStatusCode. In the OAS is defined, that this endpoint will return 200 or 400 status code. Instead, I implemented my API to always return 403. Other fuzzers can detect this difference between real status code and specified status code as bug. What about RestTestGen? Is this possible? Or is it already implemented and I just don't get the information out of the results?

Thanks for your great work so far :)

[davidecorradini]

Hello!

Often, specifications are incomplete, and, in most cases, they do not provide any output schema information or just the schema for the 200/201 responses. For this reason, we decided not to implement such a check because it would raise a lot of false positives.

Nevertheless, I believe that such a check can be implemented in a subclass of Oracle in less than 5 minutes! Do you want me to implement it? Do you want to try to implement it yourself? Let me know!

Thanks!

Davide

[henning410]

Thank you for your quick reply.

Okay, your explanation makes sense.
I see in StatusCodeOracle.java that you test what type of status code comes back and set the test result. I have tried to implement this here, but I don't know how to access the expected status codes from the specification, which I need to compare with the status code we actually get back.
Maybe you can help me out here?

To avoid a lot of false positives, I may be able to implement an option in my version to disable or enable this type of detection.

[davidecorradini]

Let me implement it for you. I'll be back in 5.

[davidecorradini]

Here is it! Please note:

1. I did not test this code.
2. You should explicitly call this Oracle in the testing strategy you are running. You should probably call it in the same place the standard StatusCodeOracle is called!

Let me know if it works!

package io.resttestgen.implementation.oracle;

import io.resttestgen.core.testing.Oracle;
import io.resttestgen.core.testing.TestResult;
import io.resttestgen.core.testing.TestSequence;

import java.util.Set;

public class SpecificationStatusCodeOracle extends Oracle {

    @Override
    public TestResult assertTestSequence(TestSequence testSequence) {

        TestResult testResult = new TestResult();

        if (!testSequence.isExecuted()) {
            return testResult.setError("One or more interaction in the sequence have not been executed.");
        }

        // The received status code as string
        String receivedStatusCode = testSequence.get(0).getResponseStatusCode().toString();
        
        // The set of status codes defined in the specification for the operation of this interaction
        Set<String> specificationStatusCodes = testSequence.get(0).getFuzzedOperation().getOutputParameters().keySet();
        
        if (specificationStatusCodes.contains(receivedStatusCode)) {
            testResult.setPass("The observed status code is defined in the specification.");
        } else {
            testResult.setFail("The observed status code is not defined in the specification.");
        }
        
        testSequence.addTestResult(this, testResult);
        return testResult;
    }
}

[henning410]

Thanks for your implementation.

It works, yes 👍🏻
But somehow, on other endpoints where an 404 is defined according to my specification, it still says:

"SpecificationStatusCodeOracle": {
      "result": "FAIL",
      "message": "The observed status code is not defined in the specification."
    }

Here you can see my Swagger, where 200, 400 and 404 is defined.

I think, the Set does not contain the correct status codes. I added some line to check what's inside the set:

// The set of status codes defined in the specification for the operation of this interaction
Set<String> specificationStatusCodes = testSequence.get(0).getFuzzedOperation().getOutputParameters().keySet();
System.out.println("SET: " + specificationStatusCodes);

As you can see in the following picture, the set often contains no status codes or sometimes just one.

[davidecorradini]

Mmm, that's strange!

Could you share the OpenAPI specification you are using so I can check what's going on with a debugger?

Thanks.

[henning410]

Sure, here

api.json

[davidecorradini]

Hello!

I've checked with the debugger and I can see that RTG ignores responses without a schema. In particular, in your specification you only define a status code and a description for responses, without defining a schema.

To quickly fix this, I would suggest to add some kind of schemas (empty or fake) to your specification, while I fix the OpenAPI parser to take into account also responses without a defined schema.

Hope this helps!

Best,
Davide

[henning410]

Ah okay, that sounds understandable.

That helps me a lot, I will change my OAS. Thank you very much for your efforts!

Best regards
Henning

[davidecorradini]

Thank you!

In the next release, I plan to fix the parser to include responses with empty schemas, and I will include the so-called SpecificationStatusCodeOracle.

Davide