Generating and checking

About

In the hs-test library, tests are a set of test cases. One test case is an object of the TestCase class. The generate method should generate a list of objects of the TestCase class and return it. It's a low-level mechanism of writing tests.

In the previous sections, you saw tests with dynamically generated input, but it is possible to create tests with static predetermined input. If the program to test is rather simple you can use this kind of tests to check it.

Dynamic testing has some benefits over tests with static input:

1. You are able to test small parts of the output compared to checking the whole output. Parsing the whole output to determine if the program written correctly may be more error-prone rather than parsing small pieces of the output.
2. Some programs aren't testable with static input because the input may be dependent on the previous output (for example, Tic-Tac-Toe game).

TestCase class

One test case is a single run of the user program. Before creating an object of the TestCase class, it is necessary to import it.

Java:

import org.hyperskill.hstest.testcase.TestCase;

Python:

from hstest.test_case import TestCase

The TestCase class contains one constructor - in Java, it is an empty one. To parameterize a test case, semi-builder pattern is used - each method returns a reference to the object itself, and method calls can be joined into a call chain. In Python, constructor uses named arguments to parametrize test case.

Java

new TestCase()
    .setInput(text)
    .setAttach(object)
    .setFile(src, content)
    .setFile(src2, content2)

Python

TestCase(
    stdin=text,
    attach=object,
    files={src: content, src2: content2}
)

Input

The first and most frequently used method is setInput(string). It is intended to simulate input using the keyboard. The method takes one parameter - what exactly should be entered from the keyboard during the user program execution.

Let's look at this example: test case looks like this:

Java

testCase.setInput("15\n23")

Python

TestCase(stdin="15\n23")

In this case, two user programs below will run normally, count two numbers, and print their sum as if the user had entered these numbers using the keyboard. If you run these programs directly, you will actually have to enter these two numbers from the keyboard. And you can use the setInput method to prepare this data in advance.

Java:

...
public static void main(String[] args) {
    Scanner scanner = new Scanner(System.in);
    int firstNum = scanner.nextInt();
    int secondNum = scanner.nextInt();
    System.out.println(firstNum + secondNum);
}
...

Python:

line = input()
first, second = line.split()
print(int(first) + int(second))

---

Note: these examples use static input. Static input means it's determined before the start of the program, but this method of testing revealed serious weaknesses and limitations for testing complex programs. Use static input only when the program to be tested is very simple. In other cases, it's strongly recommended to use dynamic testing.

---

Arguments

The test case can also be parameterized by command line arguments. These arguments are passed to the program when starting the program from the console. For example:

$ java Renamer --from old --to new --all

$ python renamer.py --from old --to new --all

In this case, the Java program will get an array of five arguments: [--from, old, --to, new, --all]. A Python program will get a list of six arguments: [renamer.py, --from, old, --to, new, --all] - the first parameter in Python is always the launched file.

An example of how to set up a command-line arguments for test cases can be seen below:

Java

testCase.setArguments("--from", "old", "--to", "new", "--all")

Python

TestCase(args=["--from", "old", "--to", "new", "--all"])

An example of using command line arguments in user code can be seen below:

Java

...
public static void main(String[] args) {
    System.out.println(args.length);
    for (String arg : args) {
        System.out.println(arg);
    }
}
...

Python

import sys

print(len(sys.argv))
for arg in sys.argv:
    print(arg)

Files

You can also create external files for testing. Before starting the test case, these files are created on the file system so that the student's program can read them. After the user program finishes, these will be deleted from the file system.

An example of how to set up external files for test cases can be seen below:

Java

testCase
    .setFile("file1.txt", "File content")
    .setFile("file2.txt", "Another content")

Python

TestCase(
    files={
        'file1.txt': 'File content',
        'file2.txt': 'Another content'
    }
)

Below you can see examples of using these external files in students' code:

Java

...
public static void main(String[] args) throws Exception {
    Scanner sc = new Scanner(new File("file1.txt"));
    System.out.println(sc.nextLine());
    sc.close();
}
...

Python

with open('file1.txt') as f:
    print(f.read())

Time limit

By default, the user program has 15 seconds to do all the necessary work. When the time runs out, the program shuts down and the user is shown a message that the time limit has been exceeded.

You can make the limit bigger, smaller, or you can turn it off. You can do this using the setTimeLimit() method in Java or the time_limit argument in Python. Notice, that you need to specify the limit in milliseconds (so, by default it's 15000). If you want to disable time limit, make it negative, for example -1.

Java

testCase.setTimeLimit(10000)

Python

TestCase(time_limit=10000)

Attach

You can also attach any object to the test case. This is an internal object, it will be available during checking inside the check method. You can put in the test input, feedback, anything that can help you to check this particular test case.

Below you can see how to attach the object to the test case. Note that in the case of Java, it is necessary to apply generics.

Java

import sample.Main;
import org.hyperskill.hstest.stage.StageTest;
import org.hyperskill.hstest.testcase.CheckResult;
import org.hyperskill.hstest.testcase.TestCase;

import java.util.List;

class Attach {
    String feedback;
    int count;
    Attach(String feedback, int count) {
        this.feedback = feedback;
        this.count = count;
    }
}

public class SampleTest extends StageTest<Attach> {

    public SampleTest() {
        super(Main.class);
    }

    @Override
    public List<TestCase<Attach>> generate() {
        return List.of(
            new TestCase<Attach>().setAttach(
                new Attach("Sample feedback", 1)),

            new TestCase<Attach>().setAttach(
                new Attach("Another feedback", 2))
        );
    }

    @Override
    public CheckResult check(String reply, Attach attach) {
        if (!reply.contains(Integer.toString(attach.count))) {
            return CheckResult.wrong(attach.feedback);
        }
        return CheckResult.correct();
    }
}

Python

from hstest.stage_test import *
from hstest.test_case import TestCase


class SampleTest(StageTest):
    def generate(self) -> List[TestCase]:
        return [
            TestCase(attach=('Sample feedback', 1)),
            TestCase(attach=('Another feedback', 2))
        ]

    def check(self, reply: str, attach) -> CheckResult:
        feedback, count = attach
        if str(count) not in reply:
            return CheckResult.wrong(feedback)
        return CheckResult.correct()


if __name__ == '__main__':
    SampleTest('sample.sample').run_tests()

Generate method

You can generate these test cases within the generate method. In the case of Java, this method must return List, in the case of Python it must return a list. You can see an example of using the method in the example of Attach (previous example).

Check method

The check method is used to check the student's solution. This method has 2 parameters - reply and attach. The first parameter is the output of the student program to the standard output. Below you can see examples of student programs that print data to the standard output.

Java

...
public static void main(String[] args) {
    System.out.println("Hello World!");
    System.out.println("Second line also");
}
...

Python

print('Hello world!')
print('Second line also')

Everything that the student's program has outputted to the standard output is passed as the first argument to the check method. Therefore, in the following examples, the reply variable will be equal to "Hello world!\nSecond line also\n".

This method must return the CheckResult object. If the user's solution is incorrect, it is necessary to explain why - for example, in the example below in the check method it is checked that the user printed exactly 2 lines, and if the user not printed 2 lines, then inform the user what went wrong during checking.

Java

import sample.Main;
import org.hyperskill.hstest.stage.StageTest;
import org.hyperskill.hstest.testcase.CheckResult;
import org.hyperskill.hstest.testcase.TestCase;

import java.util.List;

public class SampleTest extends StageTest {
    public SampleTest() {
        super(Main.class);
    }

    @Override
    public List<TestCase> generate() {
        return List.of(
            new TestCase()
        );
    }

    @Override
    public CheckResult check(String reply, Object attach) {
        if (reply.trim().split().length == 2) {
             return CheckResult.correct();
        }
        return CheckResult.wrong("You should output exactly two lines");
    }
}

Python

from hstest.stage_test import *
from hstest.test_case import TestCase


class SampleTest(StageTest):
    def generate(self) -> List[TestCase]:
        return [TestCase()]

    def check(self, reply: str, attach) -> CheckResult:
        if len(reply.strip().split()) == 2:
            return CheckResult.correct()
        return CheckResult.wrong("You should output exactly two lines")


if __name__ == '__main__':
    SampleTest('sample.sample').run_tests()

Instead of returning CheckResult object you may throw WrongAnswer or TestPassed error to indicate about the result of the test. It can be useful if you are under a lot of methods and don't want to continue checking everything else.

Since throwing any exceptions in the check method is prohibited (it indicates that something went wrong in tests and tests should be corrected), a lot of tests forced to be written like this:

Java

... deep into method calls
    if (some_parse_problem) {
        throw new Exception(feedback);
    }
...

... check method
List<Grid> grids;
try {
     grids = Grid.parse(out);
} catch (Exception ex) {
     return CheckResult.wrong(ex.getMessage());
}
...

Python

... deep into function calls
    if some_parse_problem:
        raise Exception(feedback)
...

... check method
try:
    grids = Grid.parse(out);
except Exception ex:
    return CheckResult.wrong(ex.message)
...

Allowing to throw WrongAnswer error you can write this code in a more understandable way without many unnecessary try/catch constructions.

Java

import org.hyperskill.hstest.exception.outcomes.WrongAnswer;

... deep into method calls
    if (some_parse_problem) {
       throw new WrongAnswer(feedback);
    }
...

... check method
List<Grid> grids = Grid.parse(out);
...

Python

from hstest.exceptions import WrongAnswer

... deep into function calls
    if some_parse_problem:
        raise WrongAnswer(feedback)
...

... check method
grids = Grid.parse(out);