Chasten May Undercount or Overcount Specific Patterns, Like the Number of Test Cases

[gkapfham]

There are certain situations in which chasten over-count the presence of
specific source code patterns. For instance, when the tool uses the following
XPath expression to count the number of test cases:

checks:
  - name: "all-non-test-function-definition"
    code: "FUNC"
    id: "FUNC001"
    description: "First executable line of a non-test function, skipping over docstrings and/or comments"
    pattern: '//FunctionDef[not(contains(@name, "test_"))]/body/Expr[value/Constant]/following-sibling::*[1] | //FunctionDef[not(contains(@name, "test_"))]/body[not(Expr/value/Constant)]/*[1]'
  - name: "all-test-function-definition"
    code: "FUNC"
    id: "FUNC002"
    description: "First executable line of a test function, skipping over docstrings and/or comments"
    pattern: '//FunctionDef[starts-with(@name, "test_")]/body/Expr[value/Constant]/following-sibling::*[1] | //AsyncFunctionDef[starts-with(@name, "test_")]/body/Expr[value/Constant]/following-sibling::*[1] | //FunctionDef[starts-with(@name, "test_")]/body[not(Expr/value/Constant)]/*[1] | //AsyncFunctionDef[starts-with(@name, "test_")]/body[not(Expr/value/Constant)]/*[1]'
    count:
      min: 1
      max: 10

And a program has source code that looks like this:

def test_cached():
    with TemporaryDirectory() as cache_dir:
        updated = False

        @cached(
            cache_dir=cache_dir,
            input_dirs=["input_dir"],
            output_dirs=["output_dir"]
        )
        def test_function(input_dir: str, output_dir: str, parameter: int):
            nonlocal updated

            with open(f"{output_dir}/test.txt", 'w') as f:
                f.write(str(parameter))

            updated = True
            return parameter

        with TemporaryDirectory() as input_dir:
            with open(f"{input_dir}/test.txt", 'w') as f:
                f.write("test_file")

            with TemporaryDirectory() as output_dir:
                assert test_function(
                    input_dir=input_dir, 
                    output_dir=output_dir, 
                    parameter=3
                ) == 3
                assert updated == True

                # Don't change antything
                updated = False
                assert test_function(
                    input_dir=input_dir, 
                    output_dir=output_dir, 
                    parameter=3
                ) == 3
                assert updated == False

                # Change parameter
                assert test_function(
                    input_dir=input_dir, 
                    output_dir=output_dir, 
                    parameter=5
                ) == 5
                assert updated == True

                # Change input dependency
                updated = False
                with open(f"{input_dir}/test.txt", 'w') as f:
                    f.write("changed_test_file")

                assert test_function(
                    input_dir=input_dir, 
                    output_dir=output_dir, 
                    parameter=5
                ) == 5
                assert updated == True

                # Corrupt output
                with open(f"{output_dir}/test.txt", 'w') as f:
                    f.write("corrupted output")

                assert test_function(
                    input_dir=input_dir, 
                    output_dir=output_dir, 
                    parameter=5
                ) == 5
                assert updated == True

it will return a count of 2 test cases even though, from the perspective of
pytest there is only one executable test case! Are there any intelligent ways
in which we can improve the XPath expressions or detect when the tool is likely
to over-count or under-count the presence of specific types of AST nodes?

[simojo]

I will investigate this. I assume that there is some short-circuit approach.

[simojo]

I never got far with taking a stab at this. We will leave this issue on hold for now. Here is the single thing I figured out as I was researching this:

I think that this has more to do with how the XPath expressions are written, which gives more motivation to why we would want to warn the user in the case of an expression that could potentially over or under count. For anyone to take on this issue, they must spend time understanding how the wildcard * works.

Nevertheless, on-hold for now.