Testing

[gswifort]

I noticed a few things to improve in unit tests.

1. pytest seems better than unittest
   e.g., you can assert True instead of self.assertTrue(True)

2. Collecting tests can be done more simply than it is currently

   An example test runner might look like this:

   import contextlib
   import traceback
   
   import pytest
   
   
   def PyRxCmd_runtests():
       try:
           print("Running tests...")
           pytest_args = [
               ".",  # Searching for tests in the current directory
           ]
           with open("pytest.log", "w") as f:
               with contextlib.redirect_stdout(f), contextlib.redirect_stderr(f):
                   pytest.main(pytest_args)
   
       except Exception:
           traceback.print_exc()
       else:
           print("Tests finished.")

   Although you are using unittest, you can run tests using pytest, excerpts from the pytest.log file:

   ============================= test session starts =============================
   platform win32 -- Python 3.12.7, pytest-8.3.2, pluggy-1.5.0
   rootdir: d:\Users\GSWI\Documents\PROGRAMY\PyRx
   configfile: pyproject.toml
   collected 182 items
   
   UnitTestDocString.py ..                                                  [  1%]
   UnitTestMisc.py s                                                        [  1%]
   UnitTestPyAcGe.py Fssss..............................ssss........        [ 27%]
   UnitTestPyActiveX.py ........................                            [ 40%]
   UnitTestPyBrep.py FFFFFF                                                 [ 43%]
   UnitTestPyDatabase.py sss.FFFFFF...FF.FFFFF..F..s                        [ 58%]
   UnitTestPyDbAssocPersSubentIdPE.py s                                     [ 59%]
   UnitTestPyDbEntity.py s......s.....sF........sFFFFFFFFF                  [ 77%]
   UnitTestPyDbEval.py ..                                                   [ 78%]
   UnitTestPyDbObject.py ...F....                                           [ 82%]
   UnitTestPyDbObjectId.py ..                                               [ 84%]
   UnitTestPyEditor.py .sF..Fs.s.                                           [ 89%]
   UnitTestPyRxObject.py ......                                             [ 92%]
   UnitTestPyWx.py .                                                        [ 93%]
   UnitTestResbuf.py .......                                                [ 97%]
   UnitTestSheetSet.py sFFss                                                [100%]
   
   ================================== FAILURES ===================================
   __________________ TestGe.test_CompositeCurve3d_getCurveList __________________
   
   self = <UnitTestPyAcGe.TestGe testMethod=test_CompositeCurve3d_getCurveList>
   
       def test_CompositeCurve3d_getCurveList(self):
           objHnd = Db.Handle("2c92e2")
   >       objId = dbc.dbs["06457"].getObjectId(False, objHnd)
   E       KeyError: '06457'
   
   UnitTestPyAcGe.py:364: KeyError
   
   (...)
   
   ================= 36 failed, 122 passed, 24 skipped in 1.98s ==================
   

   All such functions are unnecessary:

   def pydbeval(): 
   try:
       suite = unittest.TestLoader().loadTestsFromTestCase(TestPyDbEvalVariant)
       if testcfg.logToFile:
           with open(testcfg.logFileName, "a") as f:
               f.write("\n{:*^60s}\n".format("TestDbEval"))
               runner = unittest.TextTestRunner(f, verbosity=testcfg.testVerbosity)
               runner.run(suite)
       else:
           print('TestDbEval')
           print(unittest.TextTestRunner(verbosity=testcfg.testVerbosity).run(suite))
   except Exception as err:
       print(err)

3. Assuming automatic test collection, the imports will need to be corrected:

   if "BRX" in host:
       import PyBrxBim as Bm

BTW, this is a really great idea, now I can build a test runner for all platforms

I can help with this, you just need to present what concept you had in mind.

[CEXT-Dan]

The test runner is something that’s morphed from just quick thing to print results on the command line to the current kludge
I have no problems with improving it or changing to pytest, the output of pytest looks much better

concept you had in mind.

1, have a global flag where I can disable skips, I want to be able to test new CAD versions to see what’s been fixed. I had started it with TEST_LEVEL and makeSkip

2, Have the “read only” test media open for the session, like dbc.dbs["06457"]

3, I was planning to add a folder that I can use to test object creation, i.e. a new database or sheet set, but is ignored by GIT. Now, the objects are just spammed to the current document.

4, I had turned off testing for older CAD versions, I don’t know if that was a good idea as some people have perpetual licenses and will probably want to know if something works. Then there’s the whole can of worms of when to drop support for CAD versions. I might want to add this back in somehow with a flag

Also, I was thinking about how to use TEST_LEVEL or another flag to disable long running tests, or separate regression from unit testing

[gswifort]

Ok, so where in all this is sys.argv?

[CEXT-Dan]

I want to create an application that will launch each CAD platform and run tests unattended. My thought is to use the command line arguments to pass bit-wise flags that will setup TEST_LEVEL and other flags. OnPyInitApp in UnitTestRunner can check sys.argv and do the setup.

I hope to make the applications test the logs and somehow report pass fail for each app.

[gswifort]

1, have a global flag where I can disable skips, I want to be able to test new CAD versions to see what’s been fixed. I had started it with TEST_LEVEL and makeSkip

I don't know if it's the best way, but it's probably one of the canonical ones:

# test_a.py

import pytest

@pytest.mark.known_fail_zrx25
def test_a():
    raise RuntimeError("This test should fail")

# conftest.py

import pytest

def pytest_addoption(parser):
    parser.addoption(
        "--run-known-fail-zrx25",
        action="store_true",
        default=False,
        help="Run tests marked as known_fail_zrx25",
    )


def pytest_collection_modifyitems(config, items):
    if not config.getoption("--run-known-fail-zrx25"):
        skip_known_fail = pytest.mark.skip(reason="Skipping tests marked as known_fail_zrx25")
        for item in items:
            if "known_fail_zrx25" in item.keywords:
                item.add_marker(skip_known_fail)


def pytest_configure(config):
    # register an additional marker
    config.addinivalue_line("markers", "known_fail_zrx25: This test is known to fail on zrx25")

# test_runner.py

import pytest

FLAGS = 1  # from sys.argv

if __name__ == "__main__":
    if FLAGS & 1:
        pytest.main(["-vv"])  # Don't run tests marked as known_fail_zrx25 etc.
    elif FLAGS & 2:
        pytest.main(["-vv", "--run-known-fail-zrx25"])  # run tests marked as known_fail_zrx25 etc.

FLAGS = 1

=================================================== test session starts ===================================================
(...)                                                                                                        

tests/test_a.py::test_a SKIPPED (Skipping tests marked as known_fail_zrx25)                                          [100%]

=================================================== 1 skipped in 0.02s ==================================================== 

FLAGS = 2

=================================================== test session starts ===================================================
(...)                                                                                                  

tests/test_a.py::test_a FAILED                                                                                       [100%]

======================================================== FAILURES ========================================================= 
_________________________________________________________ test_a __________________________________________________________ 

    @pytest.mark.known_fail_zrx25
    def test_a():
>       raise RuntimeError("This test should fail")
E       RuntimeError: This test should fail

tests\test_a.py:6: RuntimeError
================================================= short test summary info ================================================= 
FAILED tests/test_a.py::test_a - RuntimeError: This test should fail
==================================================== 1 failed in 0.17s ====================================================

[CEXT-Dan]

Wondering it there’s a better way to organized tests. Thinking out loud

Have a root folder pyrxtest, that holds the media, results, and configuration
Then have a folder tests, that has sub folders, modules names, then those would have the test files.

The test runner would traverse the sub folders, and automatically add test files it finds, and runs them
Each test file would be based off a template with the proper imports and maybe a setup and tear down

Setup a whole new test folder ‘pyrxtest’, then the old tests could be migrated over

-pyrxtest
--testmedia
--testResults
---tests
----PyDb
-----test_pydbpoint.py
-----test_pydbcurve.py
-----test_pydbline.py
----PyAp
-----test_appstuff.py
----PyGe
-----test_getstuff.py

adding a test would be as simple as adding a new file, the runner would pick it up

[CEXT-Dan]

I guess this is kind of what I was thinking of https://docs.pytest.org/en/stable/explanation/goodpractices.html#python-test-discovery

[gswifort]

Yes, exactly, pytest will collect tests if you name them appropriately.
Do you want me to take care of this task?

[CEXT-Dan]

That would be awesome!

[gswifort]

There will probably be tools that are not ARX-based, so there is no need to test them on all hosts (but they can still be tested in the host environment). Do you think it's a good idea to keep these tests next to tests that are host-dependent, but marked e.g. with @pytest.mark.host_independent?

[CEXT-Dan]

Good idea!

[gswifort]

I don't understand how this works:

PyRx/unitTests/testcfg.py

Lines 37 to 47 in 0801469

	def makeSkip(flags: ETFlags):
	if TEST_LEVEL == 1:
	return (False, "Full test")
	if ETFlags.eARX & flags:
	return (True, "known failure in ARX")
	if ETFlags.eBRX & flags:
	return (True, "known failure in BRX")
	if ETFlags.eGRX & flags:
	return (True, "known failure in GRX")
	if ETFlags.eZRX & flags:
	return (True, "known failure in ZRX")

It looks like you're skipping a test on all hosts if it fails on at least one.

Do you change this code manually depending on the host?

[CEXT-Dan]

Looks like I messed that up lol, I was attempting to make a bitwise flag to make the skip decorator easier

@unittest.skipIf(*testcfg.makeSkip(testcfg.ETFlags.eGRX|testcfg.ETFlags.eZRX))
#skip if this is in GRX or ZRX

[CEXT-Dan]

I think I have it fixed, thanks for pointing that out!

[CEXT-Dan]

The old tests are in disarray, so it will take some time to get them organized and moved. I can do the tedious stuff.
I read that there’s a way to output the results in HTML. I wonder if this is something worthwhile?
My thought is, I can run the tests each release, and publish the results

[gswifort]

The old tests are in disarray, so it will take some time to get them organized and moved. I can do the tedious stuff.

It's probably a good idea, you know the tests and their structure better. But I would probably have to clarify the instructions/guidelines.

I read that there’s a way to output the results in HTML. I wonder if this is something worthwhile?

I think this can be useful as long as the test name (what is displayed in the results) clearly indicates what functionality is or isn't working.

[CEXT-Dan]

I tried your commit, did sub-directories too, totally awesome!!!
The runner should open the test media files, so they can be cached, you can create a copy of testmedia folder while the new runner ins under construction.

Let me know if there are any flags I need to catch/process. There’s flags in config that may be useful dev_mode, tracemalloc, parser_debug, not sure. I can also put CAD in a “test” state, possibly enable stuff inside C++... TBD

[gswifort]

Let me know if there are any flags I need to catch/process. There’s flags in config that may be useful dev_mode, tracemalloc, parser_debug, not sure. I can also put CAD in a “test” state, possibly enable stuff inside C++...

I don't know. And I don't think I fully understand the question.

[gswifort]

I think you should create a testing branch in your repo and merge it with main once we're done migrating the tests.

[CEXT-Dan]

I don't know. And I don't think I fully understand the question.

I get the command line arguments in before Python is initialized, I would like the option to setup certain flags that may be helpful, for example

./run_tests –tf0xFFFFFFFF --hosts acad, where I can unpack the hex to setup certain conditions. I.e.
PYMEM_ALLOCATOR_DEBUG : default memory allocators with debug hooks, so I can detect memory leaks.
This would allow me to setup allocator flags in my code as well.

Of course, release compiles should test with no flags so we’re testing what the client gets.
I guess I’m asking if it’s possible to forward flags for me to use.

If not no worries, I can use my .INI, command like is cooler though

I think you should create a testing branch

I don’t think it’s necessary, it’s already isolated. I’ll stay out of it until you say it’s ready
I should have made a branch for my ActiveX task lol. I’ll be pushing hard on that for the next couple weeks

[gswifort]

Yes, it can be passed. If you need only one hex argument, it could be for example --pyinit <hex>. I have also thought about passing arguments to pytest, but they must be separated from other arguments. I think the best way is

run_tests <runner_and_pyinit_args> --pytest <only_pytest_args_after_this_mark>

[gswifort]

#177

[gswifort]

Moving unitTests to tests

• Test filenames should have the prefix test_ (this can be configured in pytest.ini if needed).

• Test classes do not have to be derived from unittest.TestCase, but they can be.

• Instead of using self.assert*(...), the built-in assert should be used (it's not necessary to change this for existing tests, what's there now will work fine too, it's just less elegant).

• I think that current tests should be stored in tests/test_core to separate them from tests for python tools.

• For better understanding of what is done in a test case, the assert statement should contain a description (especially for known failures).

  def test_something():
      expected = 1
      assert numsheets == expected, f"sheet count should be {expected} when (...)"

  ========================================== FAILURES ========================================== 
  _______________________________________ test_something _______________________________________ 
  
      def test_something():
          expected = 1
  
  >       assert numsheets == expected, f"the number of sheets should be {expected} when (...)"  
  E       AssertionError: the number of sheets should be 1 when (...)
  E       assert 2 == 1
  
  demo.py:9: AssertionError
  ================================== short test summary info =================================== 
  FAILED demo.py::test_something - AssertionError: the number of sheets should be 1 when (...)  
  

• Databases should be loaded using @pytest.fixture. If they are global fixtures (scope="session"), they should be placed in conftest.py. If they are specific to a module, they can be placed in that module. It's as simple as this:

  # conftest.py
  @pytest.fixture(scope="session")
  def db_06457():
      db = _load_db("06457.dwg")
      yield db
  
  # test_Database.py
  class TestDatabase:
      def test_dbopbjectforread(self, db_06457: Db.Database):  # db_06457 is passed by pytest
          ...

  pytest will load the database on demand (so if it's not needed, for example, if you run tests in a limited scope, it won't be loaded) and will cache it for other test functions (according to scope="session"). After yield, you can perform cleanup, but in this case, it's probably unnecessary.

[CEXT-Dan]

Really excited about this, the last company I worked for used CppUnit, the setup was a bit complex to add tests, people (me) wrote minimal tests. I need to have good coverage, because I’m really good a breaking things : )

[gswifort]

Enable this option in vscode settings to get automatic type hint detection for pytest fixtures.

"python.analysis.inlayHints.pytestParameters": true,

[CEXT-Dan]

Excellent, I get the hex flags!

[gswifort]

I'll look at the tests in more detail, maybe I'll find some things to improve (simplify). And I understand that you want to take care of the transfer then..?

[CEXT-Dan]

I understand that you want to take care of the transfer

Yes, I want to take advantage of the new flexibility the framework has to offer.
Are there any (practical) limits on the folder depth or number of files?

• I would like the tests to also serve as documentation, as the tests will always be in sync with the current behavior.
• I would like to be able to scan the directory tree so I can get a bird’s-eye view of test coverage.

I’m thinking one class, one test file, I.e.
test_dbobject
test_entity
test_curve
test_line
…

Easy for users to find to better understand behavior
Easy for contributors, they know where to add the test
Of course, there must be higher level tests where classes interact

Is this overkill? I may not be able to get every method, of every class. Though I can start off with constructors and the commonly used methods.

Thoughts?

[gswifort]

I think that the number of modules is like the frequency of commits - it's hard to overdo it :)

[gswifort]

I assume this is a bad test:

PyRx/unitTests/UnitTestMisc.py

Line 28 in c45c8fc

self.assertTrue(img.IsOk, True)

You probably meant:

self.assertTrue(img.IsOk(), True)

unittest checks something like this here:

assert bool(img.IsOk) is True

and almost every object in python gives a boolean value True.

If you used

assert img.IsOk is True

then pytest would catch the error (if IsOk is a method and not a property).

I know, I can check it, but I wanted to write it anyway :)

[CEXT-Dan]

I'll check that!

[CEXT-Dan]

One thing I noticed is that
from pyrx import Ap

Doesn’t work for me, but this does
from pyrx_imp import Ap, Ax

When I work in C++, I have everything uninstalled, and I set my stub path to the repository, I guess this is the reason.
seems If I want to do TDD, then I need to use pyrx_imp

"terminal.integrated.env.windows": {
        "PYTHONPATH": "M:/Dev/Projects/PyRxGit/pyrx/;${env:PYTHONPATH}"
},

[gswifort]

But that's still interesting. vscode should automatically add the working directory to the Path (extraPaths). So what's your root in vscode?

[CEXT-Dan]

Tests that fail in an application context have pretty bad behavior; I have to go to the task manager and kill the process because it’s locked.
This was the same with the old test, when they fail, they fail spectacularly lol
Lucky there’s only a few of these tests.

If I want to disable a test for multiple platforms, I would do something like this, right?

    @pytest.mark.known_failure_ZRX
    @pytest.mark.known_failure_BRX
    @pytest.mark.known_failure_GRX
    def test_getset_active_document(self):
        Ap.DocManager().executeInApplicationContext(active_document, None)

[gswifort]

Yes, markers are OK.

[CEXT-Dan]

But that's still interesting. vscode should automatically add the working directory to the Path (extraPaths). So what's your root in vscode?

I have no idea. I had worked with the Pylance team in issue [#23], one of the work arounds was to use a workspace.
I can’t be bother with that, I just want to double click on a python file and have it work, lol

[gswifort]

Ok, if you are not working in a workspace then it will not work as I wanted. But I think it should not be as troublesome for you as you think. Starting a workspace is simply calling code . from the work directory, in your case

M:\Dev\Projects\PyRxGit> code .

Then when you run files from the explorer and vscode detects that a given file is in this workspace (in root or subdirectory) it will open it in the workspace session (it will not open a new window but a file in an already open session).

Or even simpler, place in the main directory (M:\Dev\Projects\PyRxGit), the file PyRx.code-workspace with the content:

{
	"folders": [
		{
			"path": "."
		}
	],
	"settings": {}
}

and run it by double-clicking.

[CEXT-Dan]

Seems there’s tests that will need to run under a specific context
Example. test_AxApplication -> test_getset_active_document fails in AutoCAD if its not run in a session context.
I added the session flag, but that might have side effects to, or might require a document lock.

The old test probably also fails in session

[CEXT-Dan]

scratch that, I decided to use executeInApplicationContext when needed, 90% of tests are in a command context