(CMSIS-NN Integration) Testing strategy

[AdrianLundell]

Let's use this issue to align on the testing for the Cortex-M backend. I suggest the following tests, but I am happy to hear other opinions:

Kernel tests

All kernels should have GTests similar to the quantization ops to separate testing kernel logic from the full executorch framework. These tests should be easily be built and run on an FVP through some helper script. This should simplify debugging of kernels significantly.

Operator integration tests

We should write a cortex_m implementation of https://github.com/pytorch/executorch/blob/main/backends/test/harness/tester.py to get the kernel integration tests for free, stealing some code from the arm backend to run pte:s on FVP using semihosting.

Python code tests

We should have some level of unittesting of the python code. We can reuse the tester for easily counting the expected number of ops before/ after passes. For testing the dialect ops we should write some helper functions for asserting that they produce expected shapes and results, if they do not exist already?

Network tests

In principle we can reuse the tester framework also for running full network tests instead of having to keep integrating cortex_m support into the run.sh script/ arm_aot_compiler. We should still have some proper full flow test which is also useful as an example for users, I would suggest something close to the https://github.com/pytorch/executorch/blob/main/examples/arm/ethos_u_minimal_example.ipynb but for cortex_m in that case.

Thoughts?

@digantdesai @psiddh @per @zingo

[digantdesai]

We follow the same setup in XNNPACK. Obviously except running on an FVP. Given I want to eventually convert backends/cortex_m into backends/mcu (i.e. generic), let's build the testing abstractions such a way that we can layer different ISA or different optimized library without too much code dedup. This also goes for Simulator setup and adding it as a stage in the pipeline.

[digantdesai]

Meta: if you want @AdrianLundell you can sketch up a small design doc and we can put it up as RFC here as well. Or just add more details in this post.

[AdrianLundell]

Sure, I'll do that

[psiddh]

@AdrianLundell At the moment, I am using this dedicated script (based off run.sh) to test the newly added custom kernels (with CMSIS-NN integ)