Test failures on Mac with gfortran

[YaqiWang]

Hello,

I am trying to compile NJOY and run its test on my Mac. Its Chip is Apple M1 Ultra, macOS is Ventura 13.6.1. The compiler I am using is gfortran 11.0.1. Because a compiling error, I turned off errorm module. I am able to build NJOY successfully. But when I run tests with make test, I get this at the end of the screen output:

69% tests passed, 24 tests failed out of 78

Total Test time (real) = 581.50 sec

The following tests FAILED:
	  4 - Test04 (Failed)
	  5 - Test05 (Failed)
	 15 - Test15 (Failed)
	 16 - Test16 (Failed)
	 17 - Test17 (Failed)
	 18 - Test18 (Failed)
	 20 - Test20 (Failed)
	 21 - Test21 (Failed)
	 24 - Test24 (Failed)
	 25 - Test25 (Failed)
	 27 - Test27 (Failed)
	 32 - Test32 (Failed)
	 34 - Test34 (Failed)
	 46 - Test46 (Failed)
	 47 - Test47 (Failed)
	 48 - Test48 (Failed)
	 55 - Test55 (Failed)
	 59 - Test59 (Failed)
	 65 - Test65 (Failed)
	 67 - Test67 (Failed)
	 68 - Test68 (Failed)
	 69 - Test69 (Failed)
	 70 - Test70 (Failed)
	 74 - Test74 (Failed)

Among those failed tests, it appears tests 4/5/15/16/17/18/20/27/34/46/47/65 use errorm module, so their failures are expected. There are several other tests failed due to gold file diffs. The full test report file is attached FYI.

LastTest.log

Are those failures severe? What are the cause of those failures and is it possible to fix them?

Thank you!

[whaeck]

I have been seeing test failures on M1 systems (arm64 architecture) due to roundoff issues myself (I have an M1 too). As far as I can see, none of the differences are large enough to warrant caution. You may want to try using -DCMAKE_BUILD_TYPE=Debug to see if these test failures persist (on my M1, no tests fail in Debug mode but there are failures in Release, which makes me think the issue is in optimisation).

I assume that you have not been unable to upgrade your compiler to gcc-12 or higher? Changing to a different version of gcc other than gcc-11 would still be the best solution to solve the ERRORR problem. I don't think it would change the failure of the other tests though.

[YaqiWang]

Thank you very much for the prompt reply!

I attempted to set a looser relative tolerance in execute.py under the tests folder from 1e-9 to 1e-5 and still saw some failures. For example in Test 69, I saw:

1.784555E-05 and 1.785413E-05 are not equal
3.177341E-05 and 3.176969E-05 are not equal
4.462906E-05 and 4.462512E-05 are not equal
6.248081E-05 and 6.247912E-05 are not equal
8.925846E-05 and 8.925942E-05 are not equal

But I guess this is ok.

I confirmed that debug version indeed does not have those diffs. Only Test 48 and 59 are still failing with different numbers of lines in the reference and trial files.

I am using a pre-build conda package (https://mooseframework.inl.gov/getting_started/installation/conda.html) that includes more than compilers for convenience. I did not attempt to set up my own environment. Turning off ERRORR should be a temporary solution until the package is upgraded.

Thanks again.