more detailed test output (see #909)

[rbri]

i know this is a bit of a hack but i like to focus on fixing this cases instead of making super cool code here

[gbrail]

Thanks -- this is helpful.

How do you feel about having it always print out the list of tests that didn't pass in all cases, versus only printing if a system property or something is set? We'll have to look at those messages for quite a long time as I suspect that some of them will take a while to fix.

[rbri]

@tonygermano, @tuchida, @p-bakker what do you think?

[tuchida]

Isn't Optional<String> getMessage better?
https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html

[p-bakker]

Very useful indeed.

Do wonder if a different format would be better (and doable), one where the results aren't encoded in a string like fails in strict mode but passes in non-strict mode for all opt level(s) -1, 0, 9, but where I could take to output and parse it (for example a csv format and pasting it into Excel) and then analyse it.

For example to find all tests that fail on all optLevels in either strict or non-strict mode or find all tests that always fail in non-interpreted mode

[gbrail]

Thinking more about this...

I'm afraid that if we add lots of output to a "successful" build then we'll end up masking other problems because people will just ignore everything.

I would prefer if this instead printed out one message like this:

Warning:
27 tests failed in strict mode but passed in non-strict mode
99 tests failed for some optimization levels and passed for others
See buildGradle/reports/inconsistent-tests.txt for detail

[rbri]

#930 is the future