Failed (parallel) test run is considered a success

[foobear]

After a (parallel) geordi tests run (project has RSpec + Cucumber), I get this output:

Failing Scenarios:
cucumber foo.feature:9 # Scenario: Foo
cucumber bar.feature:27 # Scenario: Bar

233 scenarios (2 failed, 231 passed)
3864 steps (1 failed, 22 skipped, 3841 passed)

Took 250.528129601 seconds

> Successfully ran tests.

It claims tests were successful while they failed.

Also (probably as a result of that), the exit code is 0 (success) which would not stop chained commands after a test run.

[codener]

The tests command itself relies on parallel_tests to return the correct exit code, so I believe that's the reason.

Am 06.02.2015 um 15:13 schrieb Arne Hartherz notifications@github.com:

After a parallel cucumber run, I get this output:

Failing Scenarios:
cucumber foo.feature:9 # Scenario: Foo
cucumber bar.feature:27 # Scenario: Bar

233 scenarios (2 failed, 231 passed)
3864 steps (1 failed, 22 skipped, 3841 passed)

Took 250.528129601 seconds

Successfully ran tests.
But my tests failed.
Also, the exit code is 0 (success) which would not stop chained commands after a test run.

�
Reply to this email directly or view it on GitHub.

[codener]

I think I know what's wrong: tests calls geordi test with system and not exec, so the exit status gets lost.

Am 06.02.2015 um 15:13 schrieb Arne Hartherz notifications@github.com:

After a parallel cucumber run, I get this output:

Failing Scenarios:
cucumber foo.feature:9 # Scenario: Foo
cucumber bar.feature:27 # Scenario: Bar

233 scenarios (2 failed, 231 passed)
3864 steps (1 failed, 22 skipped, 3841 passed)

Took 250.528129601 seconds

Successfully ran tests.
But my tests failed.
Also, the exit code is 0 (success) which would not stop chained commands after a test run.

�
Reply to this email directly or view it on GitHub.

[codener]

Similarly all other proxy binaries.

Am 06.02.2015 um 15:13 schrieb Arne Hartherz notifications@github.com:

After a parallel cucumber run, I get this output:

Failing Scenarios:
cucumber foo.feature:9 # Scenario: Foo
cucumber bar.feature:27 # Scenario: Bar

233 scenarios (2 failed, 231 passed)
3864 steps (1 failed, 22 skipped, 3841 passed)

Took 250.528129601 seconds

Successfully ran tests.
But my tests failed.
Also, the exit code is 0 (success) which would not stop chained commands after a test run.

�
Reply to this email directly or view it on GitHub.

[triskweline]

One more thing: By default Thor exits with error 0 when there is a failure

This sounds like a horrible default. We could change this by this

class Geordi::CLI < Thor
  def self.exit_on_failure?
    true
  end
end

I'm not sure if this would negatively affect any scripts. I have seen at least one instance where we're abusing this: The tests command doesn't short-circuit if e.g. RSpec fails before Cucumber.

[triskweline]

On the other side, it looks like @codener expected tests to short-circuit since he prints "Successfully ran tests" in the last line.

[codener]

For me, geordi tests has always failed early, as expected, when rspec or cucumber tests fail.

Am 06.02.2015 um 16:44 schrieb Henning Koch notifications@github.com:

On the other side, it looks like @codener expected tests to short-circuit since he prints "Successfully ran tests" in the last line.

�
Reply to this email directly or view it on GitHub.

[codener]

Ok, this is probably not Geordi's fault. See cucumber/common#635.