Fixing tests

[samcoppock]

as my recent questions show I am trying to fix tests.

We are on winter 1.2
The background is we have a project which started as a October v1 project and is now a winter 1.2 project.

The TESTS DO WORK but they use the bootstrap.php, pluginTestCase.php and testcase.php files which were originally found in the /tests folder ( we still have this folder in the project even thou it has been removed from winter 1.2 )

I am attempting to get the test to work using the new files found on 'modules/system/tests'

PluginTestCase.php has been significantly changed and does not perfectly match the october v1 or winter v1 / v1.1 versions of this file.

TestCase.php is identical to the copy found in winter 1.0

bootstrap.php has been significantly changed and does not perfectly match the october v1 or winter v1 / v1.1 versions of this file.
( was this file replaced with modules/system/tests/bootstrap/app.php )?

we have a file called 'AbstractBackendControllerTestCase.php' which our functional tests extend ( instead of PluginTestCase.php ) and it extends 'PluginTestCase.php '

With that in mind....

I have 2 questions in this post

before we upgraded to winter 1.2 we used the phpunit command to run tests - something like this

'php -dauto_prepend_file=./vendor/winter/storm/src/Support/helpers.php ./vendor/bin/phpunit -c plugins/xxx/xxx/phpunit.xml || EXIT_CODE=$?

note the '-dauto_prepend_file=' option

If we set '-dauto_prepend_file=' to ANY of the following files

• ./vendor/winter/storm/src/Support/helpers.php
• 'modules/system/tests/bootstrap/app.php'
• ./tests/bootstrap.php ( our modified version which I mentioned above and may be the same as 'modules/system/tests/bootstrap/app.php' )
• ./tests/bootstrap.php ( october v1 version )
• ./tests/bootstrap.php ( winter 1 version )

In the case of the bootstrap.php files the following block of code needs to be removed

$loader->addDirectories([
'modules',
'plugins'
]);

Then the tests work.

If -dauto_prepend_file= is not used to specify one of these files then certain tests fail. They are ALWAYS tests which make a http call to an admin page and always a 'status 500 when status 200 was expected' fail.

Attempting to run the tests using the 'winter:test' command causes the same issue.

Obviously i cannot expect anyone on here to know anything about the modified versions of the files but based on the non modified one which cause it to work is anyone able to tell me what using -dauto_prepend_file may be doing to make the tests which fail without it work.

currently the functional tests extend the '/tests/PluginTestCase.php file - and when we do this they work.
however if I change then to extend the 'modules/system/tests/bootstrap/PluginTestCase.php' version of the file they all throw errors.

Using die() statements i have discovered that this is because the 'Artisan::call('winter:up');' call which is made in the first version of the file works fine but when the same command is run from the second ( current ) version of the file it terminates during this call. I proved this by inserting die statements before and after the call in both version of the file.
This results in our tests failing because some of our code which is supposed to execute after this point does not get executed ( remember i said we have a custom file called 'AbstractBackendControllerTestCase.php' which all of the functional tests extend which in turn extends PluginTestCase )

EXTRA NOTE

in a previous ticket i said

"In the most recent version of Winter "tests/bootstrap.php" no longer exists and appears to have been replaced with "modules/system/tests/bootstrap/app.php"

is this a correct assessment?

I based this on huge identical/near identical blocks of code which i found in both files and is what lead me to try using bootstrap/app.php ad the file which -dauto_prepend_file= is pointed to.

[LukeTowers]

@samcoppock if you have modifications to the bootstrap.php file then you're going to have to figure out what modifications specifically are required for your use case before you can switch to using the newer one entirely.

is anyone able to tell me what using -dauto_prepend_file may be doing to make the tests which fail without it work

There's no way to tell without actually looking at your code directly, but it sounds like they were doing something wrong with how they were loading unit tests and applied that option as a way to fix whatever it is that they broke (similar to some of the other issues you've raised here previously with the codebase). That option has never been required or recommended to run unit tests in October / Winter as far as I can recall.

currently the functional tests extend the '/tests/PluginTestCase.php file - and when we do this they work.
however if I change then to extend the 'modules/system/tests/bootstrap/PluginTestCase.php' version of the file they all throw errors.

Sounds like you have a modified version of PluginTestCase, in which case you're going to need to extend the core version of that class with your own and apply the necessary modifications there, and then extend that class instead of the core one in your other tests.

In the most recent version of Winter "tests/bootstrap.php" no longer exists and appears to have been replaced with "modules/system/tests/bootstrap/app.php"

Close enough, but again, without seeing the actual modifications present in your codebase (that's not an invitation to share them here btw, this is a complex mess and it's not going to be solved piecemeal in github comments) I can't guarantee that you can swap one for the other.

Overall it sounds like you could benefit from some dedicated support time to unravel this mess; generally speaking you can't expect to get free support to untangle this kind of mess when those solutions have little to no chance of actually being useful for anyone else.

[bennothommo]

@samcoppock I remember with October, there used to be a race condition for the helpers defined in the Rain library versus those defined with Laravel. Part of PHPUnit's bootstrapping is loading Composer dependencies before the app itself being tested, but October and Winter have their own bootstrap that load their own helpers first (from Rain and Storm respectively) before loading Composer dependencies, which meant that the helpers defined in Rain and Storm took precedence over those defined in Laravel. You can see it here. Hence why you needed the -d flag with the php command to load those helpers first, and reserve their function names, before PHPUnit loaded the Composer dependencies and thus took on Laravel's helpers.

This was one of the reasons we introduced the php artisan winter:test command - to automatically do this and not require the user to make such overrides happen.

I would assume that the reason the tests work when you include any of the 5 files you listed is because all of those will try to load the Winter/October helpers first before any Composer dependencies are autoloaded. But this is purely an assumption, given that you have specified that you've created your own custom test bootstrappers.

To answer your second part, yes, we've moved the entire test framework inside the modules, and yes, the base Test classes now reside in the System module. Our intention is that for people to use the php artisan winter:test command going forward, to ensure that the correct bootstrapping occurs for Winter in order to run your tests. Hence, you no longer need to define the bootstrap in your phpunit.xml config, however, we still do allow people to write custom bootstrappers if they are willing to take on all risk of ensuring their bootstrappers work and are up-to-date by defining a bootstrap value.

If you are encountering errors with our command and our bootstrap process, please give us the error messages and any context you can provide, as this will assist us to ascertain the problem.