"Creating directory failed" in GitLab CI

[JanWennrichPCSG]

When executing phive install in GitLab CI as a non-root user, a "creating directory failed" error occurs.

You can see an example here: https://dev.quiqqer.com/quiqqer/quiqqer/-/jobs/6108

I tried to debug this and came to the following conclusion:

• the repository/project is automatically cloned by GitLab CI with the root user as the owner and chmod 666 (see example)
• the user executing phive install is www-data (see example)
• if the phive.xml already exists, the method PharIo\FileSystem\Directory->ensureExists() tries to chmod the project directory (see code)
• as the executing user is www-data and thus does not own the project directory, the chmod fails due to lack of permission

Am I doing something wrong or is this a bug?
Is PHIVE even meant to be used in CI to install tooling?
Is it correct to use the phive.xml file or should I use a different file for that?

[theseer]

Am I doing something wrong or is this a bug?

I'm leaning to the conclusion of considering that a bug.

In general, I'd say the idea of updating the phive.xml in CI runs is rather useless default behavior and we should change that. It can be avoided by specifying --temporary. That should also fix the problem you have.

Is PHIVE even meant to be used in CI to install tooling?

It depends on your workflow and dev process - and the type of project at hand:

Quiqqer is an application. As such it is likely to have a bunch of dependencies, for runtime as well as development time needs. How are these updated? If that's an explicit decision made by devs, there is no reason to repeat the installation of these dependencies at CI and/or deployment times. Rather, these dependencies should be checked in along with the project into its repository. Making it self-contained.

That moves all dep solving issues as well as availability of upstream sources into the development time - where a potential unavailability of say github is not a serious problem.

It may be considered differently for libraries or other types of components meant to be reused by others. One option here would be to use .gitattributes and exclude certain files and directories from snapshots and releases. As tools like composer usually do not clone a repository but extract a release archive, that would work nicely.

It may also be considered differently, if you need different versions of things for a CI matrix run. Then, reinstalling things might actually be the only way to make that work. But then, you're not exactly testing the same software but many slightly different variants..

So, long story short: Yes, PHIVE should work without issues in CI. I so far never had permission problems in CI with PHIVE but that might be just luck.

Is it correct to use the phive.xml file or should I use a different file for that?

The phive.xml file is correct. You could alternatively store it in .phive/phars.xml. This would make sense if you'd need additional files - for auth keys for instance when accessing private repositories - and in the hopefully near future local resolving information storage alongside the phars to install. PHIVE will keep understanding both locations and use whatever it finds. (It will complain if there are both..)

[JanWennrichPCSG]

Thank you for your answer, @theseer!

Using --temporary indeed resolves the root permission issue, thanks!
But we're now running into problems with caching the PHAR files between CI runs:

tl;dr: PHIVE home directory has to be placed in project folder and requires a "complicated" command. Is there a simpler way to do this?

Quiqqer is an application. As such it is likely to have a bunch of dependencies, for runtime as well as development time needs. How are these updated?

QUIQQER is somewhat of a hybrid. It's an application (CMS) but also some kind of a framework.
It can be extended with various plugins/packages.
At the moment there are no real development-time dependencies, as we just started to integrate various tooling (e.g. PHPUnit, PHPStan, etc.) into our development processes.

Putting those dependencies into the repository would definetly work and simplify the workflow for developers and in CI.
But we'd also like to use similar tooling (e.g. PHPUnit) in all the plugins/packages.
Uploading various PHAR files in around 100 repositories seems unfeasible.

Thus we'd prefer to use PHIVE, to just have a phive.xml in every repository that specifies the required PHAR files.

For example we're now using the following phive.xml:

<?xml version="1.0" encoding="UTF-8"?>
<phive xmlns="https://phar.io/phive">
  <phar name="phpunit" version="^10.5.5" installed="10.5.5" location="./tools/phpunit" copy="false"/>
  <phar name="phpstan" version="^1.10.54" installed="1.10.54" location="./tools/phpstan" copy="false"/>
</phive>

In CI, we're now using the following command:

phive --no-progress install --temporary --copy --trust-gpg-keys 4AA394086372C20A,51C67305FFC2E5C0

This works great and puts the specified PHARs into the tools/ folder.

The only issue we've got now is caching the PHARs between CI runs:
The tools/ directory is marked to be cached and the PHARs are successfully restored on every CI run.
But the above mentioned CI command makes PHIVE reinstall all PHAR files, even though they already exist.
For an example see: https://dev.quiqqer.com/quiqqer/quiqqer/-/jobs/6622#L27

I assume this happens because PHIVE does not remember which PHARs are installed because it's home directory is not cached?

As a workaround I tried putting the PHIVE home directory into the project directory to cache it as well:

phive --no-progress --temporary --home .phive-home install --trust-gpg-keys 4AA394086372C20A,51C67305FFC2E5C0,033E5F8D801A2F8D

This does indeed work, but makes the command fairly long and "complicated" (compared to an composer install and caching vendor/).
Is there a simpler way to achieve our desired behaviour?

Maybe with the feature you mentioned to be available in the future?

local resolving information storage alongside the phars to install

[theseer]

Using --temporary indeed resolves the root permission issue, thanks!

Glad it helped.

But we're now running into problems with caching the PHAR files between CI runs:

tl;dr: PHIVE home directory has to be placed in project folder and requires a "complicated" command. Is there a simpler way to do this?

You can set an environment variable PHIVE_HOME as an alternative means to specifying it. Not sure if that makes it any easier.

Thus we'd prefer to use PHIVE, to just have a phive.xml in every repository that specifies the required PHAR files.

For example we're now using the following phive.xml:

<?xml version="1.0" encoding="UTF-8"?>
<phive xmlns="https://phar.io/phive">
  <phar name="phpunit" version="^10.5.5" installed="10.5.5" location="./tools/phpunit" copy="false"/>
  <phar name="phpstan" version="^1.10.54" installed="1.10.54" location="./tools/phpstan" copy="false"/>
</phive>

If you set copy to true here, you don't have to pass it as a CLI arg.

In CI, we're now using the following command:

phive --no-progress install --temporary --copy --trust-gpg-keys 4AA394086372C20A,51C67305FFC2E5C0

This works great and puts the specified PHARs into the tools/ folder.

The only issue we've got now is caching the PHARs between CI runs: The tools/ directory is marked to be cached and the PHARs are successfully restored on every CI run. But the above mentioned CI command makes PHIVE reinstall all PHAR files, even though they already exist. For an example see: https://dev.quiqqer.com/quiqqer/quiqqer/-/jobs/6622#L27

Given you all files copied and cached, the CI command to install tools only needs to be run when the phive.xml changed.
Why don't you implement that step with a rules condition? That way, every build that has a changed phive.xml runs the update on CI and otherwise, the step is skipped.

I'm not convinced that is something phive should be doing by itself.

I assume this happens because PHIVE does not remember which PHARs are installed because it's home directory is not cached?

Partially. This foremost is a philosophical decision: As we do not technically control the tools directory, we have no idea what state it is in when the phive install command is run. You could, for instance, have manually copied a phar into it, replacing a phive installed one, modified the symlink to point to somewhere else,...

Instead of spending a lot of (development) effort on figuring out, whether or not that is the case, we decided to simply enforce the installation as defined by the phive.xml by overwriting whatever is there.

As a workaround I tried putting the PHIVE home directory into the project directory to cache it as well:

phive --no-progress --temporary --home .phive-home install --trust-gpg-keys 4AA394086372C20A,51C67305FFC2E5C0,033E5F8D801A2F8D

This does indeed work, but makes the command fairly long and "complicated" (compared to an composer install and caching vendor/). Is there a simpler way to achieve our desired behaviour?

This is "just" symlinking - given you had copy=false in the phive.xml. Technically it's still doing the same operation as before - meaning we replace any existing installation in tools.

Maybe with the feature you mentioned to be available in the future?

As written above the main point of potentially doing a redundant install is that we want to ensure the correct files to be in place no matter what has happend to the local "copy". We could of course enhance that by trying to understand the state of the files. But that, for instance, would require us to do a sha-something on the file content to catch modifed phars, resolv symlinks to see if they point to the correct location.. Can be done, but hasn't been on my agenda to be honest.

[JanWennrichPCSG]

Thanks again, for that extensive answer, @theseer!

Why don't you implement that step with a rules condition? That way, every build that has a changed phive.xml runs the update on CI and otherwise, the step is skipped.

That's a good idea! I'll try that out and see if this makes our CI simpler.

---

Should this issue stay open to fix the permission bug without the --temporary flag?
If not, the issue can be closed in my opinion.

Thank you for your help and suggestions!

[theseer]

I'll keep it open for now :)