Roadmap for pyansys/actions: looking for a common CI/CD strategy

[jorgepiloto]

📝 Background
As pyansys/actions get adopted by projects, more features are requested. For example:

• The ability to skip repository checkouts
• Avoid Python installations
• Selecting desired artifacts

The more flexibility we provide, the more different our CI/CD pipelines may look.

Projects may be different but they operate over some common points: server installation, testing, documentation, and artifacts publishing.

Experience demonstrates that is better to delegeate artifact creation to projects while artifacts should be managed by pyansys/actions.

⚙️ A common strategy for our CI/CD pipelines

With all previous information, I would like to start the discussion about a common strategy for our CI/CD pipelines. By strategy I mean:

• Should we encourage the usage of a single workflow YML file or multiple ones?

• How should this workflow file be named? Examples: ci_cd.yml, ci.yml, main.yml...

• How do we structure our CI/CD pipelines? Example:

  `style` -> `doc build` -> `doc deploy` -> `build` -> `release`
          -> `tests smoke` -> `test slow` ->
  

• How many servers should we test against to? Which is the optimal way of testing this? Cartessian product of n_servers * n_python_versions? Too much CI/CD consumption?

📌 Mission
All previous questions define the roadmap of pyansys/actions. The transition to the agreed roadmap needs to be smooth and solid.

Projects that are not compliant with pyansys/actions should not be able to use the latest features.

Individual projects should adapt to pyansys/actions, not the opposite. This is the reason behind looking for a common CI/CD strategy.

[germa89]

Pinging @pyansys/pyansys-core for visibility/feedback

[MaxJPRey]

How should this workflow file be named? Examples: ci_cd.yml, ci.yml, main.yml...

I am becoming quite an advocate of ci_cd.yml. I think that the name reflects the idea that in this workflow we are handling every aspects of devops: code style/tests/doc/packaging/release.
It is very subjective and I am open to any other suggestions.

[RobPasMue]

I'm fine with ci_cd.yml as long as there are no other 80 yml files doing CI/CD operations 😄

[MaxJPRey]

How do we structure our CI/CD pipelines? Example:
I think the global idea of having one 'thread' for the documentation and one for testing like pygeometry for instance is easy to follow and logical:
code style -> build and smoke tests ->testing
documentation style -> documentation build

I would say that for the doc to be deployed we need to be sure that the tests are passing. Otherwise we are deploying content for wrong source code introducing known issues and defects.
And then at the end the deployment, packaging and release jobs.

[RobPasMue]

PyGeometry pipelines are more complex than that... and there is always a reason.

The key concept is setting up dependent stages in such a way that the workflow is easy to follow and that the dependencies make sense:

• We start with two independent threads for code (Path A) and documentation (Path B) style checks.

Path A

• If code checks pass, the smoke tests are triggered on Path A. Smoke tests are also the ones that generate the wheels for each OS. In these stages what we should do is build the Python package and do a simple __version__ import. This will let us know that all the dependencies are resolvable by pip and that the package has no apparent issues.
• Smoke tests should only be run on each pipeline for Linux (and/or Windows if needed). macOS smoke tests should only be run when releasing.
• If smoke tests went fine, then we head onto the testing stage. This stage should basically run our unit tests and measure the coverage. If needed, it should be tested against multiple servers. Nothing else to add here.

Path B

• If the docs style checks went fine, we build the documentation.

Final Path

• Once Path A and Path B have successfully run, we head onto the packaging stage. This stage basically builds the wheel for our package.
• If the packaging went fine, then we deploy our dev docs (this should only occur for whatever commit we do on the main branch, and not on release branches)
• If the packaging went fine and we are on the process of releasing, then we head onto the releasing stage. We should release to PyPI (private or public) and to GitHub. All artifacts that are meaningful should be added to the GitHub release. In regards to PyPI, we should ensure that the wheels for each Python version and OS are also uploaded - some repos are not doing that.
• If the release process went fine, then and only then you publish the release docs. Taking down documentation is a pain, and we should only publish them if everything went fine. That's why it's on the last stage.

Misc

I hope I made myself clear. One of the key concepts here, as I already mentioned are "dependencies". Dependent jobs allow to control the workflow and avoid running useless stages. For example, if the code style checks fail (which only take seconds to run) the workflow stops.

For public repos it might not be an issue, but for private/internal repos, whatever time we spent running our workflows on GitHub runners, we are charged for it. So we should try to avoid useless resource consumption.

Even if it's only for environmental reasons, it's more than enough to set up dependencies on your workflow...

[germa89]

I definitely agree with this. We should reduce resource consumption as much as possible.

I'm aware that PyMAPDL is quite generous in that regards, and it is because of the high failure rate we have with some jobs (local mainly). I would like to fix it... eventually. But I can't for now.

[MaxJPRey]

Yes no worries @RobPasMue that is actually what I meant when I highlighted the usage of concurrent jobs when mentioning "threads" and when I talked about job cancellation for doc it was just an example for the "dependencies concept."
The reference to PyGeoemetry was an easy way for me to explain my main thoughts with a visual workflow graph .

[germa89]

If it difficult to impose an action on all repos, less to impose a whole CICD workflow. It is going to be difficult. We can enforce "stages" (doc style, tests, etc) and their APIs ("doc dev deploy" will expect a zipped file named documentation-html in ./documentation). But going further seems to me imposing "corsets" on projects. #My2Cents

[MaxJPRey]

Agreed for instance, not all products will have container and servers.

[RobPasMue]

I probably wasn't clear with my message. I wasn't trying to impose PyGeometry's stages on the rest of the projects. I was explaining the reasons behind having a dependent-like workflow, and what the stages should look like. I understand that every project will have its own "pecularities". But in general, they should look alike.

However, I do think that setting up dependencies is a "must" in terms of billable time... It's the only way to control costs.

[germa89]

No no... definitely I'm with you on that. We should have clear dependencies between jobs, and clear requirements so we avoid consuming resources as much as possible.

My comments wasn't exactly a response to yours, rather an open question about the nature of this discussion. Although I think it will be fruitful, I'm not sure if it will be very applicable.