[JENKINS-76325] Organization Folder: Support creating multiple projects from a single repository based on multiple Jenkinsfile paths

[xinnj]

Description:

This PR implements the feature requested in JENKINS-76325, enabling Organization Folders to create multiple projects from a single repository when multiple Jenkinsfiles are detected.

Changes:

1. UI Enhancement:​ Added configuration options to the Organization Folder:

   • Checkbox to enable/disable multi-project mode

   • Regex pattern field to extract project name suffix from Jenkinsfile paths

   

2. Core Logic:​ Modified project recognition to create separate projects for each valid Jenkinsfile found, rather than stopping at the first match.

Testing done

1. Automation unit tests added
2. Manual testing verified correct project creation and naming
3. Backward compatibility maintained when feature is disabled

Submitter checklist

• Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• Ensure that the pull request title represents the desired changelog entry
• Please describe what you did
• Link to relevant issues in GitHub or Jira
• Link to relevant pull requests, esp. upstream and downstream changes
• Ensure you have provided tests that demonstrate the feature works or the issue is fixed

[jtnord]

what would happen if I had a repository called repo1-win as well as one called repo1 containing Jenkinsfile-win in the same organization?
Additionally this would seem to introduce conflicts with things like checks (esp GH where each job would send the same check (continuous-integration/jenkins/pr-head) which can be seen here?

Currently if users setup multiple orgainzation folders for each of their marker files they can use https://plugins.jenkins.io/github-scm-trait-notification-context/ to set a specific trait on the root folder, but if this is all configured in one folder how would a user be able to differntiate? (I would guess similar issues exists for Gerrit and other branch sources that report back using a "well known" id)

[jglick]

Cannot speak for @jtnord but I am not inclined to accept any significant enhancements to the multibranch-related plugins at this point, only safe-looking fixes for clear bugs. See the (lengthy) discussions in linked Jira issues. The broader issue is that multibranch was deliberately designed to be quite opinionated, in a way that works well for common cases but is impractical to extend to more complex and idiosyncratic repository structures (especially monorepos). Something along the lines of jenkinsci/job-dsl-plugin#2072 could offer a far more extensible system, for those users willing to accept trickier setup, without requiring plugin code changes and thus making Jenkins developers the bottleneck.

[xinnj]

@jtnord Thank you for your feedback. Here are some additional clarifications:

what would happen if I had a repository called repo1-win as well as one called repo1 containing Jenkinsfile-win in the same organization?

The outcome depends on the order in which repositories are returned by the SCM source. The repository scanned last will take precedence. During testing on a Gitea server, I observed that repositories created later are scanned later. For example, if repo1 is created after repo1-win, the resulting Jenkins project will be named repo1-win, but its SCM source will point to repo1.

Although this behavior is not ideal, it is at least predictable. Repository maintainers should be aware of this dynamic when enabling the feature, and can avoid potential conflicts by carefully choosing repository names, Jenkinsfile names, and regex patterns accordingly.

Additionally this would seem to introduce conflicts with things like checks (esp GH where each job would send the same check (continuous-integration/jenkins/pr-head) which can be seen here?

This is a known issue that can occur whenever multiple Jenkins projects share the same SCM resource, regardless of how they are created. The feature described should not be enabled if this conflict is a concern for your use case.

Currently if users setup multiple orgainzation folders for each of their marker files they can use https://plugins.jenkins.io/github-scm-trait-notification-context/ to set a specific trait on the root folder, but if this is all configured in one folder how would a user be able to differntiate? (I would guess similar issues exists for Gerrit and other branch sources that report back using a "well known" id)

I haven't personally used this plugin, but based on its description, the configuration appears to be applied at the SCM level. The feature I described only modifies the Jenkins project (display) name and folder structure. Since it doesn't alter the underlying SCM configuration or how Jenkins identifies repositories, it likely wouldn't affect the GitHub notification context behavior you're referring to.

Let me know if you need further details on any specific aspect.

[xinnj]

@jglick Thank you for your feedback as well. This plugin has been incredibly helpful for setting up a dozen Jenkins pipelines for our large product with multiple repositories.

Unfortunately, Jenkinsfile and DSL configurations only take effect after a Jenkins project has already been created. The specific challenge we encountered occurs during the initial project creation phase, which is where this plugin provides the solution we needed.

[jtnord]

Although this behaviour is not ideal, it is at least predictable

Depending on the server, ordering from servers can be unstable (a list of repositories is a list, which may or may not be in any order and whose order may or may not remain stable over time), or depend on the time the query was created. When coupled with creating the org folder as code and DR restoring from this backup you may well have inconsistent and or unpredictable results.

Repository maintainers should be aware of this dynamic when enabling the feature, and can avoid potential conflicts by carefully choosing repository names, Jenkinsfile names, and regex patterns accordingly.

Repository maintainers may not even be aware. Organisation folders can filter repositories by access so it would not be a problem today but tomorrow you could have a re-org and introduce a large amount of conflicts. So today I maybe not able to even see the existence of the foo-jdk22 repository and be the owner of the foo repository

Additionally this would seem to introduce conflicts with things like checks (esp GH where each job would send the same check (continuous-integration/jenkins/pr-head) which can be seen here?

This is a known issue that can occur whenever multiple Jenkins projects share the same SCM resource, regardless of how they are created.

This is a solved problem today, for example you apply the trait mentioned above in the case of GitHub.

I haven't personally used this plugin, but based on its description, the configuration appears to be applied at the SCM level

The option is defined in the MBP and or Organsisation folder Item. As such with this solution you can not apply it so that different jobs from the same repo but different Jenkinsfile get different checks.
With this solution you can not set a trait in the job configuration for each different Jenkinsfile and so it becomes a problem that you do not have if you created separate organization folders (or MBPs) or each Jenkinsfile

--

There are also other project recognizers, not all recognizers use a file in the SCM). e.g. https://github.com/jenkinsci/remote-file-plugin/blob/9c0cac6de2053acb1af25916f9dd25febe8694de/src/main/java/org/jenkinsci/plugins/workflow/multibranch/extended/RemoteJenkinsFileWorkflowMultiBranchProjectFactory.java#L24 / https://github.com/jenkinsci/inline-pipeline-plugin/ causing the reflection used here to break and this recognizer to not be available.

Due to these issues that I know about (and I am sure there will be several that I do not know about) , its does not appear that this will integrate well into in the wider ecosystem, causing enhanced maintenance effort. This is somewhat echoed (more strongly) by Jesse's comment. At this current moment in time I am inclined to agree with Jesse that this is not something I am willing to accept at this time.