added the docs directory with design report for kpm sparse checkout #334
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # To get started with Dependabot version updates, you'll need to specify which | ||
| # package ecosystems to update and where the package manifests are located. | ||
| # Please see the documentation for all configuration options: | ||
| # https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates | ||
|
|
||
| version: 2 | ||
| updates: | ||
| - package-ecosystem: "gomod" # See documentation for possible values | ||
| directory: "/" # Location of package manifests | ||
| schedule: | ||
| interval: "weekly" | ||
| commit-message: | ||
| prefix: "Chore: " | ||
| include: "scope" | ||
| ignore: | ||
| - dependency-name: k8s.io/* | ||
| - package-ecosystem: "github-actions" | ||
| directory: "/" | ||
| schedule: | ||
| interval: "weekly" | ||
| commit-message: | ||
| prefix: "chore: " | ||
| include: "scope" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| ## KPM Sparse-Checkout feature | ||
Peefy marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ## 1. Introduction | ||
| kpm which is the package management tool for KCL, does not support sparse-checkout, which means an issue arises when dealing with monorepos, which may contain many KCL packages. We need to get a solution to download specific packages instead of all in a monorepo. | ||
|
|
||
| ## 2. User Worklow | ||
| - Adding a Subdirectory Dependency via Command Line | ||
| - Specifying a Subdirectory in `kcl.mod` | ||
| - Adding a Subdirectory Dependency via Configuration File | ||
|
|
||
| ## 3. Command Line Inteface | ||
| We will try to keep the command line interface as simple and straightforward as possible for adding a Git repository subdirectory as a dependency. The below steps show how a user can use the feature: | ||
|
|
||
| - Adding a Dependency : | ||
| The following command will lead to the addition of a dependency: | ||
| `kcl mod add <repository_url> <subdirectory_path>` | ||
|
|
||
| This command will add the specified subdirectory as a dependency in the current KCL project. | ||
|
|
||
| - Non-interactive Method : | ||
| In addition to the command line interface, users can specify subdirectory dependencies directly in the `kcl.mod` file, which can be particularly useful for automated environments. | ||
|
|
||
| ## 4. Example Use-case | ||
| Considering the nginx-ingres module, on typing the command | ||
|
|
||
| ``` | ||
| kcl mod add https://github.com/kcl-lang/modules/tree/main/nginx-ingress /restrict-ingress-anotations | ||
|
There was a problem hiding this comment. I just have a question, why don't you just merge these two parameters, something like this? I'm not sure what the problems will be behind such a merging, so you need to consider it or have a research. There was a problem hiding this comment. Done the changes, i did some research and gave thought to it but i don't think there won't be much of any problem. If any problems arise during implementation, i'll adjust accordingly. |
||
| ``` | ||
|
|
||
| The following command will lead to the addition of restrict-ingress-anotations package to the current KCL project and will update the kcl.mod accordingly. | ||
|
|
||
| Alternatively, for non-interactive environments, users can directly edit the `kcl.mod` file: | ||
| ``` | ||
| [package] | ||
| name = "test" | ||
| edition = "v0.9.0" | ||
| version = "0.0.1" | ||
|
|
||
| [dependencies] | ||
| my_dependency = { git = "https://github.com/example/repo", subdirectory = "path/to/package" } | ||
|
There was a problem hiding this comment. The problem here is the same as above. Any problem like this ? |
||
|
|
||
| ``` | ||
|
|
||
| ## 5. Specifying how kcl.mod will specify the added subdirectories | ||
| To support the specification of a subdirectory for dependencies that are sourced from Git repositories, we need to extend the kcl.mod file structure. This involves adding an optional `subdirectory` field under each dependency. This can be manually edited or automatically updated by the kcl mod add command. | ||
|
|
||
| A sample kcl mod for the same would look like this. | ||
|
|
||
| ``` | ||
| [package] | ||
| name = "test" | ||
| edition = "v0.9.0" | ||
| version = "0.0.1" | ||
|
|
||
| [dependencies] | ||
| my_dependency = { git = "https://github.com/example/repo", subdirectory = "path/to/package" } | ||
| ``` | ||
|
|
||
| ## 6. Integration and the use of go-getter to download the specific subdirectories | ||
|
There was a problem hiding this comment. This chapter I think could need more details. There was a problem hiding this comment. Ok, I'll add some on my own, though can you still point out some specific details (if any) you're looking for. |
||
|
|
||
| The repoUrl field in the struct `CloneOptions` in kpm/pkg/git/git.go will be given the subdir url accordingly, which then downloads each selected subdirectory one by one. KCL currently uses go-getter to download using URL's with ease. | ||
|
|
||
| We can also provide a fallback mechanism to download the entire repo if the subdirectory download fails repeatedly by something like - | ||
|
|
||
| ```Go | ||
| err := downloadSubdir(repoUrl, subdir) | ||
| if err != nil { | ||
| log.Printf("Failed to download subdirectory, falling back to full repository: %v", err) | ||
| return downloadRepo(repoUrl) | ||
| } | ||
|
|
||
| ``` | ||
|
|
||
| ## 7. Conclusion | ||
| By extending the kcl.mod file to include a subdirectory field for Git-based dependencies, users can now specify and manage dependencies that reside in specific subdirectories of a Git repository. This enhancement will enable kpm to download only the necessary parts of large repositories, significantly improving performance when dealing with monorepos. The user interface remains intuitive, and the implementation leverages Git's sparse-checkout feature to achieve the desired functionality even when used in automated CI/CD pipelines. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.