Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Remote Cache for golangci-lint #4986

Closed
2 of 3 tasks
ryancurrah opened this issue Sep 5, 2024 · 6 comments · Fixed by #5098
Closed
2 of 3 tasks

Support Remote Cache for golangci-lint #4986

ryancurrah opened this issue Sep 5, 2024 · 6 comments · Fixed by #5098
Assignees
Labels
area: cache enhancement New feature or improvement

Comments

@ryancurrah
Copy link
Member

ryancurrah commented Sep 5, 2024

Welcome

Your feature request related to a problem? Please describe

Yes, this feature request is related to the problem of long linting times, especially in large codebases or CI environments where there are multiple runners/agents.

Currently, golangci-lint lacks a mechanism to cache results remotely, which could help in reducing the overall time taken for linting when running it repeatedly over the same or similar code. This lack of caching increases build times and resource usage, especially when dealing with distributed and large build environments.

Describe the solution you'd like

I propose introducing a feature in golangci-lint to support remote caching by abstracting the cache mechanism. Similar to how Go has implemented the GOCACHEPROG environment variable, which allows a custom cache implementation via a child process that communicates using JSON over stdin/stdout, golangci-lint could introduce a GOLANGCICACHEPROG environment variable. This would allow users to specify a binary responsible for handling cache operations, thereby enabling remote caching similar to https://go-review.googlesource.com/c/go/+/486715.

Describe alternatives you've considered

Persisting cache on individual agents. In our environment, multiple agents are used, and they may not always have an up-to-date cache, leading to inconsistencies and longer build times.

Additional context

Willing to implement this myself if it is deemed an acceptable proposal. Remote caching has proven for us to reduce build times in Go projects. By using a similar approach in golangci-lint, users could experience similar reductions in linting times, especially in CI/CD environments or large projects when linting a lot of code. The ability to cache results remotely would lead to more efficient use of resources and faster feedback loops during builds.

Supporter

@ryancurrah ryancurrah added the enhancement New feature or improvement label Sep 5, 2024
@ldez ldez added no decision No decision to fix or not proposal and removed enhancement New feature or improvement labels Sep 5, 2024
@ryancurrah
Copy link
Member Author

Note this feature relies on golang/go#64876

@ryancurrah ryancurrah self-assigned this Oct 29, 2024
@ldez ldez self-assigned this Oct 31, 2024
@ldez
Copy link
Member

ldez commented Oct 31, 2024

Just to clarify why I put no-decision: I'm working on a re-sync of the internal code related to cache.

It requires a lot of time to ensure it is non-breaking and provides a reviewable PR.

My priority is maintainability, so the new feature (if not added by the sync) will be evaluated after the sync because this a really really really long and complex job to just re-sync the code.

The changes in the original code have never been documented, I already added a "summary" but this is not enough to work quickly on this.
I don't want (because I'm not able to do it) to follow 2 things on the same complex topic at the same time.

@ryancurrah
Copy link
Member Author

ryancurrah commented Oct 31, 2024

Did not realize you were refactoring or reimplementing the caching behaviour. Will wait for that and then we can see how to extend it for remote cache.

@ldez ldez added enhancement New feature or improvement area: cache and removed no decision No decision to fix or not proposal labels Oct 31, 2024
@ldez
Copy link
Member

ldez commented Oct 31, 2024

@ryancurrah I updated the internals inside #5098.

As a side effect, the programmatic cache is supported but I didn't test it.

If you plan to create a cache implementation, I'm interested in playing with it or contributing.

@ldez
Copy link
Member

ldez commented Nov 1, 2024

I didn't saw before but there is a test implementation: https://github.com/bradfitz/go-tool-cache/

$ go install github.com/bradfitz/go-tool-cache/cmd/go-cacher@latest
go: downloading github.com/bradfitz/go-tool-cache v0.0.0-20230425225207-ef6c7b1b26e9

$ GOLANGCI_LINT_CACHEPROG="$(which go-cacher) --verbose" ./golangci-lint run
2024/11/01 19:11:42 Defaulting to cache dir /home/ldez/.cache/go-cacher ...
2024/11/01 19:12:03 cacher: closing; 8009 gets (0 hits, 8009 misses, 0 errors); 9233 puts (0 errors)

$ GOLANGCI_LINT_CACHEPROG="$(which go-cacher) --verbose" ./golangci-lint run
2024/11/01 19:12:05 Defaulting to cache dir /home/ldez/.cache/go-cacher ...
2024/11/01 19:12:06 cacher: closing; 153 gets (153 hits, 0 misses, 0 errors); 0 puts (0 errors)

And it works as expected, it's funny.

@ldez ldez closed this as completed in #5098 Nov 2, 2024
@ryancurrah
Copy link
Member Author

Crazy good sir. Sorry I didn't get to review it. I will definitely try it out though. We have a large staging environment for testing all our CI tools. I can test this release on 30k builds in varying projects.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: cache enhancement New feature or improvement
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants