Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementation of C++ metrics parallelization #734

Merged
merged 9 commits into from
Jun 17, 2024

Conversation

dbukki
Copy link
Collaborator

@dbukki dbukki commented Apr 12, 2024

Fixes #733

Added parallelization support to the C++ metrics parser plugin:

  • Using getFilterPathsQuery, we can now perform the same check as what cc::util::isRootedUnderAnyOf used to do, but on the ODB-side. This is good because:
    • The number of records retrieved for manual processing by the client's handler is now drastically reduced.
    • The filePath members can now be removed from views that only needed them for this purpose. This further reduces the amount of data needed to be transferred per record as well.
  • The parallelCalcMetric method of the metrics parser now serves as the backbone of all parallel computations.
  • Existing metrics have been refactored to perform their outermost loop using the new parallelization mechanism.

Further considerations:

  • Buffering: Preload all records to already know the size on startup. (This is a memory overhead, but it might be worth it.)
  • Progress reporting: Building on the idea of buffering, report the current progress of each metric onto the log to increase interactivity. (Reporting at regular intervals in time and/or in steps.)
  • Caching & Flushing: Collect all computed metrics into a single shared thread-safe container and persist everything at once at the end. This could eliminate the "one transaction for each worker" phenomenon currently observed. (Depending on the size of the overhead of opening and closing transactions, this could potentially give us a speed boost.)

@dbukki dbukki self-assigned this Apr 12, 2024
@mcserep mcserep added Kind: Enhancement 🌟 Plugin: C++ Issues related to the parsing and presentation of C++ projects. Plugin: Metrics Issues related to the code metrics plugin. labels Apr 15, 2024
@mcserep mcserep requested review from intjftw and mcserep April 24, 2024 14:03
@mcserep mcserep added this to the Upcoming Release milestone Apr 29, 2024
@mcserep
Copy link
Collaborator

mcserep commented Jun 4, 2024

For some reason the diff contains several changes from unrelated PRs, which were already merged. This makes quite hard to review this PR, and the changes should be rebased to master.

@mcserep
Copy link
Collaborator

mcserep commented Jun 17, 2024

For some reason the diff contains several changes from unrelated PRs, which were already merged. This makes quite hard to review this PR, and the changes should be rebased to master.

Any insights on this @dbukki? Could you refactor this PR to master? I would love to merge this, but it is quite hard to review it like this.

@dbukki
Copy link
Collaborator Author

dbukki commented Jun 17, 2024

Any insights on this @dbukki? Could you refactor this PR to master? I would love to merge this, but it is quite hard to review it like this.

Sorry for not noticing your comment earlier. My branches must have been out of sync by the time I rebased this branch onto my latest version of master to add the parallelization of bumpy road. The funky diff behavior is probably a result of that. I synced my fork, the diff seems alright now.

@mcserep mcserep merged commit e86e58b into Ericsson:master Jun 17, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Kind: Enhancement 🌟 Plugin: C++ Issues related to the parsing and presentation of C++ projects. Plugin: Metrics Issues related to the code metrics plugin.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Parallel parsing in C++ metrics
2 participants