feat: Group and chunk user points for efficient processing #292
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This Pull Request introduces an enhancement to the way we handle user points in our system. Previously, we processed user points individually, which was not efficient when dealing with large volumes of data.
In this PR, we have implemented a new approach where we first group the points by user. This ensures that all points for a specific user are processed together, improving the coherence of our data handling.
Furthermore, we have introduced a chunking mechanism. The chunking mechanism divides the grouped points into chunks of 30 or more. The key aspect of this chunking mechanism is that it ensures all points for a user are in the same chunk. This means that the chunk size doesn't have to be strictly 30 but the first possible size above 30 that can accommodate all points for the last processed user.
This new approach not only improves the efficiency of our data processing but also ensures that all points for a specific user are handled together, thereby maintaining user-specific context.
Changes include:
This PR is expected to significantly improve the efficiency and reliability of our user points handling process.