Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Group and chunk user points for efficient processing #292

Merged
merged 1 commit into from
May 24, 2024

Conversation

halaprix
Copy link
Member

This Pull Request introduces an enhancement to the way we handle user points in our system. Previously, we processed user points individually, which was not efficient when dealing with large volumes of data.

In this PR, we have implemented a new approach where we first group the points by user. This ensures that all points for a specific user are processed together, improving the coherence of our data handling.

Furthermore, we have introduced a chunking mechanism. The chunking mechanism divides the grouped points into chunks of 30 or more. The key aspect of this chunking mechanism is that it ensures all points for a user are in the same chunk. This means that the chunk size doesn't have to be strictly 30 but the first possible size above 30 that can accommodate all points for the last processed user.

This new approach not only improves the efficiency of our data processing but also ensures that all points for a specific user are handled together, thereby maintaining user-specific context.

Changes include:

  • Grouping points by user before processing
  • Dividing grouped points into chunks of size 30 or more
  • Ensuring all points for a user are in the same chunk
  • Adjusting chunk size to accommodate all points for the last processed user

This PR is expected to significantly improve the efficiency and reliability of our user points handling process.

@halaprix halaprix merged commit 16ca1c3 into dev May 24, 2024
2 checks passed
@halaprix halaprix deleted the kk/split-points-by-user-chunks branch May 24, 2024 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants