feat(flow): implement adaptive throttler #196
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… throughput control
- Fix gocritic issues (elseif, ifElseChain) - Fix gosec G115 integer overflow - Fix revive unused parameter issues - Fix unconvert unnecessary conversions - Fix line length violations (>120 chars) - Rename CPUUsageModeRusage → CPUUsageModeReal - Move goroutineHeuristicSampler to dedicated file - Update gopsutil from v4.25.10 → v4.24.6 for Go 1.21 compatibility
…mmented-out configuration examples
- Rename CPUUsageModeReal to CPUUsageModeMeasured for clarity - Add clarifying comment to setSystemMemoryReader function The new name better reflects that even gopsutil measurements are approximations rather than absolute 'real' values.
…ed environments - Add cgroup v2/v1 auto-detection for accurate memory stats in containers - Normalize CPU usage sampling by core count for better accuracy - Support custom memory readers in ResourceMonitor for flexible deployments - Add comprehensive system_memory tests with mock file systems - Improve test reliability with polling-based waiting instead of fixed sleeps
fix(flow): emit partial sliding window on early close (reugn#192)
- Implement a demo application showcasing the adaptive throttler in action
reugn
reviewed
Nov 20, 2025
Co-authored-by: Eugene R. <reugpro@gmail.com>
Co-authored-by: Eugene R. <reugpro@gmail.com>
reugn
reviewed
Nov 20, 2025
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #196 +/- ##
===========================================
- Coverage 99.53% 89.07% -10.47%
===========================================
Files 13 20 +7
Lines 651 1227 +576
===========================================
+ Hits 648 1093 +445
- Misses 2 95 +93
- Partials 1 39 +38 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
reugn
changed the title
reugn
reviewed
Nov 21, 2025
…/sysmonitor package Refactor CPU and memory monitoring code from the flow package into a dedicated internal/sysmonitor package. This improves code organization, testability, and platform-specific implementation support. Key changes: - Move CPU sampling logic to internal/sysmonitor with platform-specific implementations for Darwin, Linux, Windows, and fallback - Move system memory monitoring to internal/sysmonitor with support for: - Linux: /proc/meminfo, cgroup v1, and cgroup v2 detection - Darwin: Mach kernel APIs with CGO - Windows: GlobalMemoryStatusEx API - Fallback: Error handling for unsupported platforms - Move heuristic CPU sampler from flow/cpu_sampler_heuristic.go to internal/sysmonitor/heuristic.go - Update adaptive_throttler and resource_monitor to use new package - Remove flow/cpu_sampler.go, flow/system_memory.go
Updated the getProcessCPUTimes function to use GetCurrentProcess for the current process, enhancing reliability and eliminating the need to open a process handle. This change simplifies the code and improves performance when retrieving CPU times.
Author
|
@reugn Changes applied. Ready for re-review. |
reugn
reviewed
Nov 22, 2025
reugn
reviewed
Nov 22, 2025
…at errors - Implemented a safety check to ensure the number of CPUs is at least 1 in both Darwin and Linux CPU sampling functions. - Updated memory reading functions to handle potential errors gracefully, defaulting to 0 for unavailable stats.
…ed cpuTimesStat struct
…sCPUTimes function
…rottling process Co-authored-by: Eugene R. <reugpro@gmail.com>
…/go-streams into feature/adaptive-throttler
Co-authored-by: Eugene R. <reugpro@gmail.com>
…quirement Co-authored-by: Eugene R. <reugpro@gmail.com>
Co-authored-by: Eugene R. <reugpro@gmail.com>
…ource statistics collection Co-authored-by: Eugene R. <reugpro@gmail.com>
…its functionality Co-authored-by: Eugene R. <reugpro@gmail.com>
…ctions to resource_monitor.go
…in NewAdaptiveThrottler calls
…Pointer for improved type safety
…cessary mutex locking
…hod on AdaptiveThrottlerConfig
… documentation and improve code clarity
…tor registry with reference counting - Replace singleton ResourceMonitor with shared registry pattern for efficient resource sharing - Add dynamic interval updates and improved CPU mode switching - Enhance memory sampling with better fallback handling - Add InDelta assertion helper for precise float comparisons - Comprehensive test refactoring with improved mocking and coverage - Update adaptive throttler tests to use new resource monitor architecture
kxrxh
force-pushed
the
feature/adaptive-throttler
branch
from
16077b7 to
6e24e80
Compare
Author
|
@reugn Can you please re-run workflows (codecov, golangci-lint)? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Modifications
ResourceMonitorwith pluggablecpuUsageSampler(heuristic & measured modes) andmemoryReader(cgroup v1/v2 & host support).AdaptiveThrottlerConfigfor tuning thresholds, adaptation steps, and sampling intervals.Verify change