Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactored and optimized segmentation metrics eval #395

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Mintas
Copy link

@Mintas Mintas commented Nov 26, 2024

Refactored and optimized segmentation metrics evaluation.

  1. Refactoring:
    key purpose is to make some of the methods more useful for extension in client libraries, make code more concise, reducing code duplications, thus also increasing possible community support

  2. Optimisation:
    here we note, that some code duplications lead us to recalculate some preparative steps for metric calculation. Reusing common logical steps helped to significantly increase performance. In our microbenchmark of segment.eval() we have achieved 1.58 speedup!
    This may be crucial in some applications that may require intensive metric calculation (like Hyperparameter Optimization)

  3. Benchmarking details:
    We took some symbolic music files (midis) with known expert markup of musical segments. There were 20 data points considered. We ran 300 iterations of eval method over all data provided to get better averaging.

Benchmarking results:
Before: time: 24.702 and latency: 0.0823, avg per data point: 0.0041
After: time: 15.615 and latency: 0.0520, avg per data point: 0.0026
Speedup is: 1.582 (~ -37%)

@Mintas
Copy link
Author

Mintas commented Dec 10, 2024

@bmcfee @craffel
This PR is getting stale, would you consider reviewing and merging it?
TY

@bmcfee
Copy link
Collaborator

bmcfee commented Dec 10, 2024

Sorry I won't have time to review this for a few weeks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants