feat: add StringWithConcurrency function #202
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposal for adding
StringWithConcurrency
function tofile_instance
.When testing metro2 file generation with a large amount of segments with moov metro2, I noticed that the
String()
function infile_instance
can take a decent amount of time (i.e. ~15 minutes on my machine for close to ~200k segments).To alleviate this, I implemented a simple
StringWithConcurrency
function in our fork that takes in a desired # of concurrent goroutines, splits up the base segment processing into that amount of chunks, processes them in parallel, and merges them at the end.In my testing, leveraging this function has allowed us to reduce the
String()
generation time by over tenfold.We are now using this function in our fork for furnishment process - I figured I would open a PR if moov thinks it is valuable to the main repo. Please let me know what you think - thanks!