Add LLM support to Brute Search #769
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Merged
model_analyzer/config/generate/perf_analyzer_config_generator.py
Outdated
Show resolved
Hide resolved
nv-hwoo
reviewed
Oct 11, 2023
model_analyzer/config/generate/perf_analyzer_config_generator.py
Outdated
Show resolved
Hide resolved
model_analyzer/config/generate/perf_analyzer_config_generator.py
Outdated
Show resolved
Hide resolved
model_analyzer/config/generate/perf_analyzer_config_generator.py
Outdated
Show resolved
Hide resolved
model_analyzer/config/generate/perf_analyzer_config_generator.py
Outdated
Show resolved
Hide resolved
model_analyzer/config/generate/perf_analyzer_config_generator.py
Outdated
Show resolved
Hide resolved
model_analyzer/config/generate/perf_analyzer_config_generator.py
Outdated
Show resolved
Hide resolved
model_analyzer/config/generate/perf_analyzer_config_generator.py
Outdated
Show resolved
Hide resolved
Co-authored-by: Hyunjae Woo <107147848+nv-hwoo@users.noreply.github.com>
Co-authored-by: Hyunjae Woo <107147848+nv-hwoo@users.noreply.github.com>
Co-authored-by: Hyunjae Woo <107147848+nv-hwoo@users.noreply.github.com>
Co-authored-by: Hyunjae Woo <107147848+nv-hwoo@users.noreply.github.com>
|
Need to keep an eye on triton-inference-server/tutorials#59. This will affect input tensor names (e.g. "PROMPT" -> "text_input"). |
nv-hwoo
reviewed
Oct 11, 2023
nv-hwoo
approved these changes
Oct 12, 2023
nv-braf
added a commit
that referenced
this pull request
Nov 3, 2023
* Adding new options for LLM (#768) * Update README and versions for 23.09 branch (#761) (#767) * Adding new options for LLM * Fixing codeQL issues * Fixing codeQL issue --------- Co-authored-by: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> * Add LLM support to Brute Search (#769) * Initial coding complete * First unit test passing * Adding test for prompt length * Refactor PACG methods * Further refactoring * Ensure early exit isn't enabled for LLM models * Fix type checking errors * Attempt at fixing codeql issue * Revert "Attempt at fixing codeql issue" This reverts commit 2619b83. * Attempt at codeQL fix * Adding deepcopy back in * Removing deepcopy in an attempt to fix codeQL errors * Update model_analyzer/config/input/config_command_profile.py Co-authored-by: Hyunjae Woo <107147848+nv-hwoo@users.noreply.github.com> * Update model_analyzer/config/generate/perf_analyzer_config_generator.py Co-authored-by: Hyunjae Woo <107147848+nv-hwoo@users.noreply.github.com> * Update model_analyzer/config/generate/perf_analyzer_config_generator.py Co-authored-by: Hyunjae Woo <107147848+nv-hwoo@users.noreply.github.com> * Update model_analyzer/config/generate/perf_analyzer_config_generator.py Co-authored-by: Hyunjae Woo <107147848+nv-hwoo@users.noreply.github.com> * Moving location of method * Changing parameter to inference load * Changing parameter to inference load * Changing prompt length to text input length * Changing max_tokens to use request-parameter * Fix input-data typo * Changing non-parameter to parameter --------- Co-authored-by: Hyunjae Woo <107147848+nv-hwoo@users.noreply.github.com> * New LLM record types (#770) * New measurement fields created. * Fixing omission in llm_metric_table * Changing name to be avg_token_to_token... * New config options based on live run (#775) * Added new config options and modified existing options * Refactoring model parameter setting * Removing magic numbers * Capture LLM metrics from PA (#774) * Initial code for aggregation of new LLM metrics * New measurement fields created. * Fixing PA unit tests * Adding hooks in metrics to capture new LLM fields * Fixing codeQL errors * Fixing type checking errors * Changes needed post-merge from other branches * Revert naming mistake (due to merge). * Changes uncovered during live testing * Fixes based on hwoo review * Fixing typo * Change to use lists and mean() * Changes based on hwoo review * Correct how periodic concurrency works in PACG (#777) * Created a new class ConfigRangeNumeric and using it for periodic-concurrency * Fixes and defaults for periodic concurrency * First unit test passing * PACG chagnes complete. Unit tests updated and passing * Removing uneeded class * Fixing codeQL and hwoo's review suggestions * Adding missing else * Llm testing live run (#778) * Created a new class ConfigRangeNumeric and using it for periodic-concurrency * Fixes and defaults for periodic concurrency * First unit test passing * PACG chagnes complete. Unit tests updated and passing * Removing uneeded class * Changes to fix live run * Minor refactor and cleanup * Removing json files * Changing to use f-string * More cleanup from hwoo CR * Removing stale code for request period * Fix nit * Changes to get LLM summary reports working (#779) * Changes to get LLM summary reports working * Addressing hwoo's CR * Adding illegal LLM checks w/ unit testing + some minor cleanup (#781) * Adding illegal LLM checks w/ unit testing + some minor cleanup * Updated with TMA * Misc LLM cleanup (#782) * General cleanup * Add ticket nums to todos * Fix for non-LLM breaking bug introduced. * summary table in progress --------- Co-authored-by: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> Co-authored-by: Hyunjae Woo <107147848+nv-hwoo@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The bulk of the changes are to the Perf Analyzer Config Generator (PACG). This has undergone a fairly significant refactor in an attempt to cleanly support LLM config generation.
For now the assumption is that early exit is not an option for LLM models, but this can be added a later time once we have added support for LLM objectives to the measurement class.
I've updated unit testing and added a couple new tests that check generation of PA configs for LLM models.