Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improving Mixed precision experience #985

Merged
merged 13 commits into from
Mar 11, 2024
Merged

Improving Mixed precision experience #985

merged 13 commits into from
Mar 11, 2024

Conversation

ofirgo
Copy link
Collaborator

@ofirgo ofirgo commented Mar 10, 2024

Pull Request Description:

Several minor modifications to Mixed Precision to improve the user experience and stability:

  • Disabling Hessian-based weights for a mixed precision metric by default (reduce run-time significantly and improve consistency)
  • Modifying distance metric to take the maximal value between the maximal configuration value and other computed values with additional epsilon, for numerical stability.
  • Adding a convenient API for choosing a distance metric weighting method (MpDistanceWeighting enum) - enables to use the last-layer metric which, in some cases, can produce more consistent results.
  • Improve results logging.
  • Remove default mp config object (unused and can cause confusion).
  • Slight modification to mixed precision tests.

Checklist before requesting a review:

  • I set the appropriate labels on the pull request.
  • I have added/updated the release note draft (if necessary).
  • I have updated the documentation to reflect my changes (if necessary).
  • All function and files are well documented.
  • All function and classes have type hints.
  • There is a licenses in all file.
  • The function and variable names are informative.
  • I have checked for code duplications.
  • I have added new unittest (if necessary).

Ofir Gordon added 6 commits March 7, 2024 15:34
Copy link
Collaborator

@eladc-git eladc-git left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some small comments.

@ofirgo ofirgo requested a review from eladc-git March 11, 2024 08:08
@ofirgo ofirgo merged commit f411aac into sony:main Mar 11, 2024
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants