pytorch-optimizer v2.11.1

Change Log

Feature

• Implement Tiger optimizer (#192)
  • A Tight-fisted Optimizer
• Implement CAME optimizer (#196)
  • Confidence-guided Adaptive Memory Efficient Optimization
• Implement loss functions (#198)
  • Tversky Loss : Tversky loss function for image segmentation using 3D fully convolutional deep networks
  • Focal Tversky Loss
  • Lovasz Hinge Loss : The Lovász-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks

Diff

2.11.0...2.11.1

pytorch-optimizer v2.11.0

Change Log

Feature

• Implement PAdam optimizer (#186)
  • Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Implement LOMO optimizer (#188)
  • Full Parameter Fine-tuning for Large Language Models with Limited Resources
• Implement loss functions (#189)
  • BCELoss
  • BCEFocalLoss
  • FocalLoss : Focal Loss for Dense Object Detection
  • FocalCosineLoss : Data-Efficient Deep Learning Method for Image Classification Using Data Augmentation, Focal Cosine Loss, and Ensemble
  • DiceLoss : Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations
  • LDAMLoss : Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
  • JaccardLoss
  • BiTemperedLogisticLoss : Robust Bi-Tempered Logistic Loss Based on Bregman Divergences

Diff

2.10.1...2.11.0

pytorch-optimizer v2.10.1

Change Log

Feature

• Implement Prodigy optimizer (#183)
  • An Expeditiously Adaptive Parameter-Free Learner

Fix

• perturb isn't multiplied by -step_size in SWATS optimizer. (#179)
• chebyshev step has size of T while the permutation is 2^T. (#168, #181)

Diff

2.10.0...2.10.1

pytorch-optimizer v2.10.0

Change Log

Feature

• Implement Amos optimizer (#174)
  • An Adam-style Optimizer with Adaptive Weight Decay towards Model-Oriented Scale
• Implement SignSGD optimizer (#176)
  • Compressed Optimisation for Non-Convex Problems
• Implement AdaHessian optimizer (#176)
  • An Adaptive Second Order Optimizer for Machine Learning
• Implement SophiaH optimizer (#173, #176)
  • A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
• Implement re-usable functions to compute hessian in BaseOptimizer (#176, #177)
  • two types of distribution are supported (Gaussian, Rademacher).
• Support AdamD feature for AdaHessian optimizer (#177)

Diff

2.9.1...2.10.0

Contributions

thanks to @i404788

pytorch-optimizer v2.9.1

Change Log

Fix

• fix weight decay in Ranger21 (#170)

Diff

2.9.0...2.9.1

pytorch-optimizer v2.9.0

Change Log

Feature

• Implement AdaMax optimizer, #148
  • A variant of Adam based on the infinity norm
• Implement Gravity optimizer, #151
  • a Kinematic Approach on Optimization in Deep Learning
• Implement AdaSmooth optimizer, #153
  • An Adaptive Learning Rate Method based on Effective Ratio
• Implement SRMM optimizer, #154
  • Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates
• Implement AvaGrad optimizer, #155
  • Domain-independent Dominance of Adaptive Methods
• Implement AdaShift optimizer, #157
  • Decorrelation and Convergence of Adaptive Learning Rate Methods
• Upgrade to D-Adaptation v3, #158, #159
• Implement AdaDelta optimizer, #160
  • An Adaptive Learning Rate Method

Docs

• Fix readthedocs build issue, #156
• Move citations into table, #156

Refactor

• Refactor validation logic, #149, #150
• Rename amsbound, amsgrad terms into ams_bound, #149
• Return gradient instead of the parameter, AGC. #149
• Refactor duplicates (e.g. rectified step size, AMSBound, AdamD, AdaNorm, weight decay) into re-usable functions, #150
• Move pytorch_optimizer.experimental under pytorch_optimizer.*.experimental

Diff

2.8.0...2.9.0

pytorch-optimizer v2.8.0

Change Log

Feature

• Implement A2Grad optimizer, #136
  • Optimal Adaptive and Accelerated Stochastic Gradient Descent
• Implement Accelerated SGD optimizer, #137
  • Accelerating Stochastic Gradient Descent For Least Squares Regression
• Implement Adaptive SGD optimizer, #139
  • Adaptive Gradient Descent without Descent
• Implement SGDW optimizer, #139
  • Decoupled Weight Decay Regularization
• Implement Yogi optimizer, #140
  • Adaptive Methods for Nonconvex Optimization
• Implement SWATS optimizer, #141
  • Improving Generalization Performance by Switching from Adam to SGD
• Implement Fromage optimizer, #142
  • On the distance between two neural networks and the stability of learning
• Implement MSVAG optimizer, #143
  • Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients
• Implement AdaMod optimizer, #144
  • An Adaptive and Momental Bound Method for Stochastic Learning
• Implement AggMo optimizer, #145
  • Aggregated Momentum: Stability Through Passive Damping
• Implement QHAdam, QHM optimizers, #146
  • Quasi-hyperbolic momentum and Adam for deep learning
• Implement PID optimizer, #147
  • A PID Controller Approach for Stochastic Optimization of Deep Networks

Bug

• Fix update in Lion optimizer, #135
• Fix momentum_buffer in SGDP optimizer, #139

pytorch-optimizer v2.7.0

Change Log

Features

• Implement AdaNorm optimizer, #133
  • AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs
• Implement RotoGrad optimizer, #124, #134
  • RotoGrad: Gradient Homogenization in Multitask Learning
• Implement D-Adapt Adan optimizer, #134
• Support AdaNorm variant, #133, #134
  • AdaBelief
  • AdamP
  • AdamS
  • AdaPNM
  • diffGrad
  • Lamb
  • RAdam
  • Ranger
  • Adan
• Support AMSGrad variant, #133, #134
  • diffGrad
  • AdaFactor
• Support degenerated_to_sgd, #133
  • Ranger
  • Lamb

Refactor

• Rename adamd_debias_term to adam_debias, #133
• Merge the rectified version with the original, #133
  • diffRGrad + diffGrad -> diffGrad
  • RaLamb + Lamb -> Lamb
  • now you can simply use with rectify=True

Fix

• Fix previous_grad deepcopy issue in Adan optimizer. #134

pytorch-optimizer v2.6.1

Change Log

Fix

• variables are not located on the same device with the gradients, #132 (related to #131) (thanks to @Bing-su)
• fix approximate_sq_grad() in Adafactor optimizer, #132

pytorch-optimizer v2.6.0

Change Log

Feature

• Implement SM3 optimizer, #130
  • Memory-Efficient Adaptive Optimization
• Tweak Scalable Shampoo optimizer, #128, #129
  • implement a new preconditioner type, OUTPUT.
  • optimize speed/memory usage of coupled Newton iteration and power iteration methods.
  • use in-place operation (SQRT-N Grafting).
  • clean-up shampoo_utils more readable.
  • support skip_preconditioning_rank_lt parameter to skip preconditioning in case of the low-rank gradient.
  • set default value for preconditioning_compute_steps to 1000.
  • set default value for start_preconditioning_step to 25.