-
Notifications
You must be signed in to change notification settings - Fork 40
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Adaptive Regularisation with Cubics (ARC) Solver (#277)
* Add the adaptive regularisation with cubics (ARC) solver with different sub solvers, both Lanczos and a gradient based one. * Update changelog. --------- Co-authored-by: mathiasrm1 <103418574+mathiasrm1@users.noreply.github.com>
- Loading branch information
1 parent
da64a03
commit 0ab53d6
Showing
31 changed files
with
1,469 additions
and
49 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,64 @@ | ||
# [Adaptive regularization with Cubics](@id ARSSection) | ||
|
||
|
||
|
||
```@meta | ||
CurrentModule = Manopt | ||
``` | ||
|
||
```@docs | ||
adaptive_regularization_with_cubics | ||
adaptive_regularization_with_cubics! | ||
``` | ||
|
||
## State | ||
|
||
```@docs | ||
AdaptiveRegularizationState | ||
``` | ||
|
||
## Sub solvers | ||
|
||
There are several ways to approach the subsolver. The default is the first one. | ||
|
||
## Lanczos Iteration | ||
|
||
```@docs | ||
Manopt.LanczosState | ||
``` | ||
|
||
## (Conjugate) Gradient Descent | ||
|
||
There are two generic functors, that implement the sub problem | ||
|
||
```@docs | ||
AdaptiveRegularizationCubicCost | ||
AdaptiveRegularizationCubicGrad | ||
``` | ||
|
||
Since the sub problem is given on the tangent space, you have to provide | ||
|
||
``` | ||
g = AdaptiveRegularizationCubicCost(M, mho, σ) | ||
grad_g = AdaptiveRegularizationCubicGrad(M, mho, σ) | ||
sub_problem = DefaultProblem(TangentSpaceAt(M,p), ManifoldGradienObjective(g, grad_g)) | ||
``` | ||
|
||
where `mho` is the hessian objective of `f` to solve. | ||
Then use this for the `sub_problem` keyword | ||
and use your favourite gradient based solver for the `sub_state` keyword, for example a | ||
[`ConjugateGradientDescentState`](@ref) | ||
|
||
## Additional Stopping Criteria | ||
|
||
```@docs | ||
StopWhenAllLanczosVectorsUsed | ||
StopWhenFirstOrderProgress | ||
``` | ||
|
||
## Literature | ||
|
||
```@bibliography | ||
Pages = ["solvers/adaptive-regularization-with-cubics.md"] | ||
Canonical=false | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
function set_manopt_parameter!(M::TangentSpaceAtPoint, ::Val{:p}, v) | ||
M.point .= v | ||
return M | ||
end | ||
function (f::Manopt.AdaptiveRegularizationCubicCost)(M::TangentSpaceAtPoint, X) | ||
## (33) in Agarwal et al. | ||
return get_cost(base_manifold(M), f.mho, M.point) + | ||
inner(base_manifold(M), M.point, X, f.X) + | ||
1 / 2 * inner( | ||
base_manifold(M), | ||
M.point, | ||
X, | ||
get_hessian(base_manifold(M), f.mho, M.point, X), | ||
) + | ||
f.σ / 3 * norm(base_manifold(M), M.point, X)^3 | ||
end | ||
function (grad_f::Manopt.AdaptiveRegularizationCubicGrad)(M::TangentSpaceAtPoint, X) | ||
# (37) in Agarwal et | ||
return grad_f.X + | ||
get_hessian(base_manifold(M), grad_f.mho, M.point, X) + | ||
grad_f.σ * norm(base_manifold(M), M.point, X) * X | ||
end | ||
function (grad_f::Manopt.AdaptiveRegularizationCubicGrad)(M::TangentSpaceAtPoint, Y, X) | ||
get_hessian!(base_manifold(M), Y, grad_f.mho, M.point, X) | ||
Y .= Y + grad_f.X + grad_f.σ * norm(base_manifold(M), M.point, X) * X | ||
return Y | ||
end | ||
function (c::StopWhenFirstOrderProgress)( | ||
dmp::AbstractManoptProblem{<:TangentSpaceAtPoint}, | ||
ams::AbstractManoptSolverState, | ||
i::Int, | ||
) | ||
if (i == 0) | ||
c.reason = "" | ||
return false | ||
end | ||
#Update Gradient | ||
TpM = get_manifold(dmp) | ||
nG = norm(base_manifold(TpM), TpM.point, get_gradient(dmp, ams.p)) | ||
nX = norm(base_manifold(TpM), TpM.point, ams.p) | ||
if (i > 0) && (nG <= c.θ * nX^2) | ||
c.reason = "The algorithm has reduced the model grad norm by $(c.θ).\n" | ||
return true | ||
end | ||
return false | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
0ab53d6
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@JuliaRegistrator register
0ab53d6
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Registration pull request created: JuliaRegistries/General/90162
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via: