Optimization.LBFGS errors with Float32

[CarloLucibello]

The following code works fine when setting T=Float64, and errors on Float32:

using Random, Statistics, LinearAlgebra
using Zygote
using OptimizationOptimJL

T = Float32
ps = randn(T, 2)

function callback(state, l)
    println("Iteration: $(state.iter), Loss: $l")
    return false
end

opt_func = OptimizationFunction(AutoZygote()) do ps, data
    mean(ps.^2)
end

opt_prob = OptimizationProblem(opt_func, ps)

## this works
# using Optimisers, OptimizationOptimisers
# res = Optimization.solve(opt_prob, Optimisers.Adam(); callback, maxiters=10)

res = Optimization.solve(opt_prob, Optimization.LBFGS(); callback, maxiters=10)

ERROR: MethodError: Cannot `convert` an object of type Vector{Float32} to an object of type Float64
The function `convert` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  convert(::Type{T}, ::T) where T
   @ Base Base.jl:126
  convert(::Type{<:Real}, ::T) where T<:SparseConnectivityTracer.AbstractTracer
   @ SparseConnectivityTracer ~/.julia/packages/SparseConnectivityTracer/aHPgm/src/overloads/conversion.jl:10
  convert(::Type{N}, ::D) where {N<:Real, P, T, D<:SparseConnectivityTracer.Dual{P, T}}
   @ SparseConnectivityTracer ~/.julia/packages/SparseConnectivityTracer/aHPgm/src/overloads/conversion.jl:31
  ...

Stacktrace:
 [1] Base.RefValue{Float64}(x::Vector{Float32})
   @ Base ./refvalue.jl:8
 [2] convert(::Type{Ref{Float64}}, x::Vector{Float32})
   @ Base ./refpointer.jl:111
 [3] cconvert(T::Type, x::Vector{Float32})
   @ Base ./essentials.jl:687
 [4] setulb(n::Int64, m::Int64, x::Vector{…}, l::Vector{…}, u::Vector{…}, nbd::Vector{…}, f::Float64, g::Vector{…}, factr::Float64, pgtol::Float64, wa::Vector{…}, iwa::Vector{…}, task::Vector{…}, iprint::Int64, csave::Vector{…}, lsave::Vector{…}, isave::Vector{…}, dsave::Vector{…})
   @ LBFGSB ~/.julia/packages/LBFGSB/UZibA/src/subroutine.jl:74
 [5] (::LBFGSB.L_BFGS_B)(func::Optimization.var"#16#28"{…}, grad!::OptimizationZygoteExt.var"#grad#14"{…}, x0::Vector{…}, bounds::Matrix{…}; m::Int64, factr::Float64, pgtol::Float64, iprint::Int64, maxfun::Int64, maxiter::Int64)
   @ LBFGSB ~/.julia/packages/LBFGSB/UZibA/src/wrapper.jl:58
 [6] __solve(cache::OptimizationCache{…})
   @ Optimization ~/.julia/packages/Optimization/lRtl8/src/lbfgsb.jl:246
 [7] solve!(cache::OptimizationCache{…})
   @ SciMLBase ~/.julia/packages/SciMLBase/JxpZc/src/solve.jl:234
 [8] solve(::OptimizationProblem{…}, ::Optimization.LBFGS; kwargs::@Kwargs{…})
   @ SciMLBase ~/.julia/packages/SciMLBase/JxpZc/src/solve.jl:131
 [9] top-level scope
   @ ~/work/MaskedDiffusion1L/test.jl:22
Some type information was truncated. Use `show(err)` to see complete types.

[CarloLucibello]

Optim.LBFGS() instead works fine

[ChrisRackauckas]

Optimization.LBFGS, which is actually OptimizationLBFGSB.LBFGSB now after some refactorization / fixes, is https://github.com/Gnimuc/LBFGSB.jl, a wrapper over a Fortran code that is, to my understanding, hardcoded to work with Float64s only. Because of this, this is not going to be fixed because it's not solvable from the baseline code.

Now I think this needs a mention of some other changes. Optimization.LBFGS was incorrectly added to "the base" package here which made it a necessary dependency of all solvers (including the binary) and also placed it as a default over other implementations. That just wasn't good to say the least. This has been rectified very recently, with #986 it was moved to a subpackage OptimizationLBFGSB which is currently being registered JuliaRegistries/General#139567. There is then a refactor going on so that OptimizationBase.jl is a proper Base package following the general SciML development rules #1037. This will then lead to the Optimization.jl v5 breaking update that these solvers will be removed from Optimization.jl, and the solvers won't depend on it either, so it will lose its "preferred status".

@Moelf I think will be very happy about that because he has some benchmarks showing it underperforming anyways. That's not the reason why we did this, the reason was to be more in line with proper dev practices, but it's just another nail in the coffin to just move it to a side library.

Note that Sophia and AugLag are also moving to being solver sublibraries with this.

So in short, close as not planned since this method just has limitations. But we're in progress for major changes in this repo, first refactoring it to match the dev styles of everything else, then starting to build out SimpleOptimization.jl for small problems/ / GPUs / static compilation, then building out pure Julia implementations for big problems. Basically, the same path as NonlinearSolve.jl, just a few years behind it and doing the exact same steps in the playbook, and LBFGSB will be a nice thing to benchmark against but not much more than that.