Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GR backend interaction with system libraries #540

Open
EetuReijonen opened this issue Feb 21, 2024 · 4 comments
Open

GR backend interaction with system libraries #540

EetuReijonen opened this issue Feb 21, 2024 · 4 comments

Comments

@EetuReijonen
Copy link

Hi, I have run into a weird issue when trying to use Plots.jl with GRBackend and Gurobi.jl (the optimization solver) with multiple workers.

The minimal working example is here:

using Plots
gr()
plot()

using Distributed
addprocs(4)
@everywhere begin
    using Gurobi
    Gurobi.Env()
end

This code will produce an error, which, according to my understanding, indicates that the Gurobi library/program cannot be found.

Plots.GRBackend()

4-element Vector{Int64}:
 2
 3
 4
 5

Set parameter Username
Academic license - for non-commercial use only - expires 2024-05-21
ERROR: On worker 2:
could not load library "libgurobi100.dylib"
dlopen(libgurobi100.dylib, 0x0001): tried: 'libgurobi100.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibgurobi100.dylib' (no such file), '/Applications/Julia-1.9.1.app/Contents/Resources/julia/lib/julia/libgurobi100.dylib' (no such file), '/Applications/Julia-1.9.1.app/Contents/Resources/julia/lib/julia/../libgurobi100.dylib' (no such file), '/Applications/Julia-1.9.1.app/Contents/Resources/julia/lib/libgurobi100.dylib' (no such file), '/usr/lib/libgurobi100.dylib' (no such file, not in dyld cache), 'libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/5902919ba30b97bafa388ca950a351ea32516d26/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/92b949e2f3a66439c69a8d334fc95810fbd9df9b/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/5b338c8fa90c05e6faea86e54d2996cca76cfbbe/lib/libgurobi100.dylib' (no such file), '/Applications/Julia-1.9.1.app/Contents/Resources/julia/lib/julia/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/d18fd9a0903c01ba831674a057a5ef7caf6aeec6/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/9410bad2635eda2239b4a72ba4316c4aa8f5b76e/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/95f332911a8dd5ab82620ba58c217193bd0d515d/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/10b2b258c07d7a76b2e3331f1ed70d8a8eb6d71c/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/9a76a401f82e0e3cafce618fb8d2d5c307ab2836/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/4aea97e4407a13e6d2ef7e5a05864ed1db642163/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/c65e07e3da4f1bf519bc432389dbbd61df320457/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/4ec62d729213a748d2300dd0832ebe8ed2292093/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/e6b9fb44029423f5cd69e0cbbff25abcc4b32a8f/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/b2108f561a8812e376eb80e71a24a3678a24d231/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/ada2a202928dd4cb2fc4bd18c4efa9d5455ec742/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/df3881e810714d6a09467fe85a6fde79385fe702/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/3b3d0bcaf14a9b239a4f4dc20ef7b9e63030a47e/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/abf161ac3d4df76ae74bbf5432b7e061b3876236/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/4260cf51a368d8e305a5de3669e32539e1e6cc72/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/fc7ba632b72ce7d852c1924aa2bbfe244a71c780/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/413111420faa4e2aeaa383c075eaa213402d939c/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/ca2831bf6edc5088aec5b329ea98364951d6cad0/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/3fe6bf926e57cc4be598151cd40832221de2e894/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/c325a23bc1f6521474cef5f634f18c8ab311bb02/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/0db9c3f6cf936a0da49e2ba954ba3e10bed6ad72/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/1a7e22e66b523d9cb884cf85c3ec065b5fb3e5c3/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/4a64372f135495d207e45abb25f29f9f66d46d7a/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/4609432e7098d8434a7a4c7876dd5b9e09b2a5e7/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/6095fcd268ea712c0f786f5ff1a45bf0eb7b005e/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/8dfccc7f1aa9d156f12a6e7db599014b805b11f2/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/b80d54484d815d1ace4cb475d53da633579d7d92/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/b450526929615030746974fd622effa333c2c87a/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/fb653fdc268e50c0be4391f6ca2f2fce2be7f15b/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/9a9f59eab237f7454fee1d6ab112a254032540b7/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/2107e7bc404f11b178cb9724cb371ef704995727/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/4b3b2d79556cc3aef6e3d8a234649cc85b91bb87/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtConcurrent.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtCore.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtDBus.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtGui.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtNetwork.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtOpenGL.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtPrintSupport.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtSql.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtTest.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtWidgets.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtXml.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/f1af74bc0a21f699d4974bdceac10d4638e61e74/lib/libgurobi100.dylib' (no such file), '/Applications/Julia-1.9.1.app/Contents/Resources/julia/bin/../lib/julia/libgurobi100.dylib' (no such file), '/Applications/Julia-1.9.1.app/Contents/Resources/julia/bin/../lib/libgurobi100.dylib' (no such file), '/libgurobi100.dylib' (no such file)
Stacktrace:
  [1] GRBemptyenv
    @ ~/.julia/packages/Gurobi/yvF0V/src/gen100/libgrb_api.jl:813 [inlined]
  [2] _
    @ ~/.julia/packages/Gurobi/yvF0V/src/MOI_wrapper/MOI_wrapper.jl:108
  [3] Env
    @ ~/.julia/packages/Gurobi/yvF0V/src/MOI_wrapper/MOI_wrapper.jl:102
  [4] top-level scope
    @ ~/Desktop/GAMMA-OPT/Gogeta.jl/examples/neural_networks_parallel.jl:34
  [5] eval
    @ ./boot.jl:370
  [6] #invokelatest#2
    @ ./essentials.jl:816
  [7] invokelatest
    @ ./essentials.jl:813
  [8] #114
    @ /Applications/Julia-1.9.1.app/Contents/Resources/julia/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:301
  [9] run_work_thunk
    @ /Applications/Julia-1.9.1.app/Contents/Resources/julia/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
 [10] run_work_thunk
    @ /Applications/Julia-1.9.1.app/Contents/Resources/julia/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:79
 [11] #100
    @ ./task.jl:514

...and 3 more exceptions.

Stacktrace:
 [1] sync_end(c::Channel{Any})
   @ Base ./task.jl:445
 [2] macro expansion
   @ ./task.jl:477 [inlined]
 [3] remotecall_eval(m::Module, procs::Vector{Int64}, ex::Expr)
   @ Distributed /Applications/Julia-1.9.1.app/Contents/Resources/julia/share/julia/stdlib/v1.9/Distributed/src/macros.jl:219
 [4] top-level scope
   @ /Applications/Julia-1.9.1.app/Contents/Resources/julia/share/julia/stdlib/v1.9/Distributed/src/macros.jl:203

However, by using a different backend (e.g. PyPlot), the code works just fine:

using Plots
pyplot()
plot()

using Distributed
addprocs(4)
@everywhere begin
    using Gurobi
    Gurobi.Env()
end

Output:

Plots.PyPlotBackend()

4-element Vector{Int64}:
 2
 3
 4
 5

Set parameter Username
Academic license - for non-commercial use only - expires 2024-05-21
      From worker 5:    Set parameter Username
      From worker 4:    Set parameter Username
      From worker 4:    Academic license - for non-commercial use only - expires 2024-05-21
      From worker 3:    Set parameter Username
      From worker 3:    Academic license - for non-commercial use only - expires 2024-05-21
      From worker 2:    Set parameter Username
      From worker 2:    Academic license - for non-commercial use only - expires 2024-05-21
      From worker 5:    Academic license - for non-commercial use only - expires 2024-05-21

I'm using Julia version 1.9.1, Plot.jl v1.39.0, Gurobi.jl v1.2.1 on a Macbook M2 Air with macOS 14.2

Could this issue be caused by some kind of alteration to the system environment before it's passed to the workers? Or possibly a namespace conflict (Gurobi is also abbreviated GR in some contexts)?

@jheinen
Copy link
Owner

jheinen commented Feb 21, 2024

I have no idea, why this happens. I don't think, that there are name conflicts (GR vs GRB).

Could please find the missing library (find ~/.julia/artifacts -name libgurobi100.dylib) and try the following command:

otool -L <path from above find command>libgurobi100.dylib

You could also try to run the otool command within your Julia session.

@EetuReijonen
Copy link
Author

For some reason I cannot find the Gurobi library in my artifacts folder (or any .dylib file with gurobi in the name).

eetureijonen@Eetus-MacBook-Air artifacts % find ~/.julia/artifacts -type f \( -name "*gurobi*" -a -name "*.dylib" \)

I found it in my /Library folder and this is the output from the otool-command.

eetureijonen@Eetus-MacBook-Air lib % otool -L /Library/gurobi1001/macos_universal2/lib/libgurobi100.dylib
/Library/gurobi1001/macos_universal2/lib/libgurobi100.dylib (architecture x86_64):
	/Library/gurobi1001/macos_universal2/lib/libgurobi100.dylib (compatibility version 10.0.0, current version 10.0.1)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1770.255.0)
	/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
	/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 1109.60.2)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 59754.60.13)
/Library/gurobi1001/macos_universal2/lib/libgurobi100.dylib (architecture arm64):
	/Library/gurobi1001/macos_universal2/lib/libgurobi100.dylib (compatibility version 10.0.0, current version 10.0.1)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1770.255.0)
	/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
	/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 1109.60.2)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 59754.60.13)

@jheinen
Copy link
Owner

jheinen commented Feb 21, 2024

/Library/gurobi1001/macos_universal2/lib is not in the dlopen search path, so you could try to explicitly set it before starting Julia:

export DYLD_LIBRARY_PATH="/Library/gurobi1001/macos_universal2/lib:${DYLD_LIBRARY_PATH}"
julia ...

@EetuReijonen
Copy link
Author

Thank you, this seems to solve the acute problem for me.

eetureijonen@Eetus-MacBook-Air examples % export DYLD_LIBRARY_PATH="/Library/gurobi1001/macos_universal2/lib:${DYLD_LIBRARY_PATH}"
eetureijonen@Eetus-MacBook-Air examples % julia --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.1 (2023-06-07)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Plots

julia> gr()
Plots.GRBackend()

julia> plot()

julia> using Distributed

julia> addprocs(4)
4-element Vector{Int64}:
 2
 3
 4
 5

julia> @everywhere begin
           using Gurobi
           Gurobi.Env()
       end
Set parameter Username
Academic license - for non-commercial use only - expires 2024-05-21
      From worker 2:    Set parameter Username
      From worker 3:    Set parameter Username
      From worker 3:    Academic license - for non-commercial use only - expires 2024-05-21
      From worker 2:    Academic license - for non-commercial use only - expires 2024-05-21
      From worker 5:    Set parameter Username
      From worker 4:    Set parameter Username
      From worker 5:    Academic license - for non-commercial use only - expires 2024-05-21
      From worker 4:    Academic license - for non-commercial use only - expires 2024-05-21

However, I think there is still something going on as other backends do not cause the same issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants