Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try migration to OpenSSL #53891

Open
wants to merge 19 commits into
base: master
Choose a base branch
from
Open

Try migration to OpenSSL #53891

wants to merge 19 commits into from

Conversation

fxcoudert
Copy link
Contributor

This is a test of migrating the source build to OpenSSL.

@inkydragon inkydragon added building Build system, or building Julia or its dependencies external dependencies Involves LLVM, OpenBLAS, or other linked libraries labels Mar 29, 2024
@giordano giordano added the JLLs label Mar 29, 2024
@giordano giordano added this to the 1.12 milestone Mar 29, 2024
@fxcoudert
Copy link
Contributor Author

OK I'm learning about stdlib as I go along, feedback is very welcome. I think I've done what is needed to add OpenSSL_jl to stdlib. It allows me to build locally.

There is one thing I think I missed, but I don't know how to fix it:

julia> import MbedTLS_jll
┌ Info: Precompiling MbedTLS_jll [c8ffd9c3-330d-5841-b78e-0817d7145fa1] 
└ @ Base loading.jl:2879

julia> import Open<hit tab>
OpenBLAS_jll
OpenLibm_jll

I can't import OpenSSL_jll. I think it's built (because I had a bug in it at some point, and it was complaining which meant it tried to compile it, and now it doesn't complain anymore)… but I'm not sure.

@fxcoudert
Copy link
Contributor Author

OK, somehow the OpenSSL_jll sources are not getting copied to the right directory. I see this message (not an error, but worrying):

Stdlibs ─────   8.848309 seconds 27.013%
Total ───────  32.755706 seconds
find: /Users/fx/Desktop/julia/usr/share/julia/stdlib/v1.12/OpenSSL_jll/src: No such file or directory
 cd /Users/fx/Desktop/julia/base && if ! JULIA_BINDIR=/Users/fx/Desktop/julia/usr/bin WINEPATH="/Users/fx/Desktop/julia/usr/bin;$WINEPATH" JULIA_LOAD_PATH='@stdlib' JULIA_PROJECT= JULIA_DEPOT_PATH=':' JULIA_NUM_THREADS=1  /Users/fx/Desktop/julia/usr/bin/julia -O3 -C "apple-m1;clone_all"  --output-o /Users/fx/Desktop/julia/usr/lib/julia/sys-o.a.tmp  --startup-file=no --warn-overwrite=yes --sysimage /Users/fx/Desktop/julia/usr/lib/julia/sys.ji /Users/fx/Desktop/julia/contrib/generate_precompile.jl 1; then echo '*** This error is usually fixed by running `make clean`. If the error persists, try `make cleanall`. ***'; false; fi 

@ViralBShah
Copy link
Member

I think you need to add it to stdlib/Makefile too. That might fix that error.

@fxcoudert fxcoudert force-pushed the openssl branch 2 times, most recently from df3f08b to bbc918d Compare March 29, 2024 17:41
@fxcoudert
Copy link
Contributor Author

I think you need to add it to stdlib/Makefile too. That might fix that error.

Thanks, that works.

@giordano
Copy link
Contributor

Once we finish JuliaPackaging/Yggdrasil#8377 (and JuliaPackaging/Yggdrasil#8386) and the corresponding builds are updated here, we can remove mbedTLS_jll as stdlib from here, since it isn't used for anything else, I think.

@fxcoudert
Copy link
Contributor Author

Build works for me locally, but on CI it fails with:

ERROR: LoadError: ArgumentError: Package OpenSSL_jll [458c3c95-2e84-50aa-8efc-19380b2a3a95] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

@giordano
Copy link
Contributor

I'm not sure, but maybe that's during documentation building?

@fxcoudert
Copy link
Contributor Author

fxcoudert commented Mar 29, 2024

Yes, it's during docs building, which I'm not doing locally (not sure what the make invocation for that is, if any). Somehow it's not finding the OpenSSL_jll, which it has just built:

Failed to precompile Documenter [e30172f5-a6a5-5a46-863b-614d45cd2de4] to "/cache/build/tester-amdci5-8/julialang/julia-master/doc/deps/compiled/v1.12/Documenter/jl_ZfVk9U".
ERROR: LoadError: ArgumentError: Package OpenSSL_jll [458c3c95-2e84-50aa-8efc-19380b2a3a95] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.
[ … stacktrace … ]
in expression starting at /cache/build/tester-amdci5-8/julialang/julia-master/doc/deps/packages/Git_jll/D8ZTt/src/wrappers/x86_64-linux-gnu.jl:6
in expression starting at /cache/build/tester-amdci5-8/julialang/julia-master/doc/deps/packages/Git_jll/D8ZTt/src/Git_jll.jl:2
in expression starting at stdin:4

@giordano
Copy link
Contributor

@IanButterworth @KristofferC do I understand correctly we need to update doc/Manifest.toml with the new stdlibs?

@IanButterworth
Copy link
Member

@giordano yeah any manifests may need to be re-resolved

@fxcoudert
Copy link
Contributor Author

yeah any manifests may need to be re-resolved

May I ask how this is done? I couldn't figure it out from the doc.

@fxcoudert fxcoudert force-pushed the openssl branch 2 times, most recently from 7c1c2de to 44859e1 Compare March 31, 2024 10:16
@fxcoudert
Copy link
Contributor Author

Manually updated doc/Manifest.toml, will see how things go

@IanButterworth
Copy link
Member

It usually requires Pkg to resolve it to be accurate

julia --project=doc -e "import Pkg; Pkg.resolve()"

I just pushed the result of that.

I also checked for other manifests in this repo that need updating and there doesn't appear to be any.

Surprisingly there's only one in Pkg https://github.com/IanButterworth/Pkg.jl/blob/205309092c861d58bf787628ccbff4f5a41a0840/test/test_packages/ShouldPreserveSemver/Manifest.toml#L90

@fxcoudert
Copy link
Contributor Author

fxcoudert commented Mar 31, 2024

Thanks @IanButterworth

That does not seem to pass CI (https://buildkite.com/julialang/julia-master/builds/35131#018e94a6-3c00-4b01-8b2a-8797e5b2ddf0), but… I can now run that command from my branch. I don't get the same output as you, here is the diff between your version and mine:

diff --git a/doc/Manifest.toml b/doc/Manifest.toml
index 3c8e5dbca3..12b2e3962e 100644
--- a/doc/Manifest.toml
+++ b/doc/Manifest.toml
@@ -10,9 +10,9 @@ uuid = "a4c015fc-c6ff-483c-b24f-f7ea428134e9"
 version = "0.0.1"
 
 [[deps.AbstractTrees]]
-git-tree-sha1 = "faa260e4cb5aba097a73fab382dd4b5819d8ec8c"
+git-tree-sha1 = "2d9c9a55f9c93e8887ad391fbae72f8ef55e1177"
 uuid = "1520ce14-60c1-5f80-bbc7-55ef81b5835c"
-version = "0.4.4"
+version = "0.4.5"
 
 [[deps.ArgTools]]
 uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f"
@@ -66,21 +66,21 @@ version = "1.11.0"
 
 [[deps.Git]]
 deps = ["Git_jll"]
-git-tree-sha1 = "51764e6c2e84c37055e846c516e9015b4a291c7d"
+git-tree-sha1 = "04eff47b1354d702c3a85e8ab23d539bb7d5957e"
 uuid = "d7ba0133-e1db-5d97-8f8c-041e4b3a1eb2"
-version = "1.3.0"
+version = "1.3.1"
 
 [[deps.Git_jll]]
 deps = ["Artifacts", "Expat_jll", "JLLWrappers", "LibCURL_jll", "Libdl", "Libiconv_jll", "OpenSSL_jll", "PCRE2_jll", "Zlib_jll"]
-git-tree-sha1 = "bb8f7cc77ec1152414b2af6db533d9471cfbb2d1"
+git-tree-sha1 = "12945451c5d0e2d0dca0724c3a8d6448b46bbdf9"
 uuid = "f8c6e375-362e-5223-8a59-34ff63f689eb"
-version = "2.42.0+0"
+version = "2.44.0+1"
 
 [[deps.IOCapture]]
 deps = ["Logging", "Random"]
-git-tree-sha1 = "d75853a0bdbfb1ac815478bacd89cd27b550ace6"
+git-tree-sha1 = "8b72179abc660bfab5e28472e019392b97d0985c"
 uuid = "b5f81e59-6552-4d32-b1f0-c071b021bf89"
-version = "0.2.3"
+version = "0.2.4"
 
 [[deps.InteractiveUtils]]
 deps = ["Markdown"]
@@ -169,7 +169,6 @@ version = "1.2.0"
 
 [[deps.OpenSSL_jll]]
 deps = ["Artifacts", "Libdl"]
-git-tree-sha1 = "cc6e1927ac521b659af340e0ca45828a3ffc748f"
 uuid = "458c3c95-2e84-50aa-8efc-19380b2a3a95"
 version = "3.0.13+0"
 
@@ -180,9 +179,9 @@ version = "10.43.0+0"
 
 [[deps.Parsers]]
 deps = ["Dates", "PrecompileTools", "UUIDs"]
-git-tree-sha1 = "a935806434c9d4c506ba941871b327b96d41f2bf"
+git-tree-sha1 = "8489905bcdbcfac64d1daa51ca07c0d8f0283821"
 uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0"
-version = "2.8.0"
+version = "2.8.1"
 
 [[deps.Pkg]]
 deps = ["Artifacts", "Dates", "Downloads", "FileWatching", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "Random", "SHA", "TOML", "Tar", "UUIDs", "p7zip_jll"]
@@ -195,15 +194,15 @@ weakdeps = ["REPL"]
 
 [[deps.PrecompileTools]]
 deps = ["Preferences"]
-git-tree-sha1 = "03b4c25b43cb84cee5c90aa9b5ea0a78fd848d2f"
+git-tree-sha1 = "5aa36f7049a63a1528fe8f7c3f2113413ffd4e1f"
 uuid = "aea7be01-6a6a-4083-8856-8a6e6704d82a"
-version = "1.2.0"
+version = "1.2.1"
 
 [[deps.Preferences]]
 deps = ["TOML"]
-git-tree-sha1 = "00805cd429dcb4870060ff49ef443486c262e38e"
+git-tree-sha1 = "9306f6085165d270f7e3db02af26a400d580f5c6"
 uuid = "21216c6a-2e73-6563-6e65-726566657250"
-version = "1.4.1"
+version = "1.4.3"
 
 [[deps.Printf]]
 deps = ["Unicode"]
@@ -258,9 +257,9 @@ uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 version = "1.11.0"
 
 [[deps.TranscodingStreams]]
-git-tree-sha1 = "54194d92959d8ebaa8e26227dbe3cdefcdcd594f"
+git-tree-sha1 = "14389d51751169994b2e1317d5c72f7dc4f21045"
 uuid = "3bb67fe8-82b1-5028-8e26-92a6c54297fa"
-version = "0.10.3"
+version = "0.10.6"
 weakdeps = ["Random", "Test"]
 
     [deps.TranscodingStreams.extensions]

Some versions and commit ids are different. But maybe more importantly, it does remove the git-tree-sha1 line for OpenSSL_jll, and that seems to make make html pass for me locally. Trying to push to see if CI likes it.

@fxcoudert
Copy link
Contributor Author

fxcoudert commented Mar 31, 2024

OK, that makes the build pass on FreeBSD and macOS (both Intel and ARM). It's also failing on Windows. It's failing on x86_64-linux and variants, but it is passing on aarch64-linux, powerpc64le-linux and i686-linux. The reason for failure is: could not load library "libcrypto.so.3"

My guess is that it's because in OpenSSL.v3.0.13.x86_64-linux-gnu (from https://github.com/JuliaBinaryWrappers/OpenSSL_jll.jl/releases/tag/OpenSSL-v3.0.13%2B0) the shared libraries are in lib64/ and not lib/. I've checked the example of LibGit2-v1.8.0+1, and there the libraries are always in lib/ and not lib64/.

So: should I got back to the OpenSSL package and make it put libraries in lib/? Or is there a way to tell Julia to be smarter about finding things in lib64/ on 64-bit linux?


On Windows, the naming convention is libcrypto-3-x64.dll for 64_bit and libcrypto-3.dll for 32-bit. Really?

I'll let testing finish here, but I think Windows can be handled with:

diff --git a/stdlib/OpenSSL_jll/src/OpenSSL_jll.jl b/stdlib/OpenSSL_jll/src/OpenSSL_jll.jl
index 26eb3d243e..bba9a0a299 100644
--- a/stdlib/OpenSSL_jll/src/OpenSSL_jll.jl
+++ b/stdlib/OpenSSL_jll/src/OpenSSL_jll.jl
@@ -3,7 +3,7 @@
 ## dummy stub for https://github.com/JuliaBinaryWrappers/OpenSSL_jll.jl
 
 baremodule OpenSSL_jll
-using Base, Libdl
+using Base, Libdl, Base.BinaryPlatforms
 
 const PATH_list = String[]
 const LIBPATH_list = String[]
@@ -20,8 +20,13 @@ libssl_handle::Ptr{Cvoid} = C_NULL
 libssl_path::String = ""
 
 if Sys.iswindows()
-    const libcrypto = "libcrypto.dll"
-    const libssl = "libssl.dll"
+    if arch(HostPlatform()) == "x86_64"
+        const libcrypto = "libcrypto-3-x64.dll"
+        const libssl = "libssl-3-x64.dll"
+    else
+        const libcrypto = "libcrypto-3.dll"
+        const libssl = "libssl-3.dll"
+    end
 elseif Sys.isapple()
     const libcrypto = "@rpath/libcrypto.3.dylib"
     const libssl = "@rpath/libssl.3.dylib"

@fxcoudert
Copy link
Contributor Author

fxcoudert commented Mar 31, 2024

Status:

julia> import LibGit2

julia> repo = LibGit2.clone("https://github.com/JuliaLang/Example.jl", "/tmp/foobar")
TLS host verification: the identity of the server `github.com` could not be verified. Someone could be trying to man-in-the-middle your connection. It is also possible that the correct server is using an invalid certificate or that your system's certificate authority root store is misconfigured.
ERROR: GitError(Code:ERROR, Class:HTTP, user rejected certificate for github.com)
Stacktrace:
 [1] macro expansion
   @ ~/Desktop/julia/usr/share/julia/stdlib/v1.12/LibGit2/src/error.jl:115 [inlined]
 [2] clone(repo_url::String, repo_path::String, clone_opts::LibGit2.CloneOptions)
   @ LibGit2 ~/Desktop/julia/usr/share/julia/stdlib/v1.12/LibGit2/src/repository.jl:459
 [3] clone(repo_url::String, repo_path::String; branch::String, isbare::Bool, remote_cb::Ptr{Nothing}, credentials::Nothing, callbacks::Dict{Symbol, Tuple{Ptr{Nothing}, Any}})
   @ LibGit2 ~/Desktop/julia/usr/share/julia/stdlib/v1.12/LibGit2/src/LibGit2.jl:584
 [4] clone(repo_url::String, repo_path::String)
   @ LibGit2 ~/Desktop/julia/usr/share/julia/stdlib/v1.12/LibGit2/src/LibGit2.jl:557
 [5] top-level scope
   @ REPL[2]:1

I have to dig to understand why this used to work and doesn't anymore. It is working for me on a full source build, but not with binarybuilder. Go figure…

giordano

This comment was marked as resolved.

@giordano
Copy link
Contributor

On Windows, the naming convention is libcrypto-3-x64.dll for 64_bit and libcrypto-3.dll for 32-bit. Really?

Yes, we stumbled upon that already: JuliaPackaging/Yggdrasil#7576 (comment)

@giordano
Copy link
Contributor

giordano commented Mar 31, 2024

I have to dig to understand why this used to work and doesn't anymore. It is working for me on a full source build, but not with binarybuilder. Go figure…

It works locally for me with BinaryBuilder binaries:

julia> using LibGit2

julia> repo = LibGit2.clone("https://github.com/JuliaLang/Example.jl", mktempdir())
LibGit2.GitRepo("/tmp/jl_6yE8YE")

julia> versioninfo()
Julia Version 1.12.0-DEV.288
Commit 9aeeb2205ba* (2024-03-31 20:28 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: 8 × Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, haswell)
Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)

but it's possible libgit2/openssl is picking up from my system whatever certificates it needs. Maybe we somehow need to inform openssl/libgit2 where the certificates are (MozillaCACerts_jll.cacert)?

Comment on lines +48 to +52
# If not already done, set the environment variable `SSL_CERT_FILE` when necessary.
cert_loc = ca_roots()
if cert_loc isa String
get!(ENV, "SSL_CERT_FILE", cert_loc)
end
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@StefanKarpinski can you please check this makes sense? I followed the same logic as in

cert_loc = NetworkOptions.ca_roots()
cert_loc !== nothing && set_ssl_cert_locations(cert_loc)
in practice this should be set only on Linux and FreeBSD, unless the user set one of the relevant certificates env variables already.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be done in NetworkOptions though, so that everyone gets it?

@giordano
Copy link
Contributor

Ok, setting the environment variable doesn't work well because there are tests explicitly checking that ENV isn't modified. I'm running out of ideas and bandwidth to work on this, if someone else has concrete suggestions about how to handle this nicely, please go ahead.

@mkitti
Copy link
Contributor

mkitti commented May 13, 2024

I got some bad ideas. One is that we mangle the binary in the same way as others. The other is that we modify the source code to read the location from somewhere.

@StefanKarpinski
Copy link
Member

Ugh, we should probably schedule a call about this to figure out what to do...

@StefanKarpinski
Copy link
Member

I'm kind of tempted to use a simpler TLS library that is better isolated.

@staticfloat
Copy link
Member

I just read through this, and as far as I can tell, the only thing that is failing is teaching OpenSSL about the certificate location, is that correct?

@billatarm
Copy link

Since mbedtls-2.28 is EOL'd at the end of 2024, any way we could nudge Julia to OpenSSL 3.X as distros, like Fedora, would like to drop 2.28.

@mkitti
Copy link
Contributor

mkitti commented Sep 5, 2024

We should modify the BinaryBuilder build of OpenSSL such that these hardcoded paths are a longer placeholder path:

julia> using OpenSSL_jll

julia> @ccall(OpenSSL_jll.libssl.X509_get_default_cert_dir()::Cstring) |> unsafe_string
"/workspace/destdir/ssl/certs"

julia> @ccall(OpenSSL_jll.libssl.X509_get_default_cert_file()::Cstring) |> unsafe_string
"/workspace/destdir/ssl/cert.pem"

For example, the conda-forge builds set this as follows:

/home/conda/feedstock_root/build_artifacts/openssl_split_1717533479680/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/ssl/certs
/home/conda/feedstock_root/build_artifacts/openssl_split_1717533479680/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/ssl/cert.pem

conda, mamba, or pixi then replaces these with environment specific locations.

For example,

/path/to/env/.pixi/envs/default/ssl/certs
/path/to/env/.pixi/envs/default/ssl/cert.pem

I'm not suggesting that Julia do the mangling directly. Rather I suggest that rather we just use an extended placeholder path so that someone could easily replace it in the BinaryBuilder openssl binaries if needed.

@mkitti
Copy link
Contributor

mkitti commented Sep 5, 2024

From: https://prefix-dev.github.io/rattler-build/latest/internals/#making-packages-relocatable-with-rattler-build

Binary file prefix detection and registration: Binary files containing the build prefix can be automatically registered. The registered files will have their build prefix replaced with the install prefix at install time. This works by padding the install prefix with null terminators, such that the length of the binary file remains the same. The build prefix must be long enough to accommodate any reasonable installation prefix. On macOS and Linux, rattler-build pads the build prefix to 255 characters by appending _placehold to the end of the build directory name.

The construction of the placeholder path is as follows in Rust:
https://github.com/prefix-dev/rattler-build/blob/097be23bfca6fd6827092a5e725dd0506ed2e0ff/src/metadata.rs#L132-L144

            let placeholder_template = "_placehold";
            let mut placeholder = String::new();
            let placeholder_length: usize = 255;

            while placeholder.len() < placeholder_length {
                placeholder.push_str(placeholder_template);
            }

            let placeholder = placeholder
                [0..placeholder_length - build_dir.join("host_env").as_os_str().len()]
                .to_string();

            build_dir.join(format!("host_env{}", placeholder))

A similar Julia function might be as follows.

julia> function gen_placehold_path(
           destdir = "/workspace/destdir";
           placeholder_template = "_placehold",
           placeholder_length = 255
       )
           len = 0
           iob = IOBuffer()
           while len < placeholder_length
               len += write(iob, placeholder_template)
           end
           seekstart(iob)
           placeholder =  String(read(iob, placeholder_length - length(destdir) - 1))
           return joinpath(destdir, placeholder)
       end
gen_placehold_path (generic function with 2 methods)

julia> gen_placehold_path()
"/workspace/destdir/_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place"

@fingolfin
Copy link
Contributor

Not sure if anyone saw my comment at JuliaPackaging/Yggdrasil#9371 to I'll repeat it here:

I think we can ensure OpenSSL finds its certificates and more by setting environment variables: That is, ensure OPENSSL_CONF, OPENSSL_ENGINES, OPENSSL_MODULES are set suitably (see this issue for more info) by the JLL wrapper, as discussed here for other packages.

The devil is of course in the details: do we wan the user to override these env vars, too? In that case the code checking them should perhaps only set them if they aren't already set. But to me this sounds like a luxury problem to be worried about if it affects someone.

@mkitti
Copy link
Contributor

mkitti commented Oct 18, 2024

I think we can ensure OpenSSL finds its certificates and more by setting environment variables: That is, ensure OPENSSL_CONF, OPENSSL_ENGINES, OPENSSL_MODULES are set suitably (see this issue for more info) by the JLL wrapper, as discussed here for other packages.

I recommend that Julia works with a SSL Context and create the necessary APIs to make one globally accessible:
https://docs.openssl.org/master/man3/OSSL_LIB_CTX/#description

Environment variables can be quite fragile, especially on Windows. I do not think we should rely on them for the standard configuration.

JuliaPackaging/Yggdrasil#9371 is not meant to implement a Julia configuration. Rather it is intended for particular interoperability scenarios where other software (e.g. Python) may depend on the default configuration of the library at build time. By providing a long path name in the Julia bundled openssl libraries, we provide the opportunity for the system integrator to mangle our openssl build rather than trying to make Julia work with third-party openssl libraries.

@fingolfin
Copy link
Contributor

By providing a long path name in the Julia bundled openssl libraries, we provide the opportunity for the system integrator to mangle our openssl build rather than trying to make Julia work with third-party openssl libraries.

I don't think we want integrators to do that at all. This is completely anathema to what JLLs do. We want them to be immutable

@mkitti
Copy link
Contributor

mkitti commented Oct 21, 2024

I don't think we want integrators to do that at all. This is completely anathema to what JLLs do. We want them to be immutable

To be clear, I do not particularly want people to do this either, but

  1. Python - Julia interoperability is important to some people.
  2. Python relies on the default hard coded path in the OpenSSL library to find SSL certificates.
  3. By making OpenSSL the default SSL library for Julia, there is significant potential that Python may end up using OpenSSL from Julia / BinaryBuilder.
  4. I do not want to spend time re-engineering how the Python ecosystem deals with this.
  5. The currently installed default path baked into the OpenSSL library is completely useless to the Julia ecosystem.
  6. As established above, environment variables are an undesirable and unreliable solution across multiple platforms. Julia has tests to make sure it can operate in an empty environment.
  7. To move past the interoperability matter, I propose turning a String that is useless to Julia into a long String that is still useless to Julia but could be useful to software that is not Julia. If anything, it just makes it more obvious that some software is trying to use the default path.
  8. The long path purposefully mimics the "solution" employed by other popular ecosystems such as Conda.

If you would like to spend more time thinking about how software that is not Julia uses OpenSSL when interoperating with Julia, please do so. I'm tired of doing this, so I just settled on proposing the solution that others outside of Julia settled on. To be honest, I'm not really sure why people are so concerned about making this one useless string long.

@gbaraldi
Copy link
Member

So looking at this it doesn't look like there's really a super satisifying path forward. Though it seems @mkitti has dealt with this on the conda side and honestly we should just follow that IMO. It works well for julia and has some solutions for interop.

@mkitti
Copy link
Contributor

mkitti commented Oct 21, 2024

For Julia, we should just do what OpenSSL.jl is currently doing.

https://github.com/JuliaWeb/OpenSSL.jl/blob/74ce7df91f28ae5618f834488f8e3e5dd2ae30a6/src/ssl.jl#L134-L168

That is we should use SSL_CTX_load_verify_locations to set the necessary paths for a SSL Context. One can create an OpenSSL.SSLContext as follows.

using OpenSSL
ctx = OpenSSL.SSLContext(OpenSSL.TLSClientMethod())

The optional second argument to SSLContext isverify_file::String=MozillaCACerts_jll.cacert. The default certificate path is normpath(Sys.BINDIR, Base.DATAROOTDIR, "julia", "cert.pem") as defined in the MozillaCACerts_jll standard library. The certificate path gets loaded into the SSL context as follows.

            @assert ccall(
                (:SSL_CTX_load_verify_locations, libssl),
                Cint,
                (SSLContext, Ptr{Cchar}, Ptr{Cchar}),
                ssl_context,
                verify_file,
                C_NULL) == 1

edit: To be consistent with the prior discussion, this should change to be NetworkOptions.ca_roots(). However, we may need to modify the behavior there for macOS and Windows as ca_roots() returns nothing there.

@mkitti
Copy link
Contributor

mkitti commented Oct 21, 2024

For the PyCall / PythonCall based packages that we can control, we should modify them to use a SSL context as I mentioned earlier.

@mkitti
Copy link
Contributor

mkitti commented Oct 21, 2024

On macOS:

% openssl version
LibreSSL 3.3.6

% openssl version -d
OPENSSLDIR: "/private/etc/ssl"

@mkitti
Copy link
Contributor

mkitti commented Oct 21, 2024

Here's a fix for ScikitLearn.jl:

using OpenSSL
using ScikitLearn
using NetworkOptions
using PyCall

if length(ARGS) > 0 && ARGS[1] == "--fix"
    py"""
    import ssl
    def create_julia_https_context():
        return ssl.create_default_context(cafile=$(NetworkOptions.bundled_ca_roots()))

    ssl._create_default_https_context = create_julia_https_context
    """
end

@sk_import datasets: (fetch_california_housing, clear_data_home)
clear_data_home()
println(length(fetch_california_housing()))

credit to https://stackoverflow.com/a/54385130/10313003 for the inspiration.

Without --fix

/Users/mkitti/miniforge3-arm64/envs/sklearn-test/lib/python3.12/site-packages/sklearn/datasets/_base.py:1472: UserWarning: Retry downloading from url: https://ndownloader.figshare.com/files/5976036
  warnings.warn(f"Retry downloading from url: {remote.url}")
ERROR: LoadError: PyError ($(Expr(:escape, :(ccall(#= /Users/mkitti/.julia/packages/PyCall/1gn3u/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'urllib.error.URLError'>
URLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))

With --fix

6

@mkitti
Copy link
Contributor

mkitti commented Oct 21, 2024

I was trying to test SSL under the scenario of Python loading Julia and OpenSSL.jl via juliacall.

However, juliacall ends up loading the Python standard library ssl before I can load OpenSSL.jl or OpenSSL_jll. Thus under this scenario, the default certificate paths are the ones created by conda, mamba, or pixi.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
building Build system, or building Julia or its dependencies external dependencies Involves LLVM, OpenBLAS, or other linked libraries JLLs
Projects
None yet
Development

Successfully merging this pull request may close these issues.