could not find system library 'openssl' required by the 'openssl-sys' crate

[gabriel-frattini]

run pkg_config fail: `PKG_CONFIG_ALLOW_SYSTEM_CFLAGS="1" PKG_CONFIG_PATH="/usr/local/lib/pkgconfig" "pkg-config" "--libs" "--cflags" "openssl"` did not exit successfully: exit status: 1
2023-03-23T18:55:30.9161896Z   error: could not find system library 'openssl' required by the 'openssl-sys' crate
2023-03-23T18:55:30.9162111Z 
2023-03-23T18:55:30.9162217Z   --- stderr
2023-03-23T18:55:30.9162554Z   Package openssl was not found in the pkg-config search path.
2023-03-23T18:55:30.9162897Z   Perhaps you should add the directory containing `openssl.pc'
2023-03-23T18:55:30.9163196Z   to the PKG_CONFIG_PATH environment variable
2023-03-23T18:55:30.9163502Z   No package 'openssl' found
2023-03-23T18:55:30.9163645Z 
2023-03-23T18:55:30.9163651Z 
2023-03-23T18:55:30.9163753Z   --- stderr
2023-03-23T18:55:30.9163999Z   thread 'main' panicked at '
2023-03-23T18:55:30.9164143Z 
2023-03-23T18:55:30.9164534Z   Could not find directory of OpenSSL installation, and this `-sys` crate cannot
2023-03-23T18:55:30.9164916Z   proceed without this knowledge. If OpenSSL is installed and this crate had
2023-03-23T18:55:30.9165293Z   trouble finding it,  you can set the `OPENSSL_DIR` environment variable for the
2023-03-23T18:55:30.9165573Z   compilation process.
2023-03-23T18:55:30.9165714Z 
2023-03-23T18:55:30.9165880Z   Make sure you also have the development packages of openssl installed.
2023-03-23T18:55:30.9166307Z   For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.
2023-03-23T18:55:30.9166501Z 
2023-03-23T18:55:30.9166742Z   If you're in a situation where you think the directory *should* be found
2023-03-23T18:55:30.9167274Z   automatically, please open a bug at https://github.com/sfackler/rust-openssl
2023-03-23T18:55:30.9167651Z   and include information about your system as well as this message.
2023-03-23T18:55:30.9167848Z 
2023-03-23T18:55:30.9168006Z   $HOST = x86_64-unknown-linux-gnu
2023-03-23T18:55:30.9168295Z   $TARGET = x86_64-unknown-linux-gnu
2023-03-23T18:55:30.9168570Z   openssl-sys = 0.9.82

it seems I don't have openssl installed during CI (linux)
Can someone help me how do I install libssl-dev in the pipeline?

[messense]

Try the before-script-linux option, see #157

[renardeinside]

I've been running into this issue for quite a while, and the workaround I've found is to add this line to Cargo.toml:

openssl = { version = "0.10", features = ["vendored"] }

However it still assumes that there is OpenSSL on the host that will install the library (if I understand correctly).

[2bndy5]

It took me a while, but I was able to augment the maturin-generated CI workflow to use a combination of these tactics:

      - name: Calculate openssl-vendored
        shell: bash
        id: is-openssl-vendored
        run: |
          if [[ "${{ startsWith(matrix.target, 'x86') }}" == "true" ]]; then
            echo "enabled=" >> $GITHUB_OUTPUT
          else
            echo "enabled=--features openssl-vendored" >> $GITHUB_OUTPUT
          fi

      - name: Build wheels
        uses: PyO3/maturin-action@v1
        with:
          target: ${{ matrix.target }}
          args: --release --out dist --find-interpreter ${{ steps.is-openssl-vendored.outputs.enabled }}
          manylinux: auto
          before-script-linux: |
            case "${{ matrix.target }}" in
              "aarch64" | "armv7" | "s390x" | "ppc64le")
                # NOTE: pypa/manylinux docker images are Debian based
                sudo apt-get update
                sudo apt-get install -y pkg-config libssl-dev
                ;;
              "x86" | "x86_64")
                # NOTE: rust-cross/manylinux docker images are CentOS based
                yum update -y
                yum install -y openssl openssl-devel
                ;;
            esac

The secret sauce is in a cargo feature that I added to conditionally invoke the openssl vendored feature:

[dependencies]
openssl = { version = "0.10", features = ["vendored"], optional = true }
openssl-probe = { version = "0.1", optional = true }

[features]
openssl-vendored = ["dep:openssl", "dep:openssl-probe"]

Note

I'm a little worried this conditional feature approach won't work on piwheels because the feature is disabled by default. I'll cross that bridge if ever needed.

And use that feature to conditionally invoke openssl-probe (as recommended by the openssl crate's doc) to find the certs:

#[cfg(feature = "openssl-vendored")]
use openssl_probe;

use pyo3::prelude::*;

#[cfg(feature = "openssl-vendored")]
fn probe_ssl_certs() {
    openssl_probe::init_ssl_cert_env_vars();
}

#[cfg(not(feature = "openssl-vendored"))]
fn probe_ssl_certs() {}

#[pyfunction]
pub fn main() {
    probe_ssl_certs();
}

Tip

The implemented openssl-vendored feature was also needed for the macos job, but no need to install the openssl headers (I think they're already installed).

[lmmx]

Very glad to have found this (may solve my issue at #326) however I would note that I did not need the openssl-vendored feature for macOS, for me those wheels built fine without it - it is just aarch64 that is failing for me

P.S. it should be feature not features when used in cfg !

// See discussion 162
// https://github.com/PyO3/maturin-action/discussions/162#discussioncomment-7978369
#[cfg(feature = "openssl-vendored")]
use openssl_probe;

...

// See discussion 162
#[cfg(feature = "openssl-vendored")]
fn probe_ssl_certs() {
    openssl_probe::init_ssl_cert_env_vars();
}


// See discussion 162
#[cfg(not(feature = "openssl-vendored"))]
fn probe_ssl_certs() {}

[2bndy5]

I edited my suggestion from features to feature. Thanks for pointing that out. I forgot I had already made that change in the pkg I was working on at the time.

[lmmx]

Sweet! P.S. I think you missed the last one:

#[cfg(not(feature = "openssl-vendored"))]

I implemented it like you did, though I also notice that it seems possible to specify according to the target as done in the polars-distances plugin (though this way ain't broke so I'm not going to fix it 😄 )

https://github.com/ion-elgreco/polars-distance/blob/5cc2ff1932519f4b42e6430faf966d7d51488b15/polars_distance/Cargo.toml#L21

[target.'cfg(target_os = "linux")'.dependencies]
jemallocator = { version = "0.5", features = ["disable_initial_exec_tls"] }

[2bndy5]

I think you missed the last one:

Thanks, I updated that one too.

however I would note that I did not need the openssl-vendored feature for macOS, for me those wheels built fine without it - it is just aarch64 that is failing for me

This might be a new development related to macos runners: PyO3/maturin#2249. I'm not sure though.