Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't run on Almalinux 8 #733

Closed
5 tasks done
Charadon opened this issue Sep 20, 2023 · 27 comments · Fixed by #858
Closed
5 tasks done

Can't run on Almalinux 8 #733

Charadon opened this issue Sep 20, 2023 · 27 comments · Fixed by #858
Labels
bug Something isn't working

Comments

@Charadon
Copy link

Thanks in advance for your bug report!

  • Have you reproduced issue in safe mode?
  • Have you used the debugging guide to try to resolve the issue?
  • Have you checked our FAQs to make sure your question isn't answered there?
  • Have you checked to make sure your issue does not already exist?
  • Have you checked you are on the latest release of Pulsar?

What happened?

If you try to run Pulsar in Almalinux 8, it won't run with the following error:

/lib64/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/.org.chromium.Chromium.qZacsn)
Error: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/.org.chromium.Chromium.qZacsn)
    at process.func [as dlopen] (electron/js2c/asar_bundle.js:5:1846)
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:1138:18)
    at Object.func [as .node] (electron/js2c/asar_bundle.js:5:2073)
    at Module.load (internal/modules/cjs/loader.js:935:32)
    at Module._load (internal/modules/cjs/loader.js:776:14)
    at Function.f._load (electron/js2c/asar_bundle.js:5:12913)
    at Module.require (internal/modules/cjs/loader.js:959:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object.<anonymous> (/tmp/.mount_Linux.9vbRre/resources/app.asar/node_modules/nsfw/js/src/index.js:4:14)
    at Module._compile (internal/modules/cjs/loader.js:1078:30)

Pulsar version

1.109.0

Which OS does this happen on?

🐧 Red Hat based (Fedora, Alma, RockyLinux, CentOS Stream, etc.)

OS details

Almalinux 8

Which CPU architecture are you running this on?

64-bit(x86_64)

What steps are needed to reproduce this?

  1. Download rpm, appimage or tar.gz distribution of pulsar
  2. Try to run it.
  3. Get the above error

Additional Information:

If I had to take a guess why this happens, it's most likely because the binaries are built against the latest version of Ubuntu. But that's just from me taking a glance at the Github Actions config file.

@Charadon Charadon added the bug Something isn't working label Sep 20, 2023
@Charadon
Copy link
Author

So, I changed the CI from ubuntu-latest to ubuntu-20.04, and the GLIBC error change to a GLIBCXX error. And the linked pull request's build has the same GLIBC error.

@Daeraxa
Copy link
Member

Daeraxa commented Sep 20, 2023

Hmm, I was hoping the binaries produced from that PR might work (https://github.com/pulsar-edit/pulsar/actions/runs/6252546119?pr=734).

@Charadon
Copy link
Author

Hmm, I was hoping the binaries produced from that PR might work (https://github.com/pulsar-edit/pulsar/actions/runs/6252546119?pr=734).

Yeah, those binaries don't fix the issue, sadly =(

It's definitely an issue with the platform that's being built-on being too new. But unfortunately ubuntu-20.04 is the oldest github supports outside of self-hosting a worker.

@Charadon
Copy link
Author

From my research, this issue has actually popped up on many electron projects since github removed the ubuntu 18.04 runner.

gorilla-devs/GDLauncher#1528
arduino/arduino-ide#1983 (comment)
Axosoft/nsfw#175

I'm gonna try building Pulsar on Slackware 14.2 (Which is still supported, and came out in 2016) and see if it runs on Almalinux 8 then, which i'm fairly certain it will

@Daeraxa
Copy link
Member

Daeraxa commented Sep 21, 2023

This exact issue was reported (or at least mentioned) here so I was hoping the same fix would be applicable as a quick win.

@Charadon
Copy link
Author

The only difference I can see between these two errors is that the one link there is a linking error with sqlite, but the one here is a linking error with the nsfw binary. So maybe the fix applied only fixes the issue with sqlite but not nsfw for whatever reason.

@Charadon
Copy link
Author

I did forget to mention, the build from the fix changed the error. Instead of it complaining at /tmp/.org_chromium*, it now shows the real culprit of the linking error

/lib64/libc.so.6: version `GLIBC_2.33' not found (required by /opt/Pulsar/resources/app.asar.unpacked/node_modules/nsfw/build/Release/nsfw.node)
Error: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /opt/Pulsar/resources/app.asar.unpacked/node_modules/nsfw/build/Release/nsfw.node)
    at process.func [as dlopen] (electron/js2c/asar_bundle.js:5:1846)
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:1138:18)
    at Object.func [as .node] (electron/js2c/asar_bundle.js:5:2073)
    at Module.load (internal/modules/cjs/loader.js:935:32)
    at Module._load (internal/modules/cjs/loader.js:776:14)
    at Function.f._load (electron/js2c/asar_bundle.js:5:12913)
    at Module.require (internal/modules/cjs/loader.js:959:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object.<anonymous> (/opt/Pulsar/resources/app.asar/node_modules/nsfw/js/src/index.js:4:14)
    at Module._compile (internal/modules/cjs/loader.js:1078:30)

@Charadon
Copy link
Author

Charadon commented Sep 21, 2023

So, I got an appimage to work. Here are the key points:

  • Building on an older linux distro fixes the GLIBC error. This is because nsfw links with an older glibc, and since glibc is backwards compatible it'll work on any distribution that has a newer glibc library than it was linked to.
  • However, you'll probably then run into a GLIBCXX error, because nsfw is a C++ program. In order to fix this you either need to:
    A: Use Clang/LLVM instead of GCC. This is the method I used on the appimage I generated.
    B: Compile pulsar with the oldest GCC version you can (So you can get an older libstdc++.so). I haven't tested it though.

the tl;dr: It's definitely the fact that Ubuntu 22.04 (The version used with ubuntu-latest) has a too new C library, and any distro released before it will fail to run any generated package. I think github actions supports using a Docker container inside the ubuntu-latest runner. So maybe the solution can be to run a docker container of an older distro such as RHEL8 (Such as Rocky Linux 8, AlmaLinux 8), the oldest still supported Debian (At the time of writing this, That's Debian 8 via the Freexian project), or Slackware 14.2.

Alternatively, maybe someone can statically compile the nsfw library? But I wouldn't even know how to start going about that.

Anyways, those are my findings. The easiest thing to do would be to just switch the linux image used in github actions to ubuntu-20.04 and (maybe?) use clang (via the CXX and CC env variable), so at the very least any distro released after 2020 will work. (Should note: This will NOT fix the issue of it not running on RHEL8)

@Charadon
Copy link
Author

So, after compiling it on Debian 8, it pretty much runs on all distributions with no issues that I can see.

I did find out that the effective minimum requirement for a C compiler was Clang 5, and GCC 8, since one of the dependencies requires C++17 support. Debian 8 had Clang 6, so I used that. Python had to be manually compiled, but since it's made in C99, that was a pretty easy thing to do (Though, I had to use Python 3.6, the minimum required version of one of the dependencies due to lack of OpenSSL 1.0.1).

So at this point, I would say it's not really a bug in Pulsar, but more of a binary distribution issue. So I guess the issue can be closed and maybe moved to a github discussion instead?

@Daeraxa
Copy link
Member

Daeraxa commented Sep 24, 2023

Honestly its beyond me at this point but yeah, I suppose that it is indeed more of a question of how far back we want to, or can, support. For example in Windows land I think Windows 7 and 8 should still be working but once we update to Electron >25(?) then it will break and only work for Windows 10 and up.

Thanks for all the investigation you have done here, it is still really valuable.

@Charadon
Copy link
Author

Charadon commented Sep 25, 2023

Honestly its beyond me at this point but yeah, I suppose that it is indeed more of a question of how far back we want to, or can, support. For example in Windows land I think Windows 7 and 8 should still be working but once we update to Electron >25(?) then it will break and only work for Windows 10 and up.

Thanks for all the investigation you have done here, it is still really valuable.

In Linux land, you'd probably want to support as old as possible, so the packages that are generated work on a wide selection of distros. The general rule of thumb seems to either be:

A. Build on the oldest supported version of Debian/Ubuntu
B. Build on the oldest supported version of Red Hat & Derivatives (Currently CentOS 7 would be a good bet, but support ends in 2024)

@griffint61
Copy link

The same error occurs on Almalinux 9.

@Charadon
Copy link
Author

Charadon commented Sep 26, 2023

The same error occurs on Almalinux 9.

Yeah, Ubuntu 22.04 (The distro the binaries are built on) uses glibc 2.35, which is one version above Almalinux's 2.34. Would it be alright if I linked the binaries I built on Debian Jessie, @Daeraxa?

(As a sidenote: The title should probably be changed to "Official Binaries can't be run on distros older than Ubuntu 22.04")

@catscarlet
Copy link

Dude. Ubuntu 20.04 is a LTS and it should be supported until 2025.

I'm using Linux Mint 20.3(Also based on Ubuntu 20.04) and I just got the same issue.

cat@cat-B85M-D3H:~$ pulsar -d -f
/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/.org.chromium.Chromium.GmVykn)
Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/.org.chromium.Chromium.GmVykn)
    at process.func [as dlopen] (electron/js2c/asar_bundle.js:5:1846)
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:1138:18)
    at Object.func [as .node] (electron/js2c/asar_bundle.js:5:2073)
    at Module.load (internal/modules/cjs/loader.js:935:32)
    at Module._load (internal/modules/cjs/loader.js:776:14)
    at Function.f._load (electron/js2c/asar_bundle.js:5:12913)
    at Module.require (internal/modules/cjs/loader.js:959:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object.<anonymous> (/opt/Pulsar/resources/app.asar/node_modules/nsfw/js/src/index.js:4:14)
    at Module._compile (internal/modules/cjs/loader.js:1078:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1108:10)
    at Module.load (internal/modules/cjs/loader.js:935:32)
    at Module._load (internal/modules/cjs/loader.js:776:14)
    at Function.f._load (electron/js2c/asar_bundle.js:5:12913)
    at Module.require (internal/modules/cjs/loader.js:959:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object.<anonymous> (/opt/Pulsar/resources/app.asar/src/config-file.js:11:14)
    at Module._compile (internal/modules/cjs/loader.js:1078:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1108:10)
    at Module.load (internal/modules/cjs/loader.js:935:32)
    at Module._load (internal/modules/cjs/loader.js:776:14)
    at Function.f._load (electron/js2c/asar_bundle.js:5:12913)
    at Module.require (internal/modules/cjs/loader.js:959:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object.<anonymous> (/opt/Pulsar/resources/app.asar/src/main-process/atom-application.js:6:20)
    at Module._compile (internal/modules/cjs/loader.js:1078:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1108:10)
    at Module.load (internal/modules/cjs/loader.js:935:32)
    at Module._load (internal/modules/cjs/loader.js:776:14)
    at Function.f._load (electron/js2c/asar_bundle.js:5:12913)
    at Module.require (internal/modules/cjs/loader.js:959:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at App.<anonymous> (/opt/Pulsar/resources/app.asar/src/main-process/start.js:108:29)
    at App.emit (events.js:315:20)

@confused-Techie
Copy link
Member

confused-Techie commented Oct 12, 2023

@Charadon As you seem pretty knowledgeable in this area (meanwhile admittedly I am not), do you think this would be possible to resolve by simply updating the nsfw module? As this is one of our internal modules, and while we don't have any main C++ devs or anything on the team, if updating that one module to use a more modern glibc version, it sounds like, from my understanding, that could resolve this issue?

This isn't my area of expertise, but I would be more than happy to deep dive to try and find a fix, especially if there are any pointers you may provide on this topic.


And to clarify, as I'm reading through this again, I shouldn't have said to "use a more modern glibc version", but instead switch off of it. Such like moving to clang or LLVM, or like you said maybe statically compiling it is best, rather than make it build on the platform. This ideally could be something we can look into as a GitHub Action.

@Charadon
Copy link
Author

Charadon commented Oct 13, 2023

@Charadon As you seem pretty knowledgeable in this area (meanwhile admittedly I am not), do you think this would be possible to resolve by simply updating the nsfw module? As this is one of our internal modules, and while we don't have any main C++ devs or anything on the team, if updating that one module to use a more modern glibc version, it sounds like, from my understanding, that could resolve this issue?

This isn't my area of expertise, but I would be more than happy to deep dive to try and find a fix, especially if there are any pointers you may provide on this topic.

And to clarify, as I'm reading through this again, I shouldn't have said to "use a more modern glibc version", but instead switch off of it. Such like moving to clang or LLVM, or like you said maybe statically compiling it is best, rather than make it build on the platform. This ideally could be something we can look into as a GitHub Action.

You can't switch off of glibc unfortunately, as it's what runs everything on Linux, even if you use Clang, you'll just avoid the STDC++ error. There's a link argument for gcc called -static-libstdc++ and you can use it via the LDFLAGS env variable.

Now, that will solve the libstdc++ error (Assuming your distro of choice doesn't even has the static libstdc++ library), but you will still have to deal with glibc error. The only thing that could work is to statically compile as much as possible if you can't use an older linux distro, but I have no idea if that's possible as some C/C++ programs can't be statically compiled due to their nature.

And of course, github doesn't have a linux distro supported where you could statically link the C library, such as Alpine, so uh, that sucks. The reason for this, is that glibc can't be statically compiled because of a lotta reasons.

@confused-Techie
Copy link
Member

Well fair enough then, thanks for the response @Charadon.

I did make a PR above, where we build on the non-latest ubuntu version, which I saw mentioned here and elsewhere would at least resolve the issue for some users, or at least those users who are on modern but not bleeding edge distros.

So if anyone is able to test the binaries built from that PR it'd be appreciated. But that does only get us part of the way there

@Charadon
Copy link
Author

Honestly, the only real solution is to either:

A. Move off of Github actions and use your own clouds with Jenkins or something.
B. Hand-build the releases on something like Debian Jessie. (Like I did here: https://ftp.iotib.net/Projects/Pulsar%20Older%20Glibc/)

In the short-term, hand building them, and then signing them with GPG to make sure they aren't tampered is the easiest solution to solve the whole GLIBC error issues. I'm sorry I can't come up with anything better, it's kind of ridiculous that Github actions only supports the 2 newest Ubuntu releases and not something like the oldest RHEL or the oldest LTS of debian. =/

@Charadon
Copy link
Author

Oh, and of course a 3rd option, which is to self-host your own github runner https://testdriven.io/blog/github-actions-docker/

@DeeDeeG
Copy link
Member

DeeDeeG commented Oct 16, 2023

Hi folks, I've been away for a bit.

I got some binaries built in CI on Debian 8 10 (via Docker) on this branch here: https://github.com/pulsar-edit/pulsar/tree/CI-test-building-on-older-Linux

Binaries available to test (at your own risk, I wasn't able to test these myself) here, if you are logged in to GitHub: https://github.com/pulsar-edit/pulsar/actions/runs/6423517654 (see the ubuntu-latest Binaries .zip file). Currently x86 only, as the ARM binaries are made on Cirrus, and the steps to set it up there too would be more work. This was only tested/gotten working on GitHub Actions so far.

I wasn't able to get CI to test and pass based off of those binaries, so I'm not sure if something is wrong with them.

Either we spend more time troubleshooting why these don't pass testing in CI, or else we don't attempt to test them, and provide them "as-is" in parallel with the ones with newer GLibC. We could give them a different name and filename prefix, something like "Legacy.Linux" binaries?

If we got these binaries built from Debian 8 10 to pass automated testing in CI, then these could potentially serve as the standard "Linux" binaries and we could stop building on Ubuntu LTS latest -- unclear if we have enough native C/C++ code in Pulsar that we're re-compiling in CI that using older semver major versions of the compiler toolchain(s) would make a difference in performance or what-have-you. Since I'm willing to guess Debian 8 has older compilers than Ubuntu LTS latest does. (Or if there's any other downside to using older compilers. Notably, Atom used the ones from the oldest distro they could build on in CI, so... It was apparently not much of a problem for them!)

@majamee
Copy link

majamee commented Oct 16, 2023

Same issue with latest version of Pulsar (1.110.0) on Ubuntu 20.04:

/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/.org.chromium.Chromium.hiSo0R)
Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/.org.chromium.Chromium.hiSo0R)
    at process.func [as dlopen] (electron/js2c/asar_bundle.js:5:1846)
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:1138:18)
    at Object.func [as .node] (electron/js2c/asar_bundle.js:5:2073)
    at Module.load (internal/modules/cjs/loader.js:935:32)
    at Module._load (internal/modules/cjs/loader.js:776:14)
    at Function.f._load (electron/js2c/asar_bundle.js:5:12913)
    at Module.require (internal/modules/cjs/loader.js:959:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object.<anonymous> (/tmp/.mount_Linux.wlvdyO/resources/app.asar/node_modules/nsfw/js/src/index.js:4:14)
    at Module._compile (internal/modules/cjs/loader.js:1078:30)
libva error: /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so init failed

@majamee
Copy link

majamee commented Oct 16, 2023

Hi folks, I've been away for a bit.

I got some binaries built in CI on Debian 8 10 (via Docker) on this branch here: https://github.com/pulsar-edit/pulsar/tree/CI-test-building-on-older-Linux

Binaries available to test (at your own risk, I wasn't able to test these myself) here, if you are logged in to GitHub: https://github.com/pulsar-edit/pulsar/actions/runs/6423517654 (see the ubuntu-latest Binaries .zip file). Currently x86 only, as the ARM binaries are made on Cirrus, and the steps to set it up there too would be more work. This was only tested/gotten working on GitHub Actions so far.

I wasn't able to get CI to test and pass based off of those binaries, so I'm not sure if something is wrong with them.

Either we spend more time troubleshooting why these don't pass testing in CI, or else we don't attempt to test them, and provide them "as-is" in parallel with the ones with newer GLibC. We could give them a different name and filename prefix, something like "Legacy.Linux" binaries?

If we got these binaries built from Debian 8 10 to pass automated testing in CI, then these could potentially serve as the standard "Linux" binaries and we could stop building on Ubuntu LTS latest -- unclear if we have enough native C/C++ code in Pulsar that we're re-compiling in CI that using older semver major versions of the compiler toolchain(s) would make a difference in performance or what-have-you. Since I'm willing to guess Debian 8 has older compilers than Ubuntu LTS latest does. (Or if there's any other downside to using older compilers. Notably, Atom used the ones from the oldest distro they could build on in CI, so... It was apparently not much of a problem for them!)

Your test version seems to be working at least on Ubuntu 20.04. Please find accompanying output here:

libva error: /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so init failed
(node:216875) Electron: Loading non-context-aware native module in renderer: '/tmp/.org.chromium.Chromium.gwphCh'. This is deprecated, see https://github.com/electron/electron/issues/18397.
(Use `exe --trace-warnings ...` to show where the warning was created)
(node:216875) Electron: Loading non-context-aware native module in renderer: '/tmp/.org.chromium.Chromium.r2NI8e'. This is deprecated, see https://github.com/electron/electron/issues/18397.
(...)

@kiskoza
Copy link
Contributor

kiskoza commented Oct 16, 2023

The experimental branch fixed the issue for me as well on Ubuntu 20.04 🎉 it would be nice to see this fix in 1.111.0

As mentioned before, Ubuntu 20.04 is an LTS, it's normal maintenance will end in 2025, but it has an extended security maintenance until 2030 as you can check here: https://endoflife.date/ubuntu

Currently 20.04 has glibc 2.31-0ubuntu9.12 (source), while 22.04 has glibc 2.35-0ubuntu3.4 (source) - so you can't really change the glibc version as long as installing everything from the official repos

@Charadon
Copy link
Author

I got some binaries built in CI on Debian 8 10 (via Docker) on this branch here: https://github.com/pulsar-edit/pulsar/tree/CI-test-building-on-older-Linux

Out of curiousity, why 10 instead of 9? With the Freexian patches, Debian 9 is supported until 2026-2027.

@DeeDeeG
Copy link
Member

DeeDeeG commented Oct 17, 2023

@kiskoza we have the simple and easy fix for Ubuntu 20.04 ready to go, see: #760. (That easy fix also addresses any distros with glibc newer than Ubuntu 20.04's, of course.) I'm pretty sure we'll merge that fix for the "standard"/"non-legacy" binaries quite soon.

@Charadon No exact reason for testing this using Debian 10, other than I had quickly confirmed it was fully supported still, and appeared old enough for all folks here asking about it on the RHEL 8/9-based families of distros. (Debian 8 was so old it wouldn't install packages properly in Docker, I don't think. Maybe the package repos for that old Debian are gone/not being hosted anymore.) If we get folks asking for even older compatibility that Debian 9 would offer, I suppose that's just as much an option.

@Charadon
Copy link
Author

Charadon commented Oct 17, 2023

@Charadon No exact reason for testing this using Debian 10, other than I had quickly confirmed it was fully supported still, and appeared old enough for all folks here asking about it on the RHEL 8/9-based families of distros. (Debian 8 was so old it wouldn't install packages properly in Docker, I don't think. Maybe the package repos for that old Debian are gone/not being hosted anymore.) If we get folks asking for even older compatibility that Debian 9 would offer, I suppose that's just as much an option.

Yeah, the Debian 8 repos have since been moved to the archives. You can replace it with Freexian's repos by changing the mirror in the container at /etc/apt/sources.list to deb http://deb.freexian.com/extended-lts jessie main contrib non-free A one-liner for doing that would be: echo 'deb http://deb.freexian.com/extended-lts jessie main contrib non-free' > /etc/apt/sources.list

More info here: https://www.freexian.com/lts/extended/docs/how-to-use-extended-lts/

@DeeDeeG
Copy link
Member

DeeDeeG commented Jan 30, 2024

Our Rolling release (x64, not ARM) binaries should now work on Alma Linux 8!

https://pulsar-edit.dev/download.html#rolling-release

And then Regular Pulsar release 1.114 should have this change as well. Feel free to let us know if there's any issues with it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
8 participants