aarch64-linux: hLock: invalid argument (Invalid argument)

[573]

Describe the bug
cabal aka the build of any project using a binfmt-enabled runner (github-actions) fails with

ghc-pkg: Couldn't open database /build/setup-package.conf.d for modification: {handle: /build/setup-package.conf.d/package.cache.lock}: hLock: invalid argument (Invalid argument)

To Reproduce
Steps to reproduce the behavior:
Checkout linked (Additional context below) repository and run nix-build or run commands according to github-actions log.

Expected behavior
The [cabal] build should run at least until the linker.

System information

• Operating system Ubuntu with binfmt enabled to emulate aarch64-linux
• cabal, ghc versions:
• /nix/store/6ryfddpdwwhz8k1fiwa4c68r9lni0khj-cabal-install-3.6.2
• /nix/store/wa600czlvx0mgv3mjrppwklli3y5jjj0-ghc-8.10.7

Additional context
See here for a repository reproducing this:
https://github.com/573/aarch64-linux-ld-issue155722-reproduce/runs/5044168375?check_suite_focus=true#step:6:336

[robx]

Seems likely this is an issue with GHC; can't find any relevant reports at first glance. Results of some random digging below:

You might be able to work around this by compiling cabal with the flag lukko:

cabal/cabal-install/cabal-install.cabal

Line 36 in 9694036

Flag lukko

That would use an alternate locking library: https://hackage.haskell.org/package/lukko. The only place that hLock is called:

cabal/cabal-install/src/Distribution/Client/Store.hs

Line 272 in 9694036

hLock h ExclusiveLock

.

The hLock code in base is here (the base that shipped with GHC 8.10.7 is 4.14, so there might have been changes):

• https://hackage.haskell.org/package/base-4.16.0.0/docs/src/GHC.IO.Handle.Lock.html#hLock
• https://hackage.haskell.org/package/base-4.16.0.0/docs/src/GHC.IO.Handle.Lock.LinuxOFD.html#lockImpl
• proper source link: https://gitlab.haskell.org/I-B-3/ghc/-/blob/master/libraries/base/GHC/IO/Handle/Lock/LinuxOFD.hsc#L47
  It seems to come down to calling fcntl(fd, F_SETLK, *flock) or similar; I could see that Storable instance being affected by the architecture change? No idea really.

[Mikolaj]

Indeed, lukko helped with similar cases, but it needs work to cope with new architectures and scenarios (no idea about yours): #7313.

[phadej]

ghc-pkg: Couldn't open database /build/setup-package.conf.d for modification: {handle: /build/setup-package.conf.d/package.cache.lock}: hLock: invalid argument (Invalid argument)

ghc-pkg is not cabal-install (and doesn't use lukko).

[Mikolaj]

TIL. Thank you. Marking as not our bug, in that case. @573: please let us know when you find something about this on GHC IRC/Matrix or wherever. And please re-open if it turns out cabal is calling ghc-pkg incorrectly.

[phadej]

FYI, non-collaborators cannot re-open issues.

[Mikolaj]

That's a day full of learning.

Huh, and this is not a configurable setting AFAICS. Oh well.

[573]

ghc-pkg: Couldn't open database /build/setup-package.conf.d for modification: {handle: /build/setup-package.conf.d/package.cache.lock}: hLock: invalid argument (Invalid argument)

ghc-pkg is not cabal-install (and doesn't use lukko).

@phadej and @Mikolaj can you pls see build in this branch https://github.com/573/aarch64-linux-ld-issue155722-reproduce/runs/5079804025?check_suite_focus=true#step:7:83
Looks like cabal is affected or involved as well.

[Mikolaj]

@573: that may, or may not, again be ghc-pkg, called by cabal.

BTW, I have no clue, but do you think aarch64-linux was supposed to work already in GHC 8.10? And if so, that there are no known errors in aarch64 support in 8.10 that are fixed in GHC 9.0.2? Just asking --- if you find documents or people that can answer that, perhaps the can also help you diagnose further. Please do report back!

[sopeters]

I have the same issue using https://hub.docker.com/_/haskell/ for multi-arch builds - is there any workaround yet?

[Mikolaj]

Judging from #8232, this seems to work for some people. Do you use cabal 3.8 and a new GHC (9.2.3 or 9.4, I guess)?

[sopeters]

@Mikolaj - I'm trying to use the [8.10.7-slim-buster] image on a X86 hardware setup (in our CI/CD pipeline) for multi-arch docker compiles / builds of our application - which results in this error. Works on M1 but not with the docker buildx extension.

Seems it uses cabal 3.6.2.0. Is there plans to build an image that has cabal 3.8 (don't see an official release yet) in it? Happy to help testing

[sopeters]

I tried the new cabal (cabal-install-3.8.0.20220526) with the above docker image and same error still

#23 13.54 /opt/ghc/8.10.7/bin/ghc-pkg-8.10.7 init --global /root/.stack/snapshots/aarch64-linux/c08a8a2eb5d54c2bc1c3c88c091ebf64a52214bfd58e2d3b4460017d76dd2c5f/8.10.7/pkgdb/
--
  | {handle: /root/.stack/snapshots/aarch64-linux/c08a8a2eb5d54c2bc1c3c88c091ebf64a52214bfd58e2d3b4460017d76dd2c5f/8.10.7/pkgdb/package.cache.lock}: hLock: invalid argument (Invalid argument)
  | #23 15.61 make: *** [Makefile:22: build-server] Error 1
#23 13.54 /opt/ghc/8.10.7/bin/ghc-pkg-8.10.7 init --global /root/.stack/snapshots/aarch64-linux/c08a8a2eb5d54c2bc1c3c88c091ebf64a52214bfd58e2d3b4460017d76dd2c5f/8.10.7/pkgdb/
--
  | {handle: /root/.stack/snapshots/aarch64-linux/c08a8a2eb5d54c2bc1c3c88c091ebf64a52214bfd58e2d3b4460017d76dd2c5f/8.10.7/pkgdb/package.cache.lock}: hLock: invalid argument (Invalid argument)
  | #23 15.61 make: *** [Makefile:22: build-server] Error 1

[Mikolaj]

Seems it uses cabal 3.6.2.0. Is there plans to build an image that has cabal 3.8 (don't see an official release yet) in it? Happy to help testing

Thank you, I will let you know. We plan to release in a week or two, but binaries installable with ghcup should be available earlier on. I've just checked and #8232 got merged very recently, as a last minute fix, so it's not yet present in the prerelease 3.8.0.20220526, but will be in 3.8.1.0. However, you may also need a newer GHC, but I don't know the details.

[Mikolaj]

The command to install cabal 3.8.1.0 (not officially released) is

ghcup --no-cache install cabal -u 'https://gitlab.haskell.org/haskell/cabal/-/jobs/1124022/artifacts/raw/out/cabal-install-3.8.1.0-x86_64-linux-alpine.tar.xz?inline=false' head

[sopeters]

Thanks @Mikolaj - is there an aarch64 build by chance - since this is running on a arm64 container?

[Mikolaj]

Doh, of course I meant to give you the arm64 one. Does any of these look like it may be it?

https://gitlab.haskell.org/haskell/cabal/-/pipelines/55001

If so, please substitute in the invocation I gave you.

[sopeters]

Sadly didn't help - still getting the same error on stack build

#23 19.25 Unable to create package database at /root/.stack/snapshots/aarch64-linux/c08a8a2eb5d54c2bc1c3c88c091ebf64a52214bfd58e2d3b4460017d76dd2c5f/8.10.7/pkgdb/
#23 19.25 Received ExitFailure 1 when running
#23 19.25 Raw command: /opt/ghc/8.10.7/bin/ghc-pkg-8.10.7 init /root/.stack/snapshots/aarch64-linux/c08a8a2eb5d54c2bc1c3c88c091ebf64a52214bfd58e2d3b4460017d76dd2c5f/8.10.7/pkgdb/
#23 19.25 Standard error:
#23 19.25
#23 19.25 ghc-pkg: {handle: /root/.stack/snapshots/aarch64-linux/c08a8a2eb5d54c2bc1c3c88c091ebf64a52214bfd58e2d3b4460017d76dd2c5f/8.10.7/pkgdb/package.cache.lock}: hLock: invalid argument (Invalid argument)

[Mikolaj]

That's strange. Do the directories along the path exist? Did they exist at the time of ghc-pkg invocation?

Anyway, I'd eliminate suspects further, using GHC 9.2.3 (or even 9.2.4) instead of 8.10.7.

[phadej]

I remind you that ghc-pkg is not cabal-install (and they don't share locking code). Trying newer GHC is a good idea, but there's nothing cabal can do it fhat doesn't work either.

EDIT: and if error still happens with newer GHC, please report it there.

[sopeters]

Thanks - porting the project to a newer GHC version is what I was hoping to avoid - not sure where ghc-pkg is maintained and if some of the locking code there could be fixed if that's the case....

The directory doesn't exist before the stack build command

  | #19 0.298 /root/.stack
  | #19 0.298 /root/.stack/config.yaml
  | #19 0.298 /root/.stack/pantry
  | #19 0.298 /root/.stack/pantry/pantry.sqlite3.pantry-write-lock
  | #19 0.298 /root/.stack/pantry/pantry.sqlite3
  | #19 0.298 /root/.stack/stack.sqlite3.pantry-write-lock
  | #19 0.298 /root/.stack/stack.sqlite3

[Mikolaj]

ghc-pkg is a part of GHC. Likely, the locking code has problems due to the architecture and the problems are fixed as part of the work to port GHC to arm64 (I think, the port is successfully finished by now). If you are using stack, you could try switching to a nightly stackage snaphot based on GHC 9.2. If that still fails, rummage in issue trackers of GHC, stack and cabal, googole, ask on IRC channels, etc. If nothing turns up, open a ticket on the GHC issue tracker. Let us know how it goes.

[Mikolaj]

BTW, does it help if you manually create /root/.stack/snapshots? Probably not, but this may really be some trivial bug of any of the related tools and scripts, so experimenting may help.

[sopeters]

Tried that and doesn't really help. I found this: https://gitlab.haskell.org/ghc/ghc/-/issues/17918#note_376463

As mentioned the porting of the old project to GHC 9.X isn't really something that I can (or want to do) - so was hoping to find some workaround to make it work with 8.10.7

Thanks a million for helping