Crash when used by Kaniko in Google Cloud Build

[devjgm]

I'm using docker-credential-gcr indirectly via kaniko in Google Cloud Build. When I use a recent version of Kaniko, which uses a recent version of this, I get the following crash:

...
Step #0: INFO[0148] Pushing layer us-central1-docker.pkg.dev/jgm-cloud-cxx/google-cloud-cpp-cloudbuild-docker/fedora-image/cache:c9a94ae6a3449e0c2d330d44632921f139be607646f008d7542890696f91e26f to cache now
Step #0: fatal error: unexpected signal during runtime execution
Step #0: [signal SIGSEGV: segmentation violation code=0x1 addr=0xe5 pc=0x7fd759f3dcb4]
Step #0:
Step #0: runtime stack:
Step #0: runtime.throw(0x7e0c4a, 0x2a)
Step #0:        /usr/local/go/src/runtime/panic.go:1116 +0x72
Step #0: runtime.sigpanic()
Step #0:        /usr/local/go/src/runtime/signal_unix.go:726 +0x4ac
Step #0:
Step #0: goroutine 1 [syscall]:
Step #0: runtime.cgocall(0x6a88c0, 0xc000085a30, 0xc000010038)
Step #0:        /usr/local/go/src/runtime/cgocall.go:133 +0x5b fp=0xc000085a00 sp=0xc0000859c8 pc=0x40563b
Step #0: os/user._Cfunc_mygetpwuid_r(0x0, 0xc0000b2ed0, 0xf2f070, 0x400, 0xc000010038, 0x7fd700000000)
Step #0:        _cgo_gotypes.go:175 +0x4d fp=0xc000085a30 sp=0xc000085a00 pc=0x68af8d
Step #0: os/user.lookupUnixUid.func1.1(0x0, 0xc0000b2ed0, 0xc000088dd0, 0xc000010038, 0xc000085ad0)
Step #0:        /usr/local/go/src/os/user/cgo_lookup_unix.go:103 +0xd0 fp=0xc000085a80 sp=0xc000085a30 pc=0x68bd30
Step #0: os/user.lookupUnixUid.func1(0x7987e0)
Step #0:        /usr/local/go/src/os/user/cgo_lookup_unix.go:103 +0x45 fp=0xc000085ab8 sp=0xc000085a80 pc=0x68bda5
Step #0: os/user.retryWithBuffer(0xc000088dd0, 0xc000085b90, 0x7fd75c9a1c00, 0x20300000000000)
Step #0:        /usr/local/go/src/os/user/cgo_lookup_unix.go:247 +0x3e fp=0xc000085b10 sp=0xc000085ab8 pc=0x68babe
Step #0: os/user.lookupUnixUid(0x0, 0x0, 0x0, 0x0)
Step #0:        /usr/local/go/src/os/user/cgo_lookup_unix.go:96 +0x132 fp=0xc000085bd8 sp=0xc000085b10 pc=0x68b3d2
Step #0: os/user.current(0xc000085c58, 0x4d40bc, 0xc0000b4ee0)
Step #0:        /usr/local/go/src/os/user/cgo_lookup_unix.go:49 +0x49 fp=0xc000085c18 sp=0xc000085bd8 pc=0x68b269
Step #0: os/user.Current.func1()
Step #0:        /usr/local/go/src/os/user/lookup.go:15 +0x25 fp=0xc000085c40 sp=0xc000085c18 pc=0x68bbe5
Step #0: sync.(*Once).doSlow(0xa09c40, 0x7ecf20)
Step #0:        /usr/local/go/src/sync/once.go:66 +0xec fp=0xc000085c90 sp=0xc000085c40 pc=0x474c8c
Step #0: sync.(*Once).Do(...)
Step #0:        /usr/local/go/src/sync/once.go:57
Step #0: os/user.Current(0x47705b, 0xa41360, 0xc0000b25d0)
Step #0:        /usr/local/go/src/os/user/lookup.go:15 +0x105 fp=0xc000085cc0 sp=0xc000085c90 pc=0x68ade5
Step #0: github.com/GoogleCloudPlatform/docker-credential-gcr/util.unixHomeDir(0x4d4700, 0xc0000b4ee0)
Step #0:        /go/src/github.com/GoogleCloudPlatform/docker-credential-gcr/util/util.go:42 +0x25 fp=0xc000085cf0 sp=0xc000085cc0 pc=0x68e585
Step #0: github.com/GoogleCloudPlatform/docker-credential-gcr/util.SdkConfigPath(0x0, 0x0, 0x0, 0x0)
Step #0:        /go/src/github.com/GoogleCloudPlatform/docker-credential-gcr/util/util.go:34 +0x26 fp=0xc000085d58 sp=0xc000085cf0 pc=0x68e466
Step #0: github.com/GoogleCloudPlatform/docker-credential-gcr/store.dockerCredentialPath(0x7fd7834cb108, 0xc000088dc0, 0x7fd7834d4328, 0x18)
Step #0:        /go/src/github.com/GoogleCloudPlatform/docker-credential-gcr/store/store.go:215 +0x6d fp=0xc000085dc8 sp=0xc000085d58 pc=0x69186d
Step #0: github.com/GoogleCloudPlatform/docker-credential-gcr/store.DefaultGCRCredStore(...)
Step #0:        /go/src/github.com/GoogleCloudPlatform/docker-credential-gcr/store/store.go:84
Step #0: github.com/GoogleCloudPlatform/docker-credential-gcr/cli.(*helperCmd).Execute(0xc00000e360, 0x8392e0, 0xc000016018, 0xc00009e3c0, 0x0, 0x0, 0x0, 0x0)
Step #0:        /go/src/github.com/GoogleCloudPlatform/docker-credential-gcr/cli/dockerHelper.go:35 +0x35 fp=0xc000085e78 sp=0xc000085dc8 pc=0x6a6915
Step #0: github.com/GoogleCloudPlatform/docker-credential-gcr/vendor/github.com/google/subcommands.(*Commander).Execute(0xc000012100, 0x8392e0, 0xc000016018, 0x0, 0x0, 0x0, 0x39)
Step #0:        /go/src/github.com/GoogleCloudPlatform/docker-credential-gcr/vendor/github.com/google/subcommands/subcommands.go:209 +0x30d fp=0xc000085f20 sp=0xc000085e78 pc=0x69252d
Step #0: github.com/GoogleCloudPlatform/docker-credential-gcr/vendor/github.com/google/subcommands.Execute(...)
Step #0:        /go/src/github.com/GoogleCloudPlatform/docker-credential-gcr/vendor/github.com/google/subcommands/subcommands.go:492
Step #0: main.main()
Step #0:        /go/src/github.com/GoogleCloudPlatform/docker-credential-gcr/main.go:54 +0x63f fp=0xc000085f88 sp=0xc000085f20 pc=0x6a85bf
Step #0: runtime.main()
Step #0:        /usr/local/go/src/runtime/proc.go:204 +0x209 fp=0xc000085fe0 sp=0xc000085f88 pc=0x439c89
Step #0: runtime.goexit()
Step #0:        /usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0x46b841
Step #0: Collecting git+git://github.com/googleapis/python-storage@8cf6c62a96ba3fff7e5028d931231e28e5029f1c
Step #0:   Cloning git://github.com/googleapis/python-storage (to revision 8cf6c62a96ba3fff7e5028d931231e28e5029f1c) to /tmp/pip-req-build-2spex1i1
Step #0:   Running command git clone -q git://github.com/googleapis/python-storage /tmp/pip-req-build-2spex1i1
CANCELLED
ERROR: context canceled

I think the problem is in this code as described in GoogleContainerTools/kaniko#1604 (comment)

[jonjohnsonjr]

tl;dr if kaniko wants to build this static binary, they need to pass more flags: golang/go#24787 (comment)

[huw0]

Really disappointing that there has been no update on this from the GCP team.
I have a high memory build and so have incurred some cost from trying to fix this. I always initially assume that the problem is my own change and not a defect in the cloud platform!

Also using GCP and Cloud build and I'm seeing this same problem. Using Kaniko latest.
The change I made to trigger this was to change the base of the Dockerfile from registry.access.redhat.com/ubi8/ubi-minimal:8.3 to registry.access.redhat.com/ubi8/ubi-minimal:8.4.

For now, I'm working around the issue with the debug version of Kaniko as suggested in GoogleContainerTools/kaniko#1604