k8s:push goal fails on insecure registries when using jib build strategy

[davidecavestro]

Description

The k8s:push goal fails to publish the image on insecure registries (i.e. a registry exposed only over http with username and password) when invoked passing -Djkube.build.strategy=jib in order to avoid depending on the docker daemon.

Info

• Eclipse JKube version : 1.3.0 (maven-kubernetes-plugin)
• Maven version (mvn -v) : 3.6.3

How to reproduce

1. deploy an insecure registry with user/pass authentication
2. add its address to the insecure-registries block within docker daemon config
3. configure the pom to push the image to the insecure registry
4. add username and password to the pom, or settings.xml or as additional args from CLI
5. invoke mvn -Djkube.build.strategy=jib clean package k8s:build k8s:push in order to build (and push) using jib

The build fails complaining

java.net.SocketException: Network is unreachable (connect failed)

as it happens when you try to access the insecure registry over https i.e. with curl (it doesn't listen on port 443, only 80)

If you omit the -Djkube.build.strategy=jib part it works

[rohanKanojia]

It's quite strange since we have allowInSecureRegistries flag set to true in code:
https://github.com/eclipse/jkube/blob/1e1816840bc0e9434ceb592cac379fabccb758c1/jkube-kit/build/service/jib/src/main/java/org/eclipse/jkube/kit/service/jib/JibServiceUtil.java#L223-L228

[davidecavestro]

In this case the insecure registry also needs username/password auth for write access, could it be the cause?

[rohanKanojia]

I'm not sure. I think we need to debug whether correct credentials are getting passed there in RegistryImage object. Would it be possible for you to set a breakpoint there and check?

[davidecavestro]

I'll try...

In the meantime I've noticed that when I don't specify credentials (i.e. through -Djkube.docker.push.username/password on the CLI) there's an additional line in the log

No credentials could be retrieved for the-registry/the-image

while the remaining part doesn't change

JIB> Containerizing application with the following files:                                                                    

JIB> [                              ] 0.0% complete > building images to registry
JIB> [                              ] 0.0% complete > authenticating push to the-registry
JIB> No credentials could be retrieved for the-registry/the-image                             

JIB> [=                             ] 4.5% complete > authenticating push to the-registry
JIB> [=                             ] 4.5% complete > authenticating push to the-registry
JIB> [====                          ] 13.6% complete > authenticating push to the-registry
JIB> I/O error for image [the-registry/the-image]:                                 

JIB>     java.net.SocketException                                                                                 

JIB>     Network is unreachable (connect failed)                                                                  

JIB> [=====                         ] 18.2% complete > building images to registry
JIB> [=====                         ] 18.2% complete > processing base image layers

[davidecavestro]

Correct credentials are passed down to jib-core.
The problem arises from com.google.cloud.tools.jib.registry.RegistryEndpointCaller that during BearerAuth attempt generates an auth request url prependend with https://.
This in turn triggers a SocketException - based on the registry response - that is not recognized as an insecure registry symptom, hence skip the wwwauth attempt.

RegistryEndpointCaller.call(URL) (/jib-core-0.16.0.jar/com.google.cloud.tools.jib.registry/RegistryEndpointCaller.class:131)
RegistryEndpointCaller.call() (/jib-core-0.16.0.jar/com.google.cloud.tools.jib.registry/RegistryEndpointCaller.class:115)
RegistryClient.callRegistryEndpoint(RegistryEndpointProvider) (/jib-core-0.16.0.jar/com.google.cloud.tools.jib.registry/RegistryClient.class:624)
RegistryClient.doBearerAuth(boolean) (/jib-core-0.16.0.jar/com.google.cloud.tools.jib.registry/RegistryClient.class:320)
RegistryClient.doPushBearerAuth() (/jib-core-0.16.0.jar/com.google.cloud.tools.jib.registry/RegistryClient.class:310)
AuthenticatePushStep.call() (/jib-core-0.16.0.jar/com.google.cloud.tools.jib.builder.steps/AuthenticatePushStep.class:65)
AuthenticatePushStep.call() (/jib-core-0.16.0.jar/com.google.cloud.tools.jib.builder.steps/AuthenticatePushStep.class:35)
TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly() (/guava-29.0-jre.jar/com.google.common.util.concurrent/TrustedListenableFutureTask.class:125)
InterruptibleTask.run() (/guava-29.0-jre.jar/com.google.common.util.concurrent/InterruptibleTask.class:69)
TrustedListenableFutureTask.run() (/guava-29.0-jre.jar/com.google.common.util.concurrent/TrustedListenableFutureTask.class:78)
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) (Unknown Source:1149)
ThreadPoolExecutor$Worker.run() (Unknown Source:624)
Thread.run() (Unknown Source:748)

[rohanKanojia]

@davidecavestro : From your investigation, does it look like it might be a bug in jib core?

[davidecavestro]

Well, I've just tried with jib-maven-plugin and it fails the same way.
There's no process listening on port 443 on that server, hence the behaviour depicted above.

It seems jib had an allowHttp config property in the past, but as of now I cannot find it.