Certificate Chain Validation Fails When Running Fabric8 Kubernetes Client Inside a Pod

[fengyinqiao]

Describe the bug

Description

I am encountering an issue when trying to connect to the Kubernetes cluster using the Fabric8 Kubernetes client from within a pod. The connection fails with the error: "Certificate chain is not valid".

Interestingly, when I use kubectl with the same kubeconfig file (--kubeconfig option), I am able to connect without any issues. This problem only occurs when using Fabric8 within the pod.

Environment Details

• Java Version: 17
• Spring Boot Version: 2.5.15
• Fabric8 Kubernetes Client Version: 6.9.2
• Pod Container Information:
  • OS: CentOS Linux 7 (Core)
  • Kernel Version: 4.19.91-26.6.al7.x86_64
  • Details from /etc/os-release:

    NAME="CentOS Linux"
    VERSION="7 (Core)"
    ID="centos"
    ID_LIKE="rhel fedora"
    VERSION_ID="7"
    PRETTY_NAME="CentOS Linux 7 (Core)"
    ANSI_COLOR="0;31"
    CPE_NAME="cpe:/o:centos:centos:7"
    HOME_URL="https://www.centos.org/"
    BUG_REPORT_URL="https://bugs.centos.org/"
    

Issue Details

• kubectl works without issues: Using kubectl --kubeconfig inside the same pod works as expected, and I can connect to the Kubernetes API server without any certificate issues.
• Fabric8 client fails with certificate error: When trying to connect using the Fabric8 Kubernetes client, it fails with the "Certificate chain is not valid" error.

Steps to Reproduce

1. Run a pod using CentOS 7 with the following kernel version: 4.19.91-26.6.al7.x86_64.
2. Inside the pod, try connecting to the Kubernetes cluster using kubectl --kubeconfig. The connection should work.
3. Use the Fabric8 Kubernetes client with the same configuration, which results in a certificate chain validation error.

Expected Behavior

The Fabric8 Kubernetes client should successfully establish a connection to the Kubernetes API server, similar to the kubectl command, without any certificate validation issues.

Actual Behavior

The Fabric8 Kubernetes client fails with a certificate chain validation error.

Fabric8 Kubernetes Client version

6.9.2

Runtime

Kubernetes (vanilla)

Kubernetes API Server version

v1.26.15-aliyun.1

Environment

Linux

Fabric8 Kubernetes Client Code and Logs

code is as below:

private void initializeClient(File file) {
        try {
            String kubeConfigPath = file.getAbsolutePath();
            Config config = Config.fromKubeconfig(Files.readString(Path.of(kubeConfigPath)));
            KubernetesClient client = new KubernetesClientBuilder().withConfig(config).build();
            clients.put(file.getName(), client);
            informerFactories.put(file.getName(), initializeInformers(client));
            logger.info("Kubernetes client initialized for config: {}", file.getName());
        } catch (Exception e) {
            logger.error("Failed to initialize Kubernetes client for config: {}", file.getName());
            throw new RuntimeException("Failed to initialize Kubernetes client for config: " + file.getName(), e);
        }
    }

Caused by: java.lang.RuntimeException: Failed to initialize Kubernetes client for config: test-kubeconfig
        at com.test.k8s.KubernetesClientsManager.initializeClient(KubernetesClientsManager.java:91)
        at com.test.k8s.KubernetesClientsManager.lambda$new$0(KubernetesClientsManager.java:38)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:129)
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:122)
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.applyCommonConfiguration(HttpClientUtils.java:191)
        at io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory.newBuilder(OkHttpClientFactory.java:82)
        at io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory.newBuilder(OkHttpClientFactory.java:29)
        at io.fabric8.kubernetes.client.KubernetesClientBuilder.getHttpClient(KubernetesClientBuilder.java:90)
        at io.fabric8.kubernetes.client.KubernetesClientBuilder.build(KubernetesClientBuilder.java:79)
        at com.test.k8s.KubernetesClientsManager.initializeClient(KubernetesClientsManager.java:85)
        ... 5 common frames omitted
Caused by: java.security.KeyStoreException: Certificate chain is not valid
        at java.base/sun.security.pkcs12.PKCS12KeyStore.setKeyEntry(PKCS12KeyStore.java:646)
        at java.base/sun.security.pkcs12.PKCS12KeyStore.engineSetKeyEntry(PKCS12KeyStore.java:589)
        at java.base/sun.security.util.KeyStoreDelegator.engineSetKeyEntry(KeyStoreDelegator.java:112)
        at java.base/java.security.KeyStore.setKeyEntry(KeyStore.java:1167)
        at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:159)
        at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:288)
        at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:188)
        at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:177)
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.applyCommonConfiguration(HttpClientUtils.java:188)
        ... 10 common frames omitted

Additional context

This issue might be related to differences in how kubectl and Fabric8 handle the certificate chain or the CA bundle.

Could you please help to identify if there is an issue with the way Fabric8 is validating the certificate chain in this particular setup or suggest any workarounds that could be used here?

Thank you for your support.

No response