Kubernetes cannot pull from a Harbor Registry deployed with ClusterIP

[felixs8696]

I have a helmfile like such

# helmfile.dev.yaml
repositories:
  - name: harbor
    url: https://helm.goharbor.io

releases:
  - name: harbor
    chart: harbor/harbor
    namespace: harbor
    values:
      - expose:
          type: clusterIP
        externalURL: "http://harbor.harbor.svc.cluster.local:80"
        persistence:
          enabled: true
          persistentVolumeClaim:
            registry:
              size: 10Gi

I am able to push to it using a simple kaniko job like such

apiVersion: batch/v1
kind: Job
metadata:
  labels:
    job-name: build-hello-world-0.0.0-test-3
  name: build-hello-world-0.0.0-test-3
  namespace: default
spec:
  template:
    metadata:
      labels:
        job-name: build-hello-world-0.0.0-test-3
    spec:
      containers:
      - args:
        - --context=tar:///mnt/build-contexts/1234567890/context.tar.gz
        - --dockerfile=Dockerfile
        - --destination=harbor.harbor.svc.cluster.local:80/hello-world:0.0.0
        - --verbosity=debug
        image: gcr.io/kaniko-project/executor:latest
        imagePullPolicy: IfNotPresent
        env:
          - name: DOCKER_CONFIG
            value: /kaniko/.docker
        name: kaniko
        volumeMounts:
        - mountPath: /mnt/build-contexts
          name: build-context-pvc
        - mountPath: /kaniko/.docker
          name: build-registry-secret
      restartPolicy: Never
      volumes:
      - name: build-context-pvc
        persistentVolumeClaim:
          claimName: build-context-pvc
      - name: build-registry-secret
        secret:
          defaultMode: 420
          secretName: build-registry-creds

However, when I try to make a dimple deployment based on this image that I've successfully pushed, I get this erroring event

│ Events:                                                                                                                                                                                                                                            │
│---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------│
│ Type      Reason     Age                   From               Message                                                                                                                                                                            │
│---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------│
│ Normal    Scheduled  13m                   default-scheduler  Successfully assigned default/hello-world-864ce95b-build-57f7d5f68f-lf6vd to minikube                                                                                              │
│ Warning   Failed     10m (x4 over 13m)     kubelet            Failed to pull image "harbor.harbor.svc.cluster.local:80/hello-world:0.0.0": Error response from daemon: Get "https://harbor.harbor.svc.cluster.local:80/v2/": dial tcp: lookup harbor.harbor.svc.cluster.local: │
│                                                             Temporary failure in name resolution                                                                                                                                                 │

Here is my deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world-build
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world-build
  template:
    metadata:
      labels:
        app: hello-world-build
    spec:
      containers:
      - env:
        - name: DOCKER_CONFIG
          value: /.docker
        image: harbor.harbor.svc.cluster.local:80/hello-world:0.0.0
        imagePullPolicy: IfNotPresent
        name: hello-world
        ports:
        - containerPort: 8000
          protocol: TCP
        volumeMounts:
        - mountPath: /.docker
          name: build-registry-secret
      volumes:
      - name: build-registry-secret
        secret:
          defaultMode: 420
          secretName: build-registry-creds

[MinerYang]

you may need set proper dns mapping for your domain name harbor.harbor.svc.cluster.local

[felixs8696]

Sorry @MinerYang can you elaborate? Where can i find the information for this?

[felixs8696]

I don't thin kit's a networking problem in general since I can successfully push to the registry. I see the artifacts published from my build job

[MinerYang]

Could you provide the log of harbor-registry and proxy log while you unable to pull

[felixs8696]

@MinerYang here are the harbor registry logs. I first push using kaniko, then try to assign the built image to a kubernetes deployment

# harbor-registry pod logs

│ 10.244.0.83 - - [15/Oct/2024:04:44:10 +0000] "PUT /v2/test/hello-world/blobs/uploads/63257e84-6350-4ca1-9c8a-74b49e64989c?_state=ydida4qcYqaYxU0z31xjgdHVWWX02jpRUdLw9qGqEa57Ik5hbWUiOiJhZ2VudGV4LXRlc3QvaGVsbG8td29ybGQiLCJVVUlEIjoiNjMy │
│ NTdlODQtNjM1MC00Y2ExLTljOGEtNzRiNDllNjQ5ODljIiwiT2Zmc2V0Ijo3NjgxLCJTdGFydGVkQXQiOiIyMDI0LTEwLTE1VDA0OjQ0OjEwWiJ9&digest=sha256%3Afa5e0ab01067d8e6bd21ae0d197dbdf925e7840c1d6c01fdb87a3f1317bb7b93 HTTP/1.1" 201 0 "" "kaniko/v1.23.2"             │
│ time="2024-10-15T04:44:10.154941507Z" level=info msg="response completed" go.version=go1.22.6 http.request.contenttype="application/octet-stream" http.request.host="harbor.harbor.svc.cluster.local:80" http.request.id=641cd2ac-8797-42f3-bb1a- │
│ 5849907577f0 http.request.method=PUT http.request.remoteaddr=10.244.0.86 http.request.uri="/v2/test/hello-world/blobs/uploads/63257e84-6350-4ca1-9c8a-74b49e64989c?_state=ydida4qcYqaYxU0z31xjgdHVWWX02jpRUdLw9qGqEa57Ik5hbWUiOiJhZ2VudGV │
│ 4LXRlc3QvaGVsbG8td29ybGQiLCJVVUlEIjoiNjMyNTdlODQtNjM1MC00Y2ExLTljOGEtNzRiNDllNjQ5ODljIiwiT2Zmc2V0Ijo3NjgxLCJTdGFydGVkQXQiOiIyMDI0LTEwLTE1VDA0OjQ0OjEwWiJ9&digest=sha256%3Afa5e0ab01067d8e6bd21ae0d197dbdf925e7840c1d6c01fdb87a3f1317bb7b93" http. │
│ request.useragent="kaniko/v1.23.2" http.response.duration=68.389292ms http.response.status=201 http.response.written=0                                                                                                                            │
│ 10.244.0.83 - - [15/Oct/2024:04:44:10 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"                                                                                                                                                       │
│ time="2024-10-15T04:44:10.230152007Z" level=info msg="authorized request" go.version=go1.22.6 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="harbor.harbor.svc.cluster.local:80" http.request │
│ .id=8a048baa-2ca8-419a-83cc-15471288fe59 http.request.method=PUT http.request.remoteaddr=10.244.0.86 http.request.uri="/v2/test/hello-world/manifests/0.0.3" http.request.useragent="kaniko/v1.23.2" vars.name="test/hello-world" │
│  vars.reference=0.0.3                                                                                                                                                                                                                             │
│ time="2024-10-15T04:44:10.239998466Z" level=info msg="response completed" go.version=go1.22.6 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="harbor.harbor.svc.cluster.local:80" http.request │
│ .id=8a048baa-2ca8-419a-83cc-15471288fe59 http.request.method=PUT http.request.remoteaddr=10.244.0.86 http.request.uri="/v2/test/hello-world/manifests/0.0.3" http.request.useragent="kaniko/v1.23.2" http.response.duration=75.00275ms ht │
│ tp.response.status=201 http.response.written=0                                                                                                                                                                                                    │
│ 10.244.0.83 - - [15/Oct/2024:04:44:10 +0000] "PUT /v2/test/hello-world/manifests/0.0.3 HTTP/1.1" 201 0 "" "kaniko/v1.23.2"                                                                                                                │
│ time="2024-10-15T04:44:10.245358174Z" level=warning msg="error authorizing context: basic authentication challenge for realm "harbor-registry-basic-realm": invalid authorization credential" go.version=go1.22.6 http.request.host="harbor-regis │
│ try:5000" http.request.id=ae6de32b-fb75-4c1d-9ee8-404e0a922e1c http.request.method=GET http.request.remoteaddr="10.244.0.83:41994" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1"                                            │
│ 10.244.0.83 - - [15/Oct/2024:04:44:10 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "Go-http-client/1.1"                                                                                                                                                   │
│ time="2024-10-15T04:44:10.312096049Z" level=info msg="authorized request" go.version=go1.22.6 http.request.host="harbor-registry:5000" http.request.id=62caf6cd-d9b1-4f3a-aaa4-b7c41ace8ddc http.request.method=GET http.request.remoteaddr="10.2 │
│ 44.0.83:33558" http.request.uri="/v2/test/hello-world/manifests/sha256:8fa228bded06e131d1ecc634cd30d340a4b7756483a7a510951c5920aa058cd6" http.request.useragent=harbor-registry-client vars.name="test/hello-world" vars.referenc │
│ e="sha256:8fa228bded06e131d1ecc634cd30d340a4b7756483a7a510951c5920aa058cd6"                                                                                                                                                                       │
│ 10.244.0.83 - - [15/Oct/2024:04:44:10 +0000] "GET /v2/test/hello-world/manifests/sha256:8fa228bded06e131d1ecc634cd30d340a4b7756483a7a510951c5920aa058cd6 HTTP/1.1" 200 1734 "" "harbor-registry-client"                                   │
│ time="2024-10-15T04:44:10.313968591Z" level=info msg="response completed" go.version=go1.22.6 http.request.host="harbor-registry:5000" http.request.id=62caf6cd-d9b1-4f3a-aaa4-b7c41ace8ddc http.request.method=GET http.request.remoteaddr="10.2 │
│ 44.0.83:33558" http.request.uri="/v2/test/hello-world/manifests/sha256:8fa228bded06e131d1ecc634cd30d340a4b7756483a7a510951c5920aa058cd6" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.docker. │
│ distribution.manifest.v2+json" http.response.duration=67.536666ms http.response.status=200 http.response.written=1734                                                                                                                             │
│ time="2024-10-15T04:44:10.379290632Z" level=info msg="authorized request" go.version=go1.22.6 http.request.host="harbor-registry:5000" http.request.id=074f60f4-7ce9-4910-8e5c-9d6ce33ad12f http.request.method=GET http.request.remoteaddr="10.2 │
│ 44.0.83:33558" http.request.uri="/v2/test/hello-world/blobs/sha256:fa5e0ab01067d8e6bd21ae0d197dbdf925e7840c1d6c01fdb87a3f1317bb7b93" http.request.useragent=harbor-registry-client vars.digest="sha256:fa5e0ab01067d8e6bd21ae0d197dbdf925 │
│ e7840c1d6c01fdb87a3f1317bb7b93" vars.name="test/hello-world"                                                                                                                                                                              │
│ 10.244.0.83 - - [15/Oct/2024:04:44:10 +0000] "GET /v2/test/hello-world/blobs/sha256:fa5e0ab01067d8e6bd21ae0d197dbdf925e7840c1d6c01fdb87a3f1317bb7b93 HTTP/1.1" 200 7681 "" "harbor-registry-client"                                       │
│ time="2024-10-15T04:44:10.381717466Z" level=info msg="response completed" go.version=go1.22.6 http.request.host="harbor-registry:5000" http.request.id=074f60f4-7ce9-4910-8e5c-9d6ce33ad12f http.request.method=GET http.request.remoteaddr="10.2 │
│ 44.0.83:33558" http.request.uri="/v2/test/hello-world/blobs/sha256:fa5e0ab01067d8e6bd21ae0d197dbdf925e7840c1d6c01fdb87a3f1317bb7b93" http.request.useragent=harbor-registry-client http.response.contenttype="application/octet-stream" h │
│ ttp.response.duration=66.75375ms http.response.status=200 http.response.written=7681                                                                                                                                                              │
│ 10.244.0.1 - - [15/Oct/2024:04:44:18 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.31"                                                                                                                                                           │
│ 10.244.0.83 - - [15/Oct/2024:04:44:20 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"                                                                                                                                                       │
│ 10.244.0.1 - - [15/Oct/2024:04:44:28 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.31"                                                                                                                                                           │
│ 10.244.0.83 - - [15/Oct/2024:04:44:30 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"

I'm not running a proxy, but here are the harbor-core logs

│ 2024-10-15T04:41:09Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.cncf.helm.config.v1+json registered                                                                             │
│ 2024-10-15T04:41:09Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.cnab.manifest.v1 registered                                                                                     │
│ 2024-10-15T04:41:09Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.oci.image.index.v1+json registered                                                                              │
│ 2024-10-15T04:41:09Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.docker.distribution.manifest.list.v2+json registered                                                            │
│ 2024-10-15T04:41:09Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.docker.distribution.manifest.v1+prettyjws registered                                                            │
│ 2024-10-15T04:41:09Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.oci.image.config.v1+json registered                                                                             │
│ 2024-10-15T04:41:09Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.docker.container.image.v1+json registered                                                                       │
│ 2024-10-15T04:41:09Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.goharbor.harbor.sbom.v1 registered                                                                              │
│ 2024-10-15T04:41:09Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.wasm.config.v1+json registered                                                                                  │
│ 2024-10-15T04:41:09Z [INFO] [/pkg/reg/adapter/native/adapter.go:36]: the factory for adapter docker-registry registered                                                                                                                           │
│ 2024-10-15T04:41:09Z [INFO] [/pkg/reg/adapter/aliacr/adapter.go:44]: the factory for adapter ali-acr registered                                                                                                                                   │
│ 2024-10-15T04:41:09Z [INFO] [/pkg/reg/adapter/awsecr/adapter.go:44]: the factory for adapter aws-ecr registered                                                                                                                                   │
│ 2024-10-15T04:41:09Z [INFO] [/pkg/reg/adapter/azurecr/adapter.go:29]: Factory for adapter azure-acr registered                                                                                                                                    │
│ 2024-10-15T04:41:09Z [INFO] [/pkg/reg/adapter/dockerhub/adapter.go:40]: Factory for adapter docker-hub registered                                                                                                                                 │
│ 2024-10-15T04:41:09Z [INFO] [/pkg/reg/adapter/dtr/adapter.go:36]: the factory of dtr adapter was registered                                                                                                                                       │
│ 2024-10-15T04:41:09Z [INFO] [/pkg/reg/adapter/githubcr/adapter.go:43]: the factory for adapter github-ghcr registered                                                                                                                             │
│ 2024-10-15T04:41:09Z [INFO] [/pkg/reg/adapter/gitlab/adapter.go:33]: the factory for adapter gitlab registered                                                                                                                                    │
│ 2024-10-15T04:41:09Z [INFO] [/pkg/reg/adapter/googlegcr/adapter.go:37]: the factory for adapter google-gcr registered                                                                                                                             │
│ 2024-10-15T04:41:09Z [INFO] [/pkg/reg/adapter/huawei/huawei_adapter.go:40]: the factory of Huawei adapter was registered                                                                                                                          │
│ 2024-10-15T04:41:09Z [INFO] [/pkg/reg/adapter/jfrog/adapter.go:42]: the factory of jfrog artifactory adapter was registered                                                                                                                       │
│ 2024-10-15T04:41:09Z [INFO] [/pkg/reg/adapter/quay/adapter.go:53]: the factory of Quay adapter was registered                                                                                                                                     │
│ 2024-10-15T04:41:09Z [INFO] [/pkg/reg/adapter/tencentcr/adapter.go:55]: the factory for adapter tencent-tcr registered                                                                                                                            │
│ 2024-10-15T04:41:09Z [INFO] [/pkg/reg/adapter/volcenginecr/adapter.go:40]: the factory for adapter volcengine-cr registered                                                                                                                       │
│ 2024-10-15T04:41:09Z [INFO] [/pkg/reg/adapter/harbor/adaper.go:31]: the factory for adapter harbor registered                                                                                                                                     │
│ 2024-10-15T04:41:09Z [INFO] [/core/controllers/base.go:159]: Config path: /etc/core/app.conf                                                                                                                                                      │
│ 2024-10-15T04:41:09Z [INFO] [/core/main.go:145]: initializing cache ...                                                                                                                                                                           │
│ 2024-10-15T04:41:09Z [INFO] [/core/main.go:164]: initializing configurations...                                                                                                                                                                   │
│ 2024-10-15T04:41:09Z [INFO] [/lib/config/systemconfig.go:178]: key path: /etc/core/key                                                                                                                                                            │
│ 2024-10-15T04:41:09Z [INFO] [/lib/config/config.go:92]: init secret store                                                                                                                                                                         │
│ 2024-10-15T04:41:09Z [INFO] [/core/main.go:166]: configurations initialization completed                                                                                                                                                          │
│ 2024-10-15T04:41:09Z [INFO] [/common/dao/base.go:67]: Registering database: type-PostgreSQL host-harbor-database port-5432 database-registry sslmode-"disable"                                                                                    │
│ 2024-10-15T04:41:10Z [INFO] [/common/dao/base.go:72]: Register database completed                                                                                                                                                                 │
│ 2024-10-15T04:41:10Z [INFO] [/common/dao/pgsql.go:135]: Upgrading schema for pgsql ...                                                                                                                                                            │
│ 2024-10-15T04:41:10Z [INFO] [/common/dao/pgsql.go:138]: No change in schema, skip.                                                                                                                                                                │
│ 2024-10-15T04:41:10Z [INFO] [/core/main.go:204]: The database has been migrated successfully                                                                                                                                                      │
│ 2024-10-15T04:41:10Z [INFO] [/core/main.go:98]: User id: 1 already has its encrypted password.                                                                                                                                                    │
│ 2024-10-15T04:41:10Z [INFO] [/core/main.go:321]: Registering Trivy scanner                                                                                                                                                                        │
│ 2024-10-15T04:41:10Z [INFO] [/pkg/scan/init.go:64]: Scanner registration already exists: http://harbor-trivy:8080                                                                                                                                 │
│ 2024-10-15T04:41:10Z [INFO] [/core/main.go:343]: Setting Trivy as default scanner                                                                                                                                                                 │
│ 2024-10-15T04:41:10Z [INFO] [/pkg/scan/init.go:79]: Skipped setting Trivy as the default scanner. The default scanner is already set to http://harbor-trivy:8080                                                                                  │
│ 2024-10-15T04:41:10Z [ERROR] [/pkg/audit/forward.go:44]: failed to create audit log, error dial tcp: missing address                                                                                                                              │
│ 2024-10-15T04:41:10Z [INFO] [/core/main.go:242]: initializing notification...                                                                                                                                                                     │
│ 2024-10-15T04:41:10Z [INFO] [/pkg/notification/notification.go:77]: notification initialization completed                                                                                                                                         │
│ 2024-10-15T04:41:10Z [INFO] [/core/main.go:261]: Version: v2.11.1, Git commit: 6b7ecba1                                                                                                                                                           │
│ 2024-10-15T04:41:10Z [INFO] [/core/main.go:263]: Fix empty subiss for meta info data.                                                                                                                                                             │
│ 2024-10-15T04:41:10Z [INFO] [/pkg/oidc/fix.go:37]: Not found any records with empty subiss, good to go.                                                                                                                                           │
│ 2024/10/15 04:41:10.437 [I] [server.go:281]  http server Running on http://:8080                                                                                                                                                                  │
│ 2024-10-15T04:43:39Z [INFO] [/server/middleware/security/robot.go:71][requestID="636aefc1-e358-4f98-abcd-550306ee6dd9"]: a robot security context generated for request GET /service/token                                                        │
│ 2024-10-15T04:44:08Z [INFO] [/server/middleware/security/robot.go:71][requestID="8df43a5e-4e23-4842-8a5a-9b1b9432c4fe"]: a robot security context generated for request GET /service/token                                                        │
│ 2024-10-15T04:44:25Z [INFO] [/controller/registry/controller.go:222]: Start regular health check for registries with interval 5m0s

and here are the harbor nginx logs

│ [15/Oct/2024:04:43:29 +0000]:10.244.0.1 - "GET / HTTP/1.1" 200 785 "-" "kube-probe/1.31" 0.002 0.002 .                                                                                                                                            │
│ [15/Oct/2024:04:43:39 +0000]:10.244.0.86 - "\x16\x03\x01\x01\x1E\x01\x00\x01\x1A\x03\x03\xD7\x00\xB2\x1F\xBFF\x84o\x18\x06\x11\xE02\x8C1l\xE7\x18\xA9\xE2r2\xE0\x88QK+\x86\xB1\x91&\xD9 \x5C\xD0\x07\xB9\x9E\x1B+\xCC\x8A!\x84\xFF\xAF<uqr\xAEA\x │
│ [15/Oct/2024:04:43:39 +0000]:10.244.0.86 - "GET /v2/ HTTP/1.1" 401 76 "-" "Go-http-client/1.1" 0.005 0.006 .                                                                                                                                      │
│ [15/Oct/2024:04:43:39 +0000]:10.244.0.86 - "GET /service/token?scope=repository%3Aagentex-test%2Fhello-world%3Apush%2Cpull&service=harbor-registry HTTP/1.1" 200 747 "-" "go-containerregistry/v0.19.2" 0.028 0.027 .                             │
│ [15/Oct/2024:04:43:39 +0000]:10.244.0.86 - "POST /v2/test/hello-world/blobs/uploads/ HTTP/1.1" 202 0 "-" "go-containerregistry/v0.19.2" 0.088 0.087 .                                                                                     │
│ [15/Oct/2024:04:43:39 +0000]:10.244.0.86 - "DELETE /v2/test/hello-world/blobs/uploads/aece951e-8fa5-425d-80e5-c6f75c924201?_state=Q1n2eAIfevQEW0PKGG6caVe8mguLkb3nZpsNNrDAtIx7Ik5hbWUiOiJhZ2VudGV4LXRlc3QvaGVsbG8td29ybGQiLCJVVUlEIjoiYWV │
│ [15/Oct/2024:04:43:39 +0000]:10.244.0.1 - "GET / HTTP/1.1" 200 785 "-" "kube-probe/1.31" 0.000 0.000 .                                                                                                                                            │
│ [15/Oct/2024:04:43:39 +0000]:10.244.0.1 - "GET / HTTP/1.1" 200 785 "-" "kube-probe/1.31" 0.001 0.000 .                                                                                                                                            │
│ [15/Oct/2024:04:43:49 +0000]:10.244.0.1 - "GET / HTTP/1.1" 200 785 "-" "kube-probe/1.31" 0.000 0.000 .                                                                                                                                            │
│ [15/Oct/2024:04:43:49 +0000]:10.244.0.1 - "GET / HTTP/1.1" 200 785 "-" "kube-probe/1.31" 0.000 0.000 .                                                                                                                                            │
│ [15/Oct/2024:04:43:57 +0000]:127.0.0.1 - "GET /api/v2.0/jobservice/queues HTTP/1.1" 401 62 "http://localhost:8080/harbor/projects/2/repositories" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ │
│ [15/Oct/2024:04:43:59 +0000]:10.244.0.1 - "GET / HTTP/1.1" 200 785 "-" "kube-probe/1.31" 0.001 0.001 .                                                                                                                                            │
│ [15/Oct/2024:04:43:59 +0000]:10.244.0.1 - "GET / HTTP/1.1" 200 785 "-" "kube-probe/1.31" 0.001 0.001 .                                                                                                                                            │
│ [15/Oct/2024:04:44:08 +0000]:10.244.0.86 - "\x16\x03\x01\x01\x1E\x01\x00\x01\x1A\x03\x03\xF9/`\xABg7T\x1A\x8B\xFFM\x867B\x83\xC9qq\xAB^I2\xF0u\x9F\xD1a\xDD\xACq\xD6\x05 \xB3PN\xDD\xAE\x81\xC3\x19k\xB9\x88PA\x19\x09\xCD4!nN\xD9\x8A\xCDz\x94-\ │
│ [15/Oct/2024:04:44:08 +0000]:10.244.0.86 - "GET /v2/ HTTP/1.1" 401 76 "-" "kaniko/v1.23.2" 0.002 0.002 .                                                                                                                                          │
│ [15/Oct/2024:04:44:08 +0000]:10.244.0.86 - "GET /service/token?scope=repository%3Aagentex-test%2Fhello-world%3Apush%2Cpull&service=harbor-registry HTTP/1.1" 200 745 "-" "kaniko/v1.23.2" 0.008 0.008 .                                           │
│ [15/Oct/2024:04:44:08 +0000]:10.244.0.86 - "HEAD /v2/test/hello-world/manifests/0.0.3 HTTP/1.1" 404 0 "-" "kaniko/v1.23.2" 0.006 0.006 .                                                                                                  │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "HEAD /v2/test/hello-world/blobs/sha256:3fe349e029970270c77c6cc4e8c8159d6390de76a4c30666e585e9ac57e54212 HTTP/1.1" 200 0 "-" "kaniko/v1.23.2" 0.208 0.208 .                                    │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "HEAD /v2/test/hello-world/blobs/sha256:14aea17807c4c653827ca820a0526d96552bda685bf29293e8be90d1b05662f6 HTTP/1.1" 200 0 "-" "kaniko/v1.23.2" 0.237 0.237 .                                    │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "HEAD /v2/test/hello-world/blobs/sha256:7c371182a779ffecfece7a4a602e5ac3b60f9f6a8bcee3370c0044e4641e052e HTTP/1.1" 404 0 "-" "kaniko/v1.23.2" 0.066 0.066 .                                    │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "HEAD /v2/test/hello-world/blobs/sha256:7399277d91f4520aebebe70097c48af726818282fbbc79c1fb34b53f088eb090 HTTP/1.1" 200 0 "-" "kaniko/v1.23.2" 0.306 0.306 .                                    │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "HEAD /v2/test/hello-world/blobs/sha256:d191be7a3c9fa95847a482db8211b6f85b45096c7817fdad4d7661ee7ff1a421 HTTP/1.1" 200 0 "-" "kaniko/v1.23.2" 0.306 0.306 .                                    │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "HEAD /v2/test/hello-world/blobs/sha256:d958b5108f62e4bb8cdb1b85d6ece023b29b979ca6c46baebacc68252bb5996e HTTP/1.1" 404 0 "-" "kaniko/v1.23.2" 0.004 0.004 .                                    │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "HEAD /v2/test/hello-world/blobs/sha256:f8a41964985ed0b8e3ac999f93cb697f21d3f130141e23bcba505e47649974ca HTTP/1.1" 404 0 "-" "kaniko/v1.23.2" 0.004 0.004 .                                    │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "HEAD /v2/test/hello-world/blobs/sha256:35f10c4cf03d43a7afd688184e68c55cc3adc2049e78dffcc419a28f8ee8ac68 HTTP/1.1" 200 0 "-" "kaniko/v1.23.2" 0.196 0.196 .                                    │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "HEAD /v2/test/hello-world/blobs/sha256:bc18a638d7c518de3559b457322ba7b6b631330c37af1178d926bb4c26dff158 HTTP/1.1" 404 0 "-" "kaniko/v1.23.2" 0.003 0.003 .                                    │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "POST /v2/test/hello-world/blobs/uploads/ HTTP/1.1" 202 0 "-" "kaniko/v1.23.2" 0.134 0.134 .                                                                                                   │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "POST /v2/test/hello-world/blobs/uploads/ HTTP/1.1" 202 0 "-" "kaniko/v1.23.2" 0.192 0.192 .                                                                                                   │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "POST /v2/test/hello-world/blobs/uploads/ HTTP/1.1" 202 0 "-" "kaniko/v1.23.2" 0.196 0.196 .                                                                                                   │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "POST /v2/test/hello-world/blobs/uploads/ HTTP/1.1" 202 0 "-" "kaniko/v1.23.2" 0.130 0.130 .                                                                                                   │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "PATCH /v2/test/hello-world/blobs/uploads/cabd7213-848b-441d-bd54-ca865e3023fb?_state=AjhZZLMXyHhvr6A9ma4c-jLkkMIlkNYQOH-jkKbnl9h7Ik5hbWUiOiJhZ2VudGV4LXRlc3QvaGVsbG8td29ybGQiLCJVVUlEIjoiY2Fi │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "PATCH /v2/test/hello-world/blobs/uploads/0833e654-6091-46b4-85e5-fd37d6d00530?_state=xv8Y-zs6w1Tw5kOWl2lS2-MTyq3EA-WzM2u2Rcq_KV17Ik5hbWUiOiJhZ2VudGV4LXRlc3QvaGVsbG8td29ybGQiLCJVVUlEIjoiMDgz │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "PATCH /v2/test/hello-world/blobs/uploads/16b7b222-f5f4-4c12-b9ed-8a42892c40bc?_state=ZLA0rWCi5iV78VRyjvipHLcT84axwVC6QjfNdaDW0vx7Ik5hbWUiOiJhZ2VudGV4LXRlc3QvaGVsbG8td29ybGQiLCJVVUlEIjoiMTZi │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.1 - "GET / HTTP/1.1" 200 785 "-" "kube-probe/1.31" 0.000 0.000 .                                                                                                                                            │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.1 - "GET / HTTP/1.1" 200 785 "-" "kube-probe/1.31" 0.000 0.000 .                                                                                                                                            │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "PUT /v2/test/hello-world/blobs/uploads/0833e654-6091-46b4-85e5-fd37d6d00530?_state=hB3kgTfLI5Ht-trIXu0ym7EMTdLGQ6lc3L5H_3WqZGl7Ik5hbWUiOiJhZ2VudGV4LXRlc3QvaGVsbG8td29ybGQiLCJVVUlEIjoiMDgzM2 │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "PUT /v2/test/hello-world/blobs/uploads/cabd7213-848b-441d-bd54-ca865e3023fb?_state=7_xg7FTWXjLqHHcNlXQUWobnyGlBWPNKFLxuqDyUe9l7Ik5hbWUiOiJhZ2VudGV4LXRlc3QvaGVsbG8td29ybGQiLCJVVUlEIjoiY2FiZD │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "HEAD /v2/test/hello-world/blobs/sha256:fa5e0ab01067d8e6bd21ae0d197dbdf925e7840c1d6c01fdb87a3f1317bb7b93 HTTP/1.1" 404 0 "-" "kaniko/v1.23.2" 0.002 0.002 .                                    │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "PUT /v2/test/hello-world/blobs/uploads/16b7b222-f5f4-4c12-b9ed-8a42892c40bc?_state=RfZmedQFXPnqRjtHFTXUd-iyiQbqYnJRDUcFBhsoIBB7Ik5hbWUiOiJhZ2VudGV4LXRlc3QvaGVsbG8td29ybGQiLCJVVUlEIjoiMTZiN2 │
│ [15/Oct/2024:04:44:09 +0000]:10.244.0.86 - "PATCH /v2/test/hello-world/blobs/uploads/f0570eb4-e091-4b0b-9275-0e257be43365?_state=fQEZdjZ-a39l03jFCb8vxOj0jNiV67Qi7xkOh6vFs317Ik5hbWUiOiJhZ2VudGV4LXRlc3QvaGVsbG8td29ybGQiLCJVVUlEIjoiZjA1 │
│ [15/Oct/2024:04:44:10 +0000]:10.244.0.86 - "POST /v2/test/hello-world/blobs/uploads/ HTTP/1.1" 202 0 "-" "kaniko/v1.23.2" 0.127 0.127 .                                                                                                   │
│ [15/Oct/2024:04:44:10 +0000]:10.244.0.86 - "PUT /v2/test/hello-world/blobs/uploads/f0570eb4-e091-4b0b-9275-0e257be43365?_state=P9Se-PbbJoml3BCJxYvxue1jedNY9rRMMeEuCXNoq-V7Ik5hbWUiOiJhZ2VudGV4LXRlc3QvaGVsbG8td29ybGQiLCJVVUlEIjoiZjA1Nz │
│ [15/Oct/2024:04:44:10 +0000]:10.244.0.86 - "PATCH /v2/test/hello-world/blobs/uploads/63257e84-6350-4ca1-9c8a-74b49e64989c?_state=fi_ckbgcu1kNb7CCRr3k8tUrFJBdTbKh-wHYslC7rxt7Ik5hbWUiOiJhZ2VudGV4LXRlc3QvaGVsbG8td29ybGQiLCJVVUlEIjoiNjMy │
│ [15/Oct/2024:04:44:10 +0000]:10.244.0.86 - "PUT /v2/test/hello-world/blobs/uploads/63257e84-6350-4ca1-9c8a-74b49e64989c?_state=ydida4qcYqaYxU0z31xjgdHVWWX02jpRUdLw9qGqEa57Ik5hbWUiOiJhZ2VudGV4LXRlc3QvaGVsbG8td29ybGQiLCJVVUlEIjoiNjMyNT │
│ [15/Oct/2024:04:44:10 +0000]:10.244.0.86 - "PUT /v2/test/hello-world/manifests/0.0.3 HTTP/1.1" 201 0 "-" "kaniko/v1.23.2" 0.243 0.243 .

[MinerYang]

I don't see any pulling requesting arriving at harbor side. This above logs more like a push request.

1. Could you try to manually pulling the image?
2. If only can not pulling from kaniko, you probably need to open debug at your kaniko config and kubernetes cluster side, for example could the kubelet properly resolve the harbor domain name etc..

[jimmidyson]

The kubelet uses the host DNS config for lookups so cannot resolve service IPs by DNS lookups of the service such as harbor.harbor.svc.cluster.local. I suggest you use a stable IP for the ClusterIP service and use that in the image name to pull from. The service IP will be routeable from the kubelet even when you can't look up the service name via DNS.

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.