Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Garbage collector not clean redis keys #1605

Open
nevlkv opened this issue Sep 7, 2023 · 23 comments
Open

Garbage collector not clean redis keys #1605

nevlkv opened this issue Sep 7, 2023 · 23 comments
Assignees

Comments

@nevlkv
Copy link

nevlkv commented Sep 7, 2023

Expected behavior and actual behavior:
Garbage collector not cleanup redis keys, and failed with message

2023-09-06T02:37:23Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:414]: 2937 blobs and 812 manifests are actually deleted 2023-09-06T02:37:23Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:415]: The GC job actual frees up 54343 MB space. 2023-09-06T02:37:23Z [ERROR] [/jobservice/job/impl/gc/garbage_collection.go:445]: failed to clean registry cache error retrieving 'blobs::*' keys: WRONGPASS invalid username-password pair or user is disabled., pattern blobs::*

Steps to reproduce the problem:

Starts registry cleanup

Versions:

  • harbor version: 2.8.4

Additional context:

external redis with default user

_REDIS_URL_REG: redis://:password@harbor-redis-master:6379/2?idle_timeout_seconds=30
@wy65701436
Copy link
Contributor

the error points the auth info is incorrect, can you check that at your end?

@nevlkv
Copy link
Author

nevlkv commented Sep 11, 2023

password correct

redis-cli -n 2 -a <password>
127.0.0.1:6379[2]> info keyspace
# Keyspace
db0:keys=133200,expires=133085,avg_ttl=2212096
db1:keys=2036,expires=1748,avg_ttl=83530815
db2:keys=5784,expires=156,avg_ttl=85886397
db5:keys=63,expires=54,avg_ttl=2360250

@nevlkv
Copy link
Author

nevlkv commented Sep 11, 2023

garbage service used _REDIS_URL_REG: ?

@MinerYang
Copy link
Collaborator

Hi @nevlkv

  • Is this a fresh install harbor? Have you changed the redis password after installation?
  • Could you also check/provide other redis url, for example _REDIS_URL_CORE?

@nevlkv
Copy link
Author

nevlkv commented Sep 12, 2023

  1. current instance was upgraded from 2.7.1, redis is external and not changed
  2. we use default redis account and password always identical
kubectl get cm harbor-core -n=io  -o yaml
apiVersion: v1
data:
  _REDIS_URL_CORE: redis://:<password>@harbor-redis-master:6379/0?idle_timeout_seconds=30
  _REDIS_URL_REG: redis://:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30

i can auth with via redis-cli and flushdb

GC use same connection string

@viceice
Copy link

viceice commented Sep 19, 2023

Seeing same error:

023-09-17T00:02:59Z [ERROR] [/jobservice/job/impl/gc/garbage_collection.go:515]: failed to clean registry cache failed to scan keys: WRONGPASS invalid username-password pair or user is disabled., pattern blobs::*

I've no password for my external redis, i'm using network policies to limit redis access to harbor.

_REDIS_URL_CORE: redis://keydb:6379/0?idle_timeout_seconds=30
_REDIS_URL_REG: redis://keydb:6379/2?idle_timeout_seconds=30

Harbor version: v2.9.0

@viceice
Copy link

viceice commented Sep 19, 2023

image
It seems the redis url is not passed to the job service via env.

I only see this inside config.yml if job service:

worker_pool:
  workers: 10
  backend: "redis"
  redis_pool:
    redis_url: "redis://keydb:6379/1"
    namespace: "harbor_job_service_namespace"
    idle_timeout_second: 3600

@viceice
Copy link

viceice commented Sep 19, 2023

We probably need to set ?

_REDIS_URL_REG: "{{ template "harbor.redis.urlForRegistry" . }}"

JOBSERVICE_WEBHOOK_JOB_HTTP_CLIENT_TIMEOUT: "{{ .Values.jobservice.notification.webhook_job_http_client_timeout }}"

@nevlkv
Copy link
Author

nevlkv commented Sep 21, 2023

error stable

2023-09-21T07:55:57Z [ERROR] [/jobservice/runner/redis.go:123]: Job 'GARBAGE_COLLECTION:c09d285cf0794f361744acab' exit with error: run error: error retrieving 'blobs::*' keys: WRONGPASS invalid username-password pair or user is disabled.

@MinerYang MinerYang transferred this issue from goharbor/harbor Sep 21, 2023
@nevlkv
Copy link
Author

nevlkv commented Sep 25, 2023

Current state

2023-09-24T02:31:20Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:414]: 3177 blobs and 562 manifests are actually deleted
2023-09-24T02:31:20Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:415]: The GC job actual frees up 37506 MB space.
2023-09-24T02:31:20Z [ERROR] [/jobservice/job/impl/gc/garbage_collection.go:445]: failed to clean registry cache error retrieving 'blobs::*' keys: WRONGPASS invalid username-password pair or user is disabled., pattern blobs::*

@MinerYang
Copy link
Collaborator

Hi @nevlkv ,
Could you try to monitor the registry log when pushing images, if there's any error msg?

@nevlkv
Copy link
Author

nevlkv commented Sep 26, 2023

Sometimes on pull 404
with "blob unknown"

time="2023-09-26T11:19:16.21575386Z" level=error msg="response completed with error" auth.user.name=harbor err.code="blob unknown" err.detail=sha256:7dbc1adf280e1aa588c033eaa746aa6db327ee16be705740f81741f5e6945c86 err.message="blob unknown to registry" go.version=go1.20.7 http.request.host=REGISTRY http.request.id=cdc0816e-2548-40d7-b33c-3f2f9a26837b http.request.method=HEAD http.request.remoteaddr=*.*.*.* http.request.uri="/v2/REPO/blobs/sha256:7dbc1adf280e1aa588c033eaa746aa6db327ee16be705740f81741f5e6945c86" http.request.useragent="docker/20.10.22 go/go1.18.9 git-commit/42c8b31 kernel/5.15.0-83-generic os/linux arch/amd64 UpstreamClient(docker-compose/1.29.2 docker-py/6.1.3 Linux/5.15.0-83-generic)" http.response.contenttype="application/json; charset=utf-8" http.response.duration=104.591881ms http.response.status=404 http.response.written=157 vars.digest="sha256:7dbc1adf280e1aa588c033eaa746aa6db327ee16be705740f81741f5e6945c86" vars.name="REPO"

@nevlkv
Copy link
Author

nevlkv commented Dec 12, 2023

If default user in URI undefined rise error

$ redis-cli -u redis://:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30 info keyspace
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
AUTH failed: WRONGPASS invalid username-password pair or user is disabled.
$ redis-cli -u redis://default:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30 info keyspace 
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Keyspace
db0:keys=1732,expires=1732,avg_ttl=1826800
db1:keys=116,expires=65,avg_ttl=44559991
db2:keys=1,expires=0,avg_ttl=0
db5:keys=4,expires=0,avg_ttl=0

define user in values

   type: external
   external:
     addr: "harbor-redis-master:6379"
+    username: "default"

and

kubectl get cm harbor-core -n=io  -o yaml
apiVersion: v1
data:
  _REDIS_URL_CORE: redis://:<password>@harbor-redis-master:6379/0?idle_timeout_seconds=30
  _REDIS_URL_REG: redis://:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30

became

kubectl get cm harbor-core -n=io  -o yaml
apiVersion: v1
data:
  _REDIS_URL_CORE: redis://default:<password>@harbor-redis-master:6379/0?idle_timeout_seconds=30
  _REDIS_URL_REG: redis://default:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30

and GC success !!!

изображение

@viceice
Copy link

viceice commented Jan 15, 2024

Interestingly a manual run works, but the sheduled runs are always failing. 😕

@nevlkv
Copy link
Author

nevlkv commented Feb 13, 2024

same error after upgrade to 2.10

scheduled fail, manual success

upgraded 08/02/2024

изображение

Copy link

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

@github-actions github-actions bot added the Stale label Apr 13, 2024
@viceice
Copy link

viceice commented Apr 15, 2024

not stale 😞

@nevlkv
Copy link
Author

nevlkv commented Apr 16, 2024

successful only manual job
image

@github-actions github-actions bot removed the Stale label Apr 16, 2024
@nevlkv
Copy link
Author

nevlkv commented Apr 22, 2024

execute success only manual

изображение

@nevlkv
Copy link
Author

nevlkv commented Jun 3, 2024

If GC scheduled and settings of harbor changed after, job became failed:

registry=# select * from schedule where callback_func_name='GARBAGE_COLLECTION'
registry-# ;
  id  |       creation_time        |        update_time         |    vendor_type     | vendor_id |    cron     | callback_func_name |                                                                                                         callback_func_param                                                                                                          | cron_type |       extra_attrs        |  revision  
------+----------------------------+----------------------------+--------------------+-----------+-------------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+--------------------------+------------
 1009 | 2022-07-21 10:09:37.790778 | 2022-07-21 10:09:37.790778 | GARBAGE_COLLECTION |        -1 | 0 0 0 * * * | GARBAGE_COLLECTION | {"trigger":null,"deleteuntagged":true,"dryrun":false,"extra_attrs":{"delete_untagged":true,"dry_run":false,"redis_url_reg":"redis://redis:<any_password>@harbor-redis-master:6379/2?idle_timeout_seconds=30","time_window":2}} | Daily     | {"delete_untagged":true} | 1717372800
(1 row)

redis_url_reg static and not changes!

Copy link

github-actions bot commented Aug 2, 2024

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

@github-actions github-actions bot added the Stale label Aug 2, 2024
@wy65701436 wy65701436 removed the Stale label Aug 5, 2024
@wy65701436
Copy link
Contributor

@nevlkv can you share your settings when installing your harbor? You are seeing the error only happen at the scheduled GC, right?

@wy65701436 wy65701436 self-assigned this Aug 5, 2024
@twhiteman
Copy link

twhiteman commented Sep 24, 2024

I've run into this issue as well (upgraded Harbor from 2.6.3 to 2.10.1) where scheduled GC runs fail to clear the redis cache due to an auth issue.
image

I noticed in the harbor execution table, that the redis_url_reg field is different between a manual and scheduled run:

> select * FROM execution WHERE vendor_type = 'GARBAGE_COLLECTION';
-[ RECORD 18 ]-+
id             | 8106990
vendor_type    | GARBAGE_COLLECTION
vendor_id      | -1
status         | Error
status_message |
trigger        | SCHEDULE
extra_attrs    | {"delete_untagged":false,"dry_run":false,"freed_space":412868683,"purged_blobs":1352,"purged_manifests":668,"redis_url_reg":"redis+sentinel://172.16.1.1:26379,172.16.1.2:26379,172.16.1.3:26379/redismaster/2?idle_timeout_seconds=30","time_window":2,"workers":0}
start_time     | 2024-08-27 00:02:49.632401
end_time       | 2024-08-27 00:05:30
revision       | 4
update_time    | 2024-08-27 00:06:05

-[ RECORD 19 ]-+
id             | 8113128
vendor_type    | GARBAGE_COLLECTION
vendor_id      | -1
status         | Success
status_message |
trigger        | MANUAL
extra_attrs    | {"delete_untagged":false,"dry_run":false,"freed_space":29574525,"purged_blobs":1021,"purged_manifests":507,"redis_url_reg":"redis+sentinel://USERNAME:PASSWORD@172.16.1.1:26379,172.16.1.2:26379,172.16.1.3:26379/redismaster/1?idle_timeout_seconds=30","time_window":2,"workers":0}
start_time     | 2024-08-27 17:06:27.378128
end_time       | 2024-08-27 17:09:10
revision       | 4
update_time    | 2024-08-27 17:09:22

Failed: "redis+sentinel://172.19.6.10:26379,172.19.6.30:26379,172.19.6.50:26379/redismaster/2?idle_timeout_seconds=30"

Success: "redis+sentinel://USERNAME:PASSWORD@172.19.6.10:26379,172.19.6.30:26379,172.19.6.50:26379/redismaster/1?idle_timeout_seconds=30"

Note the successful one has a username|password AND there is different database reference (redismaster/1|2).

Also note that I have not changed the Harbor redis configuration since the initial installation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants