AWS S3 Backup fails with SSL_CTX_load_verify_locations

[fhaefemeier]

I configured automatic backups with storing the backup archive in AWS S3 (self hosted minio cluster). After upgrade the docker image from 17.1.1 to 17.3.2 the backup configuration fails with a SSL error SSL_CTX_load_verify_locations. The configuration is not changed. Server certificate of minio cluster is a Let's Encrypt certificate.

Configuration:

GITLAB_BACKUP_SCHEDULE: daily
GITLAB_BACKUP_TIME: "23:00"
GITLAB_BACKUP_SKIP: "uploads,artifacts,packages,builds,lfs,registry,ci_secure_files,terraform_state"
GITLAB_BACKUP_EXPIRY: 302400
AWS_BACKUPS: "true"
AWS_BACKUP_REGION: "de-local-1"
AWS_BACKUP_BUCKET: "gitlab-backup"
AWS_BACKUP_ACCESS_KEY_ID: <masked>
AWS_BACKUP_SECRET_ACCESS_KEY: <masked>
AWS_BACKUP_ENDPOINT: https://main.storage.<domain>

Log extract:

2024-09-23 23:00:33 +0200 -- Creating backup archive: 1727125223_2024_09_23_17.3.2_gitlab_backup.tar ... 
rake aborted!
Excon::Error::Socket: SSL_CTX_load_verify_locations: system lib (OpenSSL::SSL::SSLError)
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/ssl_socket.rb:139:in `initialize'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/ssl_socket.rb:139:in `new'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/ssl_socket.rb:139:in `initialize'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/connection.rb:474:in `new'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/connection.rb:474:in `socket'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/connection.rb:121:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/middlewares/mock.rb:57:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/middlewares/instrumentor.rb:34:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/middlewares/idempotent.rb:19:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/middlewares/base.rb:22:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/middlewares/base.rb:22:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/connection.rb:286:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-xml-0.1.3/lib/fog/xml/sax_parser_connection.rb:35:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-xml-0.1.3/lib/fog/xml/connection.rb:7:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-aws-3.18.0/lib/fog/aws/storage.rb:677:in `_request'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-aws-3.18.0/lib/fog/aws/storage.rb:672:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-aws-3.18.0/lib/fog/aws/requests/storage/initiate_multipart_upload.rb:29:in `initiate_multipart_upload'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-aws-3.18.0/lib/fog/aws/models/storage/file.rb:324:in `multipart_save'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-aws-3.18.0/lib/fog/aws/models/storage/file.rb:279:in `save'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-core-2.1.0/lib/fog/core/collection.rb:50:in `create'
/home/git/gitlab/lib/backup/remote_storage.rb:26:in `upload'
/home/git/gitlab/lib/backup/manager.rb:279:in `upload'
/home/git/gitlab/lib/backup/manager.rb:151:in `run_all_create_tasks'
/home/git/gitlab/lib/backup/manager.rb:32:in `create'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:12:in `block in create_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:75:in `lock_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:10:in `create_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:117:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'

[ymazzer]

Same here, thought it was linked to openssl3 upcoming requirements.

Any idea

[dimm0]

Same here with google s3

[ntnamazu]

I have seen the same error message and a workaround like below to add symbolic link was posted at GitLab Forum. The post said that "Ruby is looking for /usr/lib/ssl/cert.pem and it wasn’t there". This method quite worked for me.

ln -s /etc/ssl/certs/ca-certificates.crt /usr/lib/ssl/cert.pem

Also, I implemented this workaround in PR #3016 as commit d8d7804, so if the PR is approved and merged, I think this issue would be resolved.

[ymazzer]

I have seen the same error message and a workaround like below to add symbolic link was posted at GitLab Forum. The post said that "Ruby is looking for /usr/lib/ssl/cert.pem and it wasn’t there". This method quite worked for me.

ln -s /etc/ssl/certs/ca-certificates.crt /usr/lib/ssl/cert.pem

Also, I implemented this workaround in PR #3016 as commit d8d7804, so if the PR is approved and merged, I think this issue would be resolved.

Just tested, works well. Thanks for the tip!

Btw, would have been better to keep the fix in another PR.