Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync upstream configuration #2570

Draft
wants to merge 47 commits into
base: master
Choose a base branch
from
Draft

Conversation

kkimurak
Copy link
Contributor

@kkimurak kkimurak commented May 25, 2022

Draft:

This PR syncs configuration files.
Current target versions are based on gitlab-foss v15.0.0:

  • gitlab-foss: v15.0.0
  • gitaly: v15.0.0
  • gitlab-shell: v14.3.0
  • gitlab-pages: v1.58.0

This PR is based on #2569 .

TODO:

  • add update process to assets/runtime/function
  • parameterize
  • re-check
  • check compatibility for renamed parameter

@sachilles
Copy link
Collaborator

@kkimurak The ci pipeline failed at stage test by testing the image bootup. It failes due to connection issues:

  • Hostname in DNS cache was stale, zapped
  • Trying 127.0.0.1:80...
  • connect to 127.0.0.1 port 80 failed: Connection refused
  • Failed to connect to localhost port 80 after 0 ms: Connection refused

Is this related to numerous changes in the config.toml?

@sachilles
Copy link
Collaborator

Do you have an idea how to proceed or make progress? How can we work together efficiently?

When I see the amount of changes, a four-eyes review would be the best. Do you have any objections and/or suggestions for a platform for real-time exchange?

@sachilles
Copy link
Collaborator

sachilles commented Jun 2, 2022

Unfortunately, several GitLab critical security releases (15.0.1, 14.10.4, and 14.9.5) have been published yesterday (see https://about.gitlab.com/releases/2022/06/01/critical-security-release-gitlab-15-0-1-released/).

I guess we should maintain the official releases in parallel with this WIP PR. From your point of view, is there any reason not to at least merge the PR #2569 so that the security release can be prepared?

@kkimurak
Copy link
Contributor Author

kkimurak commented Jun 3, 2022

Sorry for late, I was bit busy.

@kkimurak The ci pipeline failed at stage test by testing the image bootup. It failes due to connection issues:

  • Hostname in DNS cache was stale, zapped
  • Trying 127.0.0.1:80...
  • connect to 127.0.0.1 port 80 failed: Connection refused
  • Failed to connect to localhost port 80 after 0 ms: Connection refused

Is this related to numerous changes in the config.toml?

I think so. Maybe upstream config for [gitlab]::url in assets/runtime/config/gitaly/config.toml is not able to be applied directly. /home/git/gitlab/tmp/sockets/gitlab-workhorse.socket is used in upstream config but we use /home/git/gitlab/tmp/sockets/gitlab.socket when we start gitlab workhorse.

note: I just copied and pasted upstream config, then started checking git diff.

Unfortunately, several GitLab critical security releases (15.0.1, 14.10.4, and 14.9.5) have been published yesterday (see https://about.gitlab.com/releases/2022/06/01/critical-security-release-gitlab-15-0-1-released/).

I guess we should maintain the official releases in parallel with this WIP PR. From your point of view, is there any reason not to at least merge the PR #2569 so that the security release can be prepared?

Just please merge #2569 and prepare for security release. I'm not sure v15.0.0 works but at least test is passing. And I have learned from history that the releasing is much important than missing (or stale) parameters for this project. Thank you for your work.

BTW, I'm going to work on this PR this weekend (since 2022-06-03T20:00:00+0900)

@sachilles
Copy link
Collaborator

This sounds good. I can confirm that GitLab CE works at least for me in version 15.0.0 and 15.0.1. The only downside is that I do not use all features.

@sachilles
Copy link
Collaborator

I just added an additional section about the state (important notes) of the release in the release notes.

@kkimurak
Copy link
Contributor Author

kkimurak commented Jun 4, 2022

I'm back, confirmed that gitlab works on relative URL enabled/disabled. Also simple git push/pull works.

Here're list of changes for configurations.

  • gitaly
    • config.toml
      • add commented-out config runtime_dir
      • fix value for commented-out config tls_listen_addr (lacking trailing " )
      • add commented-out config for git (fetch.fsckObjects=true)
      • add custom_hooks_dir : moved from gitlab-shell configuration. see deprecations page. TODO: parameterize?
      • move [gitlab] section before [[concurrency]] section
      • add section [gitlab.http-settings]
        • TODO: apply parameters
      • add many more commented-out configurations for resource limitation
  • gitlab-pages
    • config
      • add listen-http, pages-root, pages-domain
      • sort based on upstream config
  • gitlab-shell
    • config.yml
      • add comment for gitlab_relative_url_root (we don't use http+unix:// protocol so we have nothing to do)
      • remove deprecated http_settings::self_signed_cert : see upstream mr 602 : since v13.26.0, v14.0.0
      • add ssl_cert_dir: TODO maybe we need to set it using SSL_CERTIFICATE_PATH, but this is the path for file itself. We need directory path.
      • default log format is changed to json : see upstream MR 476 : since v13.21.0
  • gitlabhq
    • database.yml: No notable update. just added commented-out load-balancing configurations
      • stale: official documentation says that DB_POOL (pool: in database.yml) is not very practical and gitlab override the number of allowed connections in the database connection-pool based on the configured number of application threads. See docs about client-side connection pool
    • gitlab.yml: too many changes
      • Merge Request -> merge request
      • add header note for those who submitting MR
      • some URLs for docs are updated. Eg. http://doc.gitlab.com/ce is changed to https://docs.gitlab.com/ee
      • some empty line are removed
      • add commented-out config gitlab:cdn_host to support Rails asset host: see upstream MR 67710 : since v14.2.0 : TODO: parameterize?
      • remove unicorn from comment, now only Puma configurations are written.
      • add gitlab:allowed_hosts to control Rails.application.config.hosts : see upstream MR 55491 : since v13.10.0 : TODO: parameterize?
      • add commented-out config gitlab:email_smtp_secret_file: see upstream MR 67802 : since v14.2.0 : TODO: parameterize?
      • add commented-out config gitlab:application_settings_cache_seconds : see upstream MR 51889 : since v13.9.0 : TODO: parameterize?
      • add commented-out config display_initial_root_password (default to false): Security commit, no MR. see upstream commit b48d80d : since v14.5.0
      • many update on incoming_email section. They are all commented out.q
      • add object_store section on top-level as default value : TODO: parameterize? (need to check compatibility with GITLAB_OBJECT_STORE_CONNECTION_PROVIDER and so on)
      • add local_store section on top-level : see upstream MR 55470 : since v13.11.0
      • add packages:dpkg_deb_path ; see upstream MR 44029 : since v13.6.0
      • rename packages:path to packages:storage_path
      • add ci_secure_files section for secure files API: see upstream MR 78227 : since v14.8.0 : TODO: parameterize?
      • add commented-out config ldap:secret_file
      • add commented-out config ldap:servers:main:retry_empty_result_with_codes
      • add commented-out config kerberos:simple_ldap_linking_allowed_realms
      • add commented-out config omniauth:saml_message_max_byte_size
      • add forti_authenticator section for one time password method : see upstream MR 45055 : since v13.5.0
      • add forti_token_cloud section
      • add encrypted_settings section and commented-out config entrypted_settings:path
      • remove gitaly:client_path : see upstream MR 61987 : since 14.0.0
      • some additional configs for backup:upload
      • remove pseudonymizer configurations : see upstream MR 86087 : since v15.0.0
      • many update on cron_jobs (also on ee_cron_jobs that we don't use) : deleted or modified schedules ware not parameterized.
      • many more updates on non-production (develop/test) settings
    • puma.rb: No notable changes for us.
      • shorter call for Gitlab::Cluster::PumaWorkerKillerInitializer.start
      • additional behavior via environment variable PUMA_WAIT_FOR_LESS_BUSY_WORKER (default to 0.001) and DISABLE_PUMA_NAKAYOSHI_FORK
      • add low-level error handling to Sentry. See upstream MR 66712
    • resque.yml: No notable changes.
      • Merge Request -> merge request
      • DB wording: slave -> replica
      • update URL for docs
    • secrets.yml: No notable changes for configuration.
      • note: minimum secret key length is increased from 30 chars to 32 chars. first introduced: v13.12.0 : see upstream MR 53602
      • note: Do we need to parameterize openid_connect_signing_key and encrypted_settings_key_base?
    • smtp_settings.rb: NO notable changes for configuration.
      • Merge Request -> merge request
      • add commented-out configurations example for those who uses encrypted smtp credentials.
        to achieve this, add local variable secrets to refer Gitlab::Email::SmtpConfig.secrets.{user_name,password}

note: http+unix protocol is used in some configuration but I kept traditional value http://localhost:8181{GITLAB_RELATIVE_URL} for simplicity.

@kkimurak
Copy link
Contributor Author

kkimurak commented Jun 4, 2022

Additional note: I have noticed that puma dies on startup so that gitlab did not come up. I have submit a separate PR #2579

@kkimurak
Copy link
Contributor Author

kkimurak commented Jul 2, 2022

Now I can continue developing thanks to #2579 is merged

@kkimurak
Copy link
Contributor Author

kkimurak commented Jul 8, 2022

@sachilles Is there a backporting request? If so, I'll split the commit to make it easier (although it will take some time, of course).

@sachilles
Copy link
Collaborator

@kkimurak If we will be able to merge in the near future, I'd like to avoid backporting.

@sachilles
Copy link
Collaborator

@kkimurak Sorry for the extremely long delay in processing this PR. Can we try to incorporate this PR next (and before the next release of GitLab)?

@kkimurak
Copy link
Contributor Author

Hi @sachilles , I apologize for leaving the pull request for so long.
I thought a lot, but I want to split this PR into some feature-based individual PRs (and minor typo PRs). This PR contains too many changes so that make diff larger, it makes harder to review the functionality.

As usual, the new version of GitLab will be released on the 22nd. I'm sorry, but I've been busy with work this month and it may be difficult to get things done (including checking against the new release, rebase on some concurrent works e.g. #2598 #2610 ) by then. I think I will be able to work on it in early October.

…e.yml

This commit is imported from upstream.
For moredetail, see corresponding merge request.
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/32275
Just fixing commented-out config, no effect
See upstream commit:
https://gitlab.com/gitlab-org/gitaly/-/commit/4290807efcf2de64d5c1e8abce15399361da42f9

Note : I have checked which version contains the commit
using `git tag --contains <commit ID> | sort --version-sort`
v13.1.0-rc4 and v13.2.0 or above contains the commit,
but v13.1.0 does not. I have no idea why

First contains (stable) gitlay v13.2.0
Corresponding gitlab v13.2.0
This is imported from upstream.
See corresponding merge request:
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38049

First tag contains this change: v13.3.0-ee
$ git tag --contain ce37ceb7 | sort --version-sort | head -n 1
…elative_url_root

Only affected if UNIX sockets are used for gitlab_url.
sameersbn/gitlab uses http:// and is not affected.
See upstream merge request:
https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/406

First contain gitlab-shell v13.7.0
Corresponding gitlab v13.4.2
This commit is imported from upstream.
For more detail, see corresponding merge request:
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44533
…_key_base

This commit is imported from upstream.
For more detail, see corresponding merge request:
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53602
See corresponding upstream merge request:
Cleanup Puma 5 upgrade transition code
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61572
Also removes GITLAB_MONITORING_UNICORN_SAMPLER_INTERVAL

unicorn have been removed in v14.0
See corresponding merge request:
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62090
…password`

and set to false.
Squash of following MR and commits:
- Add option to disable printing of root password during DB seeding
  https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63271
- Do not display the root password by default
  https://gitlab.com/gitlab-org/gitlab/-/commit/b4b8d80d00780fbb80ecac7506dcdcfb328d4f03
  Note that the first release contains this commit is v14.5.0
… is "json"

This commit is imported from upstream.
For more detail, check corresponding merge request:
https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/476

First contain : gitlab-shell v13.21.0
Corresponding gitlab v14.3.0
This is done in gitlab 14.9 release.
See corresponding merge requests:
gitlab-shell side
https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/378

gitaly side: First introduced in MR 2066 for [gitlab-shell] scope
https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2066
Then moved to [hook] scope in MR 2187,
https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2187
but reverted and moved back to [gitlab-shell] scope.
https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2196

If the value is empty or not set, it fallback to
{gitlab-shell install directory}
See https://gitlab.com/gitlab-org/gitaly/-/blob/16b38f034eb38253006a2e69a4b4220717b45a99/internal/gitaly/config/config.go#L231-233

Also note that there was a issue for 13-0 stable about default value
(already fixed in later release)
https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2275

First contained tag: v13.0.0 (gitlab-shell, gitaly, gitlab)
See upstream merge request:
https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4415

First contain gitaly v14.10.0
Corresponding gitlab v14.10.0
Also remove corresponding configuration parameters:
- GITLAB_ARTIFACTS_OBJECT_STORE_DIRECT_UPLOAD
- GITLAB_ARTIFACTS_OBJECT_STORE_BACKGROUND_UPLOAD
- GITLAB_PACKAGES_OBJECT_STORE_DIRECT_UPLOAD
- GITLAB_PACKAGES_OBJECT_STORE_BACKGROUND_UPLOAD
- GITLAB_LFS_OBJECT_STORE_DIRECT_UPLOAD
- GITLAB_LFS_OBJECT_STORE_BACKGROUND_UPLOAD
- GITLAB_UPLOADS_OBJECT_STORE_DIRECT_UPLOAD
- GITLAB_UPLOADS_OBJECT_STORE_BACKGROUND_UPLOAD

This is introduced with v15.0 release
See corresponding merge request:
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86905
Adding pages:object_store related configurations
Still not parameterized yet
See corresponding merge request:
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/42461
@kkimurak
Copy link
Contributor Author

I'm still busy and progress is slow, but the work is definitely nearing completion. Most of the changes were cleaned up and sorted by released tag. A little work left over from v15.4 and confirmation of new additions in v15.5 is required.
Anyway, I'm really sorry I'm a month late to do just this. I hope I can work on in a week...

@kkimurak
Copy link
Contributor Author

I also want to split them into separate pull requests for easier review.
I haven't decided where to split it, but if you have any suggestions (and requests for access via configuration parameters that are currently largely untouched) I'd be happy to hear them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants