Skip to content

Commit

Permalink
T365569: release 3.0.0 (WBS Deploy 3.0.0) (#708)
Browse files Browse the repository at this point in the history 
* useless change to trigger the pipeline

* Up to the minute hash updates

* chore: job runner restart rule (#711)

* chore: job runner restart rule (#711)

* Make tests work across all releases (#712)

* T367024: release checklist (#714)

* docs: release checklist

* docs: typo

* fix: bold

* Dockerhub docs (#705)

* docs: add dockerhub readme file for wikibas

* docs: add dockerhub readmes to all images

* style: lint

* fix: link

* feat: tag quickstatements version (#719)

* Proofreading of READMEs (#720)

* T369358: product name in template.env (#723)

* fix: product name in template.env

* docs: fix remaining occurrences of "deployment kit"

* docs(wdqs): fix updater reinit help (#725)

---------

Co-authored-by: Loren Johnson <lorenjohnson@gmail.com>
Co-authored-by: Dan Shick <datn@users.noreply.github.com>
  • Loading branch information
@rti @lorenjohnson @datn
3 people authored Jul 15, 2024
1 parent d969b2e commit 66d59fa
Show file tree
Hide file tree
Showing 26 changed files with 321 additions and 226 deletions.
66 changes: 21 additions & 45 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Wikibase Suite (WBS) eases self-hosting [Wikibase](https://wikiba.se) in product

If you want to host your own WBS instance, head over to the [WBS Deploy documentation](./deploy/README.md).

If you're looking for individual WBS Images, head over to [hub.docker.com/u/wikibase](https://hub.docker.com/u/wikibase).
If you're looking for individual WBS images, head over to [hub.docker.com/u/wikibase](https://hub.docker.com/u/wikibase).

> 🔧 This document is intended for people developing WBS.
Expand All @@ -22,13 +22,13 @@ This repository contains the Wikibase Suite toolset used for:
### Build

```
# Build all Wikibase Suite Images
# Build all Wikibase Suite images
$ ./build.sh
# Build only the MediaWiki/Wikibase containers
$ ./build.sh wikibase
# Build the wdqs container without using Docker's cache
# Build the WDQS container without using Docker's cache
$ ./build.sh --no-cache wdqs
```

Expand Down Expand Up @@ -58,7 +58,7 @@ $ cd deploy
$ docker compose up --wait
```

Find more details in the [WBS Deploy documentations](./deploy/README.md).
Find more details in the [WBS Deploy documentation](./deploy/README.md).

## Development setup

Expand All @@ -70,11 +70,11 @@ $ git config core.hooksPath .githooks

## Testing

Tests are organized in suites, which can be found in `test/suites`. Each suite runs a series of specs (tests) found in the `test/specs` directory. Which specs run in each suite by default are specified in the `.config.ts` file in each suite directory under the `specs` key.
Tests are organized in suites, which can be found in `test/suites`. Each suite runs a series of specs (tests) found in the `test/specs` directory. Which specs run by default in each suite are specified in the `.config.ts` file in each suite directory under the `specs` key.

All test suites are run against the most recently built local Docker images, those with the `:latest` tag, which are also selected when no tag is specified. The `deploy` test suite runs against the remote Docker images specified in the configuration in the `./deploy` directory.

You can run the tests in the docker container locally exactly as they are run in CI by using `test.sh`.
You can run the tests in the Docker container locally exactly as they are run in CI by using `test.sh`.

## Examples usage of `./test.sh`:

Expand All @@ -85,10 +85,10 @@ You can run the tests in the docker container locally exactly as they are run in
# Run all test suites
./test.sh all

# Only run a single suite (e.g. repo)
# Only run a single suite (e.g., repo)
./test.sh repo

# Only run a specific file within the setup for any test suite (e.g., repo and the babel extension)
# Only run a specific file within the setup for any test suite (e.g., repo and the Babel extension)
./test.sh repo --spec specs/repo/extensions/babel.ts
```

Expand Down Expand Up @@ -134,47 +134,23 @@ For more information on testing, see the [README](./test/README.md).

## Release process

WBS Deployment Kit and WBS Service Images are released using this repository. The process involves defining all upstream component versions to be used, building service images, testing all the images together and finally publishing them.
WBS Deploy and WBS images are released using this repository. The process involves updating all upstream component versions to be used, building images, testing all the images together and finally publishing them.

### Release checklist Phabricator template

```
[ ] **Pending issues as subtasks**. If any open tickets need to be resolved and/or related changes need to be included in the release, add them as subtasks of this release ticket. (If this release was triggered by a MediaWiki bugfix release, consider only including bug/security issue fixes, and avoid breaking changes.)
[ ] **Create branches**
- **For bugfix releases**
- **Create a release preparation branch.** The release branch itself should already exist, e.g., `mw-1.40`. Create a release preparation branch off the existing release branch. Choose a name for the release prep branch like `releaseprep-1.40.2`. (Don't name it `mw-*`! That's the naming convention for release branches.)
- **Backport changes from main only if absolutely necessary.** Bugfix releases should contain as few changes as possible. Make sure that the `variables.env` file is not changed by incoming changes from main.
- **For major releases**
- **Create a new release branch off main,** e.g., `mw-1.44`. This branch is now equivalent to main. It will hold all bugfix releases for that major version in the future.
- **Create a release preparation branch off that release branch.** e.g., `releaseprep-1.44.0`. This second branch is meant for preparing the first release on that release branch.
[ ] **Update `variables.env`** on the release preparation branch. You can find further instructions in the [variables.env](https://github.com/wmde/wikibase-release-pipeline/blob/main/variables.env) file itself.
[ ] **Update `CHANGES.md`** on the release preparation branch. Add a section following the example of previous releases: update the different values. The spec will eventually point to the release tag's `variables.env` once the release is published.
[ ] **Create a release PR** that merges the release preparation branch, e.g., `releaseprep-1.44.0`, into the corresponding release branch (`mw-1.44` or equivalent).
[ ] **CI should be green**. Tests may need adjustments in order to pass for the new version. Bugfix releases are likely to pass without any adjustments.
[ ] **Do a sanity check by manually reviewing a running instance**. This can be done locally on your machine or by [deploying](https://docs.google.com/document/d/1BGxcqt9CHbb-8dfWjK-lZmNoNcD08urb23JqtgoVTeg/edit#heading=h.6a8ctlepqn5d) to the [test system](https://wikibase-product-testing.wmcloud.org). You can find prebuilt images from your PR on the [GitHub Container Registry](https://github.com/wmde/wikibase-release-pipeline/pkgs/container/wikibase%2Fwikibase) tagged with `dev-BRANCHNAME`, e.g., `dev-releaseprep-wmde.17`. This tag can be referenced to set up an instance running your PR version.
[ ] **Get two reviews on the release preparation PR** so that it is technically ready to be merged. Merging will eventually trigger the release to Docker Hub.
[ ] **Prepare communication** by creating a [release announcement](https://drive.google.com/drive/folders/1kHhKKwHlwq_P9x4j8-UnzV72yq0AYpsZ) using a template.
[ ] **Coordinate with ComCom on timing the publication of the release**. Talk to SCoT (ComCom, technical writer) about this.
[ ] **Publish the release** by merging the release preparation branch into the release branch. **ATTENTION: This will automatically push images to Docker Hub, thereby releasng the new version!**
[ ] **Update the [Docker install instructions](https://github.com/wmde/wikibase-release-pipeline/blob/main/deploy/README.md)** to reflect changes in the latest version. If you were making a bugfix release for an older release, this can be skipped.
[ ] **Make sure the communication is sent.**
[ ] **Update the deploy directory.** ☠️☠️☠️ Currently, for this to happen, we need to merge another PR to the release branch. This PR's only change should be making the `deploy` directory reference the new version. For this PR's pipeline to properly run, before merging, you need to manually delete from the git repo the tag of the version you just released. Merging this deploy directory update PR is technically a re-release. Hopefully this weirdness will be fixed soon.
[ ] **Merge back to main**. Now is the time to merge anything you want back to main on the release branch.
- [ ] **Pending issues as subtasks**. If any open tickets need to be resolved and/or related changes need to be included in the release, add them as subtasks of this release ticket.
- [ ] **To release breaking changes** as a new major version of WBS Deploy, create a new branch called `deploy-X`, where `X` is the new major version.
- [ ] **Create a release PR** with the following changes targeting the appropriate `deploy-X` release branch.
- [ ] **Update `variables.env`** by adjusting WBS versions and upstream versions. You can find further instructions in the [variables.env](https://github.com/wmde/wikibase-release-pipeline/blob/main/variables.env) file itself.
- [ ] **Update `CHANGES.md`** by adding a section following the example of previous releases.
- [ ] **CI should be green**. Tests may need adjustments in order to pass for the new version. Minor releases are likely to pass without any adjustments. Try re-running tests on failure -- some specs may be flaky.
- [ ] **Do a sanity check by manually reviewing a running instance using your build**. This can be done locally on your machine or on a public server. You can find built images from your release PR on the [GitHub Container Registry](https://github.com/wmde/wikibase-release-pipeline/pkgs/container/wikibase%2Fwikibase) tagged with `dev-BRANCHNAME`, e.g., `dev-releaseprep`. This tag can be used to set up an instance running your release PR version.
- [ ] **Get two reviews on the release PR** so that it is ready to be merged. **Merging to `deploy-X` later will trigger the release to Docker Hub.** Do not merge yet!
- [ ] **Prepare communication** by creating a [release announcement](https://drive.google.com/drive/folders/1kHhKKwHlwq_P9x4j8-UnzV72yq0AYpsZ) using a template.
- [ ] **Coordinate with ComCom on timing the publication of the release**. Talk to SCoT (ComCom, technical writer) about this.
- [ ] **Publish the release** by merging the release branch into the `deploy-X` branch. **ATTENTION: This will automatically push images to Docker Hub!**
- [ ] **Merge back to main in a separate PR** from `deploy-X` to have adjustments to `CHANGES.md` and the like available on `main` too. Changes from `variables.env` should only be taken from a release of the latest version so that `main` always references the build of the latest components.
You`re done. **Congratulations!**
```
1 change: 1 addition & 0 deletions build.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -245,3 +245,4 @@ done
if ! $build_target_set; then
build_all
fi

8 changes: 4 additions & 4 deletions build/Elasticsearch/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,14 @@ In order to run Wikibase Elasticsearch, you need:

### MediaWiki/Wikibase instance

We suggest to use the [WBS Wikibase Image](https://hub.docker.com/r/wikibase/wikibase) because this is the image we
We suggest using the [WBS Wikibase Image](https://hub.docker.com/r/wikibase/wikibase) because this is the image we
run all our tests against. Follow the setup instructions over there to get it up and running.

Be sure to add the `ELASTICSEARCH_HOST` environment variable to you Wikibase container.
Be sure to add the `ELASTICSEARCH_HOST` environment variable to your Wikibase container.

## Example

You can use the following example Docker Compose to setup and run the image. Your Wikibase will be available on [http://localhost](http://localhost).
You can use the following example Docker Compose configuration to setup and run the image. Your Wikibase will be available on [http://localhost](http://localhost).

```yml
services:
Expand Down Expand Up @@ -106,7 +106,7 @@ Official releases of this image can be found on [Docker Hub wikibase/elasticsear
## Tags and Versioning
This Elasticsearch Image is using [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
This Elasticsearch image is using [semantic versioning](https://semver.org/spec/v2.0.0.html).
We provide several tags that relate to the versioning semantics.
Expand Down
21 changes: 21 additions & 0 deletions build/Elasticsearch/dockerhub.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Wikibase Suite Elasticsearch Image

Wikibase Suite (WBS) eases self-hosting [Wikibase](https://wikiba.se) in production, allowing you to maintain a knowledge graph similar to [Wikidata](https://www.wikidata.org/wiki/Wikidata:Main_Page).

If you want to host your own WBS instance, head over to the [WBS Deploy documentation](https://github.com/wmde/wikibase-release-pipeline/blob/main/deploy/README.md).

# Documentation

Version specific documentation for this image is hosted in our [git repository](https://github.com/wmde/wikibase-release-pipeline/).

## Currently supported versions

- July 2024 [3.x.x](https://github.com/wmde/wikibase-release-pipeline/blob/deploy-3/build/Elasticsearch/README.md)
- July 2024 [2.x.x](https://github.com/wmde/wikibase-release-pipeline/blob/deploy-2/build/Elasticsearch/README.md)
- July 2024 [1.x.x](https://github.com/wmde/wikibase-release-pipeline/blob/deploy-1/build/Elasticsearch/README.md)

## Legacy versions

- 17 April 2024 [7.10.2-wmde.20](https://github.com/wmde/wikibase-release-pipeline/blob/wmde.20/build/Elasticsearch/README.md)
- 17 April 2024 [7.10.2-wmde.19](https://github.com/wmde/wikibase-release-pipeline/blob/wmde.19/build/Elasticsearch/README.md)
- 29 April 2024 [7.10.2-wmde.18](https://github.com/wmde/wikibase-release-pipeline/blob/wmde.18/build/Elasticsearch/README.md)
44 changes: 14 additions & 30 deletions build/QuickStatements/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
# Wikibase Suite QuickStatements Image

[QuickStatements](https://github.com/magnusmanske/quickstatements) is a tool to
batch-edit [Wikibase](https://www.mediawiki.org/wiki/Wikibase).
[QuickStatements](https://github.com/magnusmanske/quickstatements) is a tool to batch-edit [Wikibase](https://www.mediawiki.org/wiki/Wikibase).

This image contains Quickstatements and the required Magnustools library. It is
ready to be hooked up to MediaWiki OAuth on a WBS Wikibase Image.
This image contains Quickstatements and the required Magnustools library. It is ready to be hooked up to MediaWiki OAuth on a WBS Wikibase image.

> 💡 This image is part of Wikibase Suite (WBS). [WBS Deploy](https://github.com/wmde/wikibase-release-pipeline/deploy/README.md) provides everything you need to self-host a Wikibase instance out of the box.
Expand All @@ -19,10 +17,9 @@ ready to be hooked up to MediaWiki OAuth on a WBS Wikibase Image.

### MediaWiki/Wikibase instance with

#### Official WBS Wikibase Image
#### Official WBS Wikibase image

We suggest to use the [WBS Wikibase Image](https://hub.docker.com/r/wikibase/wikibase) because this is the image we
run all our tests against. Follow the setup instructions over there to get it up and running.
We suggest to use the [WBS Wikibase image](https://hub.docker.com/r/wikibase/wikibase) because this is the image we run all our tests against. Follow the setup instructions over there to get it up and running.

#### Other Wikibase instances

Expand All @@ -36,24 +33,15 @@ php /var/www/html/extensions/OAuth/maintenance/createOAuthConsumer.php \
--grants createeditmovepage --grants editpage --grants highvolume --jsonOnSuccess
```

You can pass the consumer and secret token you got from your Wikibase instance
to this container using the environment variables `OAUTH_CONSUMER_KEY` and
`OAUTH_CONSUMER_SECRET`.
You can pass the consumer and secret token you got from your Wikibase instance to this container using the environment variables `OAUTH_CONSUMER_KEY` and `OAUTH_CONSUMER_SECRET`.

### Reverse proxy

If QuickStatements and Wikibase are running on the same IP address, a reverse
proxy is required to route HTTP requests to Wikibase or QuickStatements
depending on the URL used to access them. See the [example](#Example) below for
a reverse proxy setup using [Traefik](https://doc.traefik.io/traefik/).
If QuickStatements and Wikibase are running on the same IP address, a reverse proxy is required to route HTTP requests to Wikibase or QuickStatements depending on the URL used to access them. See the [example](#Example) below for a reverse proxy setup using [Traefik](https://doc.traefik.io/traefik/).

### DNS resolution

In order to authorize QuickStatements against Wikibase via OAuth, both services
need to be accessible via DNS domain names, both, from within the docker
network as well as from the users browser. The easiest way to archive this is
by connecting both Wikibase and Quickstatements to the internet and let them
use public DNS domain names.
In order to authorize QuickStatements against Wikibase via OAuth, both services need to be accessible via DNS domain names, both from within the Docker network as well as from the user's browser. The easiest way to archive this is by connecting both Wikibase and Quickstatements to the internet and letting them use public DNS domain names.

### Environment variables

Expand All @@ -75,7 +63,7 @@ Variables in **bold** are required.

## Example

An example how to run this image together with the [WBS Wikibase Image](https://hub.docker.com/r/wikibase/wikibase) using Docker Compose.
Here's an example of how to run this image together with the [WBS Wikibase image](https://hub.docker.com/r/wikibase/wikibase) using Docker Compose.

```yml
services:
Expand Down Expand Up @@ -198,7 +186,7 @@ Official releases of this image can be found on [Docker Hub wikibase/quickstatem
## Tags and Versioning
This image uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
This image uses [semantic versioning](https://semver.org/spec/v2.0.0.html).
We provide several tags that relate to the versioning semantics.
Expand All @@ -211,23 +199,19 @@ We provide several tags that relate to the versioning semantics.
## Known Issues
Quickstatements' "Run in background" option is not supported by this image.
QuickStatements' "Run in background" option is not supported by this image.
Quickstatements' "Batches" require an database and are not supported by this image.
QuickStatements' "Batches" require an database and are not supported by this image.
## Troubleshooting
If you see an error such as `mw-oauth exception` when trying to log in, check
that you have passed the correct consumer token and secret token to
QuickStatements.
If you see an error such as `mw-oauth exception` when trying to log in, check that you have passed the correct consumer token and secret token to QuickStatements.

If you have changed the value of $wgSecretKey $wgOAuthSecretKey since you made
the consumer, you'll need to make another new consumer or reissue the secret
token for the old one.
If you have changed the value of `$wgSecretKey` or `$wgOAuthSecretKey` since you made the consumer, you'll need to make another new consumer or reissue the secret token for the old one.

## Internal filesystem layout

Hooking into the internal filesystem can be used to extend the functionality of this image.
Hooking into the internal filesystem can extend the functionality of this image.

| Directory | Description |
| ------------------------------------------- | ------------------------------ |
Expand Down
21 changes: 21 additions & 0 deletions build/QuickStatements/dockerhub.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Wikibase Suite QuickStatements Image

Wikibase Suite (WBS) eases self-hosting [Wikibase](https://wikiba.se) in production, allowing you to maintain a knowledge graph similar to [Wikidata](https://www.wikidata.org/wiki/Wikidata:Main_Page).

If you want to host your own WBS instance, head over to the [WBS Deploy documentation](https://github.com/wmde/wikibase-release-pipeline/blob/main/deploy/README.md).

# Documentation

Version specific documentation for this image is hosted in our [git repository](https://github.com/wmde/wikibase-release-pipeline/).

## Currently supported versions

- July 2024 [3.x.x](https://github.com/wmde/wikibase-release-pipeline/blob/deploy-3/build/QuickStatements/README.md)
- July 2024 [2.x.x](https://github.com/wmde/wikibase-release-pipeline/blob/deploy-2/build/QuickStatements/README.md)
- July 2024 [1.x.x](https://github.com/wmde/wikibase-release-pipeline/blob/deploy-1/build/QuickStatements/README.md)

## Legacy versions

- 17 April 2024 [wmde.20](https://github.com/wmde/wikibase-release-pipeline/blob/wmde.20/build/QuickStatements/README.md)
- 17 April 2024 [wmde.19](https://github.com/wmde/wikibase-release-pipeline/blob/wmde.19/build/QuickStatements/README.md)
- 29 April 2024 [wmde.18](https://github.com/wmde/wikibase-release-pipeline/blob/wmde.18/build/QuickStatements/README.md)
Loading

0 comments on commit 66d59fa

Please sign in to comment.