diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index 4f99e3614..4cf384ccd 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -93,7 +93,7 @@ Check that .repo files correctly points to ``@copr/copr``. And run on batcave01. .. note:: If there is a new version of copr-rpmbuild, follow the - :ref:`terminate_os_vms` and :ref:`terminate_resalloc_vms` instructions. + :ref:`terminate_resalloc_vms` instructions. Make sure expected versions of Copr packages are installed on the dev instances:: @@ -215,31 +215,8 @@ notes against Copr git repository. Schedule and announce the outage ................................ -.. warning:: - - Schedule outage even if it has to happen in the next 5 minutes! - -Get faimiliar with the `Fedora Outage SOP `_. -In general, please follow these steps: - -1. Prepare the infrastructure ticket similar to `this old one `_. - -2. Send email to `copr-devel`_ mailing list informing about an upcomming - release. We usually copy-paste text of the infrastructure ticket created in a - previous step. Don't forget to put a link to the ticket at the end of the - email. See the `example `_. - -3. Send ``op #fedora-buildsys MyIrcNick`` message to ``ChanServ`` on - libera.chat to get the OP rights, and then adjust the channel title so it - starts with message similar to:: - - Planned outage 2022-08-17 20:00 UTC - https://pagure.io/fedora-infrastructure/issue/10854 - -4. Create a new "planned" `Fedora Status SOP`_ entry. -5. Create warning banner on Copr homepage:: - - copr-frontend warning-banner --outage_time "2022-12-31 13:00-16:00 UTC" --ticket 1234 - +See a specific document :ref:`announcing_fedora_copr_outage`, namely the +"planned" outage state. Release window -------------- @@ -248,16 +225,10 @@ If all the pre-release preparations were done meticulously and everything was tested properly, the release window shouldn't take more than ten minutes. That is, if nothing goes terribly sideways... - Let users know -------------- -1. Change the "planned" `Fedora Status SOP`_ entry into an "ongoing" entry. - -2. Announce on ``#fedora-buildsys``, change title like - ``s/Planned outage ../OUTAGE NOW .../`` and send some message like - ``WARNING: The scheduled outage just begings!``. - +See :ref:`announcing_fedora_copr_outage` again, ad "ongoning" issue. Production infra tags --------------------- @@ -371,24 +342,8 @@ If schema was modified you should generate new Schema documentation. Announce the end of the release ............................... -1. Remove the "Outage" note from the ``#fedora-buildsys`` title. - -2. Send a message on ``fedora-buildsys`` that the outage is over! - -3. Send email to `copr-devel`_ mailing list. If there is some important change - you can send email to fedora devel mailing list too. Mention the link to the - "Highlights from XXXX-XX-XX release" documentation page. - -4. Propose a new "highlights" post for the `Fedora Copr Blog`_, - see `the example - `_. - -5. Close the Fedora Infra ticket. - -6. Change the "ongoing" `Fedora Status SOP`_ entry into a "resolved" one. - -7. Remove the warning banner from frontend page using - ``copr-frontend warning-banner --remove`` +See a specific document :ref:`announcing_fedora_copr_outage`, the "resolved" +section. Release packages to PyPI @@ -446,6 +401,4 @@ Fix this document to make it easy for the release nanny of the next release to u .. _`Copr release directory`: https://releases.pagure.org/copr/copr .. _`copr-devel`: https://lists.fedoraproject.org/archives/list/copr-devel@lists.fedorahosted.org/ -.. _`Fedora Status SOP`: https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/status-fedora/ .. _`example stg infra repo`: https://kojipkgs.fedoraproject.org/repos-dist/f36-infra-stg/ -.. _`Fedora Copr Blog`: https://fedora-copr.github.io/ diff --git a/doc/how_to_upgrade_persistent_instances.rst b/doc/how_to_upgrade_persistent_instances.rst index 35800d9c9..38ea3b024 100644 --- a/doc/how_to_upgrade_persistent_instances.rst +++ b/doc/how_to_upgrade_persistent_instances.rst @@ -32,6 +32,11 @@ be done post-upgrade. Avoid conducting the pre-upgrade too far in advance of the actual upgrade. Ideally, perform this phase a couple of hours or a day before. +Announce the outage +------------------- + +See a specific document :ref:`announcing_fedora_copr_outage`, namely the +"planned" outage state. Preparation ----------- @@ -197,6 +202,14 @@ Outage window When initiating this section, aim for time efficiency as the services will be down and inaccessible to users. +Let users know +-------------- + +See :ref:`announcing_fedora_copr_outage` again, ad "ongoning" issue. + +Move IPs and Volumes to the New Instances +----------------------------------------- + .. warning:: Prepare to follow the instructions provided during the playbook run. You'll need to perform manual steps such as DB backups, consistency checks, etc. @@ -285,10 +298,8 @@ option, click ``Actions``, navigate to ``Instance settings`` and then to Final steps ----------- -Remember to announce on `fedora devel`_ and `copr devel`_ mailing lists as well -as in the ``#fedora-buildsys`` channel that everything is functional again. - -Close the infrastructure ticket to complete the upgrade process. +See a specific document :ref:`announcing_fedora_copr_outage`, the "resolved" +section. .. _`Fedora Infra OpenStack`: https://fedorainfracloud.org .. _`OpenStack images dashboard`: https://fedorainfracloud.org/dashboard/project/images/ diff --git a/doc/maintenance/announce_outage.rst b/doc/maintenance/announce_outage.rst new file mode 100644 index 000000000..e65cdad30 --- /dev/null +++ b/doc/maintenance/announce_outage.rst @@ -0,0 +1,84 @@ +.. _announcing_fedora_copr_outage: + +Fedora Copr Outages +=================== + +This document is primarily intended for planning outages due to future +infrastructure updates. However, in the event of any incidents or accidents +such as networking issues, IBM Cloud problems, Fedora Rawhide repository issues, +or any other matters that affect users, it's advisable to refer to this document +(possibly jump directly to the "Ongoing State" section). + +.. warning:: + + Schedule an outage even if it needs to occur within the next 5 minutes! + +Please familiarize yourself with the `Fedora Outage SOP`_. But in general, +follow the steps outlined in this document. + +Planned outage +-------------- + +1. Prepare the infrastructure ticket similar to `this old one `_. + +2. Send email to `copr-devel`_ mailing list informing about an upcomming + release. We usually copy-paste text of the infrastructure ticket created in a + previous step. Don't forget to put a link to the ticket at the end of the + email. See the `example `_. + +3. Send ``op #fedora-buildsys MyIrcNick`` message to ``ChanServ`` on + libera.chat to get the OP rights, and then adjust the channel title so it + starts with message similar to:: + + Planned outage 2022-08-17 20:00 UTC - https://pagure.io/fedora-infrastructure/issue/10854 + +4. Ditto for the Matrix channel. TODO: we need to get admin access there. + +5. Create a new "planned" `Fedora Status SOP`_ entry. + +6. Create warning banner on Copr homepage:: + + copr-frontend warning-banner --outage_time "2022-12-31 13:00-16:00 UTC" --ticket 1234 + + +Ongoing outage +-------------- + +When the outage begins to cause real effects + +1. Change the "planned" `Fedora Status SOP`_ entry into an "ongoing" entry. + +2. Announce on ``#fedora-buildsys``, change title like + ``s/Planned outage ../OUTAGE NOW .../`` and send some message like + ``WARNING: The scheduled outage just begings!``. + +3. Announce on Matrix. + + +Resolved outage +--------------- + +1. Remove the "Outage" note from the ``#fedora-buildsys`` and Matrix title. + +2. Send a message on ``fedora-buildsys`` that the outage is over! + +3. Send email to `copr-devel`_ mailing list. If there is some important change + you can send email to fedora devel mailing list too. Mention the link to the + "Highlights from XXXX-XX-XX release" documentation page. + +4. Propose a new "highlights" post for the `Fedora Copr Blog`_, + see `the example + `_. + +5. Close the Fedora Infra ticket. + +6. Change the "ongoing" `Fedora Status SOP`_ entry into a "resolved" one. + +7. Remove the warning banner from frontend page using + ``copr-frontend warning-banner --remove`` + + +.. _`copr-devel`: https://lists.fedoraproject.org/archives/list/copr-devel@lists.fedorahosted.org/ +.. _`Fedora Outage SOP`: https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/outage/ +.. _`Fedora Status SOP`: https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/status-fedora/ +.. _`Fedora Copr Blog`: https://fedora-copr.github.io/ diff --git a/doc/maintenance_documentation.rst b/doc/maintenance_documentation.rst index 4562e6893..ce456f07b 100644 --- a/doc/maintenance_documentation.rst +++ b/doc/maintenance_documentation.rst @@ -15,6 +15,7 @@ This section contains information about maintenance topics. You may also be inte How to manage active chroots How to rename chroots Fedora Copr hypervisors + Outage announcements .. toctree::