fedora-copr · FrostyX · Dec 11, 2023 · Dec 5, 2023 · Dec 6, 2023 · Dec 7, 2023
diff --git a/doc/database_backup.rst b/doc/database_backup.rst
@@ -1,18 +1,50 @@
 .. _database_backup:
 
-Database backup
-===============
+Database backups
+================
 
-We periodically create a databse dump and offer users to download
-it. At the same time, it can be used as a database backup if something
-wrong happens. Please see ``/etc/cron.d/cron-backup-database-coprdb``.
+We periodically create two kinds of database dumps.
 
-To backup the database manually (this can be useful e.g. before
-upgrading to a new major version of PostgreSQL), run::
+Private/backup dump
+-------------------
+
+This "complete" dump is done for potential disaster-recovery situations.  It
+contains all the data (including private stuff like API tokens), and therefore
+we **never publish it or download it onto our machines**.  The dump is created in
+the ``/backups/`` directory on Copr Frontend, and it is periodically pulled by
+a rdiff-backup Fedora Infrastructure bot `configured by Ansible
+<https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/rdiff-backup.yml>`_.
+
+To generate the backup manually (this can be useful e.g. before upgrading to a
+new major version of PostgreSQL), run::
 
     [root@copr-fe ~][PROD]# su - postgres
     bash-5.0$ /usr/local/bin/backup-database coprdb
 
-Please be aware that the script does ``sleep`` for some
-undeterministic amount of time. You might want to kill the ``sleep``
-process to speed it up a little.
+.. warning::
+
+    Please be aware that the script does an initial ``sleep`` for some
+    undeterministic amount of time (to not backup all the Fedora Infra databases
+    at the same time).  You might want to kill the ``sleep`` process to speed it
+    up a little.  Still, be prepared that the dump, mostly because of the XZ
+    compression, takes more than 20 minutes!
+
+.. warning::
+
+    If you run this manually to have the :ref:`last-minute pre-upgrade dump
+    <how_to_upgrade_persistent_instances>`, you need to **keep the machine
+    running** till the upgrade is done — to keep the ``/backups`` directory
+    existing!
+
+Public dumps
+------------
+
+These dumps are `publicly available
+<https://copr.fedorainfracloud.org/db_dumps/>`_ for anyone's experiments.
+These are generated overnight via::
+
+    /etc/cron.d/cron-backup-database-coprdb
+
+Those dumps have all the private data filtered out (namely the contents of
+``_private`` tables), but still usable as-is for debugging purposes (e.g.
+spawning a testing Copr Frontend container with pre-generated database).
diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst
@@ -93,7 +93,7 @@ Check that .repo files correctly points to ``@copr/copr``. And run on batcave01.
 .. note::
 
     If there is a new version of copr-rpmbuild, follow the
-    :ref:`terminate_os_vms` and :ref:`terminate_resalloc_vms` instructions.
+    :ref:`terminate_resalloc_vms` instructions.
 
 Make sure expected versions of Copr packages are installed on the dev
 instances::
@@ -215,31 +215,8 @@ notes against Copr git repository.
 Schedule and announce the outage
 ................................
 
-.. warning::
-
-    Schedule outage even if it has to happen in the next 5 minutes!
-
-Get faimiliar with the `Fedora Outage SOP <https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/outage/>`_.
-In general, please follow these steps:
-
-1. Prepare the infrastructure ticket similar to `this old one <https://pagure.io/fedora-infrastructure/issue/10854>`_.
-
-2. Send email to `copr-devel`_ mailing list informing about an upcomming
-   release. We usually copy-paste text of the infrastructure ticket created in a
-   previous step. Don't forget to put a link to the ticket at the end of the
-   email.  See the `example <https://lists.fedoraproject.org/archives/list/copr-devel@lists.fedorahosted.org/message/FVVX3Y7IVRTFW3NYVBTWX3AK3BHNRATX/>`_.
-
-3. Send ``op #fedora-buildsys MyIrcNick`` message to ``ChanServ`` on
-   libera.chat to get the OP rights, and then adjust the channel title so it
-   starts with message similar to::
-
-        Planned outage 2022-08-17 20:00 UTC - https://pagure.io/fedora-infrastructure/issue/10854
-
-4. Create a new "planned" `Fedora Status SOP`_ entry.
-5. Create warning banner on Copr homepage::
-
-     copr-frontend warning-banner --outage_time "2022-12-31 13:00-16:00 UTC" --ticket 1234
-
+See a specific document :ref:`announcing_fedora_copr_outage`, namely the
+"planned" outage state.
 
 Release window
 --------------
@@ -248,16 +225,10 @@ If all the pre-release preparations were done meticulously and everything
 was tested properly, the release window shouldn't take more than ten
 minutes. That is, if nothing goes terribly sideways...
 
-
 Let users know
 --------------
 
-1. Change the "planned" `Fedora Status SOP`_ entry into an "ongoing" entry.
-
-2. Announce on ``#fedora-buildsys``, change title like
-   ``s/Planned outage ../OUTAGE NOW .../`` and send some message like
-   ``WARNING: The scheduled outage just begings!``.
-
+See :ref:`announcing_fedora_copr_outage` again, ad "ongoning" issue.
 
 Production infra tags
 ---------------------
@@ -371,24 +342,8 @@ If schema was modified you should generate new Schema documentation.
 Announce the end of the release
 ...............................
 
-1. Remove the "Outage" note from the ``#fedora-buildsys`` title.
-
-2. Send a message on ``fedora-buildsys`` that the outage is over!
-
-3. Send email to `copr-devel`_ mailing list.  If there is some important change
-   you can send email to fedora devel mailing list too.  Mention the link to the
-   "Highlights from XXXX-XX-XX release" documentation page.
-
-4. Propose a new "highlights" post for the `Fedora Copr Blog`_,
-   see `the example
-   <https://github.com/fedora-copr/fedora-copr.github.io/pull/55/files>`_.
-
-5. Close the Fedora Infra ticket.
-
-6. Change the "ongoing" `Fedora Status SOP`_ entry into a "resolved" one.
-
-7. Remove the warning banner from frontend page using
-   ``copr-frontend warning-banner --remove``
+See a specific document :ref:`announcing_fedora_copr_outage`, the "resolved"
+section.
 
 
 Release packages to PyPI
@@ -446,6 +401,4 @@ Fix this document to make it easy for the release nanny of the next release to u
 
 .. _`Copr release directory`: https://releases.pagure.org/copr/copr
 .. _`copr-devel`: https://lists.fedoraproject.org/archives/list/copr-devel@lists.fedorahosted.org/
-.. _`Fedora Status SOP`: https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/status-fedora/
 .. _`example stg infra repo`: https://kojipkgs.fedoraproject.org/repos-dist/f36-infra-stg/
-.. _`Fedora Copr Blog`: https://fedora-copr.github.io/
diff --git a/doc/how_to_upgrade_builders.rst b/doc/how_to_upgrade_builders.rst
@@ -8,7 +8,6 @@ This article explains how to upgrade the Copr builders images in
 - :ref:`AWS <prepare_aws_source_images>` (x86_64 and aarch64),
 - :ref:`LibVirt/OpenStack <prepare_libvirt_source_images>` (x86_64 and ppc64le),
 - :ref:`IBM Cloud <prepare_ibmcloud_source_images>` (s390x),
-- |ss| :ref:`We currently don't work with OpenStack separately <how_to_upgrade_builders_openstack>` |se|.
 
 This HOWTO is useful for upgrading images to a newer Fedora release, or for just
 updating all the packages contained within the builder images.  This image

diff --git a/doc/how_to_upgrade_builders_openstack.rst b/doc/how_to_upgrade_builders_openstack.rst