Merge pull request #56 from ULHPC/backup

Updating backup and purging information / simplification
ULHPC · Apr 19, 2024 · 8a0b260 · 8a0b260
2 parents b14476f + 6d80ff7
commit 8a0b260
Show file tree

Hide file tree

Showing 2 changed files with 17 additions and 41 deletions.
diff --git a/docs/data/backups.md b/docs/data/backups.md
@@ -5,29 +5,28 @@
     a regular basis.  **Ultimately, it is your responsibility to
     protect yourself from data loss.**
 
-ULHPC has 3 different backup targets, with different rotation policies and physical locations.
-
-The backups are only accessible by HPC staff, for disaster recovery purposes.
+The backups are only accessible by HPC staff, for disaster recovery purposes only.
 
 More precisions can be requested via a support request.
 
-## User directories on the ULHPC clusters
+## Directories on the ULHPC clusters infrastructure
+
+For computation purposes, ULHPC users can use multiple storages: home, scratch and projects. Note however that the HPC Platform does not have the infrastructure to backup all of them, see details below.
 
 | Directory        | Path           | Backup location | Frequency | Retention                                                    |
 |------------------|----------------|-----------------|-----------|--------------------------------------------------------------|
-| home directories | `$HOME`        | CDC, Belval     | Weekly    | last 7 backups, at least one per month for the last 2 months |
-| projects         | `$PROJECTWORK` | CDC, Belval     | Weekly    | one backup per week of the backup directory (`$PROJECT/backup/`) |
+| home directories | `$HOME`        | not backed up     |     |
 | scratch          | `$SCRATCH`     | not backed up   |           |                                                              |
+| projects         | `$PROJECTWORK` | CDC, Belval     | Weekly    | one backup per week of the backup directory ONLY (`$PROJECT/backup/`) |
 
 
-## Isilon project directories snapshots
+## Directories on the SIU Isilon infrastructure
 
 Projects stored on the Isilon filesystem are *snapshotted* weekly, the snapshots are kept for 10 days.
 
 !!! danger
     Snapshots are **not a real backup**. It does not protect you against a system failure, it will only permit to recover some files in case of accidental deletion
 
-
 Each project directory, in `/mnt/isilon/projects/` contains a hidden sub-directory `.snapshot`:
 
 * `.snapshot` is invisible to `ls`, `ls -a`, `find` and similar
@@ -37,19 +36,10 @@ Each project directory, in `/mnt/isilon/projects/` contains a hidden sub-directo
 * files can *only* be copied *out* of a snapshot
 
 
-## Virtual machines
-
-| Source                 | Backup location | Frequency | Retention                                          |
-|------------------------|-----------------|-----------|----------------------------------------------------|
-| Gitlab infrastructure  | CDC, Belval     | Weekly    | last 5 weekly snapshots                            |
-| Iris infrastructure    | CDC, Belval     | Weekly    | last 5 weekly snapshots                            |
-
-
 ## Services
 
 | Name                         | Backup location | Frequency | Retention                                                 |
 |------------------------------|-----------------|-----------|-----------------------------------------------------------|
-| gitlab.uni.lu                | CDC, Belval     | Daily     | last 7 daily backups, one per month for the last 6 months |
 | hpc.uni.lu (pad, privatebin) | CDC, Belval     | Daily     | last 7 daily backups, one per month for the last 6 months |
 
 
@@ -61,16 +51,6 @@ with pathnames and timestamps of the missing data.
 
 Such restore requests may take a few days to complete.
 
-## Purging
-
-!!! note
-    See [Filesystem Quotas and Purging](../filesystems/quotas.md) for detailed information about inode,
-    space quotas, and file system purge policies.
-
-!!! warning
-	`$SCRATCH` directories are **not** backed up
-
-
 ## Backup Tools
 
 In practice, the ULHPC backup infrastructure is fully puppetized and make use of several tools facilitating the operations:

diff --git a/docs/filesystems/quotas.md b/docs/filesystems/quotas.md
@@ -1,21 +1,21 @@
-# Quotas and Purging
+# Quotas 
 
 ## Overview
 
-| Directory    | Default space quota | Default inode quota | Purge time |
-| ---------------------- | ------------------- | ------------------- | ---------- |
-| `$HOME`                | 500 GB              | 1 M                 | -          |
-| `$SCRATCH`             | 10 TB               | 1 M                 | 60 days    |
-| `/work/projects/...`   | 1 TB                | 1 M                 | -          |
-| `/mnt/isilon/projects/...` | 1.14 PB globally | -                   | -          |
+| Directory    | Default space quota | Default inode quota | 
+| ---------------------- | ------------------- | ------------------- |
+| `$HOME`                | 500 GB              | 1 M                 |
+| `$SCRATCH`             | 10 TB               | 1 M                 |
+| `/work/projects/...`   | 1 TB                | 1 M                 |
+| `/mnt/isilon/projects/...` | 1.14 PB globally | -                   |
 
 ## Quotas
 
 !!! warning
 	When a quota is reached writes to that directory will fail.
 
 !!! note
-	On Isilon everyone shares one global quota. Unfortunately it is not possible to see the quota status on the cluster.
+	On Isilon everyone shares one global quota and the HPC Platform team sets up project quotas. Unfortunately it is not possible to see the quota status on the cluster.
 
 ### Current usage
 
@@ -60,9 +60,9 @@ For a more graphical approach, use `ncdu`, with the `c` option to display the ag
 
 ### Increases
 
-If you or your project needs additional space or inodes for your scratch or project directory you may request it via [ServiceNow](https://hpc.uni.lu/support/) (HPC &rarr; Storage & projects &rarr; Extend quota).
+If your project needs additional space or inodes for a specific project directory you may request it via [ServiceNow](https://hpc.uni.lu/support/) (HPC &rarr; Storage & projects &rarr; Extend quota).
 
-Quotas on the home directory cannot be increased.
+Quotas on the home directory and scratch cannot be increased.
 
 ### Troubleshooting
 
@@ -72,7 +72,3 @@ To avoid this issue, check out the `newgrp` command or set the `s` mode bit ("se
 
 To transfer data with `rsync` into a project directory, please check the [data transfer documentation](/data/transfer/#transfer-from-your-local-machine-to-a-project-directory-on-the-remote-cluster).
 
-## Purging
-Files in the scratch (`$SCRATCH`, `/mnt/lscratch`) directories are purged. This means **files that have not been read** (i.e. atime has not been updated) **within the last 60 days are automatically removed every month**.
-
-Files in `/tmp` on the compute nodes are **removed** at the end of the job.