From 6d80ff764617af24119514edc10f4a97de739334 Mon Sep 17 00:00:00 2001 From: Julien Schleich Date: Fri, 19 Apr 2024 11:29:40 +0200 Subject: [PATCH] Updating backup and purging information / simplification --- docs/data/backups.md | 34 +++++++--------------------------- docs/filesystems/quotas.md | 24 ++++++++++-------------- 2 files changed, 17 insertions(+), 41 deletions(-) diff --git a/docs/data/backups.md b/docs/data/backups.md index b8015e96..aa3b6a35 100644 --- a/docs/data/backups.md +++ b/docs/data/backups.md @@ -5,29 +5,28 @@ a regular basis. **Ultimately, it is your responsibility to protect yourself from data loss.** -ULHPC has 3 different backup targets, with different rotation policies and physical locations. - -The backups are only accessible by HPC staff, for disaster recovery purposes. +The backups are only accessible by HPC staff, for disaster recovery purposes only. More precisions can be requested via a support request. -## User directories on the ULHPC clusters +## Directories on the ULHPC clusters infrastructure + +For computation purposes, ULHPC users can use multiple storages: home, scratch and projects. Note however that the HPC Platform does not have the infrastructure to backup all of them, see details below. | Directory | Path | Backup location | Frequency | Retention | |------------------|----------------|-----------------|-----------|--------------------------------------------------------------| -| home directories | `$HOME` | CDC, Belval | Weekly | last 7 backups, at least one per month for the last 2 months | -| projects | `$PROJECTWORK` | CDC, Belval | Weekly | one backup per week of the backup directory (`$PROJECT/backup/`) | +| home directories | `$HOME` | not backed up | | | scratch | `$SCRATCH` | not backed up | | | +| projects | `$PROJECTWORK` | CDC, Belval | Weekly | one backup per week of the backup directory ONLY (`$PROJECT/backup/`) | -## Isilon project directories snapshots +## Directories on the SIU Isilon infrastructure Projects stored on the Isilon filesystem are *snapshotted* weekly, the snapshots are kept for 10 days. !!! danger Snapshots are **not a real backup**. It does not protect you against a system failure, it will only permit to recover some files in case of accidental deletion - Each project directory, in `/mnt/isilon/projects/` contains a hidden sub-directory `.snapshot`: * `.snapshot` is invisible to `ls`, `ls -a`, `find` and similar @@ -37,19 +36,10 @@ Each project directory, in `/mnt/isilon/projects/` contains a hidden sub-directo * files can *only* be copied *out* of a snapshot -## Virtual machines - -| Source | Backup location | Frequency | Retention | -|------------------------|-----------------|-----------|----------------------------------------------------| -| Gitlab infrastructure | CDC, Belval | Weekly | last 5 weekly snapshots | -| Iris infrastructure | CDC, Belval | Weekly | last 5 weekly snapshots | - - ## Services | Name | Backup location | Frequency | Retention | |------------------------------|-----------------|-----------|-----------------------------------------------------------| -| gitlab.uni.lu | CDC, Belval | Daily | last 7 daily backups, one per month for the last 6 months | | hpc.uni.lu (pad, privatebin) | CDC, Belval | Daily | last 7 daily backups, one per month for the last 6 months | @@ -61,16 +51,6 @@ with pathnames and timestamps of the missing data. Such restore requests may take a few days to complete. -## Purging - -!!! note - See [Filesystem Quotas and Purging](../filesystems/quotas.md) for detailed information about inode, - space quotas, and file system purge policies. - -!!! warning - `$SCRATCH` directories are **not** backed up - - ## Backup Tools In practice, the ULHPC backup infrastructure is fully puppetized and make use of several tools facilitating the operations: diff --git a/docs/filesystems/quotas.md b/docs/filesystems/quotas.md index ee806b21..062a6a50 100644 --- a/docs/filesystems/quotas.md +++ b/docs/filesystems/quotas.md @@ -1,13 +1,13 @@ -# Quotas and Purging +# Quotas ## Overview -| Directory | Default space quota | Default inode quota | Purge time | -| ---------------------- | ------------------- | ------------------- | ---------- | -| `$HOME` | 500 GB | 1 M | - | -| `$SCRATCH` | 10 TB | 1 M | 60 days | -| `/work/projects/...` | 1 TB | 1 M | - | -| `/mnt/isilon/projects/...` | 1.14 PB globally | - | - | +| Directory | Default space quota | Default inode quota | +| ---------------------- | ------------------- | ------------------- | +| `$HOME` | 500 GB | 1 M | +| `$SCRATCH` | 10 TB | 1 M | +| `/work/projects/...` | 1 TB | 1 M | +| `/mnt/isilon/projects/...` | 1.14 PB globally | - | ## Quotas @@ -15,7 +15,7 @@ When a quota is reached writes to that directory will fail. !!! note - On Isilon everyone shares one global quota. Unfortunately it is not possible to see the quota status on the cluster. + On Isilon everyone shares one global quota and the HPC Platform team sets up project quotas. Unfortunately it is not possible to see the quota status on the cluster. ### Current usage @@ -60,9 +60,9 @@ For a more graphical approach, use `ncdu`, with the `c` option to display the ag ### Increases -If you or your project needs additional space or inodes for your scratch or project directory you may request it via [ServiceNow](https://hpc.uni.lu/support/) (HPC → Storage & projects → Extend quota). +If your project needs additional space or inodes for a specific project directory you may request it via [ServiceNow](https://hpc.uni.lu/support/) (HPC → Storage & projects → Extend quota). -Quotas on the home directory cannot be increased. +Quotas on the home directory and scratch cannot be increased. ### Troubleshooting @@ -72,7 +72,3 @@ To avoid this issue, check out the `newgrp` command or set the `s` mode bit ("se To transfer data with `rsync` into a project directory, please check the [data transfer documentation](/data/transfer/#transfer-from-your-local-machine-to-a-project-directory-on-the-remote-cluster). -## Purging -Files in the scratch (`$SCRATCH`, `/mnt/lscratch`) directories are purged. This means **files that have not been read** (i.e. atime has not been updated) **within the last 60 days are automatically removed every month**. - -Files in `/tmp` on the compute nodes are **removed** at the end of the job.