Set additional_deletion_requests_v1 retention to 775 days

[BenWu]

Description

Per policy discussion in https://mozilla-hub.atlassian.net/browse/DSRE-1799, the retention of deletion requests should be equal to the longest retained data subject to the deletion request. That would be 760 for pings https://github.com/mozilla/probe-scraper/blob/cf4abcf2e94e1453315f44e2b2d45ab1ba8a29d2/repositories.yaml#L1336-L1342 and eventually 775 for derived data

cc @akkomar

Related Tickets & Documents

• DSRE-1799

Reviewer, please follow this checklist

┆Issue is synchronized with this Jira Task

[kik-kik]

@BenWu should we define this value inside bqetl_project.yaml which could be used as a default value when retention policy is enabled on a table? From my understanding most tables should share the same retention policy. If need be we could always allow for explicitly overwriting it in the metadata.yaml.

[BenWu]

IMO since retention is still new, it would be better to have retention explicitly set for every table rather than having a default value so there are fewer surprises. When we get to a point where we all assume derived client tables default to 775 days then we can set a default. There's also going to be a check that retention is set so all the existing tables should be covered

[dataops-ci-bot]

Integration report for "Add client_level table type"

sql.diff

Click to expand!

diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_focus_beta_derived/additional_deletion_requests_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_focus_beta_derived/additional_deletion_requests_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_focus_beta_derived/additional_deletion_requests_v1/metadata.yaml	2025-01-29 17:37:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_focus_beta_derived/additional_deletion_requests_v1/metadata.yaml	2025-01-29 17:37:37.000000000 +0000
@@ -8,6 +8,7 @@
 - akomar@mozilla.com
 labels:
   schedule: daily
+  table_type: client_level
   dag: bqetl_org_mozilla_focus_derived
   owner1: akomar
 scheduling:
@@ -18,7 +19,7 @@
     type: day
     field: submission_timestamp
     require_partition_filter: false
-    expiration_days: null
+    expiration_days: 775.0
   range_partitioning: null
   clustering: null
 workgroup_access:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_focus_derived/additional_deletion_requests_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_focus_derived/additional_deletion_requests_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_focus_derived/additional_deletion_requests_v1/metadata.yaml	2025-01-29 17:36:56.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_focus_derived/additional_deletion_requests_v1/metadata.yaml	2025-01-29 17:37:25.000000000 +0000
@@ -8,6 +8,7 @@
 - akomar@mozilla.com
 labels:
   schedule: daily
+  table_type: client_level
   dag: bqetl_org_mozilla_focus_derived
   owner1: akomar
 scheduling:
@@ -18,7 +19,7 @@
     type: day
     field: submission_timestamp
     require_partition_filter: false
-    expiration_days: null
+    expiration_days: 775.0
   range_partitioning: null
   clustering: null
 workgroup_access:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_focus_nightly_derived/additional_deletion_requests_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_focus_nightly_derived/additional_deletion_requests_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_focus_nightly_derived/additional_deletion_requests_v1/metadata.yaml	2025-01-29 17:37:02.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_focus_nightly_derived/additional_deletion_requests_v1/metadata.yaml	2025-01-29 17:37:18.000000000 +0000
@@ -8,6 +8,7 @@
 - akomar@mozilla.com
 labels:
   schedule: daily
+  table_type: client_level
   dag: bqetl_org_mozilla_focus_derived
   owner1: akomar
 scheduling:
@@ -18,7 +19,7 @@
     type: day
     field: submission_timestamp
     require_partition_filter: false
-    expiration_days: null
+    expiration_days: 775.0
   range_partitioning: null
   clustering: null
 workgroup_access:

Link to full diff