Apply 775 days retention on Messaging derived tables + some other tables missing metadata.yaml

[alekhyamoz]

Added retention on the below list of tables.

messaging_system_derived | cfr_users_last_seen_v1
messaging_system_derived | onboarding_users_daily_v1
messaging_system_derived | onboarding_users_last_seen_v1
messaging_system_derived | snippets_users_daily_v1
messaging_system_derived | snippets_users_last_seen_v1
org_mozilla_fenix_derived | baseline_daily_v1
org_mozilla_fenix_derived | clients_last_seen_v2
org_mozilla_fenix_nightly_derived | baseline_daily_v1
org_mozilla_fenix_nightly_derived | baseline_last_seen_v1
org_mozilla_firefox_derived | migrated_clients_v1
org_mozilla_vrbrowser_derived | baseline_daily_v1
org_mozilla_vrbrowser_derived | clients_daily_v1
org_mozilla_vrbrowser_derived | clients_last_seen_v1
org_mozilla_vrbrowser_derived | metrics_daily_v1

[dataops-ci-bot]

Integration report for "Merge branch 'main' into messaging_derived_retention"

sql.diff

Click to expand!

diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_mozilla_org_derived.py /tmp/workspace/generated-sql/dags/bqetl_mozilla_org_derived.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_mozilla_org_derived.py	2025-02-04 18:30:13.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_mozilla_org_derived.py	2025-02-04 18:31:57.000000000 +0000
@@ -53,8 +53,12 @@
         dataset_id="stub_attribution_service_derived",
         project_id="moz-fx-data-shared-prod",
         is_dq_check_fail=True,
-        owner="frank@mozilla.com",
-        email=["frank@mozilla.com", "telemetry-alerts@mozilla.com"],
+        owner="kwindau@mozilla.com",
+        email=[
+            "frank@mozilla.com",
+            "kwindau@mozilla.com",
+            "telemetry-alerts@mozilla.com",
+        ],
         depends_on_past=False,
         task_concurrency=1,
         parameters=["download_date:DATE:{{ds}}"],
@@ -80,8 +84,12 @@
         destination_table="dl_token_ga_attribution_lookup_v1",
         dataset_id="stub_attribution_service_derived",
         project_id="moz-fx-data-shared-prod",
-        owner="frank@mozilla.com",
-        email=["frank@mozilla.com", "telemetry-alerts@mozilla.com"],
+        owner="kwindau@mozilla.com",
+        email=[
+            "frank@mozilla.com",
+            "kwindau@mozilla.com",
+            "telemetry-alerts@mozilla.com",
+        ],
         date_partition_parameter=None,
         depends_on_past=False,
         task_concurrency=1,
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_derived: baseline_daily_v1
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_derived: clients_last_seen_v2
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly_derived: baseline_daily_v1
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly_derived: baseline_last_seen_v1
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived: baseline_daily_v1
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived: clients_daily_v1
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived: metrics_daily_v1
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/cfr_users_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/cfr_users_daily_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/cfr_users_daily_v1/metadata.yaml	2025-02-04 18:26:29.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/cfr_users_daily_v1/metadata.yaml	2025-02-04 18:27:01.000000000 +0000
@@ -5,6 +5,7 @@
 - najiang@mozilla.com
 labels:
   incremental: true
+  table_type: client_level
   dag: bqetl_messaging_system
   owner1: najiang
 scheduling:
@@ -14,7 +15,7 @@
     type: day
     field: submission_date
     require_partition_filter: false
-    expiration_days: null
+    expiration_days: 775.0
   range_partitioning: null
   clustering:
     fields:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/cfr_users_last_seen_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/cfr_users_last_seen_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/cfr_users_last_seen_v1/metadata.yaml	2025-02-04 18:26:29.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/cfr_users_last_seen_v1/metadata.yaml	2025-02-04 18:27:01.000000000 +0000
@@ -7,6 +7,7 @@
   incremental: true
   dag: bqetl_messaging_system
   owner1: najiang
+  table_type: client_level
 scheduling:
   dag_name: bqetl_messaging_system
   depends_on_past: true
@@ -15,7 +16,7 @@
     type: day
     field: submission_date
     require_partition_filter: true
-    expiration_days: null
+    expiration_days: 775.0
   range_partitioning: null
   clustering:
     fields:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/onboarding_users_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/onboarding_users_daily_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/onboarding_users_daily_v1/metadata.yaml	2025-02-04 18:26:29.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/onboarding_users_daily_v1/metadata.yaml	2025-02-04 18:27:01.000000000 +0000
@@ -5,6 +5,7 @@
 - najiang@mozilla.com
 labels:
   incremental: true
+  table_type: client_level
   dag: bqetl_messaging_system
   owner1: najiang
 scheduling:
@@ -14,7 +15,7 @@
     type: day
     field: submission_date
     require_partition_filter: false
-    expiration_days: null
+    expiration_days: 775.0
   range_partitioning: null
   clustering:
     fields:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/onboarding_users_last_seen_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/onboarding_users_last_seen_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/onboarding_users_last_seen_v1/metadata.yaml	2025-02-04 18:26:29.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/onboarding_users_last_seen_v1/metadata.yaml	2025-02-04 18:27:01.000000000 +0000
@@ -7,6 +7,7 @@
   incremental: true
   dag: bqetl_messaging_system
   owner1: najiang
+  table_type: client_level
 scheduling:
   dag_name: bqetl_messaging_system
   depends_on_past: true
@@ -15,7 +16,7 @@
     type: day
     field: submission_date
     require_partition_filter: true
-    expiration_days: null
+    expiration_days: 775.0
   range_partitioning: null
   clustering:
     fields:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/snippets_users_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/snippets_users_daily_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/snippets_users_daily_v1/metadata.yaml	2025-02-04 18:26:29.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/snippets_users_daily_v1/metadata.yaml	2025-02-04 18:27:01.000000000 +0000
@@ -5,6 +5,7 @@
 - najiang@mozilla.com
 labels:
   incremental: true
+  table_type: client_level
   dag: bqetl_messaging_system
   owner1: najiang
 scheduling:
@@ -14,7 +15,7 @@
     type: day
     field: submission_date
     require_partition_filter: false
-    expiration_days: null
+    expiration_days: 775.0
   range_partitioning: null
   clustering:
     fields:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/snippets_users_last_seen_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/snippets_users_last_seen_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/snippets_users_last_seen_v1/metadata.yaml	2025-02-04 18:26:29.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/messaging_system_derived/snippets_users_last_seen_v1/metadata.yaml	2025-02-04 18:27:01.000000000 +0000
@@ -7,6 +7,7 @@
   incremental: true
   dag: bqetl_messaging_system
   owner1: najiang
+  table_type: client_level
 scheduling:
   dag_name: bqetl_messaging_system
   depends_on_past: true
@@ -15,7 +16,7 @@
     type: day
     field: submission_date
     require_partition_filter: true
-    expiration_days: null
+    expiration_days: 775.0
   range_partitioning: null
   clustering:
     fields:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_derived/baseline_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_derived/baseline_daily_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_derived/baseline_daily_v1/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_derived/baseline_daily_v1/metadata.yaml	2025-02-04 18:27:03.000000000 +0000
@@ -0,0 +1,21 @@
+friendly_name: Fenix derived baseline daily
+description: |-
+  fenix derived baseline daily
+owners:
+- ascholtz@mozilla.com
+labels:
+  table_type: client_level
+  owner1: ascholtz
+bigquery:
+  time_partitioning:
+    type: day
+    field: submission_date
+    require_partition_filter: true
+    expiration_days: 775.0
+  range_partitioning: null
+  clustering: null
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+references: {}
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_derived/clients_last_seen_v2/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_derived/clients_last_seen_v2/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_derived/clients_last_seen_v2/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_derived/clients_last_seen_v2/metadata.yaml	2025-02-04 18:27:03.000000000 +0000
@@ -0,0 +1,21 @@
+friendly_name: Fenix clients last seen v2
+description: |-
+  fenix derived clients last seen v2
+owners:
+- ascholtz@mozilla.com
+labels:
+  table_type: client_level
+  owner1: ascholtz
+bigquery:
+  time_partitioning:
+    type: day
+    field: submission_date
+    require_partition_filter: true
+    expiration_days: 775.0
+  range_partitioning: null
+  clustering: null
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+references: {}
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly_derived/baseline_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly_derived/baseline_daily_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly_derived/baseline_daily_v1/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly_derived/baseline_daily_v1/metadata.yaml	2025-02-04 18:26:54.000000000 +0000
@@ -0,0 +1,21 @@
+friendly_name: Fenix nightly derived baseline daily
+description: |-
+  fenix nightly derived baseline daily
+owners:
+- ascholtz@mozilla.com
+labels:
+  table_type: client_level
+  owner1: ascholtz
+bigquery:
+  time_partitioning:
+    type: day
+    field: submission_date
+    require_partition_filter: true
+    expiration_days: 775.0
+  range_partitioning: null
+  clustering: null
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+references: {}
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly_derived/baseline_last_seen_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly_derived/baseline_last_seen_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly_derived/baseline_last_seen_v1/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly_derived/baseline_last_seen_v1/metadata.yaml	2025-02-04 18:26:54.000000000 +0000
@@ -0,0 +1,21 @@
+friendly_name: Fenix nightly derived baseline last seen
+description: |-
+  fenix nightly derived baseline last seen
+owners:
+- ascholtz@mozilla.com
+labels:
+  table_type: client_level
+  owner1: ascholtz
+bigquery:
+  time_partitioning:
+    type: day
+    field: submission_date
+    require_partition_filter: true
+    expiration_days: 775.0
+  range_partitioning: null
+  clustering: null
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+references: {}
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox_derived/migrated_clients_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox_derived/migrated_clients_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox_derived/migrated_clients_v1/metadata.yaml	2025-02-04 18:26:24.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox_derived/migrated_clients_v1/metadata.yaml	2025-02-04 18:26:42.000000000 +0000
@@ -11,12 +11,13 @@
   application: fenix
   schedule: daily
   owner1: frank
+  table_type: client_level
 bigquery:
   time_partitioning:
     type: day
     field: submission_date
     require_partition_filter: false
-    expiration_days: null
+    expiration_days: 775.0
   range_partitioning: null
   clustering:
     fields:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/baseline_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/baseline_daily_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/baseline_daily_v1/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/baseline_daily_v1/metadata.yaml	2025-02-04 18:26:43.000000000 +0000
@@ -0,0 +1,21 @@
+friendly_name: vrbrowswer derived baseline daily
+description: |-
+  vrbrowswer derived baseline daily
+owners:
+- ascholtz@mozilla.com
+labels:
+  table_type: client_level
+  owner1: ascholtz
+bigquery:
+  time_partitioning:
+    type: day
+    field: submission_date
+    require_partition_filter: true
+    expiration_days: 775.0
+  range_partitioning: null
+  clustering: null
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+references: {}
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/clients_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/clients_daily_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/clients_daily_v1/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/clients_daily_v1/metadata.yaml	2025-02-04 18:26:43.000000000 +0000
@@ -0,0 +1,21 @@
+friendly_name: vrbrowswer derived clients daily
+description: |-
+  vrbrowswer derived clients daily
+owners:
+- ascholtz@mozilla.com
+labels:
+  table_type: client_level
+  owner1: ascholtz
+bigquery:
+  time_partitioning:
+    type: day
+    field: submission_date
+    require_partition_filter: true
+    expiration_days: 775.0
+  range_partitioning: null
+  clustering: null
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+references: {}
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/clients_last_seen_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/clients_last_seen_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/clients_last_seen_v1/metadata.yaml	2025-02-04 18:26:32.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/clients_last_seen_v1/metadata.yaml	2025-02-04 18:26:43.000000000 +0000
@@ -5,12 +5,13 @@
 labels:
   schedule: daily
   application: firefox-reality
+  table_type: client_level
 bigquery:
   time_partitioning:
     type: day
     field: submission_date
     require_partition_filter: false
-    expiration_days: null
+    expiration_days: 775.0
   range_partitioning: null
   clustering:
     fields:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/metrics_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/metrics_daily_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/metrics_daily_v1/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/metrics_daily_v1/metadata.yaml	2025-02-04 18:26:43.000000000 +0000
@@ -0,0 +1,21 @@
+friendly_name: vrbrowswer derived metrics daily
+description: |-
+  vrbrowswer derived metrics daily
+owners:
+- ascholtz@mozilla.com
+labels:
+  table_type: client_level
+  owner1: ascholtz
+bigquery:
+  time_partitioning:
+    type: day
+    field: submission_date
+    require_partition_filter: true
+    expiration_days: 775.0
+  range_partitioning: null
+  clustering: null
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+references: {}
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/stub_attribution_service_derived/dl_token_ga_attribution_lookup_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/stub_attribution_service_derived/dl_token_ga_attribution_lookup_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/stub_attribution_service_derived/dl_token_ga_attribution_lookup_v1/metadata.yaml	2025-02-04 18:26:14.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/stub_attribution_service_derived/dl_token_ga_attribution_lookup_v1/metadata.yaml	2025-02-04 18:27:05.000000000 +0000
@@ -8,10 +8,11 @@
   ga_client_id - Available in Stub Attribution Service and GA
   stub_session_id - Available in Stub Attribution Service and GA
 owners:
-- frank@mozilla.com
+- kwindau@mozilla.com
 labels:
   incremental: true
-  owner1: frank
+  owner1: kwindau
+  table_type: client_level
   dag: bqetl_mozilla_org_derived
 scheduling:
   dag_name: bqetl_mozilla_org_derived

Link to full diff

[kwindau]

This table has no rows right now so I think we can apply and then make a note to come back and delete later.

[kwindau]

LGTM - left some notes basically saying some of these can likely be deleted in the future but for now no harm in applying retention

[scholtzan]

The VR browser ones can all be deprecated as well