Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply 775 days retention on Messaging derived tables + some other tables missing metadata.yaml #6964

Merged
merged 3 commits into from
Feb 4, 2025
Merged

Apply 775 days retention on Messaging derived tables + some other tables missing metadata.yaml #6964

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f910db0
add retention to messaging derived + some other tables client level t…
alekhyamoz Feb 4, 2025
79394cd
FIX ci issues
alekhyamoz Feb 4, 2025
e7b5088
Merge branch 'main' into messaging_derived_retention
alekhyamoz Feb 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions sql/moz-fx-data-shared-prod/messaging_system_derived/cfr_users_daily_v1/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,15 @@ owners:
- najiang@mozilla.com
labels:
incremental: true
table_type: client_level
scheduling:
dag_name: bqetl_messaging_system
bigquery:
time_partitioning:
type: day
field: submission_date
require_partition_filter: false
expiration_days: 775
clustering:
fields:
- release_channel
3 changes: 2 additions & 1 deletion sql/moz-fx-data-shared-prod/messaging_system_derived/cfr_users_last_seen_v1/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ labels:
incremental: true
dag: bqetl_messaging_system
owner1: najiang
table_type: client_level
scheduling:
dag_name: bqetl_messaging_system
depends_on_past: true
Expand All @@ -15,7 +16,7 @@ bigquery:
type: day
field: submission_date
require_partition_filter: true
expiration_days: null
expiration_days: 775
clustering:
fields:
- release_channel
Expand Down
2 changes: 2 additions & 0 deletions sql/moz-fx-data-shared-prod/messaging_system_derived/onboarding_users_daily_v1/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,15 @@ owners:
- najiang@mozilla.com
labels:
incremental: true
table_type: client_level
scheduling:
dag_name: bqetl_messaging_system
bigquery:
time_partitioning:
type: day
field: submission_date
require_partition_filter: false
expiration_days: 775
clustering:
fields:
- release_channel
3 changes: 2 additions & 1 deletion ...-fx-data-shared-prod/messaging_system_derived/onboarding_users_last_seen_v1/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ labels:
incremental: true
dag: bqetl_messaging_system
owner1: najiang
table_type: client_level
scheduling:
dag_name: bqetl_messaging_system
depends_on_past: true
Expand All @@ -15,7 +16,7 @@ bigquery:
type: day
field: submission_date
require_partition_filter: true
expiration_days: null
expiration_days: 775
clustering:
fields:
- release_channel
Expand Down
2 changes: 2 additions & 0 deletions sql/moz-fx-data-shared-prod/messaging_system_derived/snippets_users_daily_v1/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,15 @@ owners:
- najiang@mozilla.com
labels:
incremental: true
table_type: client_level
scheduling:
dag_name: bqetl_messaging_system
bigquery:
time_partitioning:
type: day
field: submission_date
require_partition_filter: false
expiration_days: 775
clustering:
fields:
- release_channel
3 changes: 2 additions & 1 deletion ...oz-fx-data-shared-prod/messaging_system_derived/snippets_users_last_seen_v1/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ labels:
incremental: true
dag: bqetl_messaging_system
owner1: najiang
table_type: client_level
scheduling:
dag_name: bqetl_messaging_system
depends_on_past: true
Expand All @@ -15,7 +16,7 @@ bigquery:
type: day
field: submission_date
require_partition_filter: true
expiration_days: null
expiration_days: 775
clustering:
fields:
- release_channel
Expand Down
15 changes: 15 additions & 0 deletions sql/moz-fx-data-shared-prod/org_mozilla_fenix_derived/baseline_daily_v1/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
friendly_name: Fenix derived baseline daily
description: |-
fenix derived baseline daily
owners:
- ascholtz@mozilla.com
labels:
incremental: false
table_type: client_level
shredder_mitigation: false
bigquery:
time_partitioning:
type: day
field: submission_date
require_partition_filter: true
expiration_days: 775
15 changes: 15 additions & 0 deletions sql/moz-fx-data-shared-prod/org_mozilla_fenix_derived/clients_last_seen_v2/metadata.yaml
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This table has no rows right now so I think we can apply and then make a note to come back and delete later.

Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
friendly_name: Fenix clients last seen v2
description: |-
fenix derived clients last seen v2
owners:
- ascholtz@mozilla.com
labels:
incremental: false
table_type: client_level
shredder_mitigation: false
bigquery:
time_partitioning:
type: day
field: submission_date
require_partition_filter: true
expiration_days: 775
15 changes: 15 additions & 0 deletions ...moz-fx-data-shared-prod/org_mozilla_fenix_nightly_derived/baseline_daily_v1/metadata.yaml
View file
Open in desktop
kwindau marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
friendly_name: Fenix nightly derived baseline daily
description: |-
fenix nightly derived baseline daily
owners:
- ascholtz@mozilla.com
labels:
incremental: false
table_type: client_level
shredder_mitigation: false
bigquery:
time_partitioning:
type: day
field: submission_date
require_partition_filter: true
expiration_days: 775
15 changes: 15 additions & 0 deletions ...fx-data-shared-prod/org_mozilla_fenix_nightly_derived/baseline_last_seen_v1/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
friendly_name: Fenix nightly derived baseline last seen
description: |-
fenix nightly derived baseline last seen
owners:
- ascholtz@mozilla.com
labels:
incremental: false
table_type: client_level
shredder_mitigation: false
bigquery:
time_partitioning:
type: day
field: submission_date
require_partition_filter: true
expiration_days: 775
5 changes: 3 additions & 2 deletions sql/moz-fx-data-shared-prod/org_mozilla_firefox_derived/migrated_clients_v1/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,14 @@ owners:
labels:
application: fenix
schedule: daily
owner1: frank
owner1: ascholtz
table_type: client_level
bigquery:
time_partitioning:
type: day
field: submission_date
require_partition_filter: null
expiration_days: null
expiration_days: 775
clustering:
fields:
- fenix_client_id
Expand Down
15 changes: 15 additions & 0 deletions sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/baseline_daily_v1/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
friendly_name: vrbrowswer derived baseline daily
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The VR browser ones can all be deprecated as well

description: |-
vrbrowswer derived baseline daily
owners:
- ascholtz@mozilla.com
labels:
incremental: false
table_type: client_level
shredder_mitigation: false
bigquery:
time_partitioning:
type: day
field: submission_date
require_partition_filter: true
expiration_days: 775
15 changes: 15 additions & 0 deletions sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/clients_daily_v1/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
friendly_name: vrbrowswer derived clients daily
description: |-
vrbrowswer derived clients daily
owners:
- ascholtz@mozilla.com
labels:
incremental: false
table_type: client_level
shredder_mitigation: false
bigquery:
time_partitioning:
type: day
field: submission_date
require_partition_filter: true
expiration_days: 775
3 changes: 2 additions & 1 deletion sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/clients_last_seen_v1/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,13 @@ description: |
labels:
schedule: daily
application: firefox-reality
table_type: client_level
bigquery:
time_partitioning:
type: day
field: submission_date
require_partition_filter: false
expiration_days: null
expiration_days: 775
clustering:
fields:
- app_name
Expand Down
15 changes: 15 additions & 0 deletions sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/metrics_daily_v1/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
friendly_name: vrbrowswer derived metrics daily
description: |-
vrbrowswer derived metrics daily
owners:
- ascholtz@mozilla.com
labels:
incremental: false
table_type: client_level
shredder_mitigation: false
bigquery:
time_partitioning:
type: day
field: submission_date
require_partition_filter: true
expiration_days: 775
5 changes: 3 additions & 2 deletions ...red-prod/stub_attribution_service_derived/dl_token_ga_attribution_lookup_v1/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,11 @@ description: |-
ga_client_id - Available in Stub Attribution Service and GA
stub_session_id - Available in Stub Attribution Service and GA
owners:
- frank@mozilla.com
- kwindau@mozilla.com
labels:
incremental: true
owner1: frank@mozilla.com
owner1: kwindau@mozilla.com
table_type: client_level
scheduling:
dag_name: bqetl_mozilla_org_derived
date_partition_parameter: null
Expand Down