[PB-5670] fix/add migration to clean up duplicate backup folders with batching #852

jzunigax2 · 2026-01-08T01:12:41Z

Cleanup script to remove duplicated backup root folders

jzunigax2 · 2026-01-08T01:14:49Z

explain result @sg-gs

QUERY PLAN
Hash Right Anti Join (cost=43.08..57.72 rows=1 width=4) (actual time=13.238..13.332 rows=62 loops=1)
Hash Cond: (files.folder_id = f.id)
-> Seq Scan on files (cost=0.00..13.93 rows=186 width=4) (actual time=0.941..1.956 rows=186 loops=1)
Filter: (NOT deleted)
Rows Removed by Filter: 7
-> Hash (cost=43.07..43.07 rows=1 width=4) (actual time=9.164..9.201 rows=84 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 11kB
-> Hash Join (cost=21.47..43.07 rows=1 width=4) (actual time=8.736..8.959 rows=84 loops=1)
Hash Cond: (((f.plain_name)::text = (folders.plain_name)::text) AND ((f.bucket)::text = (folders.bucket)::text) AND (f.user_id = folders.user_id))
Join Filter: (f.id <> (min(folders.id)))
Rows Removed by Join Filter: 21
-> Seq Scan on folders f (cost=0.00..19.57 rows=257 width=48) (actual time=0.303..0.468 rows=257 loops=1)
-> Hash (cost=21.37..21.37 rows=6 width=48) (actual time=8.141..8.146 rows=21 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Limit (cost=21.13..21.37 rows=6 width=48) (actual time=7.834..7.865 rows=21 loops=1)
-> HashAggregate (cost=21.13..21.37 rows=6 width=48) (actual time=7.789..7.815 rows=21 loops=1)
Group Key: folders.plain_name, folders.bucket, folders.user_id
Filter: (count(*) > 1)
Batches: 1 Memory Usage: 24kB
Rows Removed by Filter: 30
-> Seq Scan on folders (cost=0.00..20.86 rows=22 width=48) (actual time=0.049..7.255 rows=73 loops=1)
Filter: ((parent_id IS NULL) AND (parent_uuid IS NULL) AND (NOT deleted) AND (NOT removed) AND (plain_name IS NOT NULL) AND (created_at >= '2025-12-17 14:16:00-06'::timestamp with time zone) AND (created_at <= '2026-01-05 21:50:00-06'::timestamp with time zone))
Rows Removed by Filter: 184
Planning Time: 84.248 ms
Execution Time: 27.053 ms

sg-gs

Remember to add the tests to cover the case that produced this issue in the first place.

sg-gs · 2026-01-22T13:58:19Z

migrations/20260122030036-cleanup-duplicate-backup-folders.js

+            SELECT 1
+            FROM files
+            WHERE folder_id = f.id
+            AND deleted = false


use the status != 'DELETED' field, deleted and removed are deprecated

migrations/20260122030036-cleanup-duplicate-backup-folders.js

jzunigax2 · 2026-01-22T14:57:00Z

migrations/20260122030036-cleanup-duplicate-backup-folders.js

+      WITH duplicate_groups AS (
+        SELECT
+          plain_name,
+          bucket,
+          user_id,
+          MIN(id) as id_to_keep
+        FROM folders
+        WHERE
+          created_at >= '2025-12-17 14:16:00'
+          AND created_at <= '2026-01-05 21:50:00'
+          AND parent_id IS NULL
+          AND parent_uuid IS NULL
+          AND deleted = false
+          AND removed = false
+          AND plain_name IS NOT NULL
+        GROUP BY plain_name, bucket, user_id
+        HAVING COUNT(*) > 1
+        LIMIT ${BATCH_SIZE}
+      ),


This gets the oldest folder from each duplicated group. So we should be getting 905 folders from this cte

jzunigax2 · 2026-01-22T15:02:28Z

migrations/20260122030036-cleanup-duplicate-backup-folders.js

+      folders_to_delete AS (
+        SELECT f.id
+        FROM folders f
+        INNER JOIN duplicate_groups dg
+          ON f.plain_name = dg.plain_name
+          AND f.bucket = dg.bucket
+          AND f.user_id = dg.user_id
+        WHERE
+          f.id != dg.id_to_keep
+          AND NOT EXISTS (
+            SELECT 1
+            FROM files
+            WHERE folder_id = f.id
+            AND deleted = false
+          )
+      )


this joins all folders back to their duplicate group by matching (plain_name, bucket, user_id). So if a group has 3 duplicates, the JOIN produces 3 rows — each with that group's id_to_keep. Then we filter to only keep folders where:

id != id_to_keep (not the oldest/keeper)

No direct files (status != 'DELETED')

No child folders (deleted = false)

this should get 5863 folders

… and enhance test coverage for folder creation

…m duplicate backup folder deletion

sonarqubecloud · 2026-01-22T16:31:44Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2026-02-07T02:14:04Z

This PR is stale because it has been open for more than 15 days with no activity.

jzunigax2 requested a review from sg-gs January 8, 2026 01:12

jzunigax2 self-assigned this Jan 8, 2026

jzunigax2 temporarily deployed to development January 8, 2026 01:12 — with GitHub Actions Inactive

github-actions bot added the ready-for-preview label Jan 8, 2026

jzunigax2 force-pushed the fix/cleanup-duplicate-folders branch from 0c3ca0a to 47ce568 Compare January 12, 2026 15:10

jzunigax2 temporarily deployed to development January 12, 2026 15:10 — with GitHub Actions Inactive

jzunigax2 force-pushed the fix/cleanup-duplicate-folders branch from 47ce568 to bbd6962 Compare January 12, 2026 15:10

jzunigax2 temporarily deployed to development January 12, 2026 15:10 — with GitHub Actions Inactive

jzunigax2 force-pushed the fix/cleanup-duplicate-folders branch from bbd6962 to 4ae1045 Compare January 12, 2026 16:19

jzunigax2 temporarily deployed to development January 12, 2026 16:19 — with GitHub Actions Inactive

jzunigax2 had a problem deploying to development January 12, 2026 16:19 — with GitHub Actions Failure

jzunigax2 force-pushed the fix/cleanup-duplicate-folders branch from 4ae1045 to 37c9be2 Compare January 14, 2026 21:58

jzunigax2 temporarily deployed to development January 14, 2026 21:58 — with GitHub Actions Inactive

jzunigax2 had a problem deploying to development January 14, 2026 21:58 — with GitHub Actions Failure

jzunigax2 force-pushed the fix/cleanup-duplicate-folders branch from 37c9be2 to ccf78bc Compare January 14, 2026 21:59

jzunigax2 had a problem deploying to development January 14, 2026 21:59 — with GitHub Actions Failure

jzunigax2 temporarily deployed to development January 14, 2026 21:59 — with GitHub Actions Inactive

fix: add migration to clean up duplicate backup folders with batching

ea983ee

jzunigax2 force-pushed the fix/cleanup-duplicate-folders branch from ccf78bc to ea983ee Compare January 22, 2026 03:02

jzunigax2 temporarily deployed to development January 22, 2026 03:02 — with GitHub Actions Inactive

sg-gs requested changes Jan 22, 2026

View reviewed changes

jzunigax2 commented Jan 22, 2026

View reviewed changes

fix: update migration to refine duplicate backup folder cleanup logic…

189fb5f

… and enhance test coverage for folder creation

jzunigax2 temporarily deployed to development January 22, 2026 16:25 — with GitHub Actions Inactive

fix: refine cleanup logic in migration to exclude removed folders fro…

84025d3

…m duplicate backup folder deletion

jzunigax2 temporarily deployed to development January 22, 2026 16:28 — with GitHub Actions Inactive

jzunigax2 requested a review from sg-gs January 22, 2026 16:30

github-actions bot added stalled and removed stalled labels Feb 7, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PB-5670] fix/add migration to clean up duplicate backup folders with batching #852

[PB-5670] fix/add migration to clean up duplicate backup folders with batching #852

Uh oh!

jzunigax2 commented Jan 8, 2026

Uh oh!

jzunigax2 commented Jan 8, 2026

Uh oh!

sg-gs left a comment

Uh oh!

sg-gs Jan 22, 2026

Uh oh!

jzunigax2 Jan 22, 2026

Uh oh!

Uh oh!

Uh oh!

jzunigax2 Jan 22, 2026

Uh oh!

jzunigax2 Jan 22, 2026

Uh oh!

sonarqubecloud bot commented Jan 22, 2026

Uh oh!

github-actions bot commented Feb 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[PB-5670] fix/add migration to clean up duplicate backup folders with batching #852

Are you sure you want to change the base?

[PB-5670] fix/add migration to clean up duplicate backup folders with batching #852

Uh oh!

Conversation

jzunigax2 commented Jan 8, 2026

Uh oh!

jzunigax2 commented Jan 8, 2026

Uh oh!

sg-gs left a comment

Choose a reason for hiding this comment

Uh oh!

sg-gs Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

jzunigax2 Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jzunigax2 Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

jzunigax2 Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Jan 22, 2026

Quality Gate passed

Uh oh!

github-actions bot commented Feb 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants