Skip to content

Commit

Permalink
add helpers.if_file_exists to publish_utils
Browse files Browse the repository at this point in the history
  • Loading branch information
tiffanychu90 committed Aug 14, 2024
1 parent e27c815 commit a7d93ea
Show file tree
Hide file tree
Showing 4 changed files with 1,610 additions and 80 deletions.
15 changes: 15 additions & 0 deletions _shared_utils/shared_utils/publish_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,18 @@ def write_to_public_gcs(
os.remove(local_filename)

return


def if_exists_then_delete(filepath: str):
"""
Check if file exists in GCS and delete.
For partitioned parquets, which are saved as folders, we need
to use recursive=True.
"""
if fs.exists(filepath):
if fs.isdir(filepath):
fs.rm(filepath, recursive=True)
else:
fs.rm(filepath)

return
Loading

0 comments on commit a7d93ea

Please sign in to comment.