Tips/Python Client for GCS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3c44850b",
   "metadata": {},
   "source": [
    "![tracker](https://us-central1-vertex-ai-mlops-369716.cloudfunctions.net/pixel-tracking?path=statmike%2Fvertex-ai-mlops%2FTips&file=Python+Client+for+GCS.ipynb)\n",
    "<!--- header table --->\n",
    "<table align=\"left\">\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://colab.research.google.com/github/statmike/vertex-ai-mlops/blob/main/Tips/Python%20Client%20for%20GCS.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\">\n",
    "      <br>Run in<br>Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https%3A%2F%2Fraw.githubusercontent.com%2Fstatmike%2Fvertex-ai-mlops%2Fmain%2FTips%2FPython%2520Client%2520for%2520GCS.ipynb\">\n",
    "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\">\n",
    "      <br>Run in<br>Colab Enterprise\n",
    "    </a>\n",
    "  </td>      \n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://github.com/statmike/vertex-ai-mlops/blob/main/Tips/Python%20Client%20for%20GCS.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      <br>View on<br>GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/statmike/vertex-ai-mlops/main/Tips/Python%20Client%20for%20GCS.ipynb\">\n",
    "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
    "      <br>Open in<br>Vertex AI Workbench\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6bab0ab-aced-42e1-81cc-897bf7d6c09c",
   "metadata": {},
   "source": [
    "# Python Client for GCS\n",
    "\n",
    "Google Cloud Storage (GCS), buckets to hold information, data, files.\n",
    "\n",
    "This notebook illustrates some common interactions with GCS and provides tips on using the Python Client to:\n",
    "- List buckets\n",
    "- List files\n",
    "- create buckets\n",
    "- delete buckets\n",
    "- Download files\n",
    "- Upload files\n",
    "- more\n",
    "\n",
    "Resources:\n",
    "- [Product](https://cloud.google.com/storage)\n",
    "- [Client API](https://github.com/googleapis/python-storage)\n",
    "- [Client API Documentation](https://cloud.google.com/python/docs/reference/storage/latest)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2056c123-d8a6-46da-b990-40ef067e73a4",
   "metadata": {},
   "source": [
    "---\n",
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c60f85d2-3250-4c94-8b7a-0a7b46ffaeb3",
   "metadata": {},
   "source": [
    "inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0af04da6-d5b5-4a32-904f-cb8531fa5ba3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'statmike-mlops-349915'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project = !gcloud config get-value project\n",
    "PROJECT_ID = project[0]\n",
    "PROJECT_ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d3237c8c-e220-4db9-9af0-e11f09b58754",
   "metadata": {},
   "outputs": [],
   "source": [
    "REGION = 'us-central1'\n",
    "EXPERIMENT = 'gcs'\n",
    "SERIES = 'tips'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be41cb1d-181b-4558-84d8-dae9c74fb35b",
   "metadata": {},
   "source": [
    "packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3bdba16a-f4a1-4b10-856f-4141389b3cf4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.cloud import storage\n",
    "import os, shutil\n",
    "import glob\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a85f74e-3ebe-4a6c-96db-fef7b9caa321",
   "metadata": {},
   "source": [
    "clients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3e3271c4-dbe8-406b-8c3f-4e753a655c56",
   "metadata": {},
   "outputs": [],
   "source": [
    "gcs = storage.Client()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "379632d0-cbfa-4d02-88b1-32f31f573075",
   "metadata": {},
   "source": [
    "parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7376d6f5-beb9-4f08-be0d-5fbc42c588d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "DIR = f'temp/{EXPERIMENT}'\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4354c29b-8263-4d25-bee5-639c7f791633",
   "metadata": {},
   "source": [
    "environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "431f84d7-c7ba-4bc0-a67b-ac524234660d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DIR exists?  True\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['job-parms', 'gcs', 'multiprocess']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove directory named DIR if exists\n",
    "shutil.rmtree(DIR, ignore_errors = True)\n",
    "\n",
    "# create directory DIR\n",
    "os.makedirs(DIR)\n",
    "\n",
    "# check for existance of DIR\n",
    "print('DIR exists? ', os.path.exists(DIR))\n",
    "\n",
    "# list contents of directory one level higher than DIR\n",
    "os.listdir(DIR + '/../')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53280796-6b16-4653-a658-f22eb1822efd",
   "metadata": {},
   "source": [
    "---\n",
    "## Buckets\n",
    "\n",
    "Buckets are the actual storage locations of files.  They have resources in projects, have properties and permissions, and are located in a region."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "33d72c0d-1785-45ba-bb83-1547fe4a00e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the projects buckets directly here:\n",
      "https://console.cloud.google.com/storage/browser?forceOnBucketsSortingFiltering=false&project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f\"View the projects buckets directly here:\\nhttps://console.cloud.google.com/storage/browser?forceOnBucketsSortingFiltering=false&project={PROJECT_ID}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "608421b4-60c9-45c3-8158-d2dc5a733758",
   "metadata": {},
   "source": [
    "### Is there a bucket in this project with the same names as the project?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "92bc0ee7-cbe1-44e3-9560-11a3e84d73bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'statmike-mlops-349915'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "PROJECT_ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c6c83dd9-a8ff-4615-aa55-778e73ed566e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "google.cloud.storage.bucket.Bucket"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lookup = gcs.lookup_bucket(PROJECT_ID)\n",
    "type(lookup)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "22e30f78-5733-45cc-aa81-be4fdf13ce61",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NoneType"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lookup = gcs.lookup_bucket(PROJECT_ID+'not_real_name')\n",
    "type(lookup)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7e84623-a549-49e3-bff8-a5fba1ace9af",
   "metadata": {},
   "source": [
    "### List the buckets in the current project:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "af75a737-52c6-4347-a2ab-1c2c76119d03",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Bucket: cloud-ai-platform-a68e7f3a-fac8-47f6-9f92-fff95c09cdb8>,\n",
       " <Bucket: statmike-mlops-349915>,\n",
       " <Bucket: statmike-mlops-349915-vertex-pipelines-us-central1>]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(gcs.list_buckets())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1fbeccfa-f175-4425-bdee-9186541b15e5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Bucket: statmike-mlops-349915>,\n",
       " <Bucket: statmike-mlops-349915-vertex-pipelines-us-central1>]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(gcs.list_buckets(prefix = 'statmike'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c557a0bd-8214-4b4d-a153-f7c8fd4ae4dc",
   "metadata": {},
   "source": [
    "### Create a new bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "fecef6e8-3557-48ca-9997-a7bba81d8f95",
   "metadata": {},
   "outputs": [],
   "source": [
    "bucket = gcs.bucket(PROJECT_ID + TIMESTAMP)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bd604927-aad6-4cb8-8bc9-85a3e3422a98",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Bucket: statmike-mlops-34991520220920104702>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bucket = gcs.create_bucket(bucket, project = PROJECT_ID, location = REGION)\n",
    "bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "be2c35a3-b328-4ae3-81fb-1f01d55ebe45",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Bucket: statmike-mlops-349915>,\n",
       " <Bucket: statmike-mlops-349915-vertex-pipelines-us-central1>,\n",
       " <Bucket: statmike-mlops-34991520220920104702>]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(gcs.list_buckets(prefix = 'statmike'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "e737527d-1004-4152-aa5f-1a282da026c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the bucket directly here:\n",
      "https://console.cloud.google.com/storage/browser/statmike-mlops-34991520220920104702;tab=objects&project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f\"View the bucket directly here:\\nhttps://console.cloud.google.com/storage/browser/{PROJECT_ID + TIMESTAMP};tab=objects&project={PROJECT_ID}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "074b0455-9441-488b-bd89-e811aaf1103a",
   "metadata": {},
   "source": [
    "### Retrieve a specific bucket:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "84d1cdb6-fe49-4c04-8644-39c6f6974e59",
   "metadata": {},
   "outputs": [],
   "source": [
    "bucket = gcs.bucket(PROJECT_ID + TIMESTAMP)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "33a0de9f-2b50-4563-9b51-793a11d7c488",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Bucket: statmike-mlops-34991520220920104702>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "fae9101d-a05c-465b-ac8b-0733085bdbef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'statmike-mlops-34991520220920104702'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bucket.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7bef12d5-46f5-4d8d-9071-908e3b9e6f06",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/b/statmike-mlops-34991520220920104702'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bucket.path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6181a78e-6638-434a-978e-a0b9a9e3137f",
   "metadata": {},
   "source": [
    "## Files (blobs)\n",
    "\n",
    "Files are objects, called blobs.  The name includes and prefix you want to use to represent a folder structure.  That's right, there are no actual folders in object storage.  Just files name prefixed with folder like names to help organize and find information.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17240f3c-211b-419e-aefc-9dd4a36358f3",
   "metadata": {},
   "source": [
    "### Make some local files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "8f4c5b2c-a218-4a7c-95d6-9bca74313b6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_folders = 3\n",
    "n_files = 100\n",
    "\n",
    "for folder in range(n_folders):\n",
    "    if not os.path.exists(f'./{DIR}/folder_{folder}'): os.mkdir(f'./{DIR}/folder_{folder}')\n",
    "    for f in range(n_files):\n",
    "        with open(f'./{DIR}/folder_{folder}/myfile_{f}.txt', 'w') as file:\n",
    "            file.write(f'Creating the example file named: myfile_{f}.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "cd983d14-3c2b-4a7d-97cf-4fe5c1c33838",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['folder_1', 'folder_2', 'folder_0']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(f'./{DIR}')[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "0460b404-e2e5-4179-9d69-74f00ee0ceac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['myfile_95.txt',\n",
       " 'myfile_93.txt',\n",
       " 'myfile_65.txt',\n",
       " 'myfile_98.txt',\n",
       " 'myfile_14.txt',\n",
       " 'myfile_10.txt',\n",
       " 'myfile_49.txt',\n",
       " 'myfile_38.txt',\n",
       " 'myfile_90.txt',\n",
       " 'myfile_25.txt']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(f'./{DIR}/folder_0')[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bde97427-c17a-4464-b5a8-031254155fc6",
   "metadata": {},
   "source": [
    "### Uploading Files to Bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80343896-6ca0-4761-9939-9027072e61a5",
   "metadata": {},
   "source": [
    "Get a list of files in the local folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "8c090deb-d72b-4151-9d00-9d28c362e7f1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['./temp/gcs/folder_1/myfile_95.txt',\n",
       " './temp/gcs/folder_1/myfile_93.txt',\n",
       " './temp/gcs/folder_1/myfile_65.txt',\n",
       " './temp/gcs/folder_1/myfile_98.txt',\n",
       " './temp/gcs/folder_1/myfile_14.txt']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "glob.glob(f'./{DIR}/**/**')[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "96983d88-09e0-4229-a21d-f62814fcc109",
   "metadata": {},
   "outputs": [],
   "source": [
    "gcs_path_prefix = 'my_folder/my_subfolder/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "34eb6434-d260-4b91-9163-243b75d9436d",
   "metadata": {},
   "outputs": [],
   "source": [
    "for file in glob.glob(f'./{DIR}/**/**'):\n",
    "    file_path = ('/').join(file.split('/')[-2:]) # just the subfolder and filename\n",
    "    blob = bucket.blob(gcs_path_prefix + file_path)\n",
    "    blob.upload_from_filename(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "8c691076-634f-43af-9e23-b9c62303d089",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the bucket directly here:\n",
      "https://console.cloud.google.com/storage/browser/statmike-mlops-34991520220920104702;tab=objects&project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f\"View the bucket directly here:\\nhttps://console.cloud.google.com/storage/browser/{PROJECT_ID + TIMESTAMP};tab=objects&project={PROJECT_ID}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "432a8a2c-90c6-4298-87b6-07817f185261",
   "metadata": {},
   "source": [
    "### List Files in Bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "f829b2b5-b9ee-462a-ab26-f1c46a97f8c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Blob: statmike-mlops-34991520220920104702, my_folder/my_subfolder/folder_0/myfile_0.txt, 1663670925276172>,\n",
       " <Blob: statmike-mlops-34991520220920104702, my_folder/my_subfolder/folder_0/myfile_1.txt, 1663670923166027>,\n",
       " <Blob: statmike-mlops-34991520220920104702, my_folder/my_subfolder/folder_0/myfile_10.txt, 1663670922513872>,\n",
       " <Blob: statmike-mlops-34991520220920104702, my_folder/my_subfolder/folder_0/myfile_11.txt, 1663670923347570>,\n",
       " <Blob: statmike-mlops-34991520220920104702, my_folder/my_subfolder/folder_0/myfile_12.txt, 1663670923106588>]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(bucket.list_blobs(max_results = 5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "08a909d4-22d6-4eda-903e-9110094a7801",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Blob: statmike-mlops-34991520220920104702, my_folder/my_subfolder/folder_1/myfile_0.txt, 1663670906045754>,\n",
       " <Blob: statmike-mlops-34991520220920104702, my_folder/my_subfolder/folder_1/myfile_1.txt, 1663670902233487>,\n",
       " <Blob: statmike-mlops-34991520220920104702, my_folder/my_subfolder/folder_1/myfile_10.txt, 1663670901064714>,\n",
       " <Blob: statmike-mlops-34991520220920104702, my_folder/my_subfolder/folder_1/myfile_11.txt, 1663670902565371>,\n",
       " <Blob: statmike-mlops-34991520220920104702, my_folder/my_subfolder/folder_1/myfile_12.txt, 1663670902144210>]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(bucket.list_blobs(max_results = 5, prefix = gcs_path_prefix + 'folder_1/'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3edc5f5-a0af-4103-a164-c3ade25d2ac6",
   "metadata": {},
   "source": [
    "### Downloading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "4b0b0b83-0017-4eea-9909-bb2b5f4a00ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.makedirs(f'./{DIR}/downloaded')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "cfabf115-3e56-40f7-bffa-a90abadbfc5c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['downloaded', 'folder_1', 'folder_2', 'folder_0']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(f'./{DIR}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "790b3768-2070-42e7-a498-bf4db8a7b6ca",
   "metadata": {},
   "source": [
    "download files from one subfolder on GCS to a local folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "541244d1-0acc-44e0-b10d-37e81f19c6db",
   "metadata": {},
   "outputs": [],
   "source": [
    "for blob in bucket.list_blobs(prefix = gcs_path_prefix + 'folder_0/', delimiter = '/'):\n",
    "    blob.download_to_filename(f\"./{DIR}/downloaded/{blob.name.split('/')[-1]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "ddda75df-487a-44d3-81bb-0de0133e2b0d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['myfile_95.txt',\n",
       " 'myfile_93.txt',\n",
       " 'myfile_65.txt',\n",
       " 'myfile_98.txt',\n",
       " 'myfile_14.txt',\n",
       " 'myfile_10.txt',\n",
       " 'myfile_49.txt',\n",
       " 'myfile_38.txt',\n",
       " 'myfile_90.txt',\n",
       " 'myfile_25.txt']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(f'./{DIR}/downloaded')[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb90003e-441d-49f8-aae9-e2271224594f",
   "metadata": {},
   "source": [
    "## Buckets, Again"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af8ae7b5-1580-4a5f-bc15-f404a0252ab5",
   "metadata": {},
   "source": [
    "### Delete The Bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "158e983e-50da-4fa0-b115-c8e3e45956c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Bucket: statmike-mlops-34991520220920104702>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "bbba796d-e36e-43c8-960c-8f51b6418866",
   "metadata": {},
   "outputs": [
    {
     "ename": "Conflict",
     "evalue": "409 DELETE https://storage.googleapis.com/storage/v1/b/statmike-mlops-34991520220920104702?prettyPrint=false: The bucket you tried to delete is not empty.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mConflict\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_27432/523633142.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mbucket\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdelete\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;31m# results in error: Conflict: 409 DELETE https://path: The bucket you tried to delete is not empty.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/google/cloud/storage/bucket.py\u001b[0m in \u001b[0;36mdelete\u001b[0;34m(self, force, client, if_metageneration_match, if_metageneration_not_match, timeout, retry)\u001b[0m\n\u001b[1;32m   1514\u001b[0m             \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1515\u001b[0m             \u001b[0mretry\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mretry\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1516\u001b[0;31m             \u001b[0m_target_object\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1517\u001b[0m         )\n\u001b[1;32m   1518\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/google/cloud/storage/client.py\u001b[0m in \u001b[0;36m_delete_resource\u001b[0;34m(self, path, query_params, headers, timeout, retry, _target_object)\u001b[0m\n\u001b[1;32m    660\u001b[0m             \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    661\u001b[0m             \u001b[0mretry\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mretry\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 662\u001b[0;31m             \u001b[0m_target_object\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0m_target_object\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    663\u001b[0m         )\n\u001b[1;32m    664\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/google/cloud/storage/_http.py\u001b[0m in \u001b[0;36mapi_request\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     78\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mretry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     79\u001b[0m                 \u001b[0mcall\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mretry\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcall\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 80\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mcall\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/google/api_core/retry.py\u001b[0m in \u001b[0;36mretry_wrapped_func\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    286\u001b[0m                 \u001b[0msleep_generator\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    287\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_deadline\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 288\u001b[0;31m                 \u001b[0mon_error\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mon_error\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    289\u001b[0m             )\n\u001b[1;32m    290\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/google/api_core/retry.py\u001b[0m in \u001b[0;36mretry_target\u001b[0;34m(target, predicate, sleep_generator, deadline, on_error)\u001b[0m\n\u001b[1;32m    188\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0msleep\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msleep_generator\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    189\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 190\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mtarget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    191\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    192\u001b[0m         \u001b[0;31m# pylint: disable=broad-except\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/google/cloud/_http/__init__.py\u001b[0m in \u001b[0;36mapi_request\u001b[0;34m(self, method, path, query_params, data, content_type, headers, api_base_url, api_version, expect_json, _target_object, timeout, extra_api_info)\u001b[0m\n\u001b[1;32m    492\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    493\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;36m200\u001b[0m \u001b[0;34m<=\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstatus_code\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0;36m300\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 494\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrom_http_response\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresponse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    495\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    496\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mexpect_json\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mConflict\u001b[0m: 409 DELETE https://storage.googleapis.com/storage/v1/b/statmike-mlops-34991520220920104702?prettyPrint=false: The bucket you tried to delete is not empty."
     ]
    }
   ],
   "source": [
    "bucket.delete()\n",
    "# results in error: Conflict: 409 DELETE https://path: The bucket you tried to delete is not empty."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "3067f619-50d2-4771-ab0b-879412551377",
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Refusing to delete bucket with more than 256 objects. If you actually want to delete this bucket, please delete the objects yourself before calling Bucket.delete().",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_27432/917981759.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mbucket\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdelete\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mforce\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;31m# results in error: ValueError: Refusing to delete bucket with more than 256 objects. If you actually want to delete this bucket, please delete the objects yourself before calling Bucket.delete().\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/google/cloud/storage/bucket.py\u001b[0m in \u001b[0;36mdelete\u001b[0;34m(self, force, client, if_metageneration_match, if_metageneration_not_match, timeout, retry)\u001b[0m\n\u001b[1;32m   1495\u001b[0m                     \u001b[0;34m\"yourself before calling Bucket.delete().\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1496\u001b[0m                 ) % (self._MAX_OBJECTS_FOR_ITERATION,)\n\u001b[0;32m-> 1497\u001b[0;31m                 \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1498\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1499\u001b[0m             \u001b[0;31m# Ignore 404 errors on delete.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Refusing to delete bucket with more than 256 objects. If you actually want to delete this bucket, please delete the objects yourself before calling Bucket.delete()."
     ]
    }
   ],
   "source": [
    "bucket.delete(force = True)\n",
    "# results in error: ValueError: Refusing to delete bucket with more than 256 objects. If you actually want to delete this bucket, please delete the objects yourself before calling Bucket.delete()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "df24046b-4b88-4cd2-a63f-cf4726b52df6",
   "metadata": {},
   "outputs": [],
   "source": [
    "bucket.delete_blobs(blobs = list(bucket.list_blobs()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "524b860a-f5f0-4116-af95-d8b4d1595021",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Bucket: cloud-ai-platform-a68e7f3a-fac8-47f6-9f92-fff95c09cdb8>,\n",
       " <Bucket: statmike-mlops-349915>,\n",
       " <Bucket: statmike-mlops-349915-vertex-pipelines-us-central1>,\n",
       " <Bucket: statmike-mlops-34991520220920104702>]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(gcs.list_buckets())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "1a6ae1cd-e849-4ee4-956a-16347626afbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "bucket.delete()\n",
    "# works, because the bucket is empty"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "f8943baa-6063-400e-b1ca-381e262ec4a1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Bucket: cloud-ai-platform-a68e7f3a-fac8-47f6-9f92-fff95c09cdb8>,\n",
       " <Bucket: statmike-mlops-349915>,\n",
       " <Bucket: statmike-mlops-349915-vertex-pipelines-us-central1>]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(gcs.list_buckets())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "d72d41bd-a0ef-419b-be30-01cac2fb4297",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the projects buckets directly here:\n",
      "https://console.cloud.google.com/storage/browser?forceOnBucketsSortingFiltering=false&project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f\"View the projects buckets directly here:\\nhttps://console.cloud.google.com/storage/browser?forceOnBucketsSortingFiltering=false&project={PROJECT_ID}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20f6fa48-8fcf-434f-93a6-152a5d0d5ce8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-3.m94",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-3:m94"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}