Tips/Python Job Parameters.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "16b1b683",
   "metadata": {},
   "source": [
    "![tracker](https://us-central1-vertex-ai-mlops-369716.cloudfunctions.net/pixel-tracking?path=statmike%2Fvertex-ai-mlops%2FTips&file=Python+Job+Parameters.ipynb)\n",
    "<!--- header table --->\n",
    "<table align=\"left\">\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://colab.research.google.com/github/statmike/vertex-ai-mlops/blob/main/Tips/Python%20Job%20Parameters.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\">\n",
    "      <br>Run in<br>Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https%3A%2F%2Fraw.githubusercontent.com%2Fstatmike%2Fvertex-ai-mlops%2Fmain%2FTips%2FPython%2520Job%2520Parameters.ipynb\">\n",
    "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\">\n",
    "      <br>Run in<br>Colab Enterprise\n",
    "    </a>\n",
    "  </td>      \n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://github.com/statmike/vertex-ai-mlops/blob/main/Tips/Python%20Job%20Parameters.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      <br>View on<br>GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/statmike/vertex-ai-mlops/main/Tips/Python%20Job%20Parameters.ipynb\">\n",
    "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
    "      <br>Open in<br>Vertex AI Workbench\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d52f4436-85f7-49c8-8671-cc78cae3ec58",
   "metadata": {},
   "source": [
    "# Python Job Parameters\n",
    "\n",
    "Run your Python job `python script.py` ...\n",
    "\n",
    "But, it needs inputs, or as we say parameters ...\n",
    "\n",
    "How do we feed those parameters?\n",
    "\n",
    "Two main directions\n",
    "1. on the command line\n",
    "2. import within the job\n",
    "3. BONUS: both!\n",
    "    \n",
    "This notebook gives examples of getting parameters into Python Jobs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5cf5e14-a803-4422-bdc5-a6030f14deb4",
   "metadata": {},
   "source": [
    "---\n",
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0b3b713-e710-4d54-95f3-f54214cde4f5",
   "metadata": {},
   "source": [
    "inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0340118d-16f1-4304-9f34-1fd3b167e07b",
   "metadata": {},
   "outputs": [],
   "source": [
    "EXPERIMENT = 'job-parms'\n",
    "SERIES = 'tips'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d765d007-3681-4a28-80dd-521af8bc86cf",
   "metadata": {},
   "source": [
    "parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d0d08c80-1bdb-4863-a80e-4ec20a6a2da8",
   "metadata": {},
   "outputs": [],
   "source": [
    "DIR = f'temp/{EXPERIMENT}'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c84187ec-1ce9-4e7a-b71d-eda0a891451d",
   "metadata": {},
   "source": [
    "environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "94c6df0d-84c1-4b3f-aa42-99bdc00ea8fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf {DIR}\n",
    "!mkdir -p {DIR}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f2f7ce6-55f6-41f8-92dd-af71e60f31d8",
   "metadata": {},
   "source": [
    "---\n",
    "## Example Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "667908ea-fdc6-43ef-852e-02946b3e0835",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing ./temp/job-parms/example.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./{DIR}/example.py\n",
    "\n",
    "operation = 'add'\n",
    "a = 2\n",
    "b = 2.5\n",
    "c = 10\n",
    "\n",
    "def add(a, b, c):\n",
    "    print(f'The value is: {a + b + c}')\n",
    "    return a + b + c\n",
    "\n",
    "def multiply(a, b, c):\n",
    "    print(f'The value is: {a * b * c}')\n",
    "    return a * b * c\n",
    "\n",
    "if operation == 'add': add(a, b, c)\n",
    "elif operation == 'multiply': multiply(a, b, c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1ba1fd1d-08d7-420d-b690-283d81fc2223",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 14.5\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b39ca86f-1d29-46a6-b174-7ecebc611bb8",
   "metadata": {},
   "source": [
    "---\n",
    "## Command Line Parameters\n",
    "\n",
    "These come in several flavors:\n",
    "- positional\n",
    "    - optional\n",
    "    - required\n",
    "- flags\n",
    "    - optional\n",
    "    - required\n",
    "\n",
    "**Goal For Examples Below:**\n",
    "\n",
    "Modify the Example Job above to accept the four parameters from the command line:\n",
    "- operation - make positional and optional, (default = add)\n",
    "- a - make positional and required, int\n",
    "- b - make a flag that is requires, float\n",
    "- c - make a flag that is optional, float (default = 10) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e22de72-af69-4fb3-88e8-223d2eea8f3a",
   "metadata": {},
   "source": [
    "### Library: Argparse\n",
    "\n",
    "- Use: `import argparse`\n",
    "- [Reference](https://docs.python.org/3/library/argparse.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "67697030-7f83-4111-b499-b99b126d2887",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./temp/job-parms/example.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./{DIR}/example.py\n",
    "import argparse\n",
    "\n",
    "parser = argparse.ArgumentParser()\n",
    "parser.add_argument(dest = 'operation', nargs = '?', default = 'add') # optional positional\n",
    "parser.add_argument(dest = 'a', type = int) # required positional\n",
    "parser.add_argument('--b', dest = 'b', type = float) # required flag\n",
    "parser.add_argument('--c', dest = 'c', type = float, default = 10) # optional flag\n",
    "args = parser.parse_args()\n",
    "\n",
    "operation = args.operation\n",
    "a = args.a\n",
    "b = args.b\n",
    "c = args.c\n",
    "\n",
    "def add(a, b, c):\n",
    "    print(f'The value is: {a + b + c}')\n",
    "    return a + b + c\n",
    "\n",
    "def multiply(a, b, c):\n",
    "    print(f'The value is: {a * b * c}')\n",
    "    return a * b * c\n",
    "\n",
    "if operation == 'add': add(a, b, c)\n",
    "elif operation == 'multiply': multiply(a, b, c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "efb7e09d-e627-48d6-b805-3e7b92a42c4b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 14.5\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py 2 --b=2.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8be56da0-2438-40bf-bbbf-00f4018b378c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 14.5\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py --b=2.5 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "020a1168-9973-4934-9974-14a8440132fc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 14.5\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py add 2 --b=2.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "914b8a4f-195d-495f-90bf-0cb7bc0c6f78",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 14.5\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py --b=2.5 add 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "48f5fbe8-e2a5-4651-9fc8-3704d6c59ed8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 50.0\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py multiply 2 --b=2.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7a955cf9-34e3-4e8e-98c6-339c2125a2ff",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 5.0\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py multiply 2 --b=2.5 --c=1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7c3107b-82d2-4a08-96c2-f15593bbdd93",
   "metadata": {},
   "source": [
    "**Notes:**\n",
    "\n",
    "- The positional parameters should all go before, or after, any flag parameters.  Just don't split them between before and after."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e7c2437-ab24-4d8f-a011-24397cf9c848",
   "metadata": {},
   "source": [
    "### Library: Docopt\n",
    "\n",
    "- Use: `import docopt`\n",
    "- [Reference](https://github.com/docopt/docopt)\n",
    "- [docopt.org](http://docopt.org/)\n",
    "\n",
    "Notes:\n",
    " - uses top of file to specify the usage of command line parameters\n",
    " - `[]` is an optional parameter\n",
    " - `()` is a required parameter\n",
    " - name that are upper-case or start+end with `<name>` are positional parameters\n",
    " - options can be used to desribe parameters\n",
    "     - a description can follow the option but needs two spaces to deliniate\n",
    "     - the desription can contain a default value, helpful for optional paramters, in the format [default: value-here]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "45d69b0b-1c04-4240-97c0-93f909ddd2b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./temp/job-parms/example.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./{DIR}/example.py\n",
    "\"\"\"\n",
    "usage:\n",
    "    example.py OPERATION (A) (--b=B) [--c=C]\n",
    "    example.py (A) (--b=B) [--c=C]\n",
    "\n",
    "options:\n",
    "    --b=B\n",
    "    --c=C  Has value [default: 10]\n",
    "\"\"\"\n",
    "from docopt import docopt\n",
    "\n",
    "def main(args):\n",
    "    \n",
    "    if args['OPERATION']:\n",
    "        operation = args['OPERATION']\n",
    "    else: operation = 'add'\n",
    "    a = int(args['A'])\n",
    "    b = float(args['--b'])\n",
    "    c = float(args['--c'])\n",
    "    \n",
    "    def add(a, b, c):\n",
    "        print(f'The value is: {a + b + c}')\n",
    "        return a + b + c\n",
    "\n",
    "    def multiply(a, b, c):\n",
    "        print(f'The value is: {a * b * c}')\n",
    "        return a * b * c\n",
    "\n",
    "    if operation == 'add': add(a, b, c)\n",
    "    elif operation == 'multiply': multiply(a, b, c)\n",
    "    \n",
    "if __name__ == '__main__':\n",
    "    args = docopt(__doc__)\n",
    "    #print(args)\n",
    "    main(args)   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bcb45493-d50e-4d23-be80-87d8bf4c2dc2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 14.5\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py 2 --b=2.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "1ae2e70f-46ae-4c14-8203-0508a7a6e5b7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 14.5\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py --b=2.5 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "690ce854-7a42-4a99-a0ce-9d9986b1de09",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 14.5\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py add 2 --b=2.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "27d69264-10ab-41c3-abed-13be2ac4a14f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 14.5\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py --b=2.5 add 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d20a561f-785c-4137-814a-40b1a52fbaa3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 50.0\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py multiply 2 --b=2.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1dea4b70-94d1-4c2f-99b8-0cfcdbdbb0ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 5.0\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py multiply 2 --b=2.5 --c=1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccea348e-1f61-4abd-8757-ae91f94d9ca4",
   "metadata": {},
   "source": [
    "### Library: Click\n",
    "\n",
    "- Use: `import click`\n",
    "- [Reference](https://palletsprojects.com/p/click/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "9ac0fe65-23d6-4cb7-b6d7-a6b19c5d978a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./temp/job-parms/example.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./{DIR}/example.py\n",
    "import click\n",
    "\n",
    "@click.command()\n",
    "@click.option('--operation', default = 'add', type = click.STRING)\n",
    "@click.argument('a', type = click.INT)\n",
    "@click.option(\"--b\", type = click.FLOAT)\n",
    "@click.option(\"--c\", default = 10, type = click.FLOAT)\n",
    "\n",
    "def main(operation, a, b, c):\n",
    "    \n",
    "    def add(a, b, c):\n",
    "        print(f'The value is: {a + b + c}')\n",
    "        return a + b + c\n",
    "\n",
    "    def multiply(a, b, c):\n",
    "        print(f'The value is: {a * b * c}')\n",
    "        return a * b * c\n",
    "\n",
    "    if operation == 'add': add(a, b, c)\n",
    "    elif operation == 'multiply': multiply(a, b, c)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "f59a650c-bd10-48c4-8cb3-b408a8ff6013",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 14.5\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py 2 --b=2.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a5028ad2-97c5-4eef-8d6d-26af118ab612",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 14.5\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py --b=2.5 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "964cb76f-49c5-464b-b074-10c9a852bae9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 14.5\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py --operation=add 2 --b=2.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7cfd6dfe-f4e1-4033-b79b-b1dcb0d3459c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 14.5\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py --b=2.5 --operation=add 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "a4a0550b-c270-44ba-af0a-237ef7b171a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 50.0\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py --operation=multiply 2 --b=2.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "11f0c745-e63f-4976-8faf-01f3bc733393",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 5.0\n"
     ]
    }
   ],
   "source": [
    "!python ./{DIR}/example.py --operation=multiply 2 --b=2.5 --c=1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cae9ed2-d997-47c4-840a-198c38142516",
   "metadata": {},
   "source": [
    "---\n",
    "## Parameter Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "85a48614-20fa-4709-9194-7f2b5b4390c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters = {\n",
    "    'operation': 'add',\n",
    "    'a': 2,\n",
    "    'b': 2.5,\n",
    "    'c': 10 \n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "117a99a7-8bc3-4898-8f77-33d4a514587b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'operation': 'add', 'a': 2, 'b': 2.5, 'c': 10}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc13efef-e492-4529-b7c0-e0ad16823404",
   "metadata": {},
   "source": [
    "### JSON"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b3bcfc9-98ce-401c-8625-2aa77b84925c",
   "metadata": {},
   "source": [
    "output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "0ca88b67-d2a4-4c7e-a972-9cc4fb109412",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "with open(f'./{DIR}/example.json', 'w') as file:\n",
    "    json.dump(parameters, file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4a0bd0b-d421-4684-9ca3-4a170efc26d6",
   "metadata": {},
   "source": [
    "input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "05fc39a2-1779-4c42-9149-247605211825",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "with open(f'./{DIR}/example.json', 'r') as file:\n",
    "    data = json.load(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f36dcbc-bc27-4f36-a508-145adf25442d",
   "metadata": {},
   "source": [
    "show file contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "9b646510-f96d-444f-8e23-c05ac723fc70",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "```json\n",
       "\n",
       "{\"operation\": \"add\", \"a\": 2, \"b\": 2.5, \"c\": 10}\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Markdown\n",
    "\n",
    "with open(f'./{DIR}/example.json', 'r') as file:\n",
    "    raw_data = file.read()\n",
    "Markdown(f\"```json\\n\\n{raw_data}\\n```\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6dd40999-86f0-4969-ba3d-9f5ff7e9b6c0",
   "metadata": {},
   "source": [
    "create job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "c7e14503-c141-4081-8ffc-2df06330e7f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./temp/job-parms/example.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./{DIR}/example.py\n",
    "import json\n",
    "\n",
    "parameters = json.load(open('example.json', 'r'))\n",
    "operation = parameters['operation']\n",
    "a = parameters['a']\n",
    "b = parameters['b']\n",
    "c = parameters['c']\n",
    "\n",
    "def add(a, b, c):\n",
    "    print(f'The value is: {a + b + c}')\n",
    "    return a + b + c\n",
    "\n",
    "def multiply(a, b, c):\n",
    "    print(f'The value is: {a * b * c}')\n",
    "    return a * b * c\n",
    "\n",
    "if operation == 'add': add(a, b, c)\n",
    "elif operation == 'multiply': multiply(a, b, c)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07c0a391-d626-4adf-b39b-bf4c71e66de5",
   "metadata": {},
   "source": [
    "execute the script in the folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "d6503dae-efa8-4589-aa30-420d65fded11",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "example.json  example.py\n"
     ]
    }
   ],
   "source": [
    "!ls ./{DIR}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "df543afc-6a49-4b72-8d35-acc8a3806089",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 14.5\n"
     ]
    }
   ],
   "source": [
    "!cd ./{DIR} && python example.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "037f384b-b21d-4d3d-99c4-7db398e1534b",
   "metadata": {},
   "source": [
    "### YAML"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "867aee37-432a-4440-be55-b85dbe2420dd",
   "metadata": {},
   "source": [
    "output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "77ed79b1-bb11-4d3f-96cf-c5113acdc926",
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "\n",
    "with open(f'./{DIR}/example.yml', 'w') as file:\n",
    "    yaml.dump(parameters, file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95f5df85-be0e-450c-ba76-7cfcaf55d004",
   "metadata": {},
   "source": [
    "input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "c4963c84-25b3-4a80-8f07-e14a35aee48e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "\n",
    "with open(f'./{DIR}/example.yml', 'r') as file:\n",
    "    data = yaml.full_load(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b0641c3-2398-486f-90c6-c8722932104b",
   "metadata": {},
   "source": [
    "show file contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "3bbd3280-2689-4c56-914d-3d23d908bffa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "```json\n",
       "\n",
       "a: 2\n",
       "b: 2.5\n",
       "c: 10\n",
       "operation: add\n",
       "\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Markdown\n",
    "\n",
    "with open(f'./{DIR}/example.yml', 'r') as file:\n",
    "    raw_data = file.read()\n",
    "Markdown(f\"```json\\n\\n{raw_data}\\n```\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "077ff86f-be29-468d-aba2-27514f945ea7",
   "metadata": {},
   "source": [
    "create job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "b209ec87-9a88-4dd2-80d4-aad77ac18303",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./temp/job-parms/example.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./{DIR}/example.py\n",
    "import yaml\n",
    "\n",
    "parameters = yaml.full_load(open('example.yml', 'r'))\n",
    "operation = parameters['operation']\n",
    "a = parameters['a']\n",
    "b = parameters['b']\n",
    "c = parameters['c']\n",
    "\n",
    "def add(a, b, c):\n",
    "    print(f'The value is: {a + b + c}')\n",
    "    return a + b + c\n",
    "\n",
    "def multiply(a, b, c):\n",
    "    print(f'The value is: {a * b * c}')\n",
    "    return a * b * c\n",
    "\n",
    "if operation == 'add': add(a, b, c)\n",
    "elif operation == 'multiply': multiply(a, b, c)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8cf63bc-fdcc-4304-9ad6-8403ff5f76b8",
   "metadata": {},
   "source": [
    "execute the script in the folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "14786574-86e4-445e-9db9-dd96ef7b14d9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "example.json  example.py  example.yml\n"
     ]
    }
   ],
   "source": [
    "!ls ./{DIR}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "c04a6e91-82ad-4140-8ab0-b35e7bffbe20",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 14.5\n"
     ]
    }
   ],
   "source": [
    "!cd ./{DIR} && python example.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cbd79a6-22de-4231-a44a-7ba3c5c4ccb4",
   "metadata": {},
   "source": [
    "### Pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "812d1a13-0f5d-45c0-8fcc-3ef525c4eca1",
   "metadata": {},
   "source": [
    "output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "74759c2e-af40-4167-b7ea-2aba7d4af3ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "with open(f'./{DIR}/example.pickle', 'wb') as file:\n",
    "    pickle.dump(parameters, file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa9e20d7-e203-42f0-ba51-fdc9b3351b92",
   "metadata": {},
   "source": [
    "input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "7dff807d-c0ea-463b-89b8-5521e8bc0846",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "with open(f'./{DIR}/example.pickle', 'rb') as file:\n",
    "    data = pickle.load(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd418f8d-e648-4b51-a156-a581a1deef08",
   "metadata": {},
   "source": [
    "create job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "6eccb41b-2a0e-48a3-ba6a-528704c918b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./temp/job-parms/example.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./{DIR}/example.py\n",
    "import pickle\n",
    "\n",
    "parameters = pickle.load(open('example.pickle', 'rb'))\n",
    "operation = parameters['operation']\n",
    "a = parameters['a']\n",
    "b = parameters['b']\n",
    "c = parameters['c']\n",
    "\n",
    "def add(a, b, c):\n",
    "    print(f'The value is: {a + b + c}')\n",
    "    return a + b + c\n",
    "\n",
    "def multiply(a, b, c):\n",
    "    print(f'The value is: {a * b * c}')\n",
    "    return a * b * c\n",
    "\n",
    "if operation == 'add': add(a, b, c)\n",
    "elif operation == 'multiply': multiply(a, b, c)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a9ed549-7c61-44ae-86f5-93932de0d2c1",
   "metadata": {},
   "source": [
    "execute the script in the folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "45e996d0-060c-4579-ae32-82397b237203",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "example.json  example.pickle  example.py  example.yml\n"
     ]
    }
   ],
   "source": [
    "!ls ./{DIR}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "ba38dee2-72a6-4870-ba7e-94b7b2a8e6b5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value is: 14.5\n"
     ]
    }
   ],
   "source": [
    "!cd ./{DIR} && python example.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e258d347-7861-46bc-9f7c-9e9e615e72a5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-3.m94",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-3:m94"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}