SuaveDispatch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1><span style=\"color:red\">Welcome to the SuAVE Jupyter Notebook Server</span></h1>\n",
    "\n",
    "This is the Jupyter Notebook Dispatcher module of the SuAVE platform. This environment enables you to write and execute Python scripts to process and analyze data in SuAVE surveys and image galleries. In most included scripts, the derived data (secondary variables, image characteristics, predictive labels, etc.) can be added to SuAVE surveys for visual analysis.  \n",
    "\n",
    "Look several cells below for the types of operations supported by your selected JupyterHub.\n",
    "\n",
    "You can execute cells in sequence (by clicking 'Run', or pressing Shift-Enter). From this module (\"dispatcher\") you can launch other notebooks to perform computations, image processing,  modeling and statistical tasks.\n",
    "\n",
    "Example call parameters:\n",
    "SuaveDispatch.ipynb?surveyurl=https://suave-dev.sdsc.edu/main/file=zaslavsk_WBDH_Participants.csv&views=1110101&view=grid&user=zaslavsk&csv=zaslavsk_WBDH_Participants.csv&params=none&dzc=https://maxim.ucsd.edu/dzgen/lib-staging-uploads/063044157874ce7d5841c906fa6d8073/content.dzc&activeobject=null\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Accepting survey parameters from SuAVE \n",
    "\n",
    "The following cell will retrieve survey parameters from the URL. Execute it to continue "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%javascript\n",
    "function getQueryStringValue (key)\n",
    "{  \n",
    "    return unescape(window.location.search.replace(new RegExp(\"^(?:.*[&\\\\?]\" + escape(key).replace(/[\\.\\+\\*]/g, \"\\\\$&\") + \"(?:\\\\=([^&]*))?)?.*$\", \"i\"), \"$1\"));\n",
    "}\n",
    "IPython.notebook.kernel.execute(\"survey_url='\".concat(getQueryStringValue(\"surveyurl\")).concat(\"'\"));\n",
    "IPython.notebook.kernel.execute(\"views='\".concat(getQueryStringValue(\"views\")).concat(\"'\"));\n",
    "IPython.notebook.kernel.execute(\"view='\".concat(getQueryStringValue(\"view\")).concat(\"'\"));\n",
    "IPython.notebook.kernel.execute(\"user='\".concat(getQueryStringValue(\"user\")).concat(\"'\"));\n",
    "IPython.notebook.kernel.execute(\"csv_file='\".concat(getQueryStringValue(\"csv\")).concat(\"'\")); \n",
    "IPython.notebook.kernel.execute(\"dzc_file='\".concat(getQueryStringValue(\"dzc\")).concat(\"'\")); \n",
    "IPython.notebook.kernel.execute(\"params='\".concat(getQueryStringValue(\"params\")).concat(\"'\")); \n",
    "IPython.notebook.kernel.execute(\"active_object='\".concat(getQueryStringValue(\"activeobject\")).concat(\"'\")); \n",
    "IPython.notebook.kernel.execute(\"full_notebook_url='\" + window.location + \"'\"); "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Check if the passed parameters are correct "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Check if the parameters are correct\n",
    "import webbrowser\n",
    "import ntpath\n",
    "import os\n",
    "from IPython.display import Markdown, display\n",
    "def printmd(string):\n",
    "    display(Markdown(string))\n",
    "\n",
    "url_partitioned = full_notebook_url.partition('/SuaveDispatch')\n",
    "base_url = url_partitioned[0];\n",
    "images_available = False\n",
    "if dzc_file == \"undefined\":\n",
    "    dzc_file = \"\"\n",
    "    localdzc = \"\" \n",
    "    full_images = \"full images not available on NFS storage\"\n",
    "if len(dzc_file) > 20:\n",
    "    if \"lib-staging-uploads\" in dzc_file:\n",
    "        localdzc = dzc_file.replace(\"https://maxim.ucsd.edu/dzgen/lib-staging-uploads\",\"/lib-nfs/dzgen\")\n",
    "        full_images = localdzc.replace(\"/content.dzc\",\"/full_images/\")\n",
    "    else:\n",
    "        localdzc = \"dzc not available on NFS storage\"\n",
    "        full_images = \"full images not available on NFS storage\"\n",
    "        images_available = True\n",
    "\n",
    "printmd(\"<b><span style='color:red'>Verify survey parameters: </span></b>\")\n",
    "\n",
    "print(\"Base Survey URL: \", survey_url)\n",
    "print(\"Enabled Views: \", views)\n",
    "print(\"Default View: \", view)\n",
    "print(\"User ID: \", user)\n",
    "print(\"Additional Parameters: \", params)\n",
    "print(\"Data File: \", csv_file)\n",
    "print(\"Image Tile Collection URL: \", dzc_file)\n",
    "print(\"Active Object: \", active_object)\n",
    "print(\"Jupyter Hub URL: \", base_url)\n",
    "print(\"Local Tile Collection Path : \", localdzc)\n",
    "print(\"Local Full-size Image Path: \", full_images)\n",
    "if os.path.exists(full_images):\n",
    "    print(\"Full-size Images Available\")\n",
    "else:\n",
    "    print(\"Full-size Images Not Available\")\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2a. Optional: load notebooks from a different github repository\n",
    "\n",
    "Skip this cell if you are already running notebooks from your repo  - don't clone you repo the second time! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf myclone\n",
    "!git clone --depth 1 \"https://github.com/izaslavsky/jupyter-suave.git\" myclone\n",
    "url1 = ('{base_url}/SuaveDispatch.ipynb?'+'surveyurl=' + survey_url + '&' + 'views=' + views + '&' 'view=' + view + '&' + 'user='+user+'&'+'csv='+csv_file+'&'+'dzc='+dzc_file+\"&\"+'activeobject='+active_object).format(base_url=base_url +\"/myclone\")\n",
    "print('Click the URL to open the new dispatcher : ', url1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Retrieve the survey file for processing\n",
    "\n",
    "As before, Shift-Enter to run this cell and continue to the next one. This merely prepares the data for subsequent processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import urllib3\n",
    "urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)\n",
    "\n",
    "csv_url = survey_url.split(\"main\")[0] + \"surveys/\" + csv_file\n",
    "\n",
    "# get the survey data file\n",
    "http = urllib3.PoolManager()\n",
    "r = http.request('GET', csv_url, preload_content=False)\n",
    "\n",
    "# place the file into temp_csvs \n",
    "path = \"temp_csvs/\" + csv_file \n",
    "with open(path, 'wb') as out:\n",
    "    while True:\n",
    "        data = r.read(1024)\n",
    "        if not data:\n",
    "            break\n",
    "        out.write(data)\n",
    "\n",
    "r.release_conn()\n",
    "printmd(\"<b><span style='color:red'>Survey file retrieved. Run next cell to continue.</span></b>\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 4. Now, select a notebook to do some work\n",
    "\n",
    "Select a notebook, then continue to the next cell. Note that you will see only those operations that are supported on your selected hub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "from ipywidgets import interact, interactive, fixed, interact_manual\n",
    "import ipywidgets as widgets\n",
    "\n",
    "from collections import OrderedDict\n",
    "\n",
    "nb_menu = OrderedDict()\n",
    "nb_menu_counter = 1\n",
    "\n",
    "menulist = [\n",
    "('Arithmetic Operations','arithmetic/SuaveArithmetic.ipynb','any'),\n",
    "('Descriptive Statistics','stats/DescriptiveStats.ipynb','any'),\n",
    "('Color Statistics','colors/ColorStats.ipynb','image'),\n",
    "('Classify Images','classify/ImageClassify.ipynb','image'),\n",
    "('Generate LeNet CNN Model v2','predict/PredictiveModel_v2.ipynb','image'),\n",
    "('Extend LeNet CNN Model','predict/ExtendModel.ipynb','image'),\n",
    "('Generate SVM Model','svm/SVMPredictiveModel.ipynb','image'),\n",
    "('Extend SVM Model','svm/ExtendSVM.ipynb','image'),\n",
    "('Generate SDG Dataset','SDG/GenerateSDGDataset.ipynb','largedataset')\n",
    "]\n",
    "\n",
    "if os.path.isfile(localdzc) and not os.path.isdir(full_images):\n",
    "    print(\"ATTENTION!  This hub supports image-based processing, but full-size images are not available for this survey. \\n Full-size image operations are not available from this menu.\\n Contact the admin at zaslavsk@sdsc.edu to re-generate images from image tiles.\")\n",
    "elif not os.path.isfile(localdzc):\n",
    "    print(\"ATTENTION!  Image tiles must be available on nfs storage.\\n This hub does not support nfs mounted storage. \\n Full-size image operations are not available from this menu.\")\n",
    "elif not os.path.isdir('/lib-nfs/largedatasets'):\n",
    "    print(\"ATTENTION!  Large datasets such as the SDG database must be available on nfs storage.\\n This hub does not support nfs mounted storage.\")\n",
    "\n",
    "\n",
    "# For a setup where dzc's are only on NFS share, use this:\n",
    "\n",
    "for label, nb, nbtype in menulist:\n",
    "    if nbtype == 'any':\n",
    "        nb_menu[str(nb_menu_counter) + '. ' + label] = nb\n",
    "        nb_menu_counter +=1\n",
    "    elif nbtype == 'image':\n",
    "        if os.path.isfile(localdzc) and os.path.isdir(full_images):\n",
    "            nb_menu[str(nb_menu_counter) + '. ' + label] = nb\n",
    "            nb_menu_counter +=1\n",
    "    elif nbtype == 'largedataset':\n",
    "        if os.path.isdir('/lib-nfs/largedatasets'):\n",
    "            nb_menu[str(nb_menu_counter) + '. ' + label] = nb\n",
    "            nb_menu_counter +=1\n",
    "        \n",
    "\n",
    "def f(notebooks_menu):\n",
    "    return notebooks_menu\n",
    "out = interact(f, notebooks_menu=nb_menu.keys());\n",
    "\n",
    "printmd(\"<b><span style='color:red'>Select a Jupyter notebook and then run next cell</span></b>\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Open the selected notebook and pass survey parameters to it\n",
    "\n",
    "Once the URL is for the next notebook is constructed, click that URL to open it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chosen_nb_name = nb_menu[out.widget.result]\n",
    "import webbrowser\n",
    "url1 = ('{base_url}/operations/{nb_name}?'+'surveyurl=' + survey_url + '&' + 'views=' + views + '&' 'view=' + view + '&' + 'user='+user+'&'+'csv='+csv_file+'&'+'dzc='+dzc_file+\"&\"+'activeobject='+active_object).format(base_url=base_url, nb_name=chosen_nb_name)\n",
    "\n",
    "printmd(\"<b><span style='color:red'>Click the URL to open the selected notebook:</span></b>\")\n",
    "print(url1)\n",
    "\n",
    "# webbrowser.open(url1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}