Reproducibility 2.0.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Reproducibility and exploratory computing with a Jupyter-based workflow\n",
    "*Antonino Ingargiola*, EuroScipy 2018"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# JupyterCon Talk: \"I don't like notebooks\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": [
     "hide_input"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<blockquote class=\"twitter-tweet\" data-lang=\"en\"><p lang=\"en\" dir=\"ltr\">slides for my &quot;I Don&#39;t Like Notebooks&quot; <a href=\"https://twitter.com/hashtag/JupyterCon?src=hash&amp;ref_src=twsrc%5Etfw\">#JupyterCon</a> talk:<a href=\"https://t.co/30peBFwTbv\">https://t.co/30peBFwTbv</a></p>&mdash; Joel Grus (@joelgrus) <a href=\"https://twitter.com/joelgrus/status/1033035196428378113?ref_src=twsrc%5Etfw\">August 24, 2018</a></blockquote>\n",
       "<script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%HTML\n",
    "<blockquote class=\"twitter-tweet\" data-lang=\"en\"><p lang=\"en\" dir=\"ltr\">slides for my &quot;I Don&#39;t Like Notebooks&quot; <a href=\"https://twitter.com/hashtag/JupyterCon?src=hash&amp;ref_src=twsrc%5Etfw\">#JupyterCon</a> talk:<a href=\"https://t.co/30peBFwTbv\">https://t.co/30peBFwTbv</a></p>&mdash; Joel Grus (@joelgrus) <a href=\"https://twitter.com/joelgrus/status/1033035196428378113?ref_src=twsrc%5Etfw\">August 24, 2018</a></blockquote>\n",
    "<script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [
     "hide_input"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<blockquote class=\"twitter-tweet\" data-lang=\"en\"><p lang=\"en\" dir=\"ltr\">Notebook development guidelines at <a href=\"https://twitter.com/netflixdata?ref_src=twsrc%5Etfw\">@netflixdata</a> :<br><br>* Keep a low branching factor<br>* Short and simple is better<br>* Keep to one primary outcome<br>* Leave library functions in libraries<br>* Move complexity to libraries<br><br> <a href=\"https://twitter.com/codeseal?ref_src=twsrc%5Etfw\">@codeseal</a> at <a href=\"https://twitter.com/hashtag/jupytercon?src=hash&amp;ref_src=twsrc%5Etfw\">#jupytercon</a></p>&mdash; Caitlin Hudon👩🏼‍💻 (@beeonaposy) <a href=\"https://twitter.com/beeonaposy/status/1032693394965975040?ref_src=twsrc%5Etfw\">August 23, 2018</a></blockquote>\n",
       "<script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%HTML\n",
    "<blockquote class=\"twitter-tweet\" data-lang=\"en\"><p lang=\"en\" dir=\"ltr\">Notebook development guidelines at <a href=\"https://twitter.com/netflixdata?ref_src=twsrc%5Etfw\">@netflixdata</a> :<br><br>* Keep a low branching factor<br>* Short and simple is better<br>* Keep to one primary outcome<br>* Leave library functions in libraries<br>* Move complexity to libraries<br><br> <a href=\"https://twitter.com/codeseal?ref_src=twsrc%5Etfw\">@codeseal</a> at <a href=\"https://twitter.com/hashtag/jupytercon?src=hash&amp;ref_src=twsrc%5Etfw\">#jupytercon</a></p>&mdash; Caitlin Hudon👩🏼‍💻 (@beeonaposy) <a href=\"https://twitter.com/beeonaposy/status/1032693394965975040?ref_src=twsrc%5Etfw\">August 23, 2018</a></blockquote>\n",
    "<script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# An opinionated workflow ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 1. Prepare the environment\n",
    "\n",
    "- Use one folder for the project\n",
    "- Put folder under auto backup (Dropbox, ...)\n",
    "- Create a separate environment (pipenv, conda, docker)\n",
    "    - Save specs (one of those):\n",
    "            - requirements.txt\n",
    "            - environment.yml\n",
    "            - dockerfile\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example: create a conda environment\n",
    "\n",
    ".\n",
    "```\n",
    "(base)$ cd my_project\n",
    "(base)$ conda create -n my_project_env python=3.6 scikit-learn seaborn\n",
    "(base)$ conda activate my_project_env\n",
    "(my_project_env)$\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Folder structure\n",
    "\n",
    ".\n",
    "```\n",
    "    project_1/\n",
    "    \n",
    "        data/\n",
    "        results/\n",
    "        figures/\n",
    "        \n",
    "        analysis.ipynb\n",
    "        environment.yml\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 2. Prepare the Data\n",
    "\n",
    "- Transform your data to a common format\n",
    "\n",
    "**Example:** *tidy* vs *long-form*:\n",
    "\n",
    "![](images/tidyr-spread-gather.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 3. Notebook narrative\n",
    "\n",
    "- Narrative: \n",
    "    - *headings, equations, links, figures*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Document purpose not mechanics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Print libraries's versions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 4. Notebook (code) structure\n",
    "\n",
    "\n",
    "- Use a linear structure (low branching factor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Short and simple is better"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Keep one primary outcome"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Do \"Restart + Run-All\" often"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A starting project\n",
    "*1-notebook*\n",
    "\n",
    "![](images/empty_notebook.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A starting project\n",
    "*1-notebook*\n",
    "\n",
    "- Remove duplication (DRY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Write **functions**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Add assertion-based **tests**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Maturing Project\n",
    "*(1-2 notebooks)*\n",
    "\n",
    "- Save intermediate results (example: CSV, figures)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Consolidate functions into modules:\n",
    "    - Move functions to `.py` file\n",
    "    - Move tests to `test_*.py` file, run it with `pytest`\n",
    "    - Commit `.py` files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- For short projects, commit the full notebooks\n",
    "- Use `nbdime` for diffs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [
     "hide_input"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<blockquote class=\"twitter-tweet\" data-lang=\"en\"><p lang=\"en\" dir=\"ltr\">I had such an outdated nbdime! Now a button shows a graphical diff  between an open notebook and its latest git-committed version with one  click.    🎉  <br><br>Install or update nbdime, then follow these simple instructions to enable the extension:<a href=\"https://t.co/D1pwoP7lAT\">https://t.co/D1pwoP7lAT</a> <a href=\"https://t.co/9yJnYBsMyF\">pic.twitter.com/9yJnYBsMyF</a></p>&mdash; Antonino Ingargiola (@tritemio_sc) <a href=\"https://twitter.com/tritemio_sc/status/1034075881889849344?ref_src=twsrc%5Etfw\">August 27, 2018</a></blockquote>\n",
       "<script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%HTML\n",
    "<blockquote class=\"twitter-tweet\" data-lang=\"en\"><p lang=\"en\" dir=\"ltr\">I had such an outdated nbdime! Now a button shows a graphical diff  between an open notebook and its latest git-committed version with one  click.    🎉  <br><br>Install or update nbdime, then follow these simple instructions to enable the extension:<a href=\"https://t.co/D1pwoP7lAT\">https://t.co/D1pwoP7lAT</a> <a href=\"https://t.co/9yJnYBsMyF\">pic.twitter.com/9yJnYBsMyF</a></p>&mdash; Antonino Ingargiola (@tritemio_sc) <a href=\"https://twitter.com/tritemio_sc/status/1034075881889849344?ref_src=twsrc%5Etfw\">August 27, 2018</a></blockquote>\n",
    "<script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Maturing Project (2)\n",
    "*a few notebooks*\n",
    "\n",
    "Notebooks start to be consolidated.\n",
    "\n",
    "- Several notebooks share the same `.py` files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- If notebooks are similar, **parametrize**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Batch run:\n",
    "    - Save notebooks with no output (**template**)\n",
    "    - Save executed notebooks in sub-folder\n",
    "    - Commit template notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example:\n",
    "\n",
    ".\n",
    "    project_1/\n",
    "    \n",
    "        data/\n",
    "            experiment1.hdf5\n",
    "            experiment2.hdf5\n",
    "            experiment3.hdf5\n",
    "            \n",
    "        results/\n",
    "            results_experiment1.csv\n",
    "            results_experiment2.csv\n",
    "            results_experiment3.csv\n",
    "            \n",
    "        reports/\n",
    "            analysis_experiment1.ipynb\n",
    "            analysis_experiment2.ipynb\n",
    "            analysis_experiment3.ipynb\n",
    "            \n",
    "        index.ipynb\n",
    "        analysis_template.ipynb\n",
    "        ...\n",
    "        loader.py\n",
    "        analysis.py\n",
    "        plots.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Mature project: pipeline\n",
    "*many notebooks*\n",
    "\n",
    "- Build a **master notebook** for:\n",
    "\n",
    "    1. Batch-run the full pipeline\n",
    "        - *parametrized notebooks*\n",
    "    2. Aggregate results from multiple notebooks:\n",
    "        - *figures, CSV, etc...*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Parametrize Notebooks"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "![](images/screenshot.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Parametrize notebooks\n",
    "\n",
    "- [nbrun](https://github.com/tritemio/nbrun):\n",
    "    - from dearly yours\n",
    "    - a single-function `run_notebook()`\n",
    "    - vendorize in your project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- [papermill](https://papermill.readthedocs.io/en/latest/)\n",
    "    - from Netflix\n",
    "    - graphical parametrization\n",
    "    - save data into notebooks\n",
    "    - retrieve data, figures from notebooks for summaries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Functions -> Libraries\n",
    "\n",
    "When used by more than one project:\n",
    "\n",
    "- Create a Python package\n",
    "- Use `versioneer`: auto-version based on git commit:\n",
    "\n",
    "                     commit   \n",
    "                    ,-----,\n",
    "            0.7+12.g790526b\n",
    "            /    \\\n",
    "      last tag   # of commits after tag '0.7'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'v0.5.2+36.g5bba7b9'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import skopt\n",
    "skopt.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Numpy: 1.14.2\n",
      "Scipy: 1.0.0\n",
      "Scikit-optimize: v0.5.2+36.g5bba7b9\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import scipy\n",
    "import skopt\n",
    "print('Numpy:', np.__version__)\n",
    "print('Scipy:', scipy.__version__)\n",
    "print('Scikit-optimize:', skopt.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " - Optimized (cython) burst search loaded.\n",
      " - Optimized (cython) photon counting loaded.\n",
      "--------------------------------------------------------------\n",
      " You are running FRETBursts (version 0.7+0.g790526b.dirty).\n",
      "\n",
      " If you use this software please cite the following paper:\n",
      "\n",
      "   FRETBursts: An Open Source Toolkit for Analysis of Freely-Diffusing Single-Molecule FRET\n",
      "   Ingargiola et al. (2016). http://dx.doi.org/10.1371/journal.pone.0160716 \n",
      "\n",
      "--------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "import fretbursts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Conclusions\n",
    "\n",
    "- Curate \"environments\"\n",
    "- Build narrative\n",
    "- Move complexity to libraries\n",
    "- Automate execution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [
     "hide_input"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"images/Thankyou_slide.png\" alt=\"Thank you. Antonino Ingargiola.\" \n",
       "height=\"1024\" width=\"768\"> "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%HTML\n",
    "<img src=\"images/Thankyou_slide.png\" alt=\"Thank you. Antonino Ingargiola.\" \n",
    "height=\"1024\" width=\"768\"> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Backup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [
     "hide_input"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<blockquote class=\"twitter-tweet\" data-lang=\"en\"><p lang=\"en\" dir=\"ltr\">In light of recent discussions, here&#39;s a series of videos I made a while ago that shows my approach to reproducible data analysis in the Jupyter notebook: <a href=\"https://t.co/UdLAhx4jWq\">https://t.co/UdLAhx4jWq</a></p>&mdash; Jake VanderPlas (@jakevdp) <a href=\"https://twitter.com/jakevdp/status/1034617901389496320?ref_src=twsrc%5Etfw\">August 29, 2018</a></blockquote>\n",
       "<script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%HTML\n",
    "<blockquote class=\"twitter-tweet\" data-lang=\"en\"><p lang=\"en\" dir=\"ltr\">In light of recent discussions, here&#39;s a series of videos I made a while ago that shows my approach to reproducible data analysis in the Jupyter notebook: <a href=\"https://t.co/UdLAhx4jWq\">https://t.co/UdLAhx4jWq</a></p>&mdash; Jake VanderPlas (@jakevdp) <a href=\"https://twitter.com/jakevdp/status/1034617901389496320?ref_src=twsrc%5Etfw\">August 29, 2018</a></blockquote>\n",
    "<script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3.6 (py36)",
   "language": "python",
   "name": "py36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {
    "height": "525.867px",
    "left": "0px",
    "right": "838px",
    "top": "111.133px",
    "width": "296px"
   },
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}