dobler_111014_part1.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:df48ffc979722d7be76f6be073bdbe84158e33c888703422c6dd88f2b195134e"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Applied Data Science || Greg Dobler || November 10, 2014\n",
      "-----------------------------------------------------------"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Structure of this module\n",
      "No homeworks, problems/topics/readings/videos will be suggested <br>\n",
      "One final project which will consist of submitting code <br>\n",
      "Useful reading: <i>Programming Computer Vision with Python</i>, Jan Erik Solem\n",
      "\n",
      "<br>\n",
      "\n",
      "## Getting started with scientific python\n",
      "NumPy -- arrays (2D and 3D), vectorized operations, basic functionality <br>\n",
      "SciPy -- functions, transformations, analysis <br>\n",
      "Matplotlib -- plotting, viewing, visualizations (static and interactive)\n",
      "\n",
      "<br>\n",
      "\n",
      "## Image processing and Computer Vision\n",
      "What are images from a data perspective? <br>\n",
      "How are images loaded displayed and handled? <br>\n",
      "What are some basic image processing taks? <br>\n",
      "What can we learn from images? <br><br>\n",
      "<b>Video is not particularly <i>special</i>, it's just a series of images to be analyzed together</b>\n",
      "\n",
      "----------------------\n",
      "\n",
      "## 1D\n",
      "Let's look at a \"time series\".  First some imports that will be useful:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import os\n",
      "import sys\n",
      "import time\n",
      "import numpy as np\n",
      "import scipy.ndimage as nd\n",
      "import matplotlib.pyplot as plt"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we load in the one-dimensional data (which has been saved as a numpy array) and define a plotting function to view the data in a couple of different ways."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data = np.load('output/ml_flat.npy')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "fig0 = plt.figure(0)\n",
      "def showit(fac,sep=100.):\n",
      "    fig0.clf()\n",
      "    ax0 = fig0.add_subplot(1,1,1)\n",
      "    lin0 = ax0.plot(range(data.size/fac), \n",
      "                    data.reshape(data.size/fac,fac) + \n",
      "                    sep*np.arange(fac),lw=0.05,color='k')\n",
      "    fig0.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "showit(1)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This one-dimensional \"time series\" can be sliced and plotted stacked on top of itself.  For example, cut it in half and plot the two curves:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "showit(2,200)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "If we progressively cut further, stacking each time.."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "showit(954,10)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "fig1, ax1 = plt.subplots(num=1)\n",
      "im1 = ax1.imshow(data.reshape(data.size/954,954).T[::-1,:])\n",
      "ax1.grid(0)\n",
      "fig1.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "---\n",
      "\n",
      "# 1D vs 2D arrays\n",
      "\n",
      "Grayscale images are 2D <b>arrays</b> but in many cases can be thought of (and treated as) 1D arrays.<br>\n",
      "<br>\n",
      "Many data analysis tasks you've already seen can be handled exactly the same:<br>\n",
      ". correlations<br>\n",
      ". means and variances<br>\n",
      ". multivariate regressions<br>\n",
      "<br>\n",
      "others will be different:<br>\n",
      ". filters<br>\n",
      ". derivatives<br>\n",
      ". rotations<br>\n",
      "<br>\n",
      "But since this is a numpy array things are easily <b><i>vectorized</i></b> and fast.  For example, let's look at derivatives.  First the 1D case:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "xx = np.linspace(-20,20,1000)\n",
      "yy = np.cos(xx)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "fig3, ax3 = plt.subplots(num=3)\n",
      "lin3a, = ax3.plot(xx,yy,'darkred')\n",
      "lin3b, = ax3.plot(xx[1:],(yy[1:]-yy[:-1])/(xx[1]-xx[0]),\n",
      "                  'dodgerblue')\n",
      "fig3.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 23
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "plot(xx,yy)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 20,
       "text": [
        "[<matplotlib.lines.Line2D at 0x11ec6f310>]"
       ]
      }
     ],
     "prompt_number": 20
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Which can be used to find edges,"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "xx = np.linspace(-20,20,1000)\n",
      "yy = 1.0*(xx>4.3)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "lin3a.set_data(xx,yy)\n",
      "lin3b.set_data(xx[1:],(yy[1:]-yy[:-1])/(xx[1]-xx[0]))\n",
      "lin3b.set_visible(False)\n",
      "fig3.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 21
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "lin3b.set_visible(True)\n",
      "fig3.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 22
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The same thing is true in 2D, but now we can do all rows at once,"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "img = data.reshape(data.size/954,954).T[::-1,:]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 24
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "fig4, ax4 = plt.subplots(1,2,num=4)\n",
      "[i.grid('off') for i in ax4]\n",
      "im4a = ax4[0].imshow(img)\n",
      "im4b = ax4[1].imshow(img[:,5:]-img[:,:-5])\n",
      "fig4.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 28
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "img.shape"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 25,
       "text": [
        "(954, 640)"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "... or we can do all columns at once"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "im4b.set_data(img[5:,:]-img[:-5,:])\n",
      "fig4.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 29
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "For completeness, let's do rows and columns (not simultaneously!) take the absolute value and add them together.  This is the simplest version of an edge detector,"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "edges_simp = np.abs(img[5:,5:]-img[:-5,5:]) + \\\n",
      "    np.abs(img[5:,5:]-img[5:,:-5])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 33
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "im4b.set_data(edges_simp)\n",
      "fig4.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 31
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "which is often combined with thresholding"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "im4b.set_data(edges_simp>60)\n",
      "im4b.set_clim(0,1)\n",
      "fig4.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 60
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You can see it's not doing great, it's missing some edges and there's a lot of noise.  We'll fix this later...\n",
      "\n",
      "---"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}