dobler_112414_part3.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:ac5b25b92abbed20c2c952f6dc854bf1857a2595953d2c6e99aa61ef8fe20ea9"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "(always be aware of your imports and <b><u><i>preserve namespaces</i></u></b>!!!)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import os\n",
      "import sys\n",
      "import time\n",
      "import numpy as np\n",
      "import scipy.ndimage as nd\n",
      "import matplotlib.pyplot as plt"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "---\n",
      "\n",
      "## Multiple frames (videos)\n",
      "\n",
      "Unlike single still images, multiple images of the same scene over time let you measure **time dependent** features (e.g., motion, color changes, etc.).  We will deal with already extracted frames from a video here, but an example about how that extraction can be done in python with OpenCV is <a href=\"http://tobilehman.com/blog/2013/01/20/extract-array-of-frames-from-mp4-using-python-opencv-bindings/\">here</a>. "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### DOT camera analysis\n",
      "\n",
      "The Department of Transportation has hundreds of traffic cameras located around the city.  Using this script:"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "    import os\n",
      "    import time\n",
      "\n",
      "    nframe   = 100\n",
      "    cmd_wget = \"wget http://207.251.86.238/cctv391.jpg\"\n",
      "    cmd_mv   = \"mv cctv391.jpg images/dot/cctv391_{0:03}.jpg\"\n",
      "\n",
      "    for ii in range(nframe):\n",
      "        os.system(cmd_wget)\n",
      "        os.system(cmd_mv.format(ii))\n",
      "        time.sleep(1)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "I grabbed 100 frames from camera overlooking the Manhattan bridge (and Lower East Side):"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "path1 = 'images'\n",
      "path2 = 'dot'\n",
      "dpath = os.path.join(path1,path2)\n",
      "flist = [os.path.join(dpath,i) for i in \n",
      "         sorted(os.listdir(os.path.join(dpath)))]\n",
      "\n",
      "for i in flist:\n",
      "    print(i)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "imgs = np.array([nd.imread(i) for i in flist])\n",
      "nimg, nrow, ncol = imgs.shape[0:3]\n",
      "xs = 6.5\n",
      "ys = 2*xs*float(nrow)/float(ncol)\n",
      "\n",
      "plt.close(0)\n",
      "fig0, ax0 = plt.subplots(2,1,num=0,figsize=[xs,ys])\n",
      "fig0.subplots_adjust(0,0,1,1,0,0)\n",
      "[i.axis('off') for i in ax0]\n",
      "im0a = ax0[0].imshow(imgs[0])\n",
      "fig0.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for img in imgs:\n",
      "    im0a.set_data(img)\n",
      "    fig0.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We'd like to **roughly** count the number of cars on this section of the highway.  Let's first look at frame differences:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "im0b = ax0[1].imshow(imgs[1].mean(-1) - imgs[0].mean(-1))\n",
      "fig0.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for ii in range(1,nimg):\n",
      "    im0a.set_data(imgs[ii])\n",
      "    im0b.set_data(imgs[ii].mean(-1) - imgs[ii-1].mean(-1))\n",
      "    fig0.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Or taking the absolute value (note: if we were to count off of this we'd be roughly double counting...)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "im0b.set_clim(0,128)\n",
      "\n",
      "for ii in range(1,nimg):\n",
      "    im0a.set_data(imgs[ii])\n",
      "    im0b.set_data(np.abs(imgs[ii].mean(-1) - imgs[ii-1].mean(-1)))\n",
      "    fig0.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Let's do some thresholding:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dimg = np.zeros([nrow,ncol])\n",
      "im0b.set_clim(0,1)\n",
      "\n",
      "for ii in range(1,nimg):\n",
      "    dimg[:,:] = np.abs(imgs[ii].mean(-1) - imgs[ii-1].mean(-1))\n",
      "    im0a.set_data(imgs[ii])\n",
      "    im0b.set_data(dimg>40)\n",
      "    fig0.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Background subtraction\n",
      "\n",
      "But again, we're double counting and adding noise.  Another method is to subtract the \"mean image\" from each frame:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "mimg = imgs.mean(0)\n",
      "\n",
      "im0a.set_data(mimg.clip(0,255).astype(np.uint8))\n",
      "fig0.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for ii in range(1,nimg):\n",
      "    im0a.set_data(imgs[ii])\n",
      "    im0b.set_data(np.abs(1.0*imgs[ii] - mimg).clip(0,255).astype(np.uint8))\n",
      "    fig0.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Let's take the max of this difference in color space:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "im0b.set_clim([0,255])\n",
      "\n",
      "for ii in range(1,nimg):\n",
      "    im0a.set_data(imgs[ii])\n",
      "    im0b.set_data(np.abs(1.0*imgs[ii] - mimg).max(-1))\n",
      "    fig0.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Again, we can do some clipping, but let's take a look at the distribution of brightnesses"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "plt.close(1)\n",
      "\n",
      "fig1,ax1 = plt.subplots(num=1)\n",
      "ax1.hist(np.log10(np.abs(1.0*imgs - mimg).max(-1).flatten()+1.0), bins=255)\n",
      "fig1.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The logarithm of the difference values for the objects is something like $>1.5 \\approx 31$:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "thr = 30\n",
      "im0b.set_clim([0,1])\n",
      "\n",
      "for ii in range(1,nimg):\n",
      "    im0a.set_data(imgs[ii])\n",
      "    im0b.set_data(np.abs(1.0*imgs[ii] - mimg).max(-1)>thr)\n",
      "    fig0.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "bdilation = nd.morphology.binary_dilation\n",
      "berosion = nd.morphology.binary_erosion"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "thr = 30\n",
      "im0b.set_clim([0,1])\n",
      "fgr = np.zeros([nrow,ncol],dtype=int)\n",
      "\n",
      "for ii in range(1,nimg):\n",
      "    fgr[:,:] = bdilation(berosion(np.abs(1.0*imgs[ii] - mimg).max(-1)>thr),iterations=2)\n",
      "    im0a.set_data(imgs[ii])\n",
      "    im0b.set_data(fgr)\n",
      "    fig0.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can use our trick of a row map to generate a mask,"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "col_mask = (np.arange(nrow*ncol).reshape(nrow,ncol)%ncol)<250"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "thr = 30\n",
      "im0b.set_clim([0,1])\n",
      "fgr = np.zeros([nrow,ncol],dtype=int)\n",
      "\n",
      "for ii in range(1,nimg):\n",
      "    fgr[:,:] = bdilation(berosion(np.abs(1.0*imgs[ii] - mimg).max(-1)>thr),iterations=2)\n",
      "    im0a.set_data(imgs[ii])\n",
      "    im0b.set_data(fgr*col_mask)\n",
      "    fig0.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Lastly, we count the number of detected objects:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "count = ax0[1].text(0,0,\"# of cars: \",va='top',\n",
      "                    fontsize=20,color='white')\n",
      "fig0.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "meas_label = nd.measurements.label"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "thr = 30\n",
      "sz_thr = 20 # only count objects greater than a size threshold\n",
      "im0b.set_clim([0,1])\n",
      "fgr = np.zeros([nrow,ncol],dtype=int)\n",
      "\n",
      "for ii in range(1,nimg):\n",
      "    fgr[:,:] = bdilation(berosion(np.abs(1.0*imgs[ii] - mimg).max(-1)>thr),iterations=2)\n",
      "    labs = meas_label(fgr*col_mask)\n",
      "    ncar = sum([1*((labs[0]==lab).sum()>sz_thr) for lab in range(1,labs[1]+1)])\n",
      "\n",
      "    im0a.set_data(imgs[ii])\n",
      "    im0b.set_data(fgr*col_mask)\n",
      "    count.set_text(\"# of cars: {0}\".format(ncar))\n",
      "    fig0.canvas.draw()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "---"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Closing thoughts and recap"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Below are some of the key concepts we've worked through.  They are broadly applicable, even well beyond the field of image processing and computer vision:\n",
      "\n",
      "- best practices for filesystem handling and namespaces\n",
      "- vectorized coding (indexing arrays to eliminate for loops)\n",
      "- object oriented plotting with matplotlib (for interactive plots)\n",
      "- thresholding for data subselection and masking\n",
      "- derivatives and edge detection through array shifts\n",
      "- filtering (mean, Gaussian, median) in 1D and 2D\n",
      "- regressions using the normal equation and some linear algebra\n",
      "- combining 2D data with time series information"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "---"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}