lecture-2.ipynb

{
 "metadata": {
  "celltoolbar": "Slideshow",
  "name": "",
  "signature": "sha256:a9892875325e89cbbc55486d8724d1a6a82d14fddcc878e57a9d265c19f7875a"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Lecture 2: Norms"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "## Syllabus\n",
      "**Week 2:** Matrices, vectors, norms, ranks\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "##Recap of the previous lecture\n",
      "- Matrices are rectangular tables of numbers.\n",
      "- Matrix-by-vector product is $\\mathcal{O}(n^2)$ and matrix-by-matrix is $\\mathcal{O}(n^3)$.\n",
      "- Strassen algorithm is $\\mathcal{O}(n^{2.78..})$, but big constant\n",
      "- Block algorithms are important to use computer memory hierarchy"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Vector norm\n",
      "Vectors typically provide an (approximate) description of a physical (or some other) object. One of the main question is **how accurate** the approximation is (1%, 10%). What is an acceptable representation, of course, depends on the particular applications. For example:\n",
      "- In partial differential equations accuracies $10^{-5} - 10^{-10}$ are the typical case\n",
      "- In data mining sometimes an error of $80\\%$ is ok, since the interesting signal is corrupted by a huge noise."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "## Distances and norms\n",
      "Norm is a **qualitative measure of smallness of a vector** and is typically denoted as $\\Vert x \\Vert$\n",
      "The norm should satisfy certain properties:\n",
      "\n",
      "- $\\Vert \\alpha x \\Vert = |\\alpha| \\Vert x \\Vert$,\n",
      "- $\\Vert x + y \\Vert \\leq \\Vert x \\Vert + \\Vert y \\Vert$ (triangle inequality),\n",
      "- If $\\Vert x \\Vert = 0$ then $x = 0$.\n",
      "\n",
      "The distance between two vectors is then defined as\n",
      "$$\n",
      "   d(x, y) = \\Vert x - y \\Vert.\n",
      "$$"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Standard norms\n",
      "The most well-known and widely used norm is **euclidean norm**:\n",
      "$$\\Vert x \\Vert_2 = \\sqrt{\\sum_{i=1}^n |x_i|^2},$$\n",
      "which corresponds to the distance in our real life (the vectors might have complex elements, thus is the modulus here)."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## P-norm\n",
      "Euclidean norm, or $2$-norm, is a subclass of an important class of $p$-norms:\n",
      "$$\n",
      " \\Vert x \\Vert_p = \\Big(\\sum_{i=1}^n |x_i|^p\\Big)^p.\n",
      "$$\n",
      "There are two very important special cases:\n",
      "- Infinity norm, or Chebyshev norm which is defined as the maximal element: $\\Vert x \\Vert_{\\infty} = \\max_i | x_i|$\n",
      "- $L_1$ norm (or **Manhattan distance**) which is defined as the sum of modules of the elements of $x$: $\\Vert x \\Vert_1 = \\sum_i |x_i|$  \n",
      "<img src=\"chebyshev.jpeg\" style=\"float: left; height: 1%\">  <img src=\"manhattan.jpeg\" style=\"height\">  \n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We will give examples where Manhattan is very important: it all relates to the **compressed sensing** methods that emerged in the mid-2005 as one of the most popular research topics."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Computing norms in Python\n",
      "The numpy package has all you need for computing norms (```np.linalg.norm``` function)\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import numpy as np\n",
      "n = 100\n",
      "a = np.ones(n)\n",
      "b = a + 1e-3 * np.random.randn(n)\n",
      "print 'Relative error:', np.linalg.norm(a - b) / np.linalg.norm(b)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Relative error: 0.0009807031019\n"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Equivalence of the norms\n",
      "All norms are equivalent in the sense that\n",
      "$$\n",
      "   C_1 \\Vert x \\Vert_* \\leq  \\Vert x \\Vert_{**} \\leq C_2 \\Vert x \\Vert_*\n",
      "$$\n",
      "for some constants $C_1(n), C_2(n)$ for any pairs of norms $\\Vert \\cdot \\Vert_*$ and $\\Vert \\cdot \\Vert_{**}$. You will have some problems on that in your homework!  The equivalence of the norms basically means that if the vector is small in one norm, it is small in another norm. However, the constants can be large."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Unit disks in different norms\n",
      "A unit disk is a set of point such that $\\Vert x \\Vert \\leq 1$. For the Frobenius norm is a disk; for other norms the \"disks\" look different."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%matplotlib inline\n",
      "import prettyplotlib as ppl\n",
      "import matplotlib.pyplot as plt\n",
      "import numpy as np\n",
      "p = 3 #Which norm do we use\n",
      "M = 100000 #Number of sampling points\n",
      "a = np.random.randn(M, 2)\n",
      "b = []\n",
      "for i in xrange(M):\n",
      "    if np.linalg.norm(a[i, :], p) <= 1:\n",
      "        b.append(a[i, :])\n",
      "b = np.array(b)\n",
      "plt.fill(b[:, 0], b[:, 1])\n",
      "plt.axis('equal')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 45,
       "text": [
        "(-1.0, 1.0, -1.0, 1.0)"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEACAYAAAC08h1NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VdW9//E3GYAECEiYZyQyJpEAMohCeqGMVrCCaB+n\nguLv54C0BZSrt4TbwapVK1h6QbFyhSeCVCwCWvjVREERBCNDY5AplDkgQZAMZDi/P06IISYk5OTs\ntYfP63nysM/JztmfzTr5Zp21194bRERERERERERERERERERERERERETE4V4HTgK7rrDOPGAvsANI\nsCKUiIgEz834i3llhX8MsK5keQDwmRWhREQkuDpReeH/H2BSmccZQMtgBxIRkYqFWLCNtsDhMo+P\nAO0s2K6IiFTAisIPUKfcY59F2xURkXLCLNjGUaB9mcftSp67TJcuXXz79++3II6IiKvsB2Ku5ges\n6PGvBu4tWR4InMU/C+gy+/fvx+fzufZrzpw5xjPYbd+Kior45JNP8Pl8PPvss8TEdP3Bmyc29nqO\nHTvGoEGDqv2GmzBhQs3eqQFbEvQttG37fR+qfv36AAwadCN9+vQhNjaWVatWsWjRIrKzs/Xe9Mj+\nAV2u9n1UGz3+ZGAo0Az/WP4cILzkewvxz+gZA+wDLgA/r4VtikM8//wfCQ8Po3379uzcuYt///sQ\nCxYsICIigk6dunD4cCYTJ07k7bffrvDnd+/eQZs2ba5qmytXrqyN6DVwX9C3cPTo94fL8vLyANi8\n+dPS52677TYApk6dSteucSxduph+/foBkJaWRkJCAnXqlB95Fa+pjcJ/VzXWebQWtiM2N2/ePEJD\nw3j00UeYMyeJhx/+v8ydm8SFCxcuW++NN9647HFlRV8C8/XXu+jfvz+33HILa9asAaBdu3YcPny4\nip8Ut7NijF+AxMRE0xFq3aZNn9C06TWsWPE3GjVqxIwZM3j33XcBmDs3iblzk8wGFIDSog9w5MgR\nvvjiC/Lz81m3bh1xcXF89tkWioqKCA0NNZgyeNz4uxcoO33m85WMV4lN+Xy+0mGC/Px8unbtxl//\n+jpz5vyRTZveN5xOAhEeHk5BQQG9ew/hlVd+z+DBg01Hkmoq+Z28qlpu1XROcbinn/5vQkJCGDXq\nFkaNmki/fjfw738f4u23V6vou0BBQQEAe/acJysry3AaCTb1+KVK69atY+zYsaZjiIV27NjB0aNH\n2bp1K0uXLmPVqneIjY01HUsqUJMevwq/XOb48eNERUXRoEEDwF8AevfubTiVmNa0aVOOHz9OWFgY\nISEaKLATDfVIwNq0aUPDhg3p1SuW5ORkFX0B4MyZM8ycOZNOnbry/vvvs2rVKoqKikzHkhpSj9/j\n0tPT2bhxIxcvXqRXr14MGzbMdCRxEP3OmleTHr+mc3pYdnY2GzZsYPr06aajiEN17RrPzTf3JyYm\nhunTHyciIsJ0JKkG9fg9prCwkKysLMLDw2nRooXpOOIiYWH1adLkGvbuTadJkyam43iGxvjlipKT\nkwkPD6dt27Yq+lLrCgvzOH36OIMHJ5qOIlVQj98jtm/fXnrNFpFgW7FiBbGxsfTo0cN0FNdTj18q\n5PP5WLBggekY4iF3330P8+fPp06dOvzsZ/dw4sQJ05GkDPX4PaCwsJDw8PCqVxQJkujoaE6fPm06\nhiupxy8VWr36PdMRxONGjBjBpEmTKCwsNB1FUOF3pY0bN5KWlgbAn/70Mrff/lPDicTrkpOTOXbs\nGMnJyaxfv57i4mLTkTxNQz0u8vzzzzNr1qzSx1OmTGHx4sUGE4n8UMeOMezfn+Hay0BbTdfq8bCD\nBw/SpUsXnUkptjd27O2sXfs3QGf+1gYVfo8qKCigbt26pmOIXJXw8HAuXrxoOobj6eCux/h8PiZP\nnkLz5s1NRxG5agUFBcTGxhEf34dz586ZjuMp6vE72Pvvv8+YMWNMxxCpFfr9rxn1+D1k3759Kvri\nGpMmTTIdwVNU+B0qJSXVdASRWrN8+XLq1KnDunXrSm8DKcGjoR4HOnr0KAUFBaxcuZKZM2eajiNS\n61QLqk+zejziuut6s2/fDtMxRIIiMjKSCxcumI7hGLoRi8s99dTTDBjQX0VfXC0nJ8d0BNdTj98h\ndu3aRXx8vOkYIkG3d+9ekpNXsnDhAg4c2KdzVKqgoR6XOn/+PC1atCIvTz0h8ZYLFy4QGRlpOoat\naTqnC+Xn5xMVFaWiL55Ut25dPvroI9MxXEc9fpuLiIggLy/PdAwRowoLC3VRt0qox+8yL744T0Vf\nBFiy5E3TEVxFhd/Giot1IosIQEbGV6YjuIoKv43FxcWajiBiC88//xx79+41HcM1VPhtKjs7m9//\n/jnTMURso2vXruzevdt0DFdQ4bepxo0bA0WmY4jYyqOPPsbOnTtNx3A8FX4bOnjwIMOHj+fjjzWN\nTaSs9PSvad26tekYjqfCbzPZ2dn87ne/IyXlPdNRRGxn8OD+REVFmY7heJrHbzPr169n5MiRpmOI\n2Jbm9F9O8/hd4KWX/mw6goht3X//FJYtW86hQ4coLCw0HcexVPht5siRg6YjiNjW//7vJt5+ezmx\nsXEcPnzYdBzHUuG3kZSUFHbv3mU6hohtFRfvYc2a1fzHf4zh/PnzumFLDWmM3yamT/8lL7/8kukY\nIo5y+vRpoqOjTccwSmP8DpWenq6iL1IDa9asMR3BkVT4DSsuLmbMmFtNxxBxpH/+85+mIziSCr9h\ny5Yt49Ch/aZjiDjSJ59sNh3BkVT4DYuLizMdQcSxDhzYR506dTh79qzpKI6iwm9Y7969TUcQcbzt\n27ebjuAoKvw28Nlnn5mOIOJop06dMh3BUWqj8I8CMoC9wBMVfD8R+BZIK/l6uha26QoXL15kwoQ7\ndJctkQCMHn0r48ePNx3DUQKdxx8K7AGGA0eBz4G7gLK3y0kEfglUNXXFk/P4S+bgikgNebFulGVi\nHn9/YB+QCRQAbwHjKsoW4HZc66WX5pmOIOJYoaGhTJx4P+3adeaZZ/5gOo5jhAX4822BshfMOAIM\nKLeOD7gR2IH/U8EMID3A7TpeWloaN900hJyc70xHEXGsoqIiVq5cwgsvvMjUqQ+ajuMYgRb+6nzG\n+gJoD+QAo4F3ga4VrZiUlFS6nJiYSGJiYoDx7Mnn89GnTx/TMURc47XXXuXXv57DoUMHXX8Jh9TU\nVFJTUwN6jUCHYAYCSfgP8ALMBoqBZ6/wMweBvsCZcs97aoxfY/sitatXr0Hs3v2p6RiWMzHGvw24\nDugE1AUmAavLrdOyTKj+Jcvli77nXLx40XQEEVd5+OG7TUdwjEALfyHwKPAP/OP2y/HP6Hmo5Atg\nArAL+BL4E3BngNt0vN/85jfUrVvXdAwRV2nfvr3pCI5hp/EGTwz15OfnU79+fdMxRFwnKiqK3bt3\n065dO08NpeqyzA6g28WJBMe5c+fo0KED27ZtMx3F9lT4LZaRkWE6goiream3X1Mq/BaLiooyHUHE\n1dLTPX+aUJVU+C3WrFkz0xFEXO3kyZOmI9ienT4TeeLgLsDAgQPZsmWL6RgirtShQ2cOHTpgOoZl\ndHDX5nw+Hzt37lTRFwmiBg0i2LRpk+kYtqbCbyGfz8f1119vOoaIq331VTo7duwwHcPWVPgtFBIS\nwh/+oCsIigRTixYt+MlPfmI6hq1pjN9immomEnwffPAPRo4cYTqGJTTGLyIC9OjR3XQEW1Pht9DW\nrVtNRxDxhI4dO1JcXGw6hm2p8Fvo0KFDpiOIeMa5c+dMR7AtFX4L/f3va01HEPGM115bbDqCbdnp\nSKPrD+7qwK6IdbKzs2nSpInpGEFXk4O7dqpEKvwiUmuKi4s98TunWT02puuHiFhLV8KtnAq/RfQm\nFLHWz38+1XQE27LT5yBXD/WMGzeO1avL345YRILJzTXlEo3x25wXxhtF7MTtNQU0xm97R48eNR1B\nxDMefPBB0xFsS4XfIq+++hoDB95sOoaIZ3Tp0sV0BNuy09iDq4d6oqObc+bMadMxRDwjIyODbt26\nmY4RdBrqsakTJ06o6ItY7L331piOYFvq8Vtg165dxMfHm44h4jlurSllqcdvU3FxcUybNs10DBER\nQD1+S+zbt4/rrrvOdAwRz3FrTSlLPX6bCgsLMx1BRKSUCr8FGjdubDqCiOf06zfMdATbUuG3QFhY\nGD16xJmOIeIp27b903QE29IYv0UyMzPp3Lmz6RginuLmmnKJxvhtKjU1VUVfxGKhoTq2VhkVfgsM\nGTKEMWPGmo4h4ilFRYWmI9iWCr8F6tSpQ/36DU3HEBEBVPgt8cwzL/DOO8tNxxDxnEWLFpmOYEsq\n/BZo1CjSdAQRT4qJiTEdwZZU+C3Qp8/1piOIeFL//v1NR7AlFX4LdOjQwXQEEZFSKvwWOHv2rOkI\nIp4UGhpqOoIt6QQuC4SEhHjiRBIRuykqKiIkxN39W53AZVPvvfee6QginpSammo6gi2px2+BM2fO\nEB0dbTqGiOdkZ2fTpEkT0zGCSj1+m2ratClvvPGG6RginvPdd9+ZjmBLKvwWue+++4iJ0c1YRKw0\ne/bT5OXlmY5hOxrqscDp06dp3ry56RginnTx4kXCw8NNxwgaDfXY1KpVq0xHEPEsNxf9mlLht8CU\nKVOYO3eu6RgiIoCGeiyRm5tLZKSu1yNiglvryiWmhnpGARnAXuCJStaZV/L9HUBCLWzTUT766CPT\nEUQ8acOGDaYj2FKghT8UeAV/8e8J3AX0KLfOGCAGuA6YCvwlwG06zujRo01HEPGktWvXmY5gS4EW\n/v7APiATKADeAsaVW+dWYEnJ8hagCdAywO2KiFRpy5bPTEewpUALf1vgcJnHR0qeq2qddgFuV0Sk\nSps3byYnJ8d0DNsJ9G7E1T1qUv7AQ4U/l5SUVLqcmJhIYmJijULZTb9+iWzblmo6hognuW1iRWpq\nasDXIAp0Vs9AIAn/GD/AbKAYeLbMOv8DpOIfBgL/geChwMlyr+XaWT0jRkxgw4Z++P97RMRKeXl5\n1KtXz3SMoDExq2cb/oO2nYC6wCRgdbl1VgP3liwPBM7yw6Lvahs2/A0VfRHr/exnP6Nu3bqmY9hO\noEM9hcCjwD/wz/BZDHwFPFTy/YXAOvwze/YBF4CfB7hNEZFqCQ8Pv9QjljICLfwA75d8lbWw3ONH\na2E7IiJXRTdbr5id/hS6doz/7runsmzZq6ZjiHiSW+vKJTUZ46+NHr9UITRUl0QSsdYvgC2MHt3Y\ndBBbUuG3wP79GaYjiHhMItCU+Phc00FsSYXfAp98omv1iFjLfwGBnJxfGc5hTxrjt4BmFYiYU1BQ\nQFiYe/u4uhGLiEg5oaGhpiPYjgq/BW666SbTEUQ8S5+4f0iF3wKjR48mJGSq6RginlRcXGw6gu2o\n8FsgNzeX4uJFpmOIeFJIiMpceXb6DOTag7sxMTHs37/fdAwRT3JrXblEB3dt6vHHHzcdQUSklAq/\nBdze4xARZ3Hv5FYbWb9+vekIIp5z5513cd11vUzHsCUVfgusXbu2dDksLIzCwkKDaUS8ITY2liee\nmGU6hi1pqMcCy5cvL11W0RexxtNPP8WLL75oOoYtaVaPRbZu3cqAAQNMxxDxlH79+vH555+bjhFU\nmtVjY/379+fjjz82HUPEU6ZNm2Y6gi2p8FvkxIkTJCcnm44h4hm//e1vmThxoukYtqTCb5F58+bz\nl7/8xXQMEc8YMGAA9evXNx3DllT4LeDz+Rg+fJjpGCKekpiYaDqCbanwW+DcuXMMG6bCL2KlV199\nzXQE21Lht0Djxo356U9/ajqGiKccPXrUdATb0nROi+zYsYPevXubjiHiKW6uKZdoOqeNbdy4iTp1\nIk3HEPGM8eNvMx3BtlT4LfLBBxvx+ZJMxxDxjOnTdVXcymioxwI+n083gxCx2M6dO4mLizMdI+hq\nMtSji7RZIDc313QEEU85e/YsjRs3Nh3DttQNtUBkZCSvvbbYdAwRz9AsuitT4bdIZuZh0xFEPOPD\nDz8kPz/fdAzbUuG3yLBhQ01HEPGUBx54wHQE21Lht0irVq249trupmOIeIaujVU5zeqx0IULF2jY\nsKHpGCKud801zThwYC9NmjQxHSXodAKXzc2d+xvTEUQ8ITv7NIsXv246hm2px2+RoqIiwsI0e1bE\nCl999RXdu3tjaLUmPX4Vfovk5uYSGalLNohYJScnh4iICNMxgk5DPTYWERFBRkaG6RginpCUlOSJ\nol9TKvwWUo9fxBotW7Y2HcHWVPgtlJ2dTYMG15qOIeJ64eGhpiPYmgq/hU6cOIHP18h0DBHXO3ny\npOkItqbCb6ETJ06Qk7PDdAwRV2vcuBkzZ840HcPWVPgt1Lp1a8aMGWc6hoirnTv3DeHh4aZj2Jqm\nc1qsZOoVAD179iQ9Pd1gGhF32r59O3369DEdwxKax+8Ay5Yto3v37iQkJBAaqgNQIsHglTn8oMLv\nCIWFhYwdO57169eajiLiWl6oJZfoBC6HaNeug+kIIq51882JpiPYngq/xcLCwnjkEV0nXCRYevfu\nZzqC7WmoxwCfz0fv3gns3KmpnSK1zSt15BKrb7beFFgOdAQygTuAsxWslwmcA4qAAqB/ANt0hQUL\n/qKiLxIEjzwyzXQERwikx/8ccLrk3yeAa4AnK1jvINAXOFPF63mmx5+bm0u/fv00lVOkluXl5VGv\nXj3TMSxl9cHdW4ElJctLgPFXWNdOQ0rGRURE8NBD/8d0DBHXeP3115k06R7Onq1o0EHKC6QgZ+Pv\n5V96nTNlHpd1APgW/1DPQuDVSl7PMz1+gG3btnHjjTdSUFBgOoqI4506dYpmzZqZjmFEMMb4NwCt\nKnj+qXKPfSVfFRkMHAeal7xeBrCxohWTkpJKlxMTE0lMTKwinnP17NlTRV+klgwZMoR//etfl50Z\n71apqamkpqYG9BqB/C9lAInACaA1kAJUda+zOcB3wAsVfM9TPf78/Hzq169vOoaI473wwgvce++9\n6vFfhUDG+FcD95Us3we8W8E6kcCl6xA3AEYAuwLYpmvUq1ePVq10swiRQP3qV7/ilVcWem4aZyAC\n6fE3BVYAHbh8Omcb/OP4Y4FrgXdK1g8DlgHPVPJ6nurxA574WCpihe7de/Dll2mem9ED1vf4zwDD\nga74e/KXDqcfw1/0wX9gt3fJVyyVF31P+uMf/2g6gogrPPDAFE8W/ZrSJRsMmjBhgukIIq7wox/9\nyHQER7HTWIPnhnrAPw2tRYsWpmOIOFphYaFnL3Ouq3M6UPPmzXnwwQdNxxBxrMmTJxMSolJ2NdTj\nt4Hz588TFRVlOoaI4zz55JM884y3Dx2qx+9QOs1cpGYaNWpU9UryAyr8NnDw4EHTEUQcadmyZaYj\nOJIKvw0MGTIk4FOwRbxm9OjbefPNN03HcKRArscvtWjpUvVcRK7GunUrTUdwLPX4beKxxx5l2LDh\npmOIOIZXJ4PUBhV+m4iPj+fvf6/ockciUt706b+gsLDQdAzHUuG3kby8PLp162Y6hojtDR06hPDw\ncNMxHEuF30aio6NZuHCh6RgitvTQQ4+xd+9ejh07xrhx40zHcTQVfpsZOnQoS5Ys1QWnRMpZuHA+\ne/fupXXr1rqybYBU+G1o7drV5Ofnm44hYjsNGzY0HcEVVPhtaOHChdx9992mY4jYTo8ePUxHcAU7\nfV7y7LV6KtOtWze+/vpr0zFEbGHXrl3ExsaajmE7ulaPy6joi8Drr/+VuLi+9OzZ03QU19CZuzaW\nkZFBTk4OOTk53HTTTabjiBjRoUN7du7cZjqGq6jHb2PdunUjISGBtLQ001FEjFixYgU33HCD6Riu\no8LvABMnTuSOO+4wHUPEct27d9e9KoJAB3cdZMiQIWzcuNF0DBFLJCQMYPv2zZqzXwUd3HW5ESNG\nmI4gEnSjR99C37792bz5IxX9IFHhd5AZM2Zw7Ngx0zFEgmrUqB+zbdsWnb0eRCr8DlK/fn1at25t\nOoZI0HTv3p2+ffuajuF6KvwONHHiJNMRRIKiUaNGDB482HQM11Phd6AVK97i3LlzLFiwwHQUkVqx\nefNmzp07p1spWsROR040q+cqderUmUOHMk3HEAnYgQMH6Ny5s+kYjqRZPR6TmXkQn8/H+fPnTUcR\nqbE6depQVFRkOoanqPC7QMOGDQkL09U3xHkGDhxKXl4eMTExpqN4igq/C5w+fVr3HxXHWbx4MevX\nv0fdunVNR/EcFX4XaNasGUeOHKFHjx6sXLmSGTOeMB1JpFJxcXEUFhYyefJkGjVqZDqOJ2l8wCXa\ntm1Leno6AAcOZJoNI3IFI0eOJDQ01HQMT9OsHpfSqe5iN8nJyWzevJlbb72VYcOGmY7jGjWZ1WOn\n6qDCX8tycnJo0KCB6RgitGzZkhMnTpiO4UqazimXiYyM5PTp07qeuRj15JP/yZtvLjUdQ8pQ4Xe5\n6Ohohg5NBGDJkiVmw4gnzJ8/ny5dunDPPffw61/P4ZlnfsePfzzcdCwpQ0M9HjN//nymTXucG24Y\nyuefp5qOIy7TsGETzpzJIjw83HQUz9BQj1Rp+PDhtGnTntWrk01HERfp1WsAv/3ts3TocK2KvgOo\nx+9BxcXF+Hw+ne0rtWbt2rWMGjWKgwcP0qVLF9NxPEU9fqmWkJAQQkND2bp1K4MHjzQdRxwmPj7+\nB8+NHj2akJAQFX2HUOH3sBtuuIFNmz4gIyODoUOHmo4jDjFr1iyysrJKPzn6fD6dN+IwdmotDfUY\nVlhYyNGjR8nKyqKoqIhBgwaZjiQ2kpubS1paGr169SIqKsp0HCmhE7ikVqkXJ5cUFxfr/WBTGuOX\nWpWbm0t6ejr16kXQvHlHWrfWpXO9JCsri2+++YZvv/1WRd9l7NSa6vE7xKlTp9i2bRt33DGJ777T\nTWCcqGnTaM6c+ab0cefOnTl48CAAZ86coX79+kRERJiKJ1dBPX6xRPPmzRk5ciSzZz9pOopU0/r1\n6xk/fmLp41Wr3sHn85Gdnc3dd9/D0qVL8fl8FBcXc80116jou1wgPf6JQBLQHbgB+KKS9UYBfwJC\ngdeAZytZTz1+B8rKyuLLL78kLCyMsLAwPvzwQ9asWcPs2bNJSkpi9+7dpiN6Xq9e/dm9ewsABQUF\npKSk0LdvX6Kjow0nk9pQkx5/ILoDXYEUoE8l64QC+4BOQDjwJdCjknV9bpaSkmI6QtBUtm/Hjx/3\n3XbbT327du3yxcff6PvFL2b5AH1Z9JWVleUbN268b86cpBq1n1u4ff9K2vuqBDLUkwF8XcU6/fEX\n/kygAHgLGBfANh0rNTXVdISgqWzfWrVqxTvv/I3Y2FjWrl1OUtJTLFr0Knv27OHixYvk5+fz9ttv\nM3LkLaU/81//NafS7XTt2o0hQ35S2/EdITLSP30yJSWl9LmXX55Pv35DyczMZOXKlSQlJXHy5Ek2\nbdrEY489TvPmzXn33VUkJVX+fwrufm+C+/evJoJ9zn5b4HCZx0eAAUHepthQu3btAHjwwQcue37C\nhAlMmDCBtLQ0GjVqRExMDDNm/JKFCxdx/fXxjBz5/ZnF6en/YsyYO/nRj35CSsp7P9hG374DOH78\nFLffPpZGja7h97//7+DuVC1r3749hw8f/sHzzZq1IC1tO+Hh4bRs2ZI///nPREZGcv/99zNt2qMA\ndOzYkdtvvx2AFi1aMHjwYEuzi7NUVfg3AK0qeP4/gR/+5v2QBu2lWhISEkqXo6KimDlzBgA+n4/9\n+/cTHR1NaGgokyffQc+e3Zk1q5jDhzP54IMPePPNpUyb9hgNGjRgypSHGDRoEHfddRedOnVg6tTL\n/9Dce+99dOzYgYSEBFq1asVzzz3HnXfeSX5+EbfcMoqxY8fy/PPP89Zbf2P37jQuXMjh0KGjnDp1\nrPQ1Pv74YwoKCsjIyGDPnj3Mmzev9Htt2rRj8OBBxMfHs2jRIg4fPkybNm04dsz/89nZ2TRu3JjM\nzEyuvfZaAD799FMGDRqEz+fjm2++4d13/87999/H2rVrGTp0KE2aNLlsHx5++OHa/c8Xz6mNAwIp\nwK+o+ODuQPwHgEeVPJ4NFFPxAd59gC70ISJydfYDlp9kkwL0reR7YfhDdQLqcuWDuyIiYnO34R+/\nzwVOAO+XPN8GWFtmvdHAHvw9+tlWBhQREREREYMmAv8Ciqj8PADwTwXdCaQBW4Mfq1ZUd99G4Z8W\nuxd4woJctaUp/gP/XwPrgSaVrJeJs9quOu0xr+T7O4CEStaxq6r2LxH4Fn97pQFPW5YscK8DJ4Fd\nV1jHyW1X1f4l4pC2q84JYAAH8RcaJ6ntk9vs5jlgVsnyE8AfKlnPSW1XnfYYA6wrWR4AfGZVuFpQ\nnf1LBFZbmqr23Iy/mFdWGJ3cdlD1/iVyFW1n8lo91TkB7BI7XUyuOtx+ctutwJKS5SXA+Cus65S2\nq057lN3vLfg/6bS0KF+gqvt+c0p7lbcRyL7C953cdlD1/sFVtJ0TLtLmA/4fsA140HCW2lTRyW1t\nDWW5Wi3xf+yk5N/KfoGc1HbVaY+K1mkX5Fy1pTr75wNuxD8Usg7oaU00Szi57arjqtou2GfuBnoC\nGMBg4DjQvOT1MvD/9TPN7Se3VbZ/T5V7fKVrhdi17SpS3fYo36uyezteUp2cXwDtgRz8s/HexT9k\n6RZObbvquKq2C3bh/3EtvMbxkn9PAavwf2S1Q/EIdN+O4m+oS9rj74XYxZX27yT+PwongNZAViXr\n2bXtKlKd9ii/TruS55ygOvtX9uYK7wML8B+jORPcaJZwcttVx1W1nV2Geiobm4oEGpUsNwBGcOWj\n9nZU2b5tA67j+5PbJuGcA2urgftKlu/D37soz2ltV532WA3cW7I8EDjL90Nedled/WvJ9+/X/iXL\nbij64Oy2qw7HtF11TgC7Fv/sgy+B3TjnBDC3n9zWFP/YffnpnE5vu4ra46GSr0teKfn+Dq48G82O\nqtq/R/C31ZfAp/gLpFMkA8eAi/h/9ybjrrarav+c3HYiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIi\nIu7x/wFXd6g2AAAABElEQVRzcRWuwEGC6QAAAABJRU5ErkJggg==\n",
       "text": [
        "<matplotlib.figure.Figure at 0x11102dfd0>"
       ]
      }
     ],
     "prompt_number": 45
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Why $L_1$-norm can be important\n",
      "$L_1$ norm, as it was discovered quite recently, plays an important role in **compressed sensing**. The simplest formulation is as follows:\n",
      "- You have some observations $f$ \n",
      "- You have a linear model $Ax = f$, where $A$ is an $n \\times m$ matrix, $A$ is **known**\n",
      "- The number of equations, $n$ is less than the number of unknowns, $m$\n",
      "The question: can we find the solution?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The solution is obviously non-unique, so the natural approach is to find the solution that is minimal in the certain sense:\n",
      "$$ \\Vert x \\Vert \\rightarrow \\min, \\quad \\mbox{subject to } Ax = f$$\n",
      "\n",
      "Typical choice of $\\Vert x \\Vert = \\Vert x \\Vert_2$ leads to the **linear least squares problem** (and has been used for ages).  \n",
      "The choice $\\Vert x \\Vert = \\Vert x \\Vert_1$ leads to the [**compressed sensing**](https://en.wikipedia.org/wiki/Compressed_sensing) and what happens, it typically yields the **sparsest solution**.  \n",
      "[A short demo](tv-denoising-demo.ipynb)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "It is worth to note that formally $\\Vert x \\Vert_0$ is the number of non-zero elements in $x$, so $\\Vert x \\Vert_1$ is an approximation to the number of non-zeros."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Matrices and norms\n",
      "How to measure distances between matrices? A trivial answer is that there is no big differences between matrices and vectors, and here comes the **Frobenius** norm of the matrix:\n",
      "$$\n",
      "  \\Vert A \\Vert_F = \\Big(\\sum_{i=1}^n \\sum_{j=1}^m |a_{ij}|^2\\Big)^{1/2}\n",
      "$$\n",
      "There is a more general definition though:"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Matrix norms\n",
      "$\\Vert \\cdot \\Vert$ is called a **matrix norm** if it is a vector norm on the linear space of $n \\times m$ matrices, and it also is consistent with the matrix-by-matrix product, i.e.\n",
      "$$\\Vert A B \\Vert \\leq \\Vert A \\Vert \\Vert B \\Vert$$\n",
      "\n",
      "The multiplicative property is needed in many places, for example in the estimates for the error of solution of linear systems (we will cover this subject later).  \n",
      "Can you think of some matrix norms?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Operator norms\n",
      "The most important class of the norms is the class of **operator norms**. Mathematically, they are defined as \n",
      "$$\n",
      "    \\Vert A \\Vert_* = \\sup_{x \\ne 0} \\frac{\\Vert A x \\Vert_*}{\\Vert x \\Vert_*},\n",
      "$$\n",
      "where $\\Vert \\cdot \\Vert_*$ is a **vector norm**. It is not diffcult to show that operator norm is a matrix norm. Among all operator norms $p$-norms are used, where $p$-norm is used as the vector norm. Among all $p$-norms three norms are the most common ones:  \n",
      "- $p = 2, \\quad$ spectral norm, denoted by $\\Vert A \\Vert_2$.\n",
      "- $p = \\infty, \\quad \\Vert A \\Vert_{\\infty} = \\max_i \\sum_j |A_{ij}|$.\n",
      "- $p = 1, \\quad \\Vert A \\Vert_{1} = \\max_j \\sum_i |A_{ij}|$.\n",
      "\n",
      "Note that Frobenius norm is not an operator one."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Spectral norm\n",
      "Spectral norm, $\\Vert A \\Vert_2$ is undoubtedly the most used matrix norm. It can not be computed directly from the entries using a simple formula, like the Euclidean norm, however, there are efficient algorithm to compute it.  It is directly related to the **singular value decomposition** (SVD) of the matrix. It holds\n",
      "$$\n",
      "   \\Vert A \\Vert_2 = \\sigma_1(A)\n",
      "$$\n",
      "where $\\sigma_1(A)$ is the largest singular value of the matrix $A$. We will soon learn all about this. Meanwhile, we can already compute the norm in Python."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import numpy as np\n",
      "n = 100\n",
      "a = np.random.randn(n, n) #Random n x n matrix\n",
      "s1 = np.linalg.norm(a, 2) #Spectral\n",
      "s2 = np.linalg.norm(a, 'fro') #Frobenius\n",
      "s3 = np.linalg.norm(a, 1) #1-norm\n",
      "s4 = np.linalg.norm(a, np.inf) #It was trick to find the infinity\n",
      "print 'Spectral:', s1, 'Frobenius:', s2, '1-norm', s3, 'infinity', s4"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Spectral: 19.7629070969 Frobenius: 101.273176213 1-norm 97.6821174651 infinity 98.3147621883\n"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Take home message\n",
      "- Norms are measures of smallness, used to compute the accuracy\n",
      "- $1$, $p$ and Euclidean norms \n",
      "- Matrix norms \n",
      "- $L_1$ is used in compressed sensing as a surrogate for sparsity."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##### Questions?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from IPython.core.display import HTML\n",
      "def css_styling():\n",
      "    styles = open(\"./styles/custom.css\", \"r\").read()\n",
      "    return HTML(styles)\n",
      "css_styling()"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "skip"
      }
     },
     "outputs": [
      {
       "html": [
        "<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>\n",
        "<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>\n",
        "<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>\n",
        "<style>\n",
        "    @font-face {\n",
        "        font-family: \"Computer Modern\";\n",
        "        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\n",
        "    }\n",
        "    div.cell{\n",
        "        width:80%;\n",
        "        margin-left:auto !important;\n",
        "        margin-right:auto;\n",
        "    }\n",
        "    h1 {\n",
        "        font-family: 'Alegreya Sans', sans-serif;\n",
        "    }\n",
        "    h2 {\n",
        "        font-family: 'Fenix', serif;\n",
        "    }\n",
        "    h3{\n",
        "\t\tfont-family: 'Fenix', serif;\n",
        "        margin-top:12px;\n",
        "        margin-bottom: 3px;\n",
        "       }\n",
        "\th4{\n",
        "\t\tfont-family: 'Fenix', serif;\n",
        "       }\n",
        "    h5 {\n",
        "        font-family: 'Alegreya Sans', sans-serif;\n",
        "    }\t   \n",
        "    div.text_cell_render{\n",
        "        font-family: 'Alegreya Sans',Computer Modern, \"Helvetica Neue\", Arial, Helvetica, Geneva, sans-serif;\n",
        "        line-height: 1.2;\n",
        "        font-size: 160%;\n",
        "        width:70%;\n",
        "        margin-left:auto;\n",
        "        margin-right:auto;\n",
        "    }\n",
        "    .CodeMirror{\n",
        "            font-family: \"Source Code Pro\";\n",
        "\t\t\tfont-size: 90%;\n",
        "    }\n",
        "/*    .prompt{\n",
        "        display: None;\n",
        "    }*/\n",
        "    .text_cell_render h1 {\n",
        "        font-weight: 200;\n",
        "        font-size: 50pt;\n",
        "\t\tline-height: 100%;\n",
        "        color:#CD2305;\n",
        "        margin-bottom: 0.5em;\n",
        "        margin-top: 0.5em;\n",
        "        display: block;\n",
        "    }\t\n",
        "    .text_cell_render h5 {\n",
        "        font-weight: 300;\n",
        "        font-size: 16pt;\n",
        "        color: #CD2305;\n",
        "        font-style: italic;\n",
        "        margin-bottom: .5em;\n",
        "        margin-top: 0.5em;\n",
        "        display: block;\n",
        "    }\n",
        "    \n",
        "    .warning{\n",
        "        color: rgb( 240, 20, 20 )\n",
        "        }  \n",
        "</style>\n",
        "<script>\n",
        "    MathJax.Hub.Config({\n",
        "                        TeX: {\n",
        "                           extensions: [\"AMSmath.js\"]\n",
        "                           },\n",
        "                tex2jax: {\n",
        "                    inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
        "                    displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
        "                },\n",
        "                displayAlign: 'center', // Change this to 'center' to center equations.\n",
        "                \"HTML-CSS\": {\n",
        "                    styles: {'.MathJax_Display': {\"margin\": 4}}\n",
        "                }\n",
        "        });\n",
        "</script>\n"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 1,
       "text": [
        "<IPython.core.display.HTML at 0x104335990>"
       ]
      }
     ],
     "prompt_number": 1
    }
   ],
   "metadata": {}
  }
 ]
}