AppendixESamplingDistribution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix E Sampling Distribution "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## E.1 Sampling from the true population\n",
    "\n",
    "Most properties in statistics describe the behaviour of a procedure under repeated sampling from the true population. The problem is, in the real world, the true population is unknown. In a synthetic environment (e.g., simulation in Appendix D), we do know the true population as well as the sampling scheme. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAM1BMVEUAAABNTU1oaGh8fHyM\njIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enw8PD/AAD///89ODILAAAACXBIWXMAABJ0\nAAASdAHeZh94AAAdrElEQVR4nO3d22LiOBZGYRsDoVJA8f5P2xzTwfjsX9Le0voupjPTAxa2\nVjBGkOoCYLUq9QCAHBASIEBIgAAhAQKEBAgQEiBASIAAIQEChAQIEBIgQEiAACEBAoQECBAS\nIEBIgAAhAQKEBAgQEiBASIAAIQEChAQIEBIgQEiAACEBAoQECBASIEBIgAAhAQKEBAgQEiBA\nSIAAIQEChAQIEBIgQEiAACEBAoQECBASIEBIgAAhAQKEBAgQEiBASIAAIQEChAQIEBIgQEiA\nACEBAoQECBASIEBIgAAhAQKEBAgQEiBASIAAIQEChAQIEBIgQEiAACEBAoQECBASIEBIgAAh\nAQKEBAgQEiBASIAAIQEChAQIEBIgQEiAACEBAoQECBASIEBIgAAhAQKEBAgQEiBASIAAIQEC\nhAQIEBIgQEiAACEBAoQECBASIEBIgAAhAQKEBAgQEiBASIAAIQEChAQIEBIgQEiAACEBAoQE\nCEQIqQKcWTDL9eEk2ASgREiAACEBAoQECBASIEBIgAAhAQKEBAgQEiBASIAAIQEChAQIEBIg\nQEjAREOLvAkJmGT48xKEBEww9tEjQgLGjX6Ij5CAcYQErDf+uXJCAkYRErDehK86ISRgDCEB\nAoQECBASsMarGS42AIv9nw0hAQu9hTPWESEBnT5fFxESMFsrnZGOCAno8vkkNJQRIQGdus7m\n+jMiJKDLpNdFbzdYsI1FI7O2CWAAIQF3Eyb/yK0JCcWbOP1Hb09IyN7AJbTpAQzd+Zy7IST4\nNDDD5yUwfPeEhJwNznFFSHPvhJDg0OAsn/tksmATHf/3BVtYMCp7m4BnEUKa+UKLkODPYCrt\nf7n+0t2k/++Cu59/E4ObgGORQprzZhQhwZ3hVJQhzRhTlJsY3AT8IiRDm4BfI6mk6IiQ4A8h\nGdoEHBtJJUFHhASHCMnOJuDZSCrxOyIkeDT6nBM5I0KCT+PPOVEzIiR4Ff05ZxghwStDGRES\nIEFIgAAhAQKEBAgQEiBASIBAkpBCfNsekBIhAQIRQ5rxKRFCgjMRQ/pbExJyFfPU7rytmtP9\nHji1Q2bivkb6U1V/LoSE/ES+2HBqqu2ZkJCd6Fftvqr6m5CQm/iXv4+b8dXvhARnUryPtCMk\n5MbOEqEE3/wCqES9/H19Kmq+n3fC+0jIScSQzo83ZLePOyEk5CRiSPvqcK3pUDf3OyEk5CRi\nSPXjhqd6cyIkZCbqotXHP89NQ0jITMSQNtX59VNDSMhLxJAO1e7506lqCAlZiXn5e/9Tzzcf\no0Beor4he9y+fjrtCAk5sbOyIfImACVCAgQICRAgJKSVyRplQkJK2Sz3JySkk9EnZwgJyUz/\nokP7CAnJEFJozncqJml99a7vg05ISIWQgvO9TzFJuyPfJRESEiEk/SgSbAKpEZJ+FAk2gdQI\nST+KBJtAcjl1REhIhpCC871PMVFGHRES0iGk0JzvVEyUT0eEhKQyyYiQkFoWGRESIEFIgAAh\nAQKEBAgQEvwxeIGCkOCNyUvmhARfjL6JS0hwxeqyIkKCK4RkbBNwyexHLwgJnhCStU3Ao3ZH\ndkoiJDhCSOY2AY8Iydwm4BEhmdsEXLLaESHBFUKytgn4ZLQjQoIvhGRsE3DKZkeEBHcMZkRI\n8MhcRoQESBASIEBIgAAhAQKEBAgQEiBASIAAIQEChAQIEBIgQEiAACEBAoQECBASIEBIgAAh\nAQKEBAgQEiBASIAAIQEChLSUwS/gQDqEtIzJr4RCOoS0hNEvKUQ6hLSA1a/NRTqEtAAhoY2Q\n5jP7p0WQDiHNR0j4QEiztTuiJBDSAoSET4Q0GyHhEyHNRkj4REjz0RE+ENJ8hIQPhLQAHaGN\nkBYgJLQR0hJ0hBZCWoaM8IaQliIj/EJIgAAhAQKEBAgQEiBASIBAgpAOdbU5hN0EEFnMkI7b\nqj5cvu7vvzRhNgGkETGk472gfbU7X07bavA5iZDgTMSQdtX+ctlX9e3nc7UJsQkgkYghPRYC\nVNtf/0W9CSCR6CH9eZzTPZ6Y1JuAX71LrnysxYp6and9dfRwvp/m6TcBr3oXAXtZHRwxpHP9\nszuq4SckQipM78dS/HxeJer7SPtXPvXg8xEhFab3g5KOPkHJygYkR0hKfvYZtKqqJ5jef2FQ\nzJDOu6pqvp93wuVvPBHSPLeLDVe8j4Q37Vx+XZJyVFLEkPa3ZUHnQ31fZkdIeCKkmerHDU/1\n5kRI+EFIc2/3vOG5aQgJPwhppk31WtiwaQgJP3pzcdRRzJAO1e7506lqCAkvhDTT/mdXfI/s\nFdO7DGq9ufjpKO4bssft66fTjpDwQkih2N5nUOvNxU1HhAQTemvxkREhwYreWjxkREiABCEB\nAoSEwHycmq1FSAjKy8WCtQgJAfm5fL0WISEcR2+orkVICIeQxDcxuAmEV1XllERICIaQ1Dcx\nuAkE1+4o65IICaEUFNK/f//m34iQMEkxIf37R0gIp5CQHhlxaodgSujo9WxESAgm/5D+P6kj\nJISTeUe/XxsREsLJOaR/75cYCAkBZdvRv/aVOkJCUPlm9P4/ERICKyEjQgLm6Xn3lZCA6XoX\nMRASMNXAWiBCAqYZXFJHSBiQ3YWC5UZWphISemV56XqZ0QXehIQe2b6ZOt+Ez0kQErrlvLxn\nnkkfNyIkdCOkh4mf2iMkdMr/IxBTfCyp60VI6ERIczIiJHRrd1RgSfO+iYGQ0KX4kOZ+oQkh\noUvhIc3/XiBCQpeiQ1ry9VqEhE7ldrTsW+oICZ1KDWnhlz0SEnoU2dHSjAjJl5hTusCQlmdE\nSJ5EntSldbQmI0LyI8G8JqPpCMmJNGdaZDQVITlR4EuWaNZnREhelHo1Orw5K1MHEJIPWYeU\n8AGJMiIkJ9od5VRSwkekquhCSE4EDilhmQl/OQgzIiQngoaU8kku3dOsNCNCciJgSGnPF1OF\nJM6IkLyI0lH8koKesPaTZ0RIXhCSToCMCMmNKB3FLqm99RjbD5IRIblBSBKBMiIkP6J0FLmk\n2JsPlhEheRJgshUVUsCMCMkX+VRLHFLMM8uhjASbJqSihQtp2n3FCmlwSZ1k44RUtkAzefLd\nReloUkYrt09IZQsS0ow7jBDS8Esj1QAIqXCBO5pVkmTrLWNXGAgJEqlDCrtmdvwvVqqekQmp\ndIE7mnrJQbLplil/sZKQoCJ+SpDNzZWmvG3UHuvy0RIStE8Jurm5yrR3XwkJYS2fUyZCmrqI\ngZAQ0ppZZSCk6WuBCAnhrJtXyUOataRONlZCQsvamZW2o5krUwkJoXgOacGfrBSNlZDwbn0H\nqTpa9GWPhIQwvIa09DtTRWMlJLxpd7S2JP0IO636G2GKoRIS3ihCCrt+rsPqP260fqiEhDea\nkMKtn+sQ9CPkUxES3qhC6rhbyR19MJERIXkVbmKG6CjcmZ6RjAjJp5AvQfQhhbv2YCYjQvIo\n8EWxkB1JB2woI0JyKNjEDHT/gcZrKiNCcih0SOJnPP2p4o2xjAjJnzATs3sbyvtS3qe5jAjJ\nnxghCa8KtocruFuDGRGSOwEmZlDy8ZrMiJDcKTukpStTw4sZ0nlXVc33804G78X45Eip5JDs\nZhQ1pHN934/bx50Q0jLeQtK9pjNc0SVqSPvqcK3pUDf3OyGkhZx1pBqv7YyihlQ/bniqNydC\nWs5bSJL3vaxnFDWk1048Nw0hreCsI8F47WcUNaRNdX791BDSct5CWrtSwkNGUUM6VLvnT6eq\nIaTlnHV0WbVSwkdGcS9/73/25PfITnUyP5LxldFN5hlFfkP2uH39dNoR0iorM/JRoZ+MWNlQ\nJB9PaJ4yshSSt5fQbvnYz74yYolQWdrvQZktyVtGLBEqyWdGNkOyvKSuF0uEStGVkcWSXGbE\nEqFi9HRkbU/7rOjCEqGcvXXSF5KpXe02I5YI5es9lN6ODO1qxxmxRChX7VTsh+Q6I5YIZard\nSn9HRna184xYIpSp6SGlHumd+4wsrWyIvImsfdRiOqQMMiKkPE0PKfVIM8mIkLLU0YvVkDLJ\niJCyND2kxAPNJiNCylJXMGQUFiFlaFpIacc4eUld+qFOQkg56mrGTkOXmRmZGPEYQspR95OP\nmUk5+ZzOVPzDCClLPSdxJibk9JdGlk5FxxBSluxOwTlXGOw+ik+ElCejM3DWhbruE1SjVoa0\n+TrJhtKzCSxjcPrNvN5dUkjXhxeiJdv7zAtjk2/u20btjkw9mA8rQzr/2YVoyfQuwxLz330t\nKqSbv18bdUumd5knVqbfkkUMnyGZeCg9NBcbjrdv2jqsH83AJjCflfm3bC1QgSF9N/eH2QjG\n07cJzGZlBi5eUueqpPUhnb+uT0eb7/O1pm33DVZvAgsYmYErVqYWFdLf28WG/fHxL2SP0/AO\nc8PEDFy3wNtTSWvfR7o+GR1e37JV1YoRtTeBRQzMwNXfmVpQSNX2WzaUnk1gkeQzUPHVw+WE\ndO79f61id395kXoKaj60l/pRzMBauzytnYLrJq3qs69+OiKkTK0Lad28DZYRIRncRObWzMF1\nMzdgRpanBSFlStPR7Lkr+0KTro4sTwtCylSSkITfC+SsI0LKlqajObdUfr2Ws4wIKV+xQ9J+\nS52zjAgpY5qOJt5W/GWP3p6PCClrSybhoiks/85UQtIwv9u8mD8FF0zhAF89TEga5ndbvuZO\nYcWSuvWjSI+QcHl75po1g8NkNHcUFhAS3l9LzZjCAf+cBCFJ2N9vGWnP16kzOOxfZXHWESEV\n72PGTpvCof+40ecobAdFSKXrnrEjHUX4G2E9eYfd6HKEVLjOFyPpM2qNwv5pHiEVrjOkwfOo\niH+xsus50mhJhFS2dkfD0/T6r5P84VdCWsjmvsrRnJBSZeTiWjghlW16SK+MEsxkQlrK5K7K\n0uSQfmUUeyrPO/1MhJAKN3WO/s6IkD4RUuEmzdF/7xlFnsqEtJjFPZWr8Sn6mREhfSCk0o1N\n0Y6KUpcUc9tTEVLxBqfo83o3IY0hJPSvCPp52yjx2ZX9jggJN8MZEdI4QkK390UMic+uzHdE\nSOjUXguU/GWK7YwICV06ltSlP7uynBEh4VPnytT0IdlGSHjXt8CbjgYREn4b+pwEGQ0gJPxv\n7ONGZNSLkPCS5lN7mSAk3AX7ztRCEBIuHRlxEjcTIeHznI7LCrMRUvF6MyKlGQipcJ8vjXjr\ndQlCKlrXFQZCWoKQCjZhLRAlTURIxeq53k1IixBSoaYsqaOk6QipSP3vvhLSMoRUoCkrUwlp\nHkIqzvBaIEJahpAKM7qkjo4WIaSSTFmZSkiLEFI5Ji7wpqMlCKkUkz8mQUhLEFIZ5nzaiI4W\nIKQSzP3QHhnNRkj5W/LZVzKaiZByx0fIoyCkvJFRJISUMzKKhpDyRUYREZIbM1//k1FUhOTE\nzCvSZBQZIbkw8z1SMoqOkDyYtWqH70xNgZA8mBESGaVBSA5M/2QDFaVCSA5MDYmM0okY0owP\njBHSb+2OevYdGaUUMaQDIS0zKSQySivmqd2xbkJvIksTQiKj1KK+RjpW+9CbyNFoSGSUXtyL\nDYfqGHoTORruiIws4KqdA0MhkZENhORBb0dkZIWdkCZf0itQT0hkZEfMkM67qmq+n3fC5e85\nujoiI0sihnSu7/Ng+7gTQpqHjGyLGNK+OlxrOjzeTCKk2X5lxMpUcyKGVD9ueKo3J0Jag4wM\nirrW7vHPc9MQ0nJUZFLEkDbV+fVTQ0gLkZFRURet7p4/naqGkJYgI7NiXv7e/9TzPfJWESF1\nISPD4i5a3b5+Ou0IaSYyMs3OyobIm3DGakYsQ3kiJA8sZ0RKd4Rkn/WMSOlCSPZZzYg/kfmG\nkGwzmxEhvSMkywxnNOPL9opASGYZX1JHSG8IySjjGU39sr1iEJJJxiu6EFIbIRlkPyNCaiMk\nczxkREhthGSMj4wuXGxoISRT3GRESC2EZIijjC68IfuOkMzwlREhvSMkI7xldGHR6htCMsFh\nRjdk9IOQDHCa0Q0ZPRFSco4zwg9CSsv6kjpMREgpkVE2CCkdKsoIIaVCRlkhpDTIKDOElIIg\nI64720JI8YkyIiVLCCk2YUakZAchxaV4bcRqUYMIKSbNJQZCMoiQ4hFdqeMTdRYRUiyyC96E\nZBEhxaF736jdESWZQEgxKN9+JSSTCCk48cpUQjKJkAKTL/AmJJNcheRv1oRYUkdHFjkKyd/E\nCbMylZAschOSv6kTbIE3HRnkJSR3kyfg5yTc7YsSEFIQYT9u5GpXDPL/CF6chOTrdUH4T+35\n2A9j8ngUD4QkF+fDr9b3wjgvx3MaHyG1OzK85/kM+UROjudUhCRFRpO5OJ7TEZIQGU03fEDN\nHuJehCRDRnMMHVDDB7mXj5DsX2zgO1PnGfjVaPo49yIkBTKaqz8k0we6n5OQTO9eKpqPkAip\nhYyW6A3J+LlHLy8hWT1zJqOFpj0hmTrWg9yEZPJaDhktNvEJydThHuIoJHPvLpDRGtOekEwd\n8CGuQjKFjNYhJEK6kJFAZy2EJGV975GRRFcrTjsipAXISOYzFUJSsrz7yCgsnx0R0kxkFBoh\nCRndfyypi8FlR4Q0HRnF4i8jQpqMimLylhEhTURGGEZIE5ARxhDSKDLCOEIaQUaYgpAGkRGm\nIaQBZISpCKkXGWE6QupBRpiDkDqREeYhpA5khLkI6QMZYT5CesfKVCxCSL+RERYipP9RERYj\npBcywgqE9GAqI3+fxgEh3ZjLiJS8ISSjGZGSQMSdSEimMvL6HTomRd2LpYdkKyNC0om8H8sO\nyVpGbr9n1J7Y+7HkkMxlREg6hBRpExYz8vu3GMyJvh9LDcliRoSkk3dIf7+290e13f8NtYlJ\nzC6pIySR+DsyYkjnza/H1QTZxCRmM5Icf+q7yTqkfVX/Od5/On3X1T7EJiawW9HN2sPPE9lD\n1iHV1fHn52NVh9jEKNsZrQ3J3jlhqpFkHdLbgxl+ZIEetvWMLuuu2sadORMkHErsjkp6RnKQ\nUVYhJR1MziFdXyN9n+4/pXiN5CKjy5rpF33uzBlO9puPefm7+fXINucgm+jjJaObpQefkFJu\nPu77SPv7+0j19ivu+0ieMrpZdOjbHSUuKflgIm+8gJUN3jJaiJD6hhBnY1FuMuluw+z2QjKy\nFpKN0UTcbsyQzruqar6fdxLn8ncxGVmZujZHE0HMJUL1fYduH3cSI6SCMrqYOJnqHwwhSW5y\nt68O15oO9X2ZXfiQDC+pC8PUzCWkIDe5qx83PNWbU/iQisvokvyC89Bgkg8nuARLhM5NEzqk\nAiu6EFJSEUPaVK83YTdN0JDKzOhibNFqWR3FDOlQ7Z4/naomXEjFZnRjaN4SUoCbPOx/9uf3\nyK5dvt+LzujGzqwtqqO4b8get6+fTrsgIRWfkS3lZGRpZcP6TeSUUSbzL5OHMUE+IeWWUTlz\nMAu5hJRjRqTkSB4h5ZRR9pe7pI/KzC7KIaSsMso9JOnjMrST/IeUWUZ5LwmQPjBTe8l7SLll\nlHdI0kdmaze5DinHlantjtJPESFCWnmTIJvIMaO8Q5I+MmO7yW1IWVZ0IaQ0d7ae05ByzSjr\nkKQPzdp+chlSvhldzP2mFSKk1TeRbiLrjAgpxZ0JuAsp84wu1q5GCRHS6pvINpF/RhmHxMWG\n1TcRbaKEjC7G3rFXIqS1N5FsopCMboxMDjnp1DfVkZ+QCsroxsLc0COklTdZvYnCMsqWdOpb\n6shHSGSUD+nMN5ORh5DyXFJXMOnMN5KR/ZDICC7YDomK4ITlkMgIbtgNyVpGZs7GYZHVkCxm\nREroZTMkqxmREnqYDSn8Rqaz9R46LLIZkrHJSkgYQ0jjqoqSMIKQxhESRhHSqHZHlIRPhDSK\nkDCOkEYREsYR0ihCwjhCGldQR5k/vIAIaVwxIWX/AAMipAnK6KiExxgOIU1QREhFPMhwCGmK\nEqYYIa1CSNNkP8GKeSEYCCFNlfnsIqR1CAk37Y4oaSZCwg0hrURIuCGklQgJN4S0EiHhjo7W\nISTcEdI6hIQHOlqFkPBASKsQEp7oaA1Cwg86Wo6Q8MJT0gqEhCdeJK1BSHgipDUICQ+8kbQK\nIeEhWEhlVElIuGt3pJr8pTzBERLuwoRUzrlihiHlf9BCCBJSQa+6sgupiKMWACGtk1lIxRw3\nveAd5X1E8gqpnOOmR0irEBKegneU9RHJKqSCjlsAhLQGIeFFvesISX6TOJso6sAFod1vRR0P\nQsJv0r1W0uEgJART0uEgJIRT0NHIKaSifgO6UNDhICQEVM7RyCqkkn4DelHKwSAkBFbGocgr\npIJOJWBLZiGVcyoBW7ILqZRTCdiSYUhAfIQECBASIEBIgAAhAQKEhG5c/JyFkNCFt+NmIiR8\nYoHIbISEDyxZnI+Q8IGQ5osa0nHf3A/NZvsn1CawHh/rWiBmSF+/Ds42zCYgQEgLRAzpu9qd\nLpe/zfZyPGyq7xCbgEC7I0qaImJITXW+/eNYfV1zGn5K4tAlREhLRAzpdUCq+td/0W4CAoS0\nRMSQ6scz0vl+YAjJLEJaImJI+6r5e7mcttXuct5d/yPAJqBARwvEvGr3uPZd1efroapPQTYB\nAUJaIOr7SIdrSpuv6w/1/hxoExCgo/lY2YAPhDSfnZA4eHZwKGazE1LkTWAQGc1ESOhGRrMQ\nEiAQdWXD5JdBhARnIoZ0ICRkK+ap3bFuQm8CSCPuB/uqfehNAEnEvdhwqI6hNwGkwFU7QICQ\nAAFCAgQICRAgJECAkAABQgIECAkQMBoS4MyCWa4PR8v8AJ8Yp5aXcb6YH6/5AT4xTi0v43wx\nP17zA3xinFpexvlifrzmB/jEOLW8jPPF/HjND/CJcWp5GeeL+fGaH+AT49TyMs4X8+M1P8An\nxqnlZZwv5sdrfoBPjFPLyzhfzI/X/ACfGKeWl3G+mB+v+QE+MU4tL+N8MT9e8wN8YpxaXsb5\nYn685gf4xDi1vIzzxdt4AZMICRAgJECAkAABQgIECAkQICRAgJAAAUICBAgJECAkQICQAAFC\nAgQICRAgJECAkAAB2yEdd1W1O6Uexajzvq7q/Tn1MEYcXsfa+Gh/xnnYmB7nO9Mhfd//MkBt\nfV+e6sc4bRd/fP2NheY+2k3a0fT7Gefex9F/Mh1SXR8v5221Tz2OEbv7CPfVLvVAhhzr5wT9\nW1336vW//U08oB4/4zxWu/Pt6cn0Xv2f5ZD+3CfouapTD2TE88gv+as60Ryq5jm+ffV9ue3b\nr7QD6vH/OLcO9uovloe5q46phzBJ/TzkloO//k76maC3U9BjtU07oB7/j/P1P1ieob9YHuam\nunzV9yd4276ep3Y2f8c/HNtPnEYn6LE1snPVJBvLLDZ350NVbe8vN1OPY9ThdrWhPqQexggX\nIV1aIzvcT0QdMLs7L7c9ervYsDP9m/7u6359yfowPYZ0qm2egX4yuzsvtz16e410snul9ulw\nO7W7Bm/8KclhSOfayYmd9ZB+/8OuTXV7GXe2HvxzP9bW9+qvkTXGd+kvZnfnxc8FUCfBv121\nOxm9anf5tR9Pm8b2e9y/WT74X/cXmifz120ev+PNv9/1nKCPvfpt923uV0jf5g/8b5ZDur46\nOt9ee/xJPZAR++q2Imxvd2o++FjZ8DNO+79A31gO6Xk1zP7+bFyM8/WbfmN8tM9x7qqnxMOZ\nyPYwv5uqNv57/u6+njr1IMa8puTZ+Gh/XnISElAaQgIECAkQICRAgJAAAUICBAgJECAkQICQ\nAAFCAgQICRAgJECAkAABQgIECAkQICRAgJAAAUICBAgJECAkQICQAAFCAgQICRAgJECAkAAB\nQgIECAkQICRAgJAAAUICBAgJECAkQICQAAFCAgQICRAgJECAkAABQnKpqf5e//NvtUs9EDwR\nkkunqr7+Z12fUw8ET4Tk06H6unxVf1IPAy+E5FRTHapt6kHgByE5daqq6pR6EPhBSF7tq33q\nIeB/hOQUz0i2EJJT2+trpCb1IPCDkHz6cz2x+6oOqYeBF0Jy6Vzf30fi5M4MQnJp91zZwMmd\nFYQECBASIEBIgAAhAQKEBAgQEiBASIAAIQEChAQIEBIgQEiAACEBAoQECBASIEBIgAAhAQKE\nBAgQEiBASIAAIQEChAQIEBIgQEiAACEBAoQECBASIEBIgAAhAQKEBAgQEiBASIDAfxaOfKNW\nRnYiAAAAAElFTkSuQmCC",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 420,
       "width": 420
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "### To match our setup, we will generate the covarates and keep them fixed in the remainder of the simulation. \n",
    "set.seed(1)\n",
    "n=50;\n",
    "x=as.matrix(rnorm(n,mean=10,sd=2),ncol=n);\n",
    "beta.true=c(20,0.15)\n",
    "Ey= x*beta.true[2]+beta.true[1];\n",
    "\n",
    "error.terms= (runif(n)-0.5)*5;\n",
    "y=Ey+error.terms;\n",
    "beta.hat=lm(y~x)$coef;\n",
    "\n",
    "\n",
    "plot(x=x,y=y,pch=16,cex=2)\n",
    "abline(b=beta.hat[2],a=beta.hat[1],col='red',lwd=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Write a function:\n",
    "simulate.one.instance<-function(x,beta.true){\n",
    "  n=length(x);\n",
    "  Ey= x*beta.true[2]+beta.true[1];\n",
    "  error.terms= (runif(n)-0.5)*5;\n",
    "  y=Ey+error.terms;\n",
    "  beta.hat=lm(y~x)$coef;\n",
    "  return(beta.hat)\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Set the number of replicates to be 10000\n",
    "N.sim=1e4;\n",
    "set.seed(1)\n",
    "n=50;\n",
    "x=as.matrix(rnorm(n,mean=10,sd=2),ncol=n);\n",
    "beta.true=c(20,0.15)\n",
    "beta.sim=replicate(N.sim,simulate.one.instance(x,beta.true));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAANlBMVEUAAABNTU1oaGh8fHyM\njIyampqnp6eysrK9vb3Hx8fQ0NDT09PZ2dnh4eHp6enw8PD/AAD///+NHJ0BAAAACXBIWXMA\nABJ0AAASdAHeZh94AAAbVUlEQVR4nO3dYUPaOhuA4YKIzlfl+P//7CuoHaitM0140uS6Puyw\nE126kHtoqTC8AIsN0QcALRASZCAkyEBIkIGQIAMhQQZCggyEBBkICTIQEmQgJMhASJCBkCAD\nIUEGQoIMhAQZCAkyEBJkICTIQEiQgZAgAyFBBkKCDIQEGQgJMhASZCAkyEBIkIGQIAMhQQZC\nggyEBBkICTIQEmQgJMhASJCBkCADIUEGQoIMhAQZCAkyEBJkICTIQEiQgZAgAyFBBkKCDIQE\nGQgJMhASZCAkyEBIkIGQIAMhQQZCggyEBBkICTIQEmQgJMhASJCBkCADIUEGQoIMhAQZCAky\nEBJkICTIQEiQgZAgAyFBBkKCDIQEGQgJMhASZCAkyEBIkIGQIAMhQQZCggyEBBkICTIQEmQg\nJMhASJCBkCADIUEGQoIMhAQZCAkyEBJkICTIQEiQgZAgAyFBBkKCDIQEGQgJMhDSav3333/R\nh8BISKslpJoIabWEVBMhrZaQapIe0uPdbjja7R8zHg//TEg1SQ3psB3+usl6SPwbIdUkNaT9\nsPnzdLr1/LAZ9vkOiH8lpJqkhrQZnsbbT8Mmz8HwG0KqSWpIwzD1G65ESDXxiLRaQqrJgu+R\nHp5Pt3yPFERINUn+ouzm7Kzd9pDzkPg3QqrJgueR9qfnkTa7O88jhRBSTZwmWC0h1URIqyWk\nmrhEaLWEVBOXCK2WkGriEqHVElJNPCG7WkKqiUuEVktINfGItFpCqolLhFZLSDVxidBqCakm\nLhFardeQhnnRR9gTi71aryH9b5b79ooKLbZ/GMsTUk0WLvb9dhh2D0WnYIKQarLseaT3Mw7z\nJ+3cn2UIqSaLQtoP+8PLy/N+uC8xBfOEVJNFIW2G03nvw7AtMQXzhFSTRSF9nEeYP5/g/ixD\nSDVZFNLtR0izlwi5P8sQUk3SQ9rd3T8Mf15vHvbzZxvcn2UIqSbpIY3PEQ3DZvYSIfdnGUKq\nSfJiPz3d3+92p1MO+/lL7dyfZQipJldYbPdnGUKqiZBWS0g1EdJqCakmQlotIdVESKslpJos\nPf39Dz8p4f4sQ0g1SV3seyFFE1JN0p9H2vzr66u6P8sQUk3SF/vpX187yP1ZhpBqsmCx789e\n2q7QFMwQUk2ctVstIdVESKslpJoIabWEVBMhrZaQaiKk1RJSTYS0WkKqiZBWS0g1EdJqCakm\nQlotIdVESKslpJoIabWEVBMhrZaQaiKk1RJSTYS0WkKqiZBWS0g1EdJqCakmQlotIdVESPUa\n5gmpJkKq1zDfiZBqIqR6CWlFhFQvIa2IkOolpBURUr2EtCJCqpeQVkRI9RLSigipXkJaESHF\n+eEJ10FIKyKkOD+E8j8hrYiQ4gipIUKKI6SGCCmOkBoipDhCaoiQ4gipIUKKI6SGCCmOkBoi\npDhCaoiQ4gipIUKKI6SGCCmOkBoipDhCaoiQ4gipIUKKI6SGCCmOkBoipDhCaoiQ4gipIUKK\nI6SGCCmOkBoipDhCaoiQ4gipIUKKI6SGCCmOkBoipDhCaoiQ4hQP6SfRC9ASIcUpHtJPf370\nArRESHGE1BAhxRFSQ4QUR0gNEVIcITVESHGE1BAhxRFSQ4QUR0gNEVIcITVESHGE1BAhxRFS\nQ4QUR0gNEVIcITVESHGE1BAhxRFSQ4QUR0gNEVIcITVESHGE1BAhxRFSQ4QUR0gNEVIcITVE\nSHGE1BAhxRFSQ4QUR0gNEVIcITVESHGE1BAhxRFSQ4QUR0gNEVIcITVESHGE1BAhxRFSQ4QU\nR0gNEVIcITVESHGE1BAhxRFSQ4QUR0gNEVIcITVESHGE1BAhxRFSQ4QUR0gNEVIcITVESHGE\n1BAhxRFSQ4QUR0gNEVIcITVESHGE1BAhxRFSQ4QUR0gNEVIcITVESHGE1BAhxRFSQ4QUR0gN\nEVIcITVESHGE1BAhxRFSQ4QUR0gNEVIcITVESHGE1BAhxRFSQ4QUR0gNEVIcITVESHGE1BAh\nxRFSQ4QUR0gNEVIcITVESHGE1BAhxRFSQ4QUR0gNEVIcITVESHGE1BAhxRFSQ4QUR0gNEVIc\nITVESHGE1BAhxRFSQ4QUR0gNEVIcITVESHGE1BAhxRFSQ4QUR0gNEVIcITVESHGE1BAhxRFS\nQ4QUR0gNEVIcITVESHGE1BAhxRFSQ4QUR0gNEVIcITVESHGE1BAhxRFSQ4QUR0gNEVIcITVE\nSHGE1BAhxRFSQ4QUR0gNEVIcITUkfTEf73bD0W7/WGqKxgmpIamLedgOf90UmaJ5QmpI6mLu\nh82fp9Ot54fNsC8xRfOE1JDUxdwMT+Ptp2FTYormCakhqYs5DFO/yTZF84TUEI9IcYTUkAXf\nIz08n275HimVkBqSvJg3Z2fttociU7ROSA1Z8DzS/vQ80mZ353mkNEJqiCsb4gipIUKKI6SG\nuEQojpAa4hKhOEJqiEuE4gipIZ6QjSOkhrhEKI6QGuIRKY6QGuISoThCaohLhOIIqSEuEYoj\npIa4siGOkBpSaDGHc2WmWD8hNcQlQnGE1BCXCMURUkNcIhRHSA3xhGwcITXEJUJxhNQQj0hx\nhNQQlwjFEVJDXCJUzvATIbXDJULlLAxFSGviEqFyhNQRIZUjpI4IqRwhdURI5QipI0IqR0gd\nSb+y4Z9/UqLb+0tIHUldzHsh/UhIHUlezKfN/A9PZJhi7YTUkfTFfJq/MCjHFCtXfUg/iV7A\nNVmwWPdn160WmmLdqg/pp/HoBVwTZ+3KEVJHhFSOkDoipHKE1BEhlSOkjgipHCF1REjlCKkj\nQipHSB0RUjlC6oiQyhFSR4RUjpA6IqRyhNQRIZUjpI4IqRwhdURI5QipI0IqR0gdEVI5QuqI\nkMoRUkeEVI6QOiKkcoTUESGVI6SOCKkcIXVESOUIqSNCKkdIHRFSOULqiJDKEVJHhFSOkDpy\nvljbu+fSU3RFSB05X6xhGEq01O39IaSOnC/W4c9tiZa6vT+E1JHPi/V4t83dUrf3h5A68s1i\nPW1eH5fui07RByF15OtiPdyc3mTqX9+PL2WKTgipI58W63D3+nC0fTi81rQrNEU/hNSRi8V6\nPJ5s2L+9D1++Nz7s9v4QUkcunkd6fTC6P3wMbEpM0RUhdeTieaTdQ+kpuiKkjlw8j1R+iq4I\nqSMXi3XYH7+e2+zzFtXt/SGkjpwv1vPmdIZhGDZZr23o9v4QUkfOF+tmuD0+Fh32+U59f56i\nK0LqyOVFq59vZJ+iK0LqyPlibYa3b44OQspCSB05X6z9cPP4+p/Hm2FfaoquCKkjF4v1dpVd\nzuvsvkzREyF15HKx/uyOGWW88vvrFB0RUke8ZkM5QuqIkMoRUkeEVI6QOnKxWMcfM39TbIqe\nCKkj54t1NwxCykhIHbl8Qjbz+bqvU3RFSB359hKhclN0RUgdOV+s3VDkJ5K6vT+E1JHLH6M4\nXSJUcoquCKkjn16y2MmGjITUESGVI6SOeEK2HCF1REjlCKkjl4v1sDt+VbfL+3YU3d4fQurI\n159HOr42pBc/yUFIHTlfrPvh5vRT5vfDbakpuiKkjnx+zYb3F+QqNUVXhNSRz5cICSkfIXXk\n8kX03x6RnoZtqSm6IqSOfPM90kPmq8C7vT+E1JGLxdp5FaGchNSRr88jDbs/JafoiJA64sqG\ncoTUESGVI6SOCKkcIXXEj1GUI6SOCKkcIXXkm8V6vMn6PmNCKrTRhVST7xbr4KLVLITUkW8X\ny5d2WQipI98t1v2wKT1FF4TUke9PNtyVmqIrQurIdyFt875ycbf3h5A64gnZcoTUESGVI6SO\nTDwhm/NJ2W7vDyF1REjlCKkjl+/Yt3l4/fVx4wf7shBSRy7fse/p9N+nIes1Qt3eH0LqyLdv\nNObKhiyE1JHL17X7eETyKkI5CKkj54u1H07fI3kVoUyE1JGvr/39al9uip4IqSOXi/Xn9CpC\nDyWn6IiQOuLKhnKE1BEhlSOkjnijsXKE1BFvNFaOkDrijcbKEVJHvNFYOULqiDcaK0dIHfFG\nY+UIqSPeaKwcIXXEG42VI6SOeKOxcoTUEVc2lCOkjpwv1i7vVd/fTdEVIXXk25+QLTdFV4TU\nkc+nvwtP0RUhdeR8sQ67m8fCU3RFSB3xjn3lCKkjQipHSB1x+rscIXVESOUIqSNlXhPy2ym6\nI6SOXIZUJKdu7w8hdURI5QipI0IqR0gdEVI5QuqIkMoRUkeEVI6QOvI3pCJve3k+RXeE1BEh\nlSOkjriyoRwhdURI5QipI0IqR0gdEVK64SdLN/L8sJBqIqR0hUMR0poIKZ2QGAkpnZAYCSmd\nkBgJKZ2QGAkpnZAYCSmdkBgJKZ2QGAkpnZAYCSmdkBgJKZ2QGAkpXfMh/ST6DqiJkNI1H9JP\n49F3QE2ElE5IjISUTkiMhJROSIyElE5IjISUTkiMhJROSIyElE5IjISUTkiMhJROSIyElE5I\njISUTkiMhJROSIyElE5IjISUTkiMhJROSIyElE5IjISUTkiMhJROSIyElE5IjISUTkiM0hfj\n8W53eimZ3f6x1BSVExKj1MU4bM9elummyBTVExKj1MXYD5s/T6dbzw+bYV9iiuoJiVHqYmyG\np/H207ApMUX1hMQodTEuXmZz/jU3m11vITHyiJROSIwWfI/08Hy65XukoI0spJokL8bN2Vm7\n7aHIFLUTEqMFzyPtT88jbXZ3nkcK2chCqokrG9IJiZGQ0gmJkUuE0gmJkUuE0gmJkUuE0gmJ\nkSdk0wmJUaFLhLp4z14hMfKIlE5IjFwilE5IjFwilE5IjFwilE5IjFzZkE5IjISUTkiMFi7G\n/XYYdg9Fp6iXkBgtex7p/YzD7Ek7IRUaF1JNFoW0H/aHl5fn/XBfYorqCYnRopA2w+m892HY\nlpiiekJitCikj6t/vIpQxEYWUk0WhXT7EZJLhAI2spBqkh7S7u7+YfjzevOwd4lQxEYWUk3S\nQxqv7B6GjUuEAjaykGqSvBhPT/f3u93plMN+tiMhFRoXUk1c2ZBOSIyElE5IjISUTkiMhJRO\nSIyElE5IjISUTkiMhJROSIyElE5IjISUTkiMhJROSIyElE5IjISUTkiMhJROSIyElE5IjISU\nTkiMhJROSIyElE5IjISUTkiMhJROSIyElE5IjISUTkiMhJROSIyElE5IjISUTkiMhJROSIyE\nlE5IjISUTkiMhJROSIyElE5IjISUTkiMhJROSIyElE5IjISUTkiMhJROSIyElE5IjISUTkiM\nhJROSIyElE5IjIQ0bfhJ6Y06PyykmghpWnAo9Yf0k+g78JqENE1IC8ej78BrEtK08I04Pyyk\nmghpWvhGnB8WUk2ENC18I84PC6kmQpoWvhHnh4VUEyFNC9+I88NCqomQpoVvxPlhIdVESNPC\nN+L8sJBqIqRp4RtxflhINRHStPCNOD8spJoIaVr4RpwfFlJNhDQtfCPODwupJkKaFr4R54eF\nVBMhTQvfiPPDQqqJkKaFb8T5YSHVREjTwjfi/LCQaiKkaeEbcX5YSDUR0rTwjTg/LKSaCGla\n+EacHxZSTYQ0LXwjzg8LqSZCmha+EeeHhVQTIU0L34jzw0KqiZCmhW/E+WEh1URI08I34vyw\nkGoipGnhG3F+WEg1EdK08I04PyykmghpWvhGnB8WUk2ENC18I84PC6kmQpoWvhHnh4VUEyFN\nC9+I88NCqomQpoVvxPlhIdVESNPCN+L8sJBqIqRp4RtxflhINRHStPCNOD8spJoIaVr4Rpwf\nFlJNhDQtfCPODwupJkKaFr4R54eFVBMhTQvfiPPDQqqJkKaFb8T5YSHVREjTwjfi/LCQaiKk\naeEbcX5YSDUR0rTwjTg/LKSaCGla+EacHxZSTYQ0LXwjzg8LqSZCmha+EeeHhVQTIU0L34jz\nw0KqiZCmhW/E+WEh1URI08I34vywkGoipGnhG3F+WEg1EdK08I04PyykmghpWvhGnB8WUk2E\nNC18I84PC6kmQpoWvhHnh4VUEyFNC9+I88NCqomQpoVvxPlhIdVESNPCN+L8sJBqIqRp4Rtx\nflhINRHStPCNOD8spJoIaVr4RpwfFlJNhDQtfCPODwupJkKaFr4R54eFVBMhTQvfiPPDQqqJ\nkKaFb8T5YSHVREjTwjfi/LCQaiKkaeEbcX5YSDUR0rTwjTg/LKSa9BzS8JPojTg/LKSadB1S\n9EZbNi6kmggpcKMtGxdSTYQUuNGWjQupJkIK3GjLxoVUEyEFbrRl40KqiZACN9qycSHVREiB\nG23ZuJBqIqTAjbZsvP6QfhK9AXISUuBGWzZef0g/jUdvgJyEFLiRlo0LqSZCCtxIy8aFVBMh\nBW6kZeNCqomQAjfSsnEh1URIgRtp2biQaiKkwI20bFxINRFS4EZaNi6kmggpcCMtGxdSTYQU\nuJGWjQupJkIK3EjLxoVUEyEFbqRl40KqiZACN9KycSHVREiBG2nZuJBqIqTAjbRsXEg1EVLg\nRlo2LqSaCClwIy0bF1JNhBS4kZaNC6kmQgrcSMvGhVQTIQVupGXjQqqJkAI30rJxIdVESIEb\nadm4kGoipMCNtGxcSDURUuBGWjYupJoIKXAjLRsXUk2EFLiRlo0LqSZCCtxIy8aFVBMhBW6k\nZeNCqomQAjfSsnEh1URIgRtp2biQaiKkwI20bFxINRFS4EZaNi6kmggpcCMtGxdSTYQUuJGW\nja8/pJbe0U9IgRtp2fj6Q/ppPHqD/IaQAjfKsnEh1aTlkH780iF6oywbF1JNmg4peiOUHRdS\nTYS02nEh1URIqx0XUk2EtNpxIdVESKsdF1JNhLTacSHVREirHRdSTYS02nEh1URIqx0XUk2E\ntNpxIdVESKsdF1JNhLTacSHVZM0hNX5RqpBKbZwSVh1S9B0dOy6kmghpteNCqomQVjsupJoI\nabXj7Ye0ptd0ENJqx9sP6afxUhsrRfrBPN7tTv8s7PaPpab4QfgdGTsupFIbK0XqwRy2Zw+x\nN0Wm+FH4HRk7LqSavvRLnWw/bP48nW49P2yGfYkpen+eSEhLx1M3XtJmTfy8zfA03n4aNiWm\n6P0RR0hLx1M3XtJmTf28Yeo37//nHx5hf3zEYc5rSNGHsHKJe//7zZz4eb94RKKM15CiD4HR\ngu+RHp5Pt378HokyhFST5Ie3m7OHyO0h5yHxb4RUkwXPI+1PzyNtdnc/PI9EGUKqSVVPavEb\nQqqJkFZLSDUR0moJqSZCWi0h1URIqyWkmghptYRUEyGtlpBqIqTVElJNhLRaQqqJkFZLSDUR\n0moJqSZCWi0h1URIqyWkmghptYRUEyGtlpBqIqTVElJNhLRaQqqJkFZLSDVZc0hBL+JUCy/H\nFb0Bz1V1ML8Ufezm73v+C1UdzC9FH7v5+57/QlUH80vRx27+vue/UNXB/FL0sZu/7/kvVHUw\nvxR97Obve/4LVR3ML0Ufu/n7nv9CVQfzS9HHbv6+579Q1cH8UvSxm7/v+S9UdTC/FH3s5u97\n/gtVHcwvRR+7+fue/0JVB/NL0cdu/r7nv1DVwfxS9LGbv+/5L1R1MLBWQoIMhAQZCAkyEBJk\nICTIQEiQgZAgAyFBBkKCDIQEGQgJMhASZCAkyEBIkIGQIIN1hnT/cdhPt8Nw+xw2/2G/GTb7\nw7Vn346TRs9/djNk/lePdWzhOo7il54+3ojg4fSmBJtr35Mf8z9v3ua/bsn7v3/pm9PN7VWn\nP59/H7H+l5MeNnVs4TqO4neeNh8hbTZPL4fdsA+a//Y08364versw+3h+Jh4e/zH+PXv/3o0\nj0Hzn90Mmf9oV8m7u9RxFL9yP9y8L96f00Y+DJug+S//cyW7v5Puh4eX4yrcBc2/C/77vxz/\n8kJK9VrP+IjwFDr/+1cVVw75/ShOG/n4ReXTsAua/+vNa8//PP6jFq2Oo/iVp/Gu2w4vd5vT\nI33M/HfvX9pd9RHhzWG4CXpEPJv/y82rz38zPAtpgXEH7U7fd0bN/3J/PNuwub/6/MevLx9C\nQzrN/+Xmtee/G/7EPB5+VcdR/Na4g44nG26v/4jwcefdnU4gBTwgPW92L5Ehvc3/+ea15z99\nUSukBcYddPwe6fna53/H+e+PX9q9hnz1h6TD5ubsMK6/ld7n/3Tz6vNvj+fAhbRA5Fmzsxm3\nw/Hbs8P1Q755m3ET9fe/2X5389rz356+vBPSAu+LF3L69WzGoJCftzdvTwG/nbV7vvZZu3H+\ni5vXn38YXf0QvqrhGH5vPGt2/Cfp+fpnjS5Of1/7eayH8e/79vd/uPIT0n/nP78ZML+QFvu4\nRGfYHo7fo/wJmn8/HC/52l93I5/9uxFyZcPZ/AH/hH2dtIqMVh7S+1mz69+bH/PfBMx/e/bP\n8DZ2/tuIR4TPkwppgXHxHm6GzZWvtLuY/3T19bXn/ruRDrHzh3xp9XlSIUE7hAQZCAkyEBJk\nICTIQEiQgZAgAyFBBkKCDIQEGQgJMhASZCAkyEBIkIGQIAMhQQZCggyEBBkICTIQEmQgJMhA\nSJCBkCADIUEGQoIMhAQZCAkyEBJkICTIQEiQgZAgAyFBBkKCDIQU5+K9715//3D6vw9TH/z1\n1tzH/yj18/iWkOJ8Cml7ui+2E/fIVEhTH/+T1M/je5YzzqeHlrffTr0l6m///y8nZyHLGUdI\nDbGccc728uvNty/yxvfrvt8Om/u3wf1m2H/60m4YnnfD5u7lm48fhsN22L192s3zy6exl/3b\n26Bf/c3IW2cx48yFtDvduDmO3Rxv7T6HtDn+37tvPv70sfv3T9scPo3dvd8UUmYWM87fcw0v\n45m7918fhpvDy+FmeHh5+TNsnl6eNp9Deh2/H7bffPxp6Phpr/+5PRZ1Mfb2Z/3xpV1uljPO\nTEi74RjD4fgl2m54fDnW8Cmkx4vPOvv4t6G3TzsMm09jD6c/ayek3CxnnMsv7S5C+vwU0+fT\n338/9l8+/rsxIeVlOeMIqSGWM85sSJ8/6qeQ5j/+65iQ8rKccWZC2g3jFTxvNx/nQzr7+PcP\nvDn7Huls7O37rVsh5WY543wJ6fnl49fTqbqX++NZgYfvz9pdftbZx79/4P3xXN3+eNbuYuzt\nz3r4+DxyEVKcTyFth+Pjx9uvb88CDZvjZj89DXQ7FdKXj//4wL/PI52PvT0p9TJ+HpkIKc6n\nkB63x6399uvpaoTh9u1B4+67KxsuP+vs48cP3L8WM17ZMI7thu3pIoePzyMPIfXEN0bFWNme\nCKkYK9sTIRVjZXsipGKsLGQgJMhASJCBkCADIUEGQoIMhAQZCAkyEBJkICTIQEiQgZAgAyFB\nBkKCDIQEGQgJMhASZCAkyEBIkIGQIAMhQQZCggyEBBkICTIQEmQgJMhASJDB/wHMqCv68dgw\nEAAAAABJRU5ErkJggg==",
      "text/plain": [
       "Plot with title \"\""
      ]
     },
     "metadata": {
      "image/png": {
       "height": 420,
       "width": 420
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Draw the histogram for the estimated intercept \n",
    "hist(beta.sim[1,],xlab='Fitted intercept',main='')\n",
    "abline(v=beta.true[1],lwd=3,col='red')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "-0.0205767779027077"
      ],
      "text/latex": [
       "-0.0205767779027077"
      ],
      "text/markdown": [
       "-0.0205767779027077"
      ],
      "text/plain": [
       "[1] -0.02057678"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "0.00187424303119896"
      ],
      "text/latex": [
       "0.00187424303119896"
      ],
      "text/markdown": [
       "0.00187424303119896"
      ],
      "text/plain": [
       "[1] 0.001874243"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "0.0153776153004157"
      ],
      "text/latex": [
       "0.0153776153004157"
      ],
      "text/markdown": [
       "0.0153776153004157"
      ],
      "text/plain": [
       "[1] 0.01537762"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "0.0156214500568736"
      ],
      "text/latex": [
       "0.0156214500568736"
      ],
      "text/markdown": [
       "0.0156214500568736"
      ],
      "text/plain": [
       "[1] 0.01562145"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "1.64183502873519"
      ],
      "text/latex": [
       "1.64183502873519"
      ],
      "text/markdown": [
       "1.64183502873519"
      ],
      "text/plain": [
       "[1] 1.641835"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "1.66942762352589"
      ],
      "text/latex": [
       "1.66942762352589"
      ],
      "text/markdown": [
       "1.66942762352589"
      ],
      "text/plain": [
       "[1] 1.669428"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Using the simulation results, we can verify a few results from linear regression:\n",
    "\n",
    "### Expectation \n",
    "mean(beta.sim[1,])-beta.true[1] # intercept \n",
    "mean(beta.sim[2,])-beta.true[2] # slope \n",
    "\n",
    "### Variance \n",
    "sigma.sq = 5^2/12; # variance of the error: (runif(n)-0.5)*5;\n",
    "\n",
    "# slope:\n",
    "sigma.sq/sum(  (x-mean(x))^2  ) # theoretical value \n",
    "var(beta.sim[2,]) # variance from the simulation \n",
    "\n",
    "\n",
    "# intercept:\n",
    "sigma.sq*sum(x^2)/sum(  (x-mean(x))^2  )/n # theoretical value \n",
    "var(beta.sim[1,]) # variance from the simulation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## E.2 Sampling from observed samples\n",
    "\n",
    "In most case, the true population or true data generating mechanisms are unknown. As a work-around, we can make full use of the observed samples to *approximate* the true population. This approximation is *exact* only when the samples exhaust the true population (e.g., a mandatory survey in a class). In most cases, it remains an approximation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### E.2.1 Bootstrap \n",
    "\n",
    "There are two factors that affect the sampling distribution for any estimator: (i) the true population and (ii) the sampling scheme. Boostrap is based on a simple idea of treating the observed samples as the true population. In bootstrap, (i) the observed samples are seen as the true population, and (ii) a sampling scheme is chosen to micmic the true mechanism. To be specific, if the samples are i.i.d. (independently and identically distributed), the bootstrap procedure will draw samples with replacement with equal probability from the observed samples, where \"with replacement\" is because of independence and \"equal probability\" if because of identical distribution. The bootstrap sampling distribution approximates the true sampling distribution when the sample size is large. \n",
    "\n",
    "There are many variants of bootstrap (see, e.g., [here](https://en.wikipedia.org/wiki/Bootstrapping_populations)). We will examine the nonparametric bootstrap in this section. \n",
    "\n",
    "We will revist the confidence interval in Appendix D.3. This time we write our own bootstrap function to construct the confidence interval. It is important to note the similarity between the `simulate.one.instance` function in Appendix D.3 and the `boot.fit` function below. \n",
    "\n",
    "In practice, we use the `boot` function in the package `boot` for bootstrapping. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "set.seed(1)\n",
    "n=50;\n",
    "x=as.matrix(rnorm(n,mean=10,sd=2),ncol=n);\n",
    "beta.true=c(20,0.15)\n",
    "Ey= x*beta.true[2]+beta.true[1];\n",
    "error.terms= rnorm(n)*5;\n",
    "y=Ey+error.terms;\n",
    "\n",
    "alpha=0.05"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# How does the bootstrap function differ from the simulation function?\n",
    "# \n",
    "boot.fit<-function(covariate,outcome){\n",
    "  n=length(outcome);\n",
    "  sample_indices = sample(1:n,n,replace=TRUE) # sampling with replacement\n",
    "  # Sample covariates (x), or not?\n",
    "  covariate.boot= covariate[sample_indices]; \n",
    "  outcome.boot= outcome[sample_indices];\n",
    "   \n",
    "  \n",
    "  beta.hat=lm(outcome.boot~covariate.boot)$coef;\n",
    "  return(beta.hat)\n",
    "}\n",
    "B=1e4;\n",
    "beta.hat.boot=replicate(B,boot.fit(covariate=x,outcome=y));\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A matrix: 2 × 2 of type dbl</caption>\n",
       "<thead>\n",
       "\t<tr><th></th><th scope=col>2.5%</th><th scope=col>97.5%</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>(Intercept)</th><td>13.7991429</td><td>31.0152995</td></tr>\n",
       "\t<tr><th scope=row>covariate.boot</th><td>-0.8703025</td><td> 0.8336293</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A matrix: 2 × 2 of type dbl\n",
       "\\begin{tabular}{r|ll}\n",
       "  & 2.5\\% & 97.5\\%\\\\\n",
       "\\hline\n",
       "\t(Intercept) & 13.7991429 & 31.0152995\\\\\n",
       "\tcovariate.boot & -0.8703025 &  0.8336293\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A matrix: 2 × 2 of type dbl\n",
       "\n",
       "| <!--/--> | 2.5% | 97.5% |\n",
       "|---|---|---|\n",
       "| (Intercept) | 13.7991429 | 31.0152995 |\n",
       "| covariate.boot | -0.8703025 |  0.8336293 |\n",
       "\n"
      ],
      "text/plain": [
       "               2.5%       97.5%     \n",
       "(Intercept)    13.7991429 31.0152995\n",
       "covariate.boot -0.8703025  0.8336293"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "t(apply(beta.hat.boot,1,quantile,probs=c(0.025,0.975)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<caption>A matrix: 2 × 2 of type dbl</caption>\n",
       "<thead>\n",
       "\t<tr><th></th><th scope=col>2.5 %</th><th scope=col>97.5 %</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>(Intercept)</th><td>13.0188972</td><td>30.4775452</td></tr>\n",
       "\t<tr><th scope=row>x</th><td>-0.8086838</td><td> 0.8809413</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A matrix: 2 × 2 of type dbl\n",
       "\\begin{tabular}{r|ll}\n",
       "  & 2.5 \\% & 97.5 \\%\\\\\n",
       "\\hline\n",
       "\t(Intercept) & 13.0188972 & 30.4775452\\\\\n",
       "\tx & -0.8086838 &  0.8809413\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A matrix: 2 × 2 of type dbl\n",
       "\n",
       "| <!--/--> | 2.5 % | 97.5 % |\n",
       "|---|---|---|\n",
       "| (Intercept) | 13.0188972 | 30.4775452 |\n",
       "| x | -0.8086838 |  0.8809413 |\n",
       "\n"
      ],
      "text/plain": [
       "            2.5 %      97.5 %    \n",
       "(Intercept) 13.0188972 30.4775452\n",
       "x           -0.8086838  0.8809413"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "confint(lm(y~x+1),alpha=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### E.2.2 Permutation \n",
    "\n",
    "In hypothesis testing, the test statistic is compared to the *null distribution*, that is the *sampling distribution under the null hypothesis*. Sometimes the null hypothesis is specific enough that we can approximate the true distribution using the observed samples. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose that we observe $n$ i.i.d. samples $\\{ (x_1,y_1), (x_2,y_2), \\ldots, (x_n,y_n)\\}$ from an unknown true population. For instance, we can imagine that $x$ is the face of a coin (Head, Tail) and $y$ is the age of the tosser. We are interested in the following hypothesis\n",
    "$$\n",
    "H_0: \\mathbb{E}[y \\mid x ={\\rm Tail}] = \\mathbb{E}[y \\mid x ={\\rm Head}]\n",
    "$$\n",
    "$$\n",
    "H_1: \\mathbb{E}[y \\mid x ={\\rm Tail}] \\neq  \\mathbb{E}[y \\mid x = {\\rm Head}].\n",
    "$$\n",
    "Under the null, we know that the expectation of $y$ remains the same regardless of the value of $x$. This means that the the expectation of $y$ should remain the same if we shuffle $x$, e.g., by creating a new data set as \n",
    "$$\n",
    "\\{ (x_6,y_1), (x_{17},y_2), \\ldots, (x_3,y_n)\\},\n",
    "$$\n",
    "where the indices of $x$s are *permuted*. From the permuted data set, we can estimate the mean difference between two groups defined by the new $x$.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "set.seed(1)\n",
    "n=50;\n",
    "x=rbinom(n=50,size=1,prob=0.5)\n",
    "mu.diff=0; \n",
    "error.terms= rnorm(n)*2; # Normal errors\n",
    "y=x*mu.diff+error.terms;\n",
    "\n",
    "diff=mean(y[x==1])-mean(y[x==0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "-0.269994486002787"
      ],
      "text/latex": [
       "-0.269994486002787"
      ],
      "text/markdown": [
       "-0.269994486002787"
      ],
      "text/plain": [
       "[1] -0.2699945"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x.perm=sample(x,size=length(x),replace=FALSE);\n",
    "(mean.diff=mean(y[x.perm==1])-mean(y[x.perm==0]))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Permutation + mean diff\n",
    "permutation.diff<-function(x,y){\n",
    "  n=length(y);\n",
    "  x.perm=sample(x,size=length(x),replace=FALSE);\n",
    "  mean.diff=mean(y[x.perm==1])-mean(y[x.perm==0])\n",
    "  return(mean.diff)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "set.seed(1)\n",
    "n=50;\n",
    "x=rbinom(n=50,size=1,prob=0.5)\n",
    "mu.diff=0; \n",
    "error.terms= rnorm(n)*2; # Normal errors\n",
    "y=x*mu.diff+error.terms;\n",
    "\n",
    "diff=mean(y[x==1])-mean(y[x==0])\n",
    "\n",
    "\n",
    "mean.diff.perm=replicate(1e5,permutation.diff(x,y));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAM1BMVEUAAABNTU1oaGh8fHyM\njIyampqnp6eysrK9vb3Hx8fQ0NDT09PZ2dnh4eHp6enw8PD///8uNL8wAAAACXBIWXMAABJ0\nAAASdAHeZh94AAAeiElEQVR4nO3d22LaOhBGYZmDIWxO7/+0G9tAbDB2AyP0j7S+izYtCRLO\nrBLATcIZwMdC6g0AOSAkwAAhAQYICTBASIABQgIMEBJggJAAA4QEGCAkwAAhAQYICTBASIAB\nQgIMEBJggJAAA4QEGCAkwAAhAQYICTBASIABQgIMEBJggJAAA4QEGCAkwAAhAQYICTBASIAB\nQgIMEBJggJAAA4QEGCAkwAAhAQYICTBASIABQgIMEBJggJAAA4QEGCCkV0IIw7d+/6Jv/ZXN\nbKoQIq30cOtuK0VcMUuE9Mo/hbSvvnIAN5e1vxPSbaWYK2aJkF75p5DG76XMLUI4xLru4a27\nrRRzxSwR0itPIU2/05f2Evu6b3/60g3LB4frlVf3SKfN8vLW6qf7m3B7t926+Vpod/2Q4+VP\ny23vI4+LUF/e+lld3l7Ux9v1bRdhsT+ft1VY7ofLD67vYazHP/JUV6Hqrvp5od1l0+vhfcxx\nXYXF9uHWvQjpumJYH5/Xut+66RuUOUJ65UVIx+qaz3IQ0vL69qr9iP31XX4/ctF+wO29wr77\n2+7Px/r+d3f96+v3etvQ80feNrbvf/htoe79Bl+tPe/xd6WxFVftX1XHp7Xut27yBuWOkF55\nEdLljuJyZ3S6TMy2N2+r2+B2JVX3P94+MjQftr1M2ul8rgd9XIaz32BncH1jY/38kbc1q8vb\nLxYaPHvwvMfpkK4Wj2vdb93kDcoeIb0S+q5/0f3a/KN86kbqetHu8vv2dPmq7/L75auxn8ss\nNb9Vvx/ZzHXzCP44uKbL326b4Ty0v/2u/XB9I19oPX1k186p6/xpoWY768G1XPe4qwa3buJL\nu2rfvffuYa37rZu6Qfkr6sb+yYuQmsG7PxS6zdu6uX9q1O2/+qtu/JthvX3k7uGqu1/3g99+\n3+Hh+kbG+ukjL2tep3k1eMff5U+Da7ntcfevIf1c33v9uNb91k3coPwVdWP/5EVIm+4v1vfZ\nuf7WTtb52P5FdZuhx4sv7/BTL8N9vs9Pv90/rn99Yw/9H3/73Wv1YqGnJV5f2asVmzcWj2vd\nNztxg/JX1I39k99BGE5GfZuh49NFt7deDOnlLmrRK3M6pMFbfwopvFjIKKTwuNbMNRWiqBv7\nJ69COp9+uuemloOL7vcg1eg9UvvH5ku9xXp7+NM9UvV44fhHVv13GV3ow5B69zqDtQipVdSN\n/ZOXITV268GMrmYfI7WXLq5/PxvSavYx0tNvq/7jsNGFnpYY2+PrkNoN7dqHRYO1CKlV1I39\nkxchLe4PtG93FaeXz9qF8emcv0eaf9bu6bef7nm1n9srOs8LDa9l+/zM4shWeys2zzY0z9pt\nH9YipFZRN/ZPXoR0mfHlsX3OoTlToRnD5vf7C6DdSzVjr9F071X/PuM8NXcP1/cPIf2uuX+x\n0MP90/geH0Lq/fGqelyLkFpF3dg/eRHS/cmG5h/j9rWZ9o3b5Hcvee6u7/IwU/vbMLYTODl3\nw+v7l5Cua7Zdjy70ENL1fVb/GlJ3u7szG/prEVKrqBv7J69C6h4fLbvHMM3DhWs766r3AtOh\nOddu9zRTzV9X68PxdsLCyLXfDK7vX0I6n+rLl52r3cuFHkJ6Ph+wf2Ujf/xZhKq+Po3fX4uQ\nGkXd2G873V7Uca+wKt7A8YkgdOcBHJbZnG9GSHM4PhH8PlUwPDXIL0Kaw/GJ4P5fLbpH4xkg\npDkcnxhOm+bJsGqdyf0RIc3j+AAGCAkwQEiAAUICDBASYICQAAOEBBggJMAAIQEGCAkwQEiA\nAUICDBASYICQAAOEBBggJMAAIQEGCAkw8H5I+033Y+VWdVE/4hAY825Ip8Xvd8rpvtcoULB3\nQ6pD9dP9aN/jrsrmm+UAb3o3pKr3E7IPuXw/UeBd74Y0+PZMfK8mlI57JMDAB4+Rdu1P+OAx\nEvDB09+9728dFqf59wdy9sHrSHX7OlK12vA6EorH0wSAAUICDHCKEGCAU4QAA5wiBBjgBVnA\nAKcIAQa4RwIMcIoQYIBThAADnCIEGOBpAsAAIQEGOEUIMMApQoABThECDPCCLGAg0ilCoe/N\nJQA/vnCPREjI3xdOESIk5O8LpwgREvL3hVOECAn5+8KUExLyR0iAgQ+nfLsIYbWLugTgwGev\nI12fcZg+sYGQkL+PQqpDfTqfj3XYxlgC8OOjkKrQPu99CosYSwB+fBTS7eyf6bOACAn5+yik\n9S0kThFC4d4PabXZ7sLP5c1TzSlC3xImpN5b2d4P6f7ZC6HiFKEvCf+9xGFO6u3Dfzhst6tV\n+5RDPf3duPgM2yEkVZzZ4AohqSIkV6ZC4vFTSoTkylRIry/i3io+QnKFkFQRkiuEpOrTp7//\n4WtwPot2CEnVu4d4S0gpEJKq919Hqv71+6vyWbRDSKreP8SHf/3+qnwW7RCSqg8O8bb3re0i\nLYEHhKSKZ+1cISRVhOQKIakiJFcISRUhuUJIqgjJFUJSRUiuEJIqQnKFkFQRkiuEpIqQXCEk\nVYTkCiGpIiRXCEkVIblCSKoIyRVCUkVIrhCSKkJyhZBUEZIrhKSKkFwhJFWE5AohqSIkVwhJ\nFSG5QkiqCMkVQlJFSK4QkipCcoWQVBGSK4SkipBcISRVhOQKIakiJFcISRUhuUJIqgjJFUJS\nRUiuEJIqQnKFkFQRkiuEpIqQXCEkVYTkCiGpIiRXCEkVIblCSKoIyRVCUkVIrhCSKkJyhZBU\nEZIrhKSKkFwhJFWE5AohqSIkVwhJFSG5QkiqCMkVQlJFSK4QkipCcoWQVBGSK4SkipBcISRV\nhOQKIakiJFcISRUhuUJIqgjJFUJSRUiuEJIqQnKFkFQRkiuEpIqQXCEkVYTkCiGpIiRXCEkV\nIblCSKoIyRVCUkVIrhCSKkJyhZBUEZIrhKSKkOSEKYQkipDkTBbxZkhTUt/ePBCSnBghTV2W\n+vbmgZDkEJJHhCSHkDwiJDmE5BEhySEkjwhJDiF5REhyCMkjQpJDSB4RkhxC8oiQ5BCSR4Qk\nh5A8IiQ5hOQRIckhJI8ISQ4heURIcgjJI0KSQ0geEZIcQvKIkOQQkkeEJIeQPCIkOYTkESHJ\nISSPCEkOIXlESHIIySNCkkNIHhGSHELyiJDkEJJHhCSHkDwiJDmE5BEhySEkjwhJDiF5REhy\nCMkjQpJDSB4RkhxC8oiQ5BCSR4Qkh5A8IiQ5hOQRIckhJI8ISQ4heURIcgjJI0KSQ0geEZIc\nQvKIkOQQkkeEJIeQPCIkOYTkESHJISSPCEkOIXlESHIIySNCkkNIHhGSHELyiJDkEJJHhCSH\nkDwiJDmE5BEhySEkjwhJDiF5REhyCMkjQpJDSB4RkhxC8oiQ5BCSR4Qkh5A8IiQ5hOQRIckh\nJI8ISQ4heURIcgjJI0KSQ0geEZIcQvKIkOQQkkeEJIeQPCIkOYTkESHJISSPCEkOIXlESHII\nySNCkkNIHhGSHELyiJDkEJJHhCSHkDwiJDmE5BEhySEkj94/jPvNKjRW9T7WEmUiJI/ePYyn\nRfi1jLJEqQjJo3cPYx2qn0P71nFXhTrGEqUiJI/ePYxVONzfPoQqxhKlIiSP3j2MIbz6g9kS\npSIkj7hHkkNIHn3wGGl3bN/iMZIxQvLo7cO47D1rtzhFWaJQhOTRB68j1e3rSNVqw+tIpgjJ\nI85skENIHhGSHELyiFOE5BCSR5wiJIeQPOIUITmE5BEvyMohJI84RUgOIXnEPZIcQvKIU4Tk\nEJJHnCIkh5A84hQhOYTkEWc2yCEkjwhJDiF5xClCcgjJI04RkkNIHnGKkBxC8ogXZOUQkkeR\nThEKfW8uUSpC8oh7JDmE5BGnCMkhJI84RUgOIXnEKUJyCMkjzmyQQ0geEZIcQvKIU4TkEJJH\nnCIkh5A84hQhOYTkES/Iyvl2SFNSHws/+C5Ccr4d0tRlqY+FH9wjySEkjzhFSA4hecQpQklM\nPjAhJIc4RSiJCFNPSElxZkMShJQbQkqCkHJDSEkQUm4IKQlCyg0hJUFIuXn/zIZ/PpWEz8Yz\nQsrNu4dqS0ifIKTcvH2oDtX0f54wWCJjhJSb9w/VYfrEIIsl8kVIufngUG17561GWiJbhJQb\nnrVLgpByQ0hJEFJuCCkJQsoNISVBSLkhpCQIKTeElAQh5YaQkiCk3BBSEoSUG0JKgpByQ0hJ\nEFJuCCkJQsoNISVBSLkhpCQIKTeElAQh5YaQkiCk3BBSEoSUG0JKgpByQ0hJEFJuCCkJQsoN\nISVBSLkhpCQIKTeElAQh5YaQkiCk3BBSEoSUG0JKgpByQ0hJEFJuCCkJQsoNISVBSLkhpCQI\nKTeElAQh5YaQkiCk3PQP1WJzjL0EOoSUm/6hCiHEaInPxjNCyk3/UJ1+1jFa4rPxjJBy83io\n9puFdUt8Np4RUm5GDtWhutwvbaMuUTxCys3zodotQ2MZcQkQUm4eDtVpc7k7WuxOl5pWkZbA\nmZDyMzhU++bJhvrQXWB2EPlsPCOk3AxeR7rcGW1PtwuqGEugQ0i5GbyOtNrFXgIdQsrN4HWk\n+EugQ0i5GRyqU918PVfVtkXx2XhGSLnpH6pj1T7DEEJlem4Dn41nhJSb/qFahnVzX3Sq7Z76\nflwCHULKzfCk1cc3zJdAh5By0z9UVegeHJ0IKTZCyk3/UNVhub/8tl+GOtYS6BBSbgaHqjvL\nzvI8u6cl0CKk3AwP1c+qycjwzO/nJdAgpNzwPRuSIKTcEFIShJQbQkqCkHIzOFTNfzPvRFsC\nLULKTf9QbUIgpO8gpNwMX5A1fr7ueQl0CCk3o6cIxVsCHULKTf9QrUKU/5HEZ+MZIeVm+N8o\n2lOEYi6BDiHl5uFbFvNkw3cQUm4IKQlCyg0vyCZBSLkhpCQIKTfDQ7VbNV/VrWx/HAWfjWeE\nlJvn/4/UfG9IvvlJZISUm/6h2oZl+7/Mt2Edawl0CCk3j9+z4foNuWItgQ4h5ebxFCFC+gpC\nys3wm+h390iHsIi1BDqElJuRx0g747PA+Ww8I6TcDA7Viu8i9CWElJvn15HC6ifmEmgQUm44\nsyEJQsoNISVBSLkhpCQIKTf8N4okCCk3hJQEIeVm5FDtl6Y/Z4yQRhBSbsYO1YmTVmMjpNyM\nHiq+tIuNkHIzdqi2oYq9ROkIKTfjTzZsYi2BDiHlZiykhe13Luaz8YyQcsMLskkQUm4IKQlC\nys2LF2QtX5Tls/GMkHJDSEkQUm6GP7Gv2l1+3Vf8x77YCCk3w5/Yd2h/PwTTc4T4bDwjpNyM\n/qAxzmyIjZByM/y+drd7JL6LUGSElJv+oapD+xiJ7yIUHyHl5vl7f1/U8ZZAi5ByMzxUP+13\nEdrFXAINQsoNZzYkQUi5IaQkCCk3/KCxJAgpN/ygsSQIKTf8oLEkCCk3/KCxJAgpN/ygsSQI\nKTf8oLEkCCk3/KCxJAgpN/ygsSQIKTf8oLEkCCk3nNmQBCHlpn+oVrZnfY8tgQ4h5Wb0f8jG\nWwIdQsrN49PfkZdAh5By0z9Up9VyH3kJdAgpN/zEviQIKTeElAQh5Yanv5MgpNwQUhJeQpqS\n+hhqifM9IUeXwC8vIU1dlvoYahmGFCUnjvgzQsoNIUUz+XWR/WQTUlKEFM2XJ5uQkiKkaAip\nJIQUDSGVhJCiIaSS/IYU7TWCUo84IZWEkKIhpJJwZkM0hFQSQoqGkEpCSNEQUkkIKRpCKgkh\nRUNIJSGkaAipJIQUDSGVhJCiIaSSEFI0hFQSQoqGkEpCSNEQUkkIKRpCKgkhRUNIJSGkaAip\nJIQUDSGVhJCiIaSSEFI0hFQSQoqGkEpCSNEQUkkIKRpCKgkhRUNIJSGkaAipJIQUDSGVhJCi\nIaSSEFI0hFQSQoqGkEry/uHYb1btdzde1ftYS/hGSCV593CcFr3vFL6MsoR3hFSSdw9HHaqf\nQ/vWcVeFOsYS3hFSSd49HFU43N8+hCrGEt4RUknePRyDn/wy/WNgSj3ihFQS7pGiIaSSfPAY\naXds3+Ix0guEVJK3D8ey96zd4hRlCecIqSQfvI5Ut68jVasNryONIqSScGZDNIRUEkKKhpBK\nwilC0RBSSThFKBpCKgmnCEVDSCXhBdloCKkknCIUDSGVhHukaAipJJwiFA0hlYRThKIhpJJw\nilA0hFQSzmyIhpBKEulwhL44S8gjpJJwilA0hFQSThGKhpBKwilC0RBSSXhBNhpCKgmnCEVD\nSCXhHikaQioJpwhFQ0gl4RShaAipJJwiFA0hlYRThKIhpJIQUjSEVJIPD8d2EcJqF3UJtwip\nJJ+9jnR9xmHySTtC+s5kE1JSH4VUh/p0Ph/rsI2xhHeEVJKPQqpC+7z3KSxiLOEdIZXko5Bu\npwZxitAYQirJRyGtbyFxitAIQirJ+yGtNttd+Lm8eao5RWgMIZXk/ZDu/408hIpThEYQUkne\nPhyHw3a7WrVPOdSTHRHSdyabkJLizIZoCKkkhBQNIZWEkKIhpJIQUjSEVBJCioaQSkJI0RBS\nSQgpGkIqCSFFQ0glIaRoCKkkhBQNIZWEkKIhpJIQUjSEVBJCioaQSkJI0RBSSQgpGkIqCSFF\nQ0glIaRoCKkkhBQNIZWEkKIhpJIQUjSEVBJCioaQSkJI0RBSSQgpGkIqCSFFQ0glIaRoCKkk\nhBQNIZWEkKIhpJIQUjSEVBJCioaQSkJI0eQe0pTUx/77COkjk9P03cn+ekhTl6X+vHwfIX1E\naHqFtkJIXpdIRmh6hbZCSF6XSEZoeoW2Qkhel0hGaHqFtkJIXpdIRmh6hbZCSF6XSEZoeoW2\nQkhel0hGaHqFtkJIXpdIRmh6hbZCSF6XSEZoeoW2Qkhel0hGaHqFtkJIXpdIRmh6hbZCSF6X\nSEZoeoW2Qkhel0hGaHqFtkJIXpdIRmh6hbZCSF6XSEZoeoW2Qkhel0hGaHqFtkJIXpdIRmh6\nhbZCSF6XSEZoeoW2Qkhel0hGaHqFtkJIXpdIRmh6hbZCSF6XSEZoeoW2Qkhel0hGaHqFtkJI\nXpdIRmh6hbZCSF6XSEZoeoW2Qkhel0hGaHqFtkJIXpdIRmh6hbZCSF6XSEZoeoW2Qkhel0hG\naHqFtkJIXpdIRmh6hbZCSF6XSEZoeoW2Qkhel0hGaHqFtkJIXpdIRmh6hbZCSF6XSEZoeoW2\nQkhel0hGaHqFtkJIXpdIRmh6hbZCSF6XSEZoeoW2Qkhel0hGaHqFtkJIXpdIRmh6hbZCSF6X\nSEZoeoW2Qkhel0hGaHqFtkJIXpdIRmh6hbZCSF6XSEZoeoW2Qkhel0hGaHqFtkJIXpdIRmh6\nhbZCSF6XSEZoeoW2Qkhel0hGaHqFtkJIXpdIRmh6hbZCSF6XSEZoeoW2Qkhel0hGaHqFtkJI\nXpdIRmh6hbZCSF6XSEZoeoW2Qkhel0hGaHqFtkJIXpdIRmh6hbZCSF6XSEZoeoW2Qkhel0hG\naHqFtkJIXpdIRmh6hbZCSF6XSEZoeoW2Qkhel0hGaHqFtkJIXpdIRmh6hbZCSF6XSEZoeoW2\nQkhel0hGaHqFtkJIXpdIRmh6hbZCSF6XSEZoeoW2Qkhel0hGaHqFtkJIXpdIRmh6hbZCSF6X\nSEZoeoW2Qkhel0hGaHqFtkJIXpdIRmh6hbZCSF6XSEZoeoW2Qkhel0hGaHqFtkJIXpdIRmh6\nhbZCSF6XSEZoeoW2Qkhel4gqTNGZXqGtEJLXJaKSmlCZ5aYvm5D60xkJIc2SmlCZ5d7eZupP\nZySENEtoCglJFiHNEppCQpJFSLOEppCQZBHSLKEpJCRZhDRLaAoJSRYhzRKaQkKSRUizhKaQ\nkGQR0iyhKSQkWYQ0S2gKCUkWIc0SmkJCkkVIs4SmkJBkEdIsoSkkJFmENEtoCglJFiHNEppC\nQpJFSLOEppCQZBHSLKEpJCRZhDRLaAoJSRYhzRKaQkKSRUizhKaQkGQR0iyhKSQkWYQ0S2gK\nCUkWIc0SmkJCkkVIs4SmkJBkEdIsoSkkJFmENEtoCglJFiHNEppCQpJFSLOEppCQZBHSLKEp\nJCRZhDRLaAoJSRYhzRKaQkKSRUizhKaQkGQR0iyhKSQkWYQ0S2gKCUkWIc0SmkJCkkVIs4Sm\nkJBkEdIsoSkkJFmENEtoCglJFiHNEppCQpJFSLOEpjCHkKak/lR/gJA6U59enSnMIaSpD0s9\nBR8gpE7mE6pzGSEpL/G5zCdU5zJCUl7ic5lPqM5lhKS8xOcyn1CdywhJeYnPZT6hOpcRkvIS\nn8t8QnUuIyTlJT6X+YTqXEZIykt8LvMJ1bmMkJSX+FzmE6pzGSEpL/G5zCdU5zJCUl7ic5lP\nqM5lhKS8xOcyn1CdywhJeYnPZT6hOpcRkvISn8t8QnUuIyTlJT6X+YTqXEZIykt8LvMJ1bmM\nkJSX+FzmE6pzGSEpL/G5zCdU5zJCUl7ic5lPqM5lhKS8xOcyn1CdyyY/zPE3RikppMnP01en\niZD+flnq6ZlRVEhf/tR/9zKhrRCS1yX+DRMqcBkhKS/xb5hQgcsI6dF+s2ofXKzqfawljDGh\nApcR0tBp0XugvoyyhDkmVOCyt69S/Bm9d/dQh+rn0L513FWhjrHEW958Zq7kCf3uZVGW++aE\nvfDuHqpwuL99CFWMJd7i5VNvf5nQVgjpDx8XXv3h+jf/cMc7ee8BxPbm7I8P85sf94d7JCB/\nHzxG2h3bt2YfIwH5e/vubdm7i1ycLLcE+PPB60h1+zpStdrMvI4E5E/hCQ/APUICDBASYICQ\nAAOEBBggJMAAIQEGCAkwQEiAAUICDBASYICQAAOEBBggJMAAIQEGCAkwQEiAAUICDHgMKdE3\nb4JL3xrKL61jSWjPbGWU0F4I6TWhPbOVUUJ7IaTXhPbMVkYJ7YWQXhPaM1sZJbQXQnpNaM9s\nZZTQXgjpNaE9s5VRQnshpNeE9sxWRgnthZBeE9ozWxkltBdCek1oz2xllNBeCOk1oT2zlVFC\neyGk14T2zFZGCe2FkF4T2jNbGSW0F0ICHCEkwAAhAQYICTBASIABQgIMEBJggJAAA4QEGCAk\nwAAhAQYICTBASIABQgIMEBJggJAAAy5D2i5CVZ9S7+Jqq3AI60roiGgcksYXB0XlJv9F3f6U\ngUpjbg5f+3kHE5btEVmk3saVxCFpfHNQRG7yXxzC+tT8q7dOvZHGoRKYmn2oDs1O9qk30pI4\nJI2vDorGTf6TVbdnic/WNiwF9lGH3eXXn7BJvZGGxiFpfHVQNG7yOyQ+W6FW2McqHM/NP8Cr\n1BtpaBySHkKadArL1Fu4OEgEHYTuo0UOya8vDYrSTf6TbfvljACBqZEK6Sy0kcaXBkXpJv/F\nsZL4OuYsMTWE9Nq3BkXoJv/FqVL4wq4lMDWE9NLXBkXnJs/q/5DqZdrXTPpbEZiaipBe+dqg\n6NzkWb/Te1wsjyJbkZia7lm7o8azdmeJQ9L54qCo3OS/2Ek8YXcjMDWb9vH0LtSpN3IlcEha\n3xwUkZv8F0epjhSmRuvMBolD0vjqoGjc5D9Zh9D/0io1hX0s2uMh8++LwiE5f3lQNG7ynwRC\nenRqz/5OvYs7hUNy/vKgaNxkwDlCAgwQEmCAkAADhAQYICTAACEBBggJMEBIgAFCAgwQEmCA\nkAADhAQYICTAACEBBggJMEBIgAFCAgwQEmCAkAADhAQYICTAACEBBggJMEBIgAFCAgwQEmCA\nkAADhAQYICTAACEBBggJMEBIgAFC8qn5MXTtj6I7rUOor78iHULy6R7SKoSwuf6KdAjJp/sP\nRg3heP8V6RCST72QBn9EInwCkrkM/yZUly/I6tvjm+0iVNv2rd3la7Xux5Rf7mtW7Xvd1dXl\n3a9f2oWe36ut7z/i/H6NIZwWYTWyJmwQUjKXRzXN/O+Wza/NVK/aHpaXtzZdGnX7XlUYPABq\n3301FdLmdjW9a2w/pH5eE0YIKZnLhJ/O2+uv1eVeqHnrtAy75qKf8/mnbeN6+eL2UT+hOpwP\n1f3Jhqcv7cL1HX4ervHy5vOasEJIyYSwP/8+WdDcfTSjfrp8AXZ/h9573f5y1f55NxXSrn2H\n1eAau6t5WhNWOJjJ9BPouuh9jXbcbZajoVzfmgrpPHKNvb/lCYoYOJjJTIW0vL9FSD5wMJN5\nDul+0TostrujQUhjf0tIMXAwk3kc6lX74Ob3ovGQuvfaT4XUPYhaP18jIcXDwUzmcajb5+PO\n29tTA4fxx0i7f33WbvdwjaNrwgoHM5mnoe4eGFXH9uXS1n5wefdm+9rQejSk2/Pl7StN58E1\nElJkHMxknod6u7gk0p40dwllud/93pX0mjlvemc2jIZ0aW3RnSDxe42EFBkHMzf0kQRHPTeE\nlARHPTeElARHPTeElARHHTBASIABQgIMEBJggJAAA4QEGCAkwAAhAQYICTBASIABQgIMEBJg\ngJAAA4QEGCAkwAAhAQYICTBASIABQgIMEBJggJAAA4QEGCAkwAAhAQYICTBASIABQgIM/A82\nX5oZAls5mgAAAABJRU5ErkJggg==",
      "text/plain": [
       "Plot with title \"Histogram of mean.diff.perm\""
      ]
     },
     "metadata": {
      "image/png": {
       "height": 420,
       "width": 420
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "hist(mean.diff.perm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "0.30245"
      ],
      "text/latex": [
       "0.30245"
      ],
      "text/markdown": [
       "0.30245"
      ],
      "text/plain": [
       "[1] 0.30245"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mean(abs(diff) > abs(mean.diff.perm))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "4.1.0"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}