Probability.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7734341f-8ca7-4842-bb4d-20f43bbe2a4c",
   "metadata": {},
   "source": [
    "- https://www.youtube.com/watch?v=KHVR587oW8I"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "4920743c-d8a2-4c62-bac1-7eaea6d35ab2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-03T14:52:37.144051Z",
     "iopub.status.busy": "2024-11-03T14:52:37.143438Z",
     "iopub.status.idle": "2024-11-03T14:52:37.152871Z",
     "shell.execute_reply": "2024-11-03T14:52:37.150738Z",
     "shell.execute_reply.started": "2024-11-03T14:52:37.144007Z"
    }
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b86e05a8-d153-400b-93f2-4aefbd7e2076",
   "metadata": {},
   "source": [
    "- Probabilistic interpretation\n",
    "- likelihood & uncertainlity\n",
    "- learning a probability distribution from data\n",
    "    - then inference\n",
    "    - or synthesis\n",
    "- 频率 vs. Bayesian：关于 probability 的不同观点；\n",
    "    - 频率：硬币、骰子，repeats\n",
    "        - 无法解释天气预报的概率，比如明天天气下雨的概率是70%；\n",
    "            - 因为我们无法 repeat the measurement of tomorrow's weather to find the frequency of rain outcomes;\n",
    "            - $P(rain)=\\frac{P(rain)}{P(rain)+P(sunny)}$\n",
    "    - Bayesian：Probability is viewed as the as a **subjective degree of belief** or confidence；\n",
    "        - 明天下雨的概率是 0.7：a measurement of confidence rather that a frequency\n",
    "- high-dimensional probability distribution\n",
    "    - all images come from a high-dimensional probability distribution\n",
    "- generation\n",
    "    - Sampling from the **joint distribution** of pixels allows for generative capabilities\n",
    "        - because this distribution captures **the structure of data**；\n",
    "        - 相对地，sampling each pixels **independently** always results in noise；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55922428-eb2e-44f6-9b70-56dd4cdba796",
   "metadata": {},
   "source": [
    "### surprise => Entropy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97a18834-4a55-4389-9abb-3fbd5564a75d",
   "metadata": {},
   "source": [
    "- $h(s)$：surprisal of state $s$\n",
    "- $h(s)=f(p_s)$：function of probability\n",
    "    - $h(1/6)\\gt h(1/2)$：骰子为1的惊讶度 > 硬币正面的惊讶度\n",
    "    - rare events => high surprise\n",
    "    - common events => low surprise\n",
    "    - $p_s\\rightarrow 1, h(p_s)\\rightarrow 0$\n",
    "- when probabilities multiply（概率连乘）, the surprise should add\n",
    "    - $h(p_1\\cdot p_2)=h(p_1)+h(p_2)$ （log function）\n",
    "    - 同时 $p_s$ 越大，$h(p_s)$ 越低\n",
    "- $h(s)=\\log \\frac1{p_s}$\n",
    "- average surprise of a distribution => Entropy\n",
    "    - $\\sum_sp(s)\\log\\frac1{p_s}=-\\sum_sp_s\\log p_s$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8696c948-6bee-4ee4-9276-a099d600427b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-03T14:11:10.854505Z",
     "iopub.status.busy": "2024-11-03T14:11:10.853889Z",
     "iopub.status.idle": "2024-11-03T14:11:11.374441Z",
     "shell.execute_reply": "2024-11-03T14:11:11.372579Z",
     "shell.execute_reply.started": "2024-11-03T14:11:10.854458Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Define the surprise function h(s)\n",
    "def h(p_s):\n",
    "    return np.log(1 / p_s)\n",
    "\n",
    "# Generate probability values\n",
    "p_s_values = np.linspace(0.01, 1, 1000)  # Avoid zero to prevent log(0)\n",
    "h_values = h(p_s_values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4531dc12-1be6-47a1-a382-a48d9787b404",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-03T14:12:02.064975Z",
     "iopub.status.busy": "2024-11-03T14:12:02.064610Z",
     "iopub.status.idle": "2024-11-03T14:12:02.281116Z",
     "shell.execute_reply": "2024-11-03T14:12:02.279367Z",
     "shell.execute_reply.started": "2024-11-03T14:12:02.064950Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAc0AAAE0CAYAAABO2GjMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAoA0lEQVR4nO3deXhU9b3H8XdCMoFsIiBrZF8FBRVQdq2somy2smiLFFTw6pWrtTW0XtQWsVWhWlyuj4qWq1y1LmhxwQVtZTWICLIZZAkRwp49mSy/+8eZTCYhgZOQzDnJfF7P833mnDNnZr5zjH78nTlLGGAQERGRswp3ugEREZG6QqEpIiJik0JTRETEJoWmiIiITQpNERERmxSaIiIiNik0RUREbFJoioiI2KTQFBERsUmhKSIiYpPrQnPv3r0YY06rJUuWON2aiIiEuAinGyivX79+NGjQwD/fq1cvPv30U958800HuxIREYEwXH7B9sWLF3PdddfRpUsXp1sREZEQ57qRZqDIyEhuvvlmFi1aVOk6Ho+HqKgo/3xxcTH5+fkUFhYGo0UREQkhrvtNM9CECRNo3LgxL7/8cqXrJCYmkpGR4a+dO3cqMEVEpFa4evfsRx99hNfrZdy4cZWuU9FIMzs7OxjtiYhIiHHt7tm2bdsyfPhwJk2adMb1vF4vXq83SF2JiEgoc+3u2RkzZnDkyBFWrlzpdCsiIiKAS0MzLCyMGTNm8Morr1BUVOR0OyIiIoBLQ3P48OG0a9eOl156yelWRERE/Fx9IJCIiIibuHKkKSIi4kYKTREREZsUmiIiIjYpNEVERGxSaIqIiNik0BQREbFJoSkiImKTQlNERMQmhaaIiIhNCk0RERGbFJoiIiI2KTRFRERsUmiKiIjYpNAUERGxSaEpIiJik0JTRETEJoWmiIiITQpNERERmxSaIiIiNik0RUREbFJoioiI2KTQFBERscmVodm6dWuWLVvGsWPHyM7OZvPmzVx22WVOtyUiIiEuwukGymvcuDFr1qxh9erVjBkzhiNHjtCpUydOnTrldGsiIhLiwgDjdBOBFi5cyKBBgxg6dKjTrYiIiJThut2z48aNIykpiTfeeIO0tDS++eYbZs2aVen6Ho+HuLg4f8XGxgaxWxERCTXGTZWbm2tyc3PNggULTJ8+fcxtt91mcnJyzC9/+csK158/f74JlJ6e7vh3UKlUKlX9LNftns3PzycpKYlBgwb5lz355JP069ePgQMHnra+x+MhKirKP2+MISsrKyi9iohIaHHd7tlDhw6xffv2Mst27NhB27ZtK1zf6/WSmZnpLwWmiIjUFteF5po1a+jWrVuZZV27dmX//v0OdSQiIlLK8X3EgdW3b1/j9XpNYmKi6dSpk5k6darJysoy06ZNc7w3lUqlUoV8Od7AaTV27Fjz3XffmdzcXLN9+3Yza9Ysx3tSqVQqlcp1BwKJiIi4let+0xQREXErhaaIiIhNCk0RERGbFJoiIiI2KTRFRERsUmiKiIjYpNAUERGxSaEpIiJik0JTRETEJoWmiIiITQpNERERmxSaIiIiNik0RUREbFJoioiI2KTQFBERsUmhKSIiYpNCU0RExKYIpxuoK/55yy0Mbt/eP19YXMzxnBzW7d/P/E8+Yf+pU471JiIiwaGRZhXlFxby9cGD7Dt5khaxsUzo2ZPl06ad8/tGNmhQA92JiEht0kizitKyshjxwgsAPDdxIlN69+ai5s1p3KgRp3Jz+Z+JE+mbkEDz2FiiIiJIy8zkg127+NPnn5OZnw/AMxMmMK1PH77at49VP/zA7VdcQau4OJo89BAAN15yCbOvuILuzZtTVFzMhpQUHvr0U7YePuzY9xYREYXmOQnzPabn5ZHlC8SxPXqQX1jIvpMnifV46NCkCbdfcQUtYmO55c03y7y+X0ICA9q25Yfjx/H4Rpr/OWgQD48YAcAPx44R4/EwvHNnrmzblp89/zy7jx0L2vcTEZGyFJpV1CI2lk9mzaJxw4Z0adaMk7m53P3++xQWFwMw5qWXyowIf/+zn3Hf0KGM7d6dqIgI8gsL/c9FRUQw+bXX+Hj3bsLDwmgUGcnvhg0D4JHVq/nLl1/SIDycT2bO5LI2bbh3yBBuf+ed4H5hERHxc11ozp8/nwcffLDMssOHD9OqVStnGionKiKCfgkJ/vntaWms2bfPPz+sQwf+Z9IkOpx/Po0iI/3LIxs0oFl0NKkZGf5lPxw7xse7dwNQbAzdL7iAGI8HgHlXX828q68u89l9Az5XRESCz3WhCbBt2zaGDx/uny8qKnKwm7IOnDpF7yef5OqOHXlt6lQGtW/PkvHjmbp8Ob+4+GL+NGoUAIcyM/k+LY2m0dF0aNIEgAbhZY+7OpKVVWY+LCzMP73z6FH/b6AlTuTk1MZXEhERm1wZmoWFhaSlpTndRqWMMXy+Zw8vbNzInQMHMqZbN/omJPhHoBn5+fT+61/xFhXxxNixzPSF5mnvU25+x5Ej5BQUEB0ZyWfJyfz+44/9z13SsiUNA0auIiISfK4MzS5dupCamkp+fj4bNmxg3rx57N27t8J1PR4PUVFR/nljDFnlRnC1Zcm6ddzavz9RERHcM3gwH/l2tcZHRfHt3XfjLSoiPqC3s8ktKOCxL79k/vDh/MeAAUzs2ZPjOTm0iY+nSXQ0j37xBRtTUmrr64iIyFm47jzNDRs28Ktf/YpRo0Zx66230rJlS9auXUuTSkZriYmJZGRk+Cs1NTVovR7OzOT1774DYHS3bmxKTeXpdes4lp1NbFQUX+3bxyOrV1fpPRd/9RWz33mHTampNG7UiA5NmnA0O5sXv/6a93fsqI2vISIiNoVx+l5CV4mOjmbPnj385S9/YfHixac97+RIU0REQosrd88GysnJYevWrXTp0qXC571eL16vN8hdiYhIKHLd7tnyPB4PPXr04NChQ063IiIiIc51ofnYY48xdOhQ2rdvT//+/fnHP/5BfHw8r7zyitOtiYhIiHPd7tmEhASWL19Os2bNOHr0KOvXr+fKK6/kwIEDTrcmIiIhzvUHAomIiLiF63bPioiIuJVCU0RExCaFpoiIiE0KTREREZvqdWgOBZYAHwBXONyLiIjUffU6NC8C/gMYA1zscC8iIlL31evQ/DFguqNjXYiISH0RMqHZybEuRESkvqjXobkfKPJNa6QpIiLnql6HZgFQcstmhaaIiJyreh2aULqLtgnQ2ME+RESk7qv3obknYFqjTRERORf1PjR1MJCIiNSUkApNjTRFRORc1PvQTA6Y7upYFyIiUh/U+9DcFTDd07EuRESkPqj3oZlN6S7ai7Duui0iIlId9T40Ab73PcYBbZ1sRERE6rSQCk3QLloREam+kAjNbQHTCk0REamukAjNwJFmL8e6EBGRui4kQnMnUOib7uNgHyIiUre5OjTvv/9+jDEsXrz4nN4nj9JdtD2BRufamIiIhCTXhmbfvn257bbb2LJlS428X5LvsQFwaY28o4iIhBpXhmZMTAyvvvoqt956KydPnqyR90wKmO5bI+8oIiKhxpWh+fTTT7Ny5Uo+++yzs67r8XiIi4vzV2xsbIXrfR0wrdAUEZHqcF1oTp48mcsuu4zExERb6ycmJpKRkeGv1NTUCtfbBuT7pvvXTKsiIhJiXBWaCQkJPPnkk9x8883k5+ef/QXAwoULiY+P91ebNm0qXM8LfOOb7gY0r5GORUQklIQBxukmSowfP553332XwsJC/7KIiAiKi4spLi4mKiqK4uLiar//QuB+3/QvgH+cU7ciIhJqXBWasbGxtGvXrsyypUuXsnPnTv785z/z/fffV/JKe0YDH/qmlwB3ndO7iYhIqIlwuoFAWVlZpwVjdnY2x48fP+fABFgDFGGddjL0nN9NRERCjat+06xtmcBm3/QlQBMHexERkbrHVbtng+Fx4F7f9ETgXedaERGROiakRpoAqwOmRznWhYiI1EUhN9KMBo4DDYEDQLszry4iIuIXciPNHOAL33RbdKswERGxL+RCE+CDgOlrHetCRETqmpAPzbGOdSEiInVNyP2mWWIn1uX0ioDWwBFn2xERkTogJEeaAG/5HhsANzjZiIiI1BkhG5qvB0zf6FgXIiJSl4Ts7lko3UVbDCQAh5xtR0REXC5kR5pQOtoMR6NNERE5u5AeaXYHdvimt2Jdj1ZERKQyIT3S3Ams9U1fDPRzsBcREXG/kA5NgBcCpmc51oWIiNQFIb17FiAG6wCgOKxbh7UCsh3tSERE3CrkR5rZwHLfdBwwzcFeRETE3UJ+pAlwGbDJN70D6Ik2ioiInC7kR5oA3wBf+qZ7AGMc7EVERNxLoenzRMD0vY51ISIibqbdsz5hWLtmu/nmL8cagYqIiJTQSNPHAIsC5uc71YiIiLiWRpoBPEAycKFvvh+Q5Fw7IiLiMhppBvACCwLmH3KqERERcSXXhebs2bPZsmUL6enppKens3btWkaPHh20z38J2O+bvhYYHLRPFhERt3Pd7tnrrruOoqIikpOTAZg+fTr33Xcfl156Kdu3bw9KD78GXvRNb8LaTeuqjSQiIo4xbq/jx4+bX//610H7vHAw34Ixvprugm2gUqlUKufLdbtnA4WHhzN58mRiYmJYt25dhet4PB7i4uL8FRsbe86fWwzMDZhfiHWNWhEREceTu3z16tXLZGZmmoKCAnPy5EkzZsyYStedP3++CZSenl5jfbxN6Wjzzy7YLiqVSqVytlz3myZAZGQkbdu2pXHjxtxwww3MmjWLYcOGsWPHjtPW9Xg8REVF+eeNMWRlZdVIHx2B74GGQCHQH9hcI+8sIiJ1kStDs7xPPvmEPXv2MHv27KB/9jxKT0P5Bis4i4LehYiIuIGrf9MsERYWVmY0GUyPAd/5pi8D7nGkCxERcYMIpxsob8GCBXz44YekpKQQFxfHlClTuOqqq4J6rmagAmAWsB7r/zD+CKwCtjjSjYiIOMl1odmiRQuWLVtGq1atSE9P57vvvmP06NF8+umnjvX0NdZdUO4DorBuWn05kOtYRyIi4oQ68ZumG3iwRpuX+uafBe5wrh0REXGAQrMKumEdDBTtm78BeNu5dkREJMjqxIFAbrGLshc9eBno4UgnIiLiBI00q+FVYJpvejfWtWkznGtHRESCRKFZDY2AtUAf3/z7wASsy++JiEj9pd2z1ZALTASO++avBxY5146IiASJQrOa9gGTsc7jBLgbuNexbkREJBi0e/YcTcc6IKjEFOB1Z1oREZFappHmOXoFeCBg/u+AM9cuEhGR2qaRZg15HrjVN50HXAd85lw7IiJSCxSaNaQB8Bpwo28+B7gW+NKxjkREpKZp92wNKQJuAt7xzUcD/wR+5lhHIiJS0xSaNagQ64jaf/rmY4EPsM7hFBGRuk+hWcMKgJ8DK3zzUcA/gFucakhERGqMQrMW5GNdzP0V33wDYCnwW8c6EhGRmqDQrCVFwAzgrwHL/gy8AEQ60ZCIiJwzHT0bBPOABQHzX2CNRE840o2IiFSXQjNIbsS6clAj33wyMA7Y4VRDIiJSZdo9GyRvAFcBh33znYGNWJfdExGRukGhGUQbgf7At775WGA5sATwONSTiIjYp9AMshRgIGUv8v4fwL+Bjk40JCIitik0HZCLdWTtLKzr1ELpCHSGQz2JiMjZ6UAgh/UB3sT6jbPE28BtlN7kWkRE3MF1I83777+fjRs3kpGRQVpaGu+88w5du3Z1uq1a8y1wKfBiwLJJwFasO6WIiIh7uC40hw0bxtNPP82VV17JiBEjiIiIYNWqVURHRzvdWq3JwtpVOxE45lvWCngf+D+guUN9iYhIWa7fPdusWTOOHj3K0KFD+fe//+10O7WuJdao89qAZSeAeyi9LJ+IiDjDdSPN8s477zwATpyo+Po5Ho+HuLg4f8XGxgazvRp3GBgL3EzpqLMJ1tG2nwDdnWlLRER8jJtrxYoV5l//+lelz8+fP98ESk9Pd7znmqpmYJaBMQHlBfM4mHgX9KdSqVShVq7ePbtkyRLGjh3L4MGDSU1NrXAdj8dDVFSUf94YQ1ZWVrBaDIrRwLNA+4BlacD9WLtsXfsPUESkHnI8uSuqp556yhw4cMC0b9/e8V7cUI3A/DeYHMqOPDeAGeaC/lQqlSpEyvEGTqu//e1v5uDBg6Zz586O9+K2agvmDcoGpwGzEswlLuhPpVKp6nO5bvfs008/zbRp0xg/fjy7du3yL09PTycvL+8MrwwtV2Pdq/OSgGXFwKvAfwP7gt+SiEhIcDy5A6sy06dPd7w3t1U4mF+C2UfZUWc+mGfAXOiCHlUqlao+letGmlJ1UcAc4A9A04DlXqxTVRaikaeISE1QaNYj8cB9wN1AXMDyQuDvwCPAHgf6EhGpLxSa9VATrOC8GzgvYHkR8BbwBNa9PUVEpGoUmvVYY+A/gbnA+eWe+zfwONb1bfUHICJij0IzBMRj3ej6P7GubRtoF7AYWAbkBLkvEZG6RqEZQqKAacBvgIvKPXcK66ChZ4HdQe1KRKTuUGiGoDCsS/P9BvhZBc9/CjwDvIf1O6iIiFgUmiHuUqxdt9OARuWeOwg8Dyz1TYuIhDqFpgDWgUK3YJ3v2aXcc8VYtyV7CVgB5Ae1MxER91BoShlhwHCs0ed1QINyz58AXsMK0M3BbU1ExHEKTanUhVijz1uAjhU8/y3WUbevAxXfuE1EpH5RaMpZhQHDgF8DNwDR5Z4vBv4FLAf+gTUaFRGpjxSaUiXxwGSsAL2ygucLgI+xduG+B2QHrzURkVqn0JRq6wJMwTrytnsFz2cDHwBvAyuBzOC1JiJSKxSaUiP6YIXnFKzfQsvLxzoC9y2sEah24YpIXaTQlBoVBgwGpgI/By6oYJ1C4AusEei7wKEg9SYicq4UmlJrGmAF6A3AJKBNJeslAf/01TfoD1JE3EuhKUERBvTHCs8bgE6VrHcY6/fPf2Jdzi8rKN2JiNij0BRHXIIVoNcBl1eyTj7WbtyVWEfk6kLyIuI0haY4rjVwLVaAjuD080BLHABW+eozdDCRiASfQlNcpSFwFVaAXge0q2S9YqzfQj/BCtF1WOeIiojUJoWmuFovYBQwEhiKFaoVycLalbva9/gtVrCKiNQkhabUGQ2BIVi7cEcCvc+w7imsS/uVhOgW9IcuIjXDuKmGDBli3nvvPZOammqMMWb8+PGO96RyZ7UEczOYv4M5BMacoY6DeQfM3WAuARPmgv5VKlXdq3BcJiYmhi1btnDnnXc63Yq43GHgf4FfAa2Anli3NHsTOFpu3SbABOCvWKPO41intSRi7fYtfwNuEZGKuHr3rDGGCRMmsGLFCqdbkTomDLgI66Ciq7Hu0tLsDOsXAJuANQF1pHZbFJE6KMLpBs6Vx+MhKirKP2+MIStLp8SHOgN876unsUK0J1aAXoV1paLmAetHYt215UrgXt+yZEoDdC2wAx1cJBLqXLd7tqoSExPJyMjwV2qqbocspzPANuBvWFckaoF1l5YZwAtYgVheZ2A68LzvtaeAz4GFwESs80tFJLTU+d2zGmlKTWkKDAQG+aofEHXGV0AqsAHY6HtMQpf+E6nP6vzuWa/Xi9frdboNqQeOA+/7CqzAvBwrQPsDV3D6bc/aYF0OcJJvvhjYjhWgm7AuQL8FyKvNxkUkaOp8aIrUlnys3zLXBixrRWmA9scajcYHPB+OdUGGXsBM37JCrN2/31AapN9i3aRbROoW14VmTEwMnTt39s936NCB3r17c+LECVJSUhzsTMS69+cKX4EVkt0oDdErsC5GH/gvVgRwsa+m+5YVY12AviREvwE2A+m1276InCPX/aY5bNgwvvjii9OWv/zyy8yYMSP4DYlUUSOgD3Ap1u7dy7CO3I208do9WLtzvwt43IvL/iUVCWGuC02R+igKa6R5WUBdwtkPNALIBLZiBWhgZdZKpyJyJgpNEYdEYl2AoSREL8e6nm5lt0Yrby+lAboFK1j3AEU13qmIlFBoirhIONARKzwvCaiONl+fB+zCuqjDdkov8PAjClORmqDQFKkD4rGOyL2E0kC9GIiz+XqFqUjNUGiK1FFhQHvKhuhFQFfsHxZfEqYlQbrTN/8D1ik3IlKWQlOknonECs6LsI7aLXnsgr0jeME6JWYfVoDuojRMd2GddiMSqhSaIiEiEis4A4O0qmEKkEHFYfoDuvKR1H8KTZEQVxKmF2FdqKGkulP2akdnUwzsxwrQ5ID6AetI34Kaa1nEMQpNEalUS0oDNDBQO1C1WyQVAQcoDdHAUP0R/X7qpEhPFKOnTWfgmHEkdOpCgwYRHE/7ia3r17DihWdIO3jA6RZdRaEpIlUWhXXrtPKB2h04r4rvVQykcProtCRQc2umZalATPx5PLj0DTr2vBiAnKxMjv50kGat2hATF8+SxLmsfucNh7t0F9dde1ZE3C+f0tNWyrsAK1BLqkvAY+MK1g8H2vnqmgqeP4y1e/fHco97gYPolJlzMeuBBf7AfPeFZ3h18UKKi6wt2mfwVeRmWze663v1SCbMuoP23XsS3iCc1D3JfPjqUj5/+//87/XWzp8AWPHis8Q3bcqAkddx8mgaLy14gLSUA8z50+N06NGLfbu288zv7yH1x2QA7ly4mKsnTuZIagqv/+1xpt79O6Lj4vj8rf/j1UULuemeRK6eNJmcjAzefv5vrHp9mf8zm7Vqw9S7f0vvQcOIPa8x6cePkbR6Fcuf/AtZ6afKvP+2jWtZ//FKxv16NnGNz+f7r9fx7B9+w6ljR6u0zTTSFJGgaULZIA0M1qbVeL8CrN2+gUEaGK7Hz73leis6No6l67YRERnJ3h3f85uJIypcb+j1k7j7sSUAnDx6hAJvPs3bWDfJe23xo7z1P08BpaHpzc8j8+QJPA0bEdf4fHKyMsnJzKSosIDzml5Aw+hodn2bxLwp44DSUCvw5lNUWMTJo2m0atcBgJTk3TRp0ZKczEwuaN2GoqIi/uv6q0n9MZn4Jk15/J1VNG3RCm9+HmkH9tOqfUciIiPZv3sHv/v5tRR48wPe30txcRHHD/9E6/adAPjX+2/z5H13Vmm7VeVnCRGRc3IC64bdrwIPAb8EBgDNsAK1HzAVeAD4O/AV1o2+KxMJdAJGALcBC4HXfZ9xDOtI3y3AO8Ai4C7geqzzWqu6G7m+KQkYgB2bNlS63rS59wOw+9tNzP5Zf+ZccwXrV30AwA2z78bTsFGZ9Q/v38cdIway6L9mA1Y4H9yzmztGDOClBQ8A0K1PXzxRDcu8LtITxcMzp3LX6MEc/cn6p96ybXvunTCcu8cOJT8vlwYNGtCz3wAAxtw0g6YtWlFUVMS8qeOZe/3VPDH3dgDade3B4OsmlHn/8AYNmDdlHHeNHuLv/+IrB1dto6HdsyLiEieBJF+V1xBr920HrEsKln+sLADjKL0UYUUysI74LakD5eYPU393xYWFhfmnjan4W8Y3acoFbRIAWP/JhxQWeAFY88EKrhx5LVGNGnFh567s2bbF/5ota7+ksMDLkdTSWzlu+vIzANIO7vcvO69pU384AmSeOsmuzV8DcOxQKhe0bkPKD7s4mnoQgIzjx7mgTQLnNbsAgM69egPw09497N2+FYCNn31EXk4ODaOj6dSzN6vfft3//gd272TfTusHhYN7dgPXcl7TZra3VwmFpoi4XsmVi3ZV8vz5WAFaUai2BzyVvC6e0nudVsSLdZBSZcGa4lunLvpp7x4KCwqIiIykx2X9z/6CSoK1vJws63fQoqJC/7LcrEzfWwS8R0BoA/7fTwNfm5Ndei8f4/vfl7Byryvf12nP+2Rnlt6ttsj3u214eNV3tio0RaTOO+mrbyp4LhxoTWmQtqX0wKN2vvnKbtHmwdr92+kMn32I0jA9iBWkgY+HcefBSjlZmaz96H2GXj+Jjj0v5qZ7Eln+5F/8BwJdMmAI+Xm5HE09yAVtErhy5LWsXPYihQVeBl07HoD83FxSknc70n/yti1cNuwaWnfsTIeLLmbv9q30v2Y0UY2s3cV7vt9ylneoHoWmiNRrxVjhdRD4dwXPhwHNOT1IA+cbn+H9W/nqykqeL8QK1pIeyofqQd/zTgTrC3/8PQkdu9Cx58VMuu0uRk2dztHUFJq2bE1c4/NZkjiX1/76KHc/toSufS7nuc83ljkQ6K3nnsSb58xJQR++upThv5hGk+YteWT5Cg7v30frDtb/3uzfvYOv/vlurXyuQlNEQpoB0ny1sZJ14ql4hFoy3foM7x8BXOiryhRRGqyBYRoYsLURrNkZ6cybOo7RN93CIN/FDVp36MSJtMNs+ORDtidtIC1lP7nZ2f5TTmLi4/lx+zY+/N+XypxyEmwZJ46TOPl6ps39Hb0HDaN1h06kHz/K159bp5wUeGvnkhk65URE5Bx5gDZAgq8uLPeYgHV1pXNRjBXsP5Wr1HLzx9B/1GuTQlNEJAg8WCPSykL1Qs49WME6d/UQp4dp+ZBNr+wN5IwUmiIiLhGJFazlQ/VC3/I2WMFaE7+r5VBxoB7y1WFfnaqBz6pPFJoiInVIONalClsHVJty862BFjX0efmUBmhgmFZUoXDhfYWmiEg9FIk1Ki0fpuUD9vwa/MyTnB6kFQVtXf7d1ZWhOWfOHO677z5atWrF999/z9y5c/nqq6+cbktEpN5pRMWj1JZYp9K09FXzGvzMQuAIpYF6BOsgp4oej+Gu81xdF5o33ngjy5Yt44477mDNmjXcfvvtzJo1i4suuoiUlJSzv4GIiNS4CKzdwiUhGhio5SuuBj+3GOvC+5WFavnH2j5r1HWhuX79er755hvuuOMO/7Lt27fz7rvvMm/ePAc7ExERO2KoPFADA7cFNX+xgExKQ3QfcFMNv7+rLm4QGRnJ5ZdfzqOPPlpm+apVqxg4cGCFr/F4PERFlV4Eq7i4mOzs7FrtU0REKpcN7PHV2TTB2vV7ge+xWcB04LLmQLTNz7/AV22r1LU9rgrNZs2aERERQVpaWpnlaWlptGxZ8RlMiYmJPPjgg/75gwcPcuGFZ7r2hoiIuMUJX+2sxc+IiIigsLDw7Cva4Mr7aZa/TU1YWFilt65ZuHAh8fHx/urXrx8pKSnExMQEo9V6IzY2lvT0dGJjY51upc7QNqsebbeq0zarnpiYGFJSUsrsjTxXrhppHjt2jMLCwtNGlc2bNz9t9FnC6/Xi9Za9OU9CQkK1bvkSysLCwoiPj6/0tjpyOm2z6tF2qzpts+oJDw+v8TxwVbIUFBSwadMmRowYUWb5iBEjWLt2rUNdiYiIWFw10gRYtGgRy5YtIykpiXXr1nHbbbfRtm1bnnvuOadbExERwbit5syZY/bu3Wvy8vJMUlKSGTJkiO3XejweM3/+fOPxeBz/HnWptN20zbTd3FvaZu7Zbq47T1NERMStXPWbpoiIiJspNEVERGxSaIqIiNik0BQREbGpTobmnDlz+PHHH8nNzSUpKYnBgwefcf2hQ4eSlJREbm4ue/bs4fbbbw9Sp+5RlW02ceJEVq1axZEjR0hPT2ft2rWMHDkyiN26R1X/1koMHDiQgoICNm/eXMsduk9Vt5nH4+FPf/oT+/btIy8vj+TkZGbMmBGkbt2jqttt2rRpfPvtt2RnZ/PTTz/x0ksv0aRJkyB167whQ4bw3nvvkZqaijGG8ePHn/U1NZUFjh8WXJW68cYbTX5+vpk5c6bp3r27Wbx4scnMzDQXXnhhheu3b9/eZGVlmcWLF5vu3bubmTNnmvz8fDNp0iTHv4tbt9nixYvNfffdZ/r27Ws6d+5sFixYYPLz802fPn0c/y5u3m4lFR8fb5KTk81HH31kNm/e7Pj3cPs2e/fdd826devMNddcY9q1a2f69etnBgwY4Ph3cfN2GzRokCksLDR33XWXad++vRk0aJDZunWrefvttx3/LsGq0aNHmz/+8Y9m4sSJxhhjxo8ff8b1azALnP/yVan169ebZ555psyy7du3m0ceeaTC9R999FGzffv2MsueffZZs3btWse/i1u3WUW1bds288ADDzj+XerCdlu+fLl5+OGHzfz580MuNKu6zUaNGmVOnjxpzj//fMd7r0vb7d577zXJyclllt15553mwIEDjn8XJ8pOaNZUFtSp3bMltw5btWpVmeVnunXYgAEDTlv/448/pm/fvkREuO6CSDWuOtusvLCwMOLi4jhx4kRttOhK1d1ut9xyC506deKhhx6q7RZdpzrbbNy4cSQlJfHb3/6WgwcPsmvXLh577DEaNmwYjJZdoTrbbe3atSQkJDBmzBjAuj73z3/+c1auXFnr/dZVNZUFdSo1qnPrsJYtW1a4fmRkJM2aNePw4cO11q8bVGeblXfvvfcSExPDG2+8URstulJ1tlvnzp159NFHGTJkCEVFRcFo01Wqs806duzI4MGDycvLY+LEiTRr1oxnnnmGJk2aMHPmzGC07bjqbLd169Zx00038frrr9OwYUMiIyNZsWIFd911VzBarpNqKgvq1EizRFVuHVbZ+hUtr8+qus1KTJkyhQcffJDJkydz9OjR2mrPtexut/DwcF577TXmz5/PDz/8EKz2XKkqf2vh4eEYY7jpppv4+uuv+fDDD7nnnnu45ZZbQmq0CVXbbj169OCpp57i4Ycf5vLLL2fUqFF06NBB1+g+i5rIgjo10qzOrcMOHz5c4foFBQUcP3681np1i+pssxI33ngjL774Ir/4xS/47LPParNN16nqdouLi6Nfv35ceumlLFmyBLACITw8nIKCAkaOHMnq1auD0rtTqvO3dujQIVJTU8nIyPAv27Fjh/+WTsnJybXasxtUZ7slJiayZs0aHn/8cQC2bt1KdnY2X331FX/4wx/q/R606qipLKhTI83q3Dps3bp1p60/cuRIkpKSauxO3m5W3dutTZkyhZdffplp06bxwQcf1HabrlPV7ZaRkUGvXr3o06ePv5577jl27txJnz592LBhQ7Bad0x1/tbWrFlD69aty9w0vmvXrhQVFXHw4MFa7dctqrPdoqOjKS4uLrOs5CcB3XOzYjWZBY4f+VSVKjk0e8aMGaZ79+5m0aJFJjMz07Rt29YA5pFHHjGvvPKKf/2Sw4yfeOIJ0717dzNjxoyQPeXE7jabMmWK8Xq9Zs6cOaZFixb+io+Pd/y7uHm7la9QPHq2qtssJibGHDhwwLzxxhumR48eZsiQIWbXrl3m+eefd/y7uHm7TZ8+3Xi9XjN79mzToUMHM3DgQLNx40azfv16x79LsComJsb07t3b9O7d2xhjzNy5c03v3r39p+nUYhY4/+WrWme6ddjSpUvN6tWry6w/dOhQs2nTJpOXl2d+/PFHc/vttzv+Hdy8zVavXm0qsnTpUse/h5u3W/kKxdCszjbr1q2bWbVqlcnOzjYHDhwwjz/+uGnYsKHj38Pt2+3OO+8027ZtM9nZ2SY1NdUsW7bMtG7d2vHvEawaNmzYGf87VVtZoFuDiYiI2FSnftMUERFxkkJTRETEJoWmiIiITQpNERERmxSaIiIiNik0RUREbFJoioiI2KTQFBERsUmhKSIiYpNCU0RExCaFpoiIiE0KTREREZsUmiIiIjYpNEVERGxSaIqIiNik0BQREbFJoSkiImKTQlNERMQmhaaIiIhNCk0RERGbFJoiIiI2KTRFRERsUmiKiIjYpNAUERGxSaEpIiJik0JTRETEJoWmiIiITQpNERERmxSaIiIiNik0RUREbFJoioiI2KTQFBERsUmhKSIiYpNCU0RExCaFpoiIiE0KTREREZsUmiIiIjYpNEVERGxSaIqIiNik0BQREbHp/wFPUxHq6TBS6gAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plotting\n",
    "plt.figure(figsize=(5, 3))\n",
    "plt.plot(p_s_values, h_values, color='red', linewidth=2)\n",
    "\n",
    "# Adding text and labels\n",
    "plt.text(0.05, 5, 'Rare', color='lightcoral', fontsize=10, weight='bold')\n",
    "plt.text(0.85, 0.3, 'Common', color='lightblue', fontsize=10, weight='bold')\n",
    "plt.text(0.4, 4, r'$h(s) = \\log \\frac{1}{p_s}$', fontsize=30)\n",
    "\n",
    "# Label axes\n",
    "plt.xlabel('Probability $p_s$', fontsize=14)\n",
    "plt.ylabel('Surprise', fontsize=14)\n",
    "\n",
    "# Set limits and style\n",
    "plt.xlim(0, 1)\n",
    "plt.ylim(0, 7)\n",
    "plt.grid(visible=False)\n",
    "plt.gca().set_facecolor('black')\n",
    "plt.gca().tick_params(colors='white')\n",
    "plt.gcf().set_facecolor('black')\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5edc4bc9-44ab-4516-a10a-6dad75c8821d",
   "metadata": {},
   "source": [
    "### $p_{model}$ vs. $p_{reality}$ "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3537f0e5-984f-457f-acef-b288a48ee7f6",
   "metadata": {},
   "source": [
    "- 考虑一个非常（现实中） rigged coin，$0.99$ vs. $0.01$\n",
    "    - 抛十次，均为正面；\n",
    "    - 对于 $p_{model}$，其 surprise 为\n",
    "        - $p(10\\text{ heads})=0.5^{10}$\n",
    "        - $h(10\\text{ heads})=\\log\\frac{1}{0.5^{10}}$\n",
    "    - 对于 $p_{reality}$，其 surprise\n",
    "        - $p(10\\text{ heads})=0.99^{10}$\n",
    "        - $p(10\\text{ heads})=\\log\\frac1{0.99^{10}}$\n",
    "- your surprise is caused by believing in the wrong model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "928d3ba0-5ce2-4ff8-84c3-d19b91c4686f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-03T14:29:09.947734Z",
     "iopub.status.busy": "2024-11-03T14:29:09.947448Z",
     "iopub.status.idle": "2024-11-03T14:29:09.958367Z",
     "shell.execute_reply": "2024-11-03T14:29:09.956374Z",
     "shell.execute_reply.started": "2024-11-03T14:29:09.947716Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6.931471805599453, 0.10050335853501455)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.log(0.5**-10), np.log(0.99**(-10))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76ddf8d0-6d20-43b1-92d0-cdac031fb078",
   "metadata": {},
   "source": [
    "### cross entropy => KL div"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "502914f5-1697-4687-a296-a44742a693ec",
   "metadata": {},
   "source": [
    "$$\n",
    "H(P,Q)=\\sum_sp(s)\\log\\frac1{q_s}\n",
    "$$\n",
    "\n",
    "- Average surprise you will get by observing a random variable governed by distribution $P$, while believing in its model $Q$.\n",
    "- $p_s$: how often the state s is observed\n",
    "- $\\log\\frac1{q_s}$: how surprised you will be to see it\n",
    "\n",
    "$$\n",
    "H(P,Q)\\geq H(P)\n",
    "$$\n",
    "\n",
    "- $P=Q$ 时，$H(P,Q)=H(P)$\n",
    "\n",
    "$$\n",
    "H(P,Q)\\neq H(Q,P)\n",
    "$$\n",
    "\n",
    "- 考虑之前的例子，实际是 0.99 vs. 0.01，认为是 0.5 vs. 00.5\n",
    "    - Actual $p_{heads}$ 0.99\n",
    "    - Believed $p_{heads}$ 0.5\n",
    "    - Actual $p_{tails}$ 0.01\n",
    "    - Believed $p_{tails}$ 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "119b5e70-9e88-48ab-933b-99b4a37f4cf3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-03T14:36:38.653202Z",
     "iopub.status.busy": "2024-11-03T14:36:38.652575Z",
     "iopub.status.idle": "2024-11-03T14:36:38.665871Z",
     "shell.execute_reply": "2024-11-03T14:36:38.663646Z",
     "shell.execute_reply.started": "2024-11-03T14:36:38.653157Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6931471805599453"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "0.99 * np.log(1/0.5) + 0.01 * np.log(1/0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "183253f7-2ab4-4ff8-aaff-332549756fda",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-03T14:36:54.485353Z",
     "iopub.status.busy": "2024-11-03T14:36:54.484731Z",
     "iopub.status.idle": "2024-11-03T14:36:54.498083Z",
     "shell.execute_reply": "2024-11-03T14:36:54.495953Z",
     "shell.execute_reply.started": "2024-11-03T14:36:54.485309Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.307610260920796"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "0.5 * np.log(1/0.99) + 0.5 * np.log(1/0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8340692c-b1ca-469a-9048-ff1487c9b3a0",
   "metadata": {},
   "source": [
    "#### kl div"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "320064fa-70c4-4f8b-98d5-459b543807ae",
   "metadata": {},
   "source": [
    "$$\n",
    "H(P,Q)-H(P)=\\sum_s p_s\\log\\frac{p_s}{q_s}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecbc26db-da58-4e80-a95c-e3e6433da730",
   "metadata": {},
   "source": [
    "Measures surprise caused by the model and true distributions being different, beyond the uncertainty of the true distribution itself.\n",
    "- 排除掉原始分布本身的不确定性；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "5095068b-fd9e-4ae5-8a45-d660eaf9e061",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-03T14:45:17.853694Z",
     "iopub.status.busy": "2024-11-03T14:45:17.853059Z",
     "iopub.status.idle": "2024-11-03T14:45:17.867198Z",
     "shell.execute_reply": "2024-11-03T14:45:17.865023Z",
     "shell.execute_reply.started": "2024-11-03T14:45:17.853650Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6371456462050978"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "0.99 * np.log(1/0.5) + 0.01 * np.log(1/0.5) - (0.99 * np.log(1/0.99) + 0.01 * np.log(1/0.01))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b8958d5-1b86-402e-84a3-a10782a1707d",
   "metadata": {},
   "source": [
    "### deep learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e5eb1b5-1cc9-4a69-85a8-0cc77b187a71",
   "metadata": {},
   "source": [
    "- target distribution (is unknown, $P$) vs. model's approximation ($Q$)\n",
    "- objective\n",
    "\n",
    "$$\n",
    "\\min_Q D_{KL}(P_{data},P_{model})=H(P,Q)-H(P)\n",
    "$$\n",
    "\n",
    "- 对 $Q$ 进行优化时，$H(P)$ is constant (entropy of training data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "12ad562f-b81a-410c-936e-1f93505ced13",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-03T14:52:50.043992Z",
     "iopub.status.busy": "2024-11-03T14:52:50.043370Z",
     "iopub.status.idle": "2024-11-03T14:52:50.056683Z",
     "shell.execute_reply": "2024-11-03T14:52:50.054348Z",
     "shell.execute_reply.started": "2024-11-03T14:52:50.043943Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"./imgs/cross-entropy-loss.png\" width=\"400\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='./imgs/cross-entropy-loss.png', width=400)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}