notebooks/making_a_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validation & Descriptive Analyses\n",
    "\n",
    "In this notebook, we run some sanity checks to make sure that the ``Dataset`` class behaves as expected and we also run some descriptive analyses whose results can be used later, e.g., in the accompnaying paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "from scipy.stats import pearsonr, norm\n",
    "from numpy import linalg\n",
    "from sklearn.metrics import r2_score\n",
    "\n",
    "from simulation.dataset import Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating a Simulated Dataset \n",
    "\n",
    "The first thing to do is to generate a simulated dataset via the ``Dataset`` class. We choose to create one with:\n",
    "- 10,000 hypothetical responses\n",
    "- possible score range of $[1, 6]$\n",
    "- simulated \"true\" (or gold-standard scores) sampled from $\\mathcal{N}(\\mu=3.844, \\sigma=0.74)$.\n",
    "- 50 human raters simulated in each of 4 categories with \"low\" ($\\rho=0.4$), \"moderate\" ($\\rho=0.55$), \"average\" ($\\rho=0.65$), and \"high\" ($\\rho=0.8$) inter-rater agreement respectively (total 200 human raters).\n",
    "- 5 scoring systems simulated in each of 5 categories with \"poor\" ($R^2=0$), \"low\" ($R^2=0.4$), \"medium\" ($R^2=0.65$), \"high\" ($R^2=0.8$), and \"perfect\" ($R^2=0.99$) performance respectively (total 25 systems). For all systems $R^2$ is computed against the \"true\" or gold-standard scores.\n",
    "\n",
    "These properties can be changed by editing the `dataset.json` file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the default dataset settings from `dataset.json`\n",
    "settings = json.load(open('dataset.json', 'r'))\n",
    "\n",
    "# instantiate a dataset instance from these settings\n",
    "dataset = Dataset.from_dict(settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we populate the dataset by calling the ``generate()`` method which will generate all of the simulated scores. Once this is done, we call the ``to_frames()`` method to get pandas data frames that represent the simulated information in an organized fashion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "generating true scores ...\n",
      "generating rater scores and metadata ...\n",
      "generating system scores and metadata ...\n"
     ]
    }
   ],
   "source": [
    "# generate the dataset scores\n",
    "dataset.generate()\n",
    "\n",
    "# now get the data frames representing this dataset\n",
    "df_scores, df_rater_metadata, df_system_metadata = dataset.to_frames()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distribution of true scores\n",
    "\n",
    "Let's first check that the distribution of the simulated true scores in the dataset looks as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEGCAYAAAB1iW6ZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXTc9Xnv8fczI432XbIka7Fk432TFwwECJCENU4MKQmElN6E9BCapcnNzXbbnKRJ2jRp70maNAuhhCRNcc0WwIANgYCBYGwsb7JlW8ab9s3ad2lmnvuHxq4Qsi3bI/1meV7n6GiWn2aekUcff+f7+y6iqhhjjAl/LqcLMMYYExwW6MYYEyEs0I0xJkJYoBtjTISwQDfGmAgR49QTZ2dna0lJiVNPb4wxYWnnzp0nVTVnovscC/SSkhLKy8udenpjjAlLIlJ9pvusy8UYYyKEBboxxkQIC3RjjIkQFujGGBMhLNCNMSZCWKAbY0yEsEA3xpgIYYFujDERwgLdGGMihGMzRY2JNuu317zrtrsuK3agEhOprIVujDERwgLdGGMihAW6McZECOtDN2aSrA/chDproRtjTISwFroxF0hVaeoapL6zn4bOQfqGvAz7/Ax7/Qx5/Xh9Slysi0SPmxkpcTR3D5KV5CHGbe0oMzUs0I05D35VDjR0c6Cxm2Otvfz9U/vP6+djXML8vBSWFaazMC9liqo00coC3ZhJ8PuVHSfaee1wK219wyR53MzOSaYkK5HMpDjSEmNJiHXjdgl3rinC43YR63Yx5PXRN+SjqWuQ32+rpqa9n8r6LiobuslK8lCak8TVcyfcTcyY82aBbsw5tPcN878f2cOrh1spSE/grjXFLJqZiktkwuNT42NPX070xJDoiSEnJY6yonTKitJZuyyfqqYeNu1r5O5fv8VHVhTwz3+xlLgYNzDxyVewE7Dm3CYV6CJyE/ATwA08qKo/GHf/tcDTwPHATX9Q1e8GsU5jptWpUG3oHOD326rpHfKyrmwma0oykTME+WS5RFiYn8olM5Jp6x3ipy8fobV3iF/dvYpEj7WxzIU757tHRNzAz4HrgTpgh4hsVNUD4w59XVXXTkGNxjiirXeI37xxnBi3i/veO4eCjISgPn6s28WXb5hPcVYSX3t8L3/54HZ+e8+aoD6HiS6TaQ6sAY6o6jEAEdkArAPGB7oxYedM3Rt9Q15+u/UEfoV7riwlJyVuymq4fVUhyXFuPrd+N197rIKr52Zf9KcAE50mM36qAKgdc70ucNt4V4jIXhHZLCKLJ3ogEblXRMpFpLy1tfUCyjVm6vlVeXh7NV0DI/zVFbOmNMxPuWlJPl+/aT7PVzbx5rG2KX8+E5km00KfqKmg467vAmapaq+I3AI8Bcx91w+pPgA8ALB69erxj2FMSHjzaBsn2vr56KpCZmUlnffPn6nVf65jkzwxLMhLYfO+JooyEinKTDzv5zbRbTIt9DqgaMz1QqBh7AGq2q2qvYHLm4BYEckOWpXGTJOO/mFePNDM/NwUyorSp/W5RYTbVxWSkhDD4zvr8PmtzWPOz2QCfQcwV0RKRcQD3AlsHHuAiORJoNNPRNYEHtc+N5qwoqps3DPaVvlw2UxH+rETPTF8aNlMWnuHeOu4/QmZ83POLhdV9YrI54EXGB22+JCqVorIfYH77wduB/5GRLzAAHCnqlrzwoSVA43dVDX38MGl+WQkehyrY0FeCnNyknjpYAvLi9JtKKOZtEm9UwLdKJvG3Xb/mMs/A34W3NKMmT5+VV462Ex2chxXzMlytBYR4Zal+fzs5SO8fKiFtctmOlqPCR+2SpAxQGVDN83dQ7xvwYwzzgCdTvlpCVxaksm2Y2109A07XY4JExboJur5VfnTwWZyUuJYVpjmdDmnXbdgBgBvHD3pcCUmXFigm6i3v76Llp7QaZ2fkpYQy/LCdMpPdDAw7HO6HBMGLNBNVFNVtlS1MiMljqUFodM6P+WqudkM+/xstxEvZhIs0E1UO3ayj6buQa66JDukWuen5KclMHdGMm8ebWPIa610c3YW6CaqbT3aRqLHzfJpnkR0Pq6em0PPkJen9zSc+2AT1WyAq4laNW39HGrs5pr5OcSG8LZwc3KSmJESx7//6W28vndO77A10s1YofsuNmaK/e7NE4jA5aXOjjs/FxFh1awMajsGaO4edLocE8Is0E1U6hvy8uiOWpYUpJGaEHvuH3DYiuIMXAK7qjucLsWEMAt0E5We3tNAz5CX98wO7db5KclxMSzIS2VXbact2mXOyALdRKX1b1WzIC8lrJaoXT0rg74hL1VN3U6XYkKUBbqJOvvquthf381dlxWH1c5Ac3NTSImPody6XcwZWKCbqLP+rWoSYt3cumKijbdCl9slrChK53BzD/1DXqfLMSHIAt1ElZ7BEZ7e08CHlueTGh/6J0PHW1aYjl9HFxMzZjwLdBNVnt7TQP+wj7sum+V0KRckPy2erCQP++q7nC7FhCALdBM1VJX122tYmJ/K8hBaVfF8iAhLC9M42tpLr3W7mHEs0E3UqKjr4kBj+J0MHW9pQRoKVDZYK928kwW6iRrrt9eQ6HFza1l47wCUlxpPTnIcFXUW6OadLNBNVOgeHGHj3gY+vHwmKWF4MnSsU90uJ0720WJLAZgxLNBNVHh6dz0DIz4+viYyFrM61e2yeX+T06WYEGKBbiKeqvLw9hoWz0wNqS3mLkZuajy5qXE8W2FL6pr/YYFuIt6e2k4ONfWE/cnQ8ZYWpLHjRAdNXdbtYkZZoJuIt357DUkeN+vKwmtm6LksLRjdlOO5fY0OV2JChW1wYSLS+u01AAwM+3hqTz1lRRlsjLAdf3JS4liYn8qzFQ18+qpSp8sxIcBa6Cai7antYMSnrCnJdLqUKbF2WT67azqp6+h3uhQTAizQTcRSVd460U5BegIFGQlOlzMl1i7LB2CTdbsYLNBNBKtt76e5eyhiW+cAs7KSWFqQxrMVFujGAt1EsB3VHXhiXBEzVPFM1i7Lp6Kui+q2PqdLMQ6zQDcRaXDER0VdJ8sK0oiLdTtdzpT6YKDbxVrpZlKBLiI3iUiViBwRkW+c5bhLRcQnIrcHr0Rjzt++ui5GfMrqCO5uOaUwI5EVxekW6ObcgS4ibuDnwM3AIuDjIrLoDMf9EHgh2EUac77Kq9uZkRJHUYSeDB1v7bKZHGzs5mhrr9OlGAdNpoW+BjiiqsdUdRjYAKyb4LgvAE8ALUGsz5jzdqipm9qOAS4tyYyomaFn88Gl+YjAc9ZKj2qTCfQCoHbM9brAbaeJSAFwG3B/8Eoz5sI8sqMWtwhlRelOlzJt8tLiuXRWpq3tEuUmM1N0oiaOjrv+b8DXVdV3thaRiNwL3AtQXBwZq94ZZ52aEXqK1+dnw1u1LJqZSlJcdE2EXrs8n289Xcnh5h7m5aY4XY5xwGRa6HVA0ZjrhcD4ZsBqYIOInABuB34hIreOfyBVfUBVV6vq6pycnAss2ZgzO9DYzcCIj9UlGU6XMu1uXpKPS+DZvdZKj1aTCfQdwFwRKRURD3AnsHHsAapaqqolqloCPA58VlWfCnq1xpxD+YkO0hNjmZOT7HQp0y4nJY7LZ2fxbEUjquM/RJtocM5AV1Uv8HlGR68cBB5V1UoRuU9E7pvqAo2ZrPa+YY609rJqVgauKDkZOt7aZTM5drKPA43dTpdiHDCpceiquklV56nqHFX9p8Bt96vqu06CquonVfXxYBdqzLnsrO5AgFXF0dfdcspNS/Jwu8TGpEcpmylqIoJflV01HczNTSY90eN0OY7JTPJw5SXZPFvRYN0uUSi6hgGYiHW0tZeugRFuWZrvdCnTavwoHxhd2+Vrj1dQUdfF8igaummshW4ixN7aTuJjXSzIs+F6Ny7Ow+N2sdFGu0QdC3QT9kZ8fiobulk8M41Yt72l0xJiuW5BDhv3NuD1+Z0ux0wje/ebsHewsZshrz+qZoaey20rCmjtGWLr0TanSzHTyALdhL29tZ2kxsdQmp3kdCkhYf32Glq6h4iPdfHjFw9P2M9uIpMFuglr/cNeDjf3sqwwPWrHnk8kxu1iaUE6lQ3dDHut2yVaWKCbsLa/vhufqnW3TKCsKJ1hn58DjV1Ol2KmiQW6CWsV9Z1kJ8eRnxbvdCkhZ1ZWIumJseyp7XS6FDNNLNBN2GrrHeJ4ax9LC1KjZt3z8+ESoawwnbebe2ntGXK6HDMNLNBN2HqhshkFlhRE9ibQF6OsKB0FG5MeJSzQTdjavL+RrCQPeanW3XImM1LjKUhP4Knd9U6XYqaBBboJSx19w2w92saSgjTrbjmHsqJ09tV3caSlx+lSzBSzQDdh6Y8HmvD51bpbJmFZYRougSetlR7xLNBNWNq0r4mizARm2uiWc0qJj+XquTk8tbsBv99WYIxkFugm7PQOedl69CQ3L8m37pZJum1FAfWdA5RXdzhdiplCFugm7Pz57ZOM+JT3LZjhdClh44bFuSR63NbtEuEs0E3Y2VLVQkpcDKtmRe/OROcr0RPDjYvzeK6igcERn9PlmCligW7CiqrySlULV8/LtqVyz9NtKwroHvSyparF6VLMFLEdi0xYOdDYTXP3ENfOt+6W87F+ew0+v5ISF8NP/3SE9r4RAO66rNjhykwwWaCbsLF+e83p1mVH37AtC3ue3C5hWWEa24610z/sJdFjf/6Rxj6zmrBS1dRDQXoCKfGxTpcSlsqKM/Cpsq/eVmCMRBboJmz0D3mpae9nXq7tG3qhZqbFMyMlzlZgjFAW6CZsvN3Si4JtBH0RRISyonSq2/pp7xt2uhwTZBboJmxUNfeQ6HFTkJHgdClhbXlgMxBrpUceC3QTFnx+5XBzD/NyU2yruYuUkeihJCuJPbWdqNpSAJHEAt2Ehb11nfQP+5hv3S1BsaI4nZO9Q3ZyNMJYoJuwsOVQCwLMnZHsdCkRYcnMNNwusaUAIowFugkLr1S1UpyZaGOngyTB42ZBXgrP7G3A6/M7XY4JEvvrMCGvpXuQffVd3LAo1+lSIsrywnTWv1XD9zcd4pIxn3xs9mj4mlQLXURuEpEqETkiIt+Y4P51IlIhIntEpFxErgp+qSZabTncCmD950E2Py8Fj9vFfutHjxjnDHQRcQM/B24GFgEfF5FF4w77E7BcVcuAe4AHg12oiV5bqlrITY2zvUODLNbtYn5eCpUNXfhttEtEmEwLfQ1wRFWPqeowsAFYN/YAVe3V/xn/lATYu8MExYjPz+uHT3Ld/Bm2mcUUWFKQRt+wjxMn+5wuxQTBZAK9AKgdc70ucNs7iMhtInIIeI7RVvq7iMi9gS6Z8tbW1gup10SZ8hMd9Ax5uc42s5gS83NTiHUL+xus2yUSTCbQJ2oWvasFrqpPquoC4FbgexM9kKo+oKqrVXV1Tk7O+VVqotKWqhZi3cKVl2Q7XUpE8sS4mJebQmV9t3W7RIDJjHKpA4rGXC8EGs50sKq+JiJzRCRbVU9ebIEmOp1aGvfJ3fUUZyaycc8Z33LmIi0tSKOyoZvqtn5Ks5OcLsdchMm00HcAc0WkVEQ8wJ3AxrEHiMglEujgFJGVgAdoC3axJrp09A/T0jPE/LxUp0uJaPNzU4hxWbdLJDhnoKuqF/g88AJwEHhUVStF5D4RuS9w2F8A+0VkD6MjYu5QWyTCXKSqph5gNHDM1ImLdQe6XWy0S7ib1MQiVd0EbBp32/1jLv8Q+GFwSzPR7nBzD5lJHrKTPU6XEvGWFKRxoLGb2vZ+p0sxF8Gm/puQNOLzc7S1l3m5KTZccRosyEvB7RKbZBTmLNBNSDp+so8Rn9pmFtMkPtbN3BnJ7G/oxu+3bpdwZYFuQlJVUw+xbrFRF9NoaUEaXQMj7K2zjS/ClQW6CTmqSlVzD7Ozk4l121t0uizIS8Utwub9TU6XYi6Q/bWYkHPsZB/tfcO2GNc0S/C4uWRGMs9VNNpORmHKAt2EnFcOtQA2XNEJSwrSqO8csJ2MwpQFugk5W6pamZESR0aSDVecbgvzRycZPbev0elSzAWwQDchpXfIy/bjbdbd4pBETwxXXpLN5n1N1u0ShizQTUh548hJRnxq3S0OumVpHjXt/VQ2dDtdijlPFugmpGypaiE5LoZZWTZc0Sk3LMrDbd0uYckC3YQMVeWVQ61cPTcbt8tmhzolI8nDe+ZksXmfjXYJNxboJmQcbOyhqXvQNrMIAbcszedEWz8HGq3bJZxYoJuQ8UrV6HDFa+fZ5idOu2FRLm6XsHmfTTIKJxboJmRsqWphSUEqM2wzaMdlJcdx+exMNlm3S1ixQDchobN/mJ3VHbxvvnW3hIqbl+Rz7GQfVc09TpdiJskC3YSE194+iV/hWus/Dxk3Ls7DJbCpwka7hAsLdBMSthxqITPJw/LCdKdLMQE5KXGsKc3kOet2CRsW6MZx/7WtmucrmyjOTOSRHbWnN4g2zvvg0nyOtvbxdkuv06WYSbBAN46r7xigf9hns0ND0I1L8hCB56zbJSxMak9RY6ZSVXMPAsydkex0KQbe9QmpJCuJZ/Y28KUPzLXtAEOctdCN46qaeijKTCQxztoXoaisMJ1jJ/tsSd0wYIFuHNXSM0h954DtHRrClhSk4XG7eGp3g9OlmHOwQDeOerWqFYB51n8eshI8bq5bkMMzFQ14fX6nyzFnYYFuHLWlqpXU+Bjy02x2aCi7tayA1p4hth5tc7oUcxYW6MYxIz4/rx1uZV5uip1sC3HXLZhBSnwMT+2pd7oUcxYW6MYxO6s76Bny2u5EYSA+1s0tS/J5YX8TA8M+p8sxZ2CBbhzzp4PNeNwuLsmx4YrhYN2KmfQN+3jpYLPTpZgzsEA3jnnpYAuXz8kiLtbtdClmEi4vzSIvNZ6ndlu3S6iyQDeOONray/GTfVy/0BbjChcul7CubCavHm6lvW/Y6XLMBCzQjSNeOjD6sf19C3MdrsRMxvrtNazfXoMnxoXXr/zDxkpbcycETSrQReQmEakSkSMi8o0J7v+EiFQEvraKyPLgl2oiyUsHm1mUn0pBeoLTpZjzkJcaz4yUOPbWdjpdipnAOQNdRNzAz4GbgUXAx0Vk0bjDjgPXqOoy4HvAA8Eu1ESO9r7RzSw+YN0tYUdEWFGUTnV7v3W7hKDJtNDXAEdU9ZiqDgMbgHVjD1DVraraEbi6DSgMbpkmkrxyqAW/wvutuyUsLS8aXbN+T23HOY40020ygV4A1I65Xhe47Uw+DWye6A4RuVdEykWkvLW1dfJVmojy0sFmZqTEsbQgzelSzAVIT/QwOzuJXTWdtvFFiJlMoE80hW/Cf0URuY7RQP/6RPer6gOqulpVV+fk2M7u0ah/2MsrVS3csDgXl8tmh4arlbMyaO8bprzaWumhZDKBXgcUjbleCLxr2TURWQY8CKxTVVvwwUxoS1UrgyN+blma73Qp5iIsmZmGJ8bF4+V1TpdixphMoO8A5opIqYh4gDuBjWMPEJFi4A/A3ap6OPhlmkixaV8jWUke1pRkOl2KuQieGBdLZ6bx3L5G+oe9TpdjAs4Z6KrqBT4PvAAcBB5V1UoRuU9E7gsc9i0gC/iFiOwRkfIpq9iErcERHy8fauGGxXnEuG0KRLhbOSuD3iEvL1Q2OV2KCZjUFjGqugnYNO62+8dc/mvgr4Nbmok0//jsAfqHfSTEum1SSgSYlZVIcWYiT+ys57YVNrAtFNieX2ba7G/oJtHjpjQ7yelSTBC4RJibm8zLB1v4xStHSE/0AHDXZcUOVxa97HOvmRZDXh8HG7tZlJ+K20a3RIyVRRkosNtmjoYEC3QzLV4+2MKQ18/SQht7HkkykjyUZiexq7rDxqSHAAt0My2e2FVPanwMc2zt84izqjiDtr5hatr7nS4l6lmgmynX1jvElqoWyorScdlWcxFncUEqHreLnTbJyHEW6GbKbdzbgNevlBVnOF2KmQJxMW6WFKSxr76LYa/f6XKimgW6mXJ/2FXPkoJU8lLjnS7FTJGVs9IZ8vo50NjldClRzQLdTKnDzT3sq+/iIzZOOaKVZCWRkRjLrmob7eIkC3QzpR4rryXGJXy4bKbTpZgp5BJhZXEGR1t7qe8ccLqcqGWBbqbM4IiPx3bWcePiPLKT45wux0yxFcWjY9Kf3GULdjnFAt1MmY17G+jsH+HuK2Y5XYqZBpmBMemP76yzMekOsUA3U0JV+f2b1czLTeayUltZMVqsKs7gRFu/DWF0iAW6mRJ7ajvZV9/F3VeUIDb2PGosLkgl0ePmMVsn3REW6GZKfPeZA8TFuPB6/azfXmOrK0aJuBg3H1yaz7MVDfQN2Trp080C3QRdY9cAFXVdrCzOIC7W7XQ5ZprduaaIvmEfz1U0Ol1K1LFAN0H34OvHUZSr5mY7XYpxwMriDObkJLFhh30qm24W6Cao2vuGWb+9huWF6WQE1sc20UVEuPPSYnbVdHKkpcfpcqKKBboJqt++cZxBr49r5uU4XYpx0G0rC4hxCY/sqHW6lKhigW6CpnfIy2+3nuCGRbnMsHVbolp2chzXL8rliV31DHl9TpcTNSzQTdA89OfjdA96+ey1lzhdinHQqVFNeanxtPcN862nKp0uKWpYoJugaO8b5oHXjnHj4lyWF6U7XY4JAXNmJJOV5GHb8TanS4kaFugmKH7xyhH6h7185Yb5TpdiQoRLhMtKM6lu6+dgY7fT5UQFC3Rz0X7xyhF+u/UEK4oy2HGiwyYRmdNWzsogxiX8flu106VEBQt0c9FeOtiCAu9fOMPpUkyISfTEsLwonad219M9OOJ0ORHPAt1clIq6TnbVdPCe2Vmk27hzM4HLS7PoH/bZ+i7TwALdXDBV5TvPHCApLobrFljr3EysICOBNSWZPPTn44z4bM/RqWSBbi7Yxr0N7Kzu4MZFucTbmi3mLO67djb1nQM8s7fB6VIimgW6uSB9Q15+sPkQSwvSWDkrw+lyTIi7bv4M5uem8KtXj9nmF1PIAt1ckJ/+6W0auwb5hw8vwmXrnZtzEBE+c81sqpp7eKWqxelyIpYFujlvVU09/PrPx7ljdRGrZtluRGZyPrR8JgXpCfzs5SPWSp8ikwp0EblJRKpE5IiIfGOC+xeIyJsiMiQiXwl+mSZU+P3KN5/aR0p8DN+4eYHT5ZgwEut28TfXzmFXTScvHbRW+lSIOdcBIuIGfg5cD9QBO0Rko6oeGHNYO/C3wK1TUqUJCeu317Czup0dJzr4yIoCNu9vcrokE2buuLSIh/58nB8+f4jr5ucQ47ZOgmCazG9zDXBEVY+p6jCwAVg39gBVbVHVHYDNHIhg/UNeNu9vYlZmop0INRck1u3iqzfO50hLL0/ssnHpwTaZQC8Axi5qXBe47byJyL0iUi4i5a2trRfyEMZBz1c2MTjiY11ZgZ0INRfspiV5rChO50cvHmZg2JbWDabJBPpEf7kXdEZDVR9Q1dWqujonxzZACCflJ9opr+7gyjnZ5KXZWufmwokIf3fLQpq7h/i3lw47XU5EmUyg1wFFY64XAjY7IIoMe/1886n9pCXE8j5br8UEwaUlmdx5aRH/8fox9tV1OV1OxDjnSVFgBzBXREqBeuBO4K4prcqElF+9epRDTT3cffks4mJsRqg5fxOtwDl3RgrZyXF87YkKNn7+SmLtBOlFO+dvUFW9wOeBF4CDwKOqWiki94nIfQAikicidcCXgW+KSJ2IpE5l4WZ6vN3cw7+/fIS1y/JZmG//pCZ4EjxuvrtuCQcbu/nllqNOlxMRJtNCR1U3AZvG3Xb/mMtNjHbFmAji8ytff6KCxDg3//DhxfyxstnpkkyEae8bZllhGj9+8TDdgyPMzk7mrsuKnS4rbNlnHDOh9dtr+OKG3eyq6eT6hbkW5mbK3FZWQFayh0d21NJja6ZfFAt0M6GOvmFeqGxiXm4yZbZHqJlCcbFuPr6mmIFhH4+W1+K1JXYvmAW6eRdV5ak99YgIt5YVIDbm3Eyx/LQE1pUVcLS1j+88c8DWerlAFujmXR7fWcfbLb3ctDjPdiEy02bVrAyunpvN77dV85s3TjhdTlia1ElREz3qOvr57jMHKMlKZE2praRopteNi/NI8sTwvecOUJCRwI2L85wuKaxYC92c5vMrX350LwrcvqrIpvebaecS4cd3lLG8MJ0v/Pdu3jza5nRJYcUC3Zz2wGvHeOt4O9/58GIyk6yrxTjjyd31rF2aT3pCLJ/8zVv86/NVTpcUNizQDQC7ajr40YtVfHBpPh9ZeUFrrxkTNIlxMXzqylISPG5+s/U4R1t7nS4pLFigG9p6h/jcw7vIS4vn+7cttVEtJiSkJcRyz5WliAh3P7idhs4Bp0sKeRboUc7nV770yB7a+ob55SdWkZYY63RJxpyWnRzHp95TQlvfMOt+9gb/8dox1m+vmXBtGGOBHvX+5YVDvP72SdYuzaeirsv+WEzImZmewF9dUUJH/zC/3XqCoRFbQ/1MLNCj2MPbq/nVq8e4rDST1SU2RNGErtLsJD6+ppjGrgF+v72aEZtNOiEbhx5lTrW+q5p6+P22E8zPTWHtspkOV2XMuS3MT+UvVhby2M46NrxVw12XFduSu+PYbyMKHT/Zx/q3qslNjefONUW4XXYS1ISHFcUZfGj5TA429fDFDbtt3ZdxrIUeZarb+vjd1hOkJ3r45HtKbMMKE3aumJ2Fz69s2tdIjGsvP76jzBolARboUeTNo238ZusJUhNi+PRVpaTE24gWE56uuiSbJQWp/MvzVcS6Xfzr7ctwWahboEeLZ/Y28H8e3Ut6YiyfurKUVAtzE+Y+e+0ljHiVH790GE+M8E+3Lo36ULdAj3Cqyi9fPcq/PF/FmpJMblycR4LHullMZPjb91/CsM/Hz185is+vfP+2pcRE8YlSC/QI1jUwwlce28uLB5pZuyyf//fR5fxhV73TZRkTNCLCV26Yj9vl4qd/epvuAS8/+XhZ1J4bskCPUNuPtfHVxyto6BzgW2sX8akrS2xKv4lIIsKXr59HWkIs33v2AJ98aAf3/2V0znqO3s8mEap3yMu3n97PHQ9sA+CRz1zOPVeVWpibiPfpq0r58R3LKa9u57ZfvMHxk31OlyMllQYAAApdSURBVDTtrIUeAdZvr8Gvyu6aDv5Y2UzvkJdPXVnCV2+cT6LH/olNZDrTEhWffE8pD2+v5pafvM7HVhcyPy+Vuy4rnubqnGEt9DDn8ysVdZ387OUjPLGrnowkD/ddM4dvf2ixhbmJSqXZSXz22ktIS4jld29W80JlU9RMQLJAD1MjPj+Pltdy/Y9eZcOOWnx+5Y5Li/jMe2dTlJnodHnGOCozycPfXDuH1bMyePVwKx/91ZscaYn8NdXFqd21V69ereXl5Y48dzj7j9eOsbumg61H2+gcGGFmWjzXzJ/B4pmptmWcMRPYW9fJC5VN9A/7+PL187jnylI8MeHblhWRnaq6eqL77DN5GPD7lTePtfHfb9WweV8TPlVmZSWyrmwm83JT7ISnMWexvDCd/3PDPP7+yf38YPMhHtlRy9/fspD3L5wRcX87FughSlXZV9/FC5VNPLO3kZr2ftISYrls9uhSt3mp8U6XaEzYmJESzwN3r2JLVSv/+NwB/vo/y1lemMZnrpnDjYvzImYtGOtyCRGDIz6OtPSyr76Lbcfa2Hq0jdaeIdwu4fLZmXxsdRE3Ls6ziUHGXCSfXymvbmdvbScn2vrJT4vnQ8tn8qFlM1k8M5UNO2rf9TOhNErGulxCTGvPEAcbuznQ2M3BwNfR1j58/tH/XHNS4rhidhbvnZfD+xfMICPJ43DFxkQOt0u4rDSLH32sjBcPNPH4zjoe+vNxHnjtGGkJseSnxVOSlURJdhIF6Qlh1Xq3QJ9CgyM+qtv6OdQ0Gt4HGrrZXdNJ75D39DGn3kBXz80mPy2B/LR4spI8iAjDXj+b9zc5+AqMiVxul3DTknxuWpJPZ/8wLx9q4a3j7bx0sIVDTT0AxLiEmekJvN3Sw/LCdJYXpVOSlRiyfe+T6nIRkZuAnwBu4EFV/cG4+yVw/y1AP/BJVd11tscMxy4XVWVwxE/P4Ajdg156BkfoGfQGvkboHhyhrmOA4yf7OH6yj/qOAU79dt0uYUZK3OnQzkuLJz8t3saKGxOCegZHqG7rp7qtj7rOAVq6hxgI7GWaGh/D8qJ0lhWmnQ753Gk8p3VRXS4i4gZ+DlwP1AE7RGSjqh4Yc9jNwNzA12XALwPfp42q4lfwq+JXRU9fBp9PGfT6GBj2MTAy+jUYuNw/7KN36J3h3H368jsDu2fQi9d/9v8AU+JiKM1JYmVxBvNyU8hOjiMvNZ6clLiw+uhmTDRLiY9lSUEaSwrSAPjY6kLebumloq6TPbVdVNR1cv+rx053k+alxrOsMI0F+ankpMSRk+whOzmO7OQ4UuJj8MS4iHW78LhdU7rE72Sah2uAI6p6DEBENgDrgLGBvg74Tx1t7m8TkXQRyVfVxmAX/Pz+Rr70yB78+s4QD8a5XQHiYl3Ex7qJj3GPXo5xk5E42i0yeruLuFj36OXA/acvx7qJi3GF7McxY8yFiXG7WJifysL8VO64dPS2wREflQ3d7K3tpKKuk711XfzxQPO5H8slfOaa2Xz1xgXBr3MSxxQAY0/71vHu1vdExxQA7wh0EbkXuDdwtVdEqs6r2uDJBk469NxTyV5X+IjE1wQR+ro+EeTX9bV/hq9d+I/POtMdkwn0iZqb49vDkzkGVX0AeGASzzmlRKT8TH1Q4cxeV/iIxNcE9rqcNpn5r3VA0ZjrhUDDBRxjjDFmCk0m0HcAc0WkVEQ8wJ3AxnHHbAT+SkZdDnRNRf+5McaYMztnl4uqekXk88ALjA5bfEhVK0XkvsD99wObGB2yeITRYYufmrqSg8Lxbp8pYq8rfETiawJ7XY5ybOq/McaY4ArfNSSNMca8gwW6McZEiKgKdBF5SERaRGS/07UEk4gUicgrInJQRCpF5ItO13SxRCReRN4Skb2B1/Qdp2sKJhFxi8huEXnW6VqCRUROiMg+EdkjIuG1rscZBCZJPi4ihwJ/X1c4XdPZRFUfuoi8F+hldFbrEqfrCRYRyQfyVXWXiKQAO4Fbxy3PEFYC6wMlqWqviMQCfwa+qKrbHC4tKETky8BqIFVV1zpdTzCIyAlgtapGzMQiEfkd8LqqPhgY5Zeoqp1O13UmUdVCV9XXgHan6wg2VW08tRiaqvYABxmdqRu2dNSpTSBjA18R0foQkULgg8CDTtdizkxEUoH3Ar8GUNXhUA5ziLJAjwYiUgKsALY7W8nFC3RL7AFagBdVNexfU8C/MTrzO9K2olfgjyKyM7DMR7ibDbQCvwl0jz0oIklOF3U2FugRRESSgSeAL6lqt9P1XCxV9alqGaMzj9eISNh3k4nIWqBFVXc6XcsUuFJVVzK6+urnAl2c4SwGWAn8UlVXAH3AN5wt6ews0CNEoJ/5CeBhVf2D0/UEU+Bj7hbgJodLCYYrgQ8H+ps3AO8Tkf9ytqTgUNWGwPcW4ElGV2oNZ3VA3ZhPho8zGvAhywI9AgROIP4aOKiqP3K6nmAQkRwRSQ9cTgA+ABxytqqLp6r/V1ULVbWE0WU0XlbVv3S4rIsmIkmBE/IEuiVuAMJ6NJmqNgG1IjI/cNP7eeey4SEnqrbLEZH/Bq4FskWkDvi2qv7a2aqC4krgbmBfoM8Z4O9UdZODNV2sfOB3gQ1WXMCjqhoxQ/wiUC7wZGAvgBhgvao+72xJQfEF4OHACJdjhPiyJlE1bNEYYyKZdbkYY0yEsEA3xpgIYYFujDERwgLdGGMihAW6McZECAt0E1YCq9991uk6jAlFFugm3KQDEwZ6YMx6SBGRqJrrYZxlgW7CzQ+AOYE1t/9VRK4NrAW/ntGJVSVj17sXka+IyD8ELs8RkecDi0e9LiILxj+4iFwTeOw9gQWZTs1+/Fpgre+9IvKDwG1lIrJNRCpE5EkRyQjcvkVEvi8irwJfFJFVIvJq4HlfCCx3bEzQWevBhJtvAEsCi3YhItcyumbIElU9Hlht8kweAO5T1bdF5DLgF8D7xh3zFeBzqvpGYLGzQRG5GbgVuExV+0UkM3DsfwJfUNVXReS7wLeBLwXuS1fVawJr7LwKrFPVVhG5A/gn4J6L+SUYMxELdBMJ3lLV42c7IBDO7wEeC0xPB4ib4NA3gB+JyMPAH1S1TkQ+APxGVfsBVLVdRNIYDe1XAz/3O+CxMY/zSOD7fGAJ8GLged1A4/m+QGMmwwLdRIK+MZe9vLMrMT7w3QV0nmrZn4mq/kBEngNuAbYFwlw4/801TtUkQKWqhvTWZSYyWB+6CTc9QMpZ7m8GZohIlojEAWsBAuvDHxeRj8LoCpUisnz8D4vIHFXdp6o/BMqBBcAfgXtEJDFwTKaqdgEdInJ14EfvZrRrZbwqIOfUXpQiEisii8//ZRtzbtZCN2FFVdtE5I3Aic/NwHPj7h8J9GdvB47zziV3PwH8UkS+yeiWdhuAveOe4ksich3gY3Sp1M2qOiQiZUC5iAwDm4C/A/4XcH8g6CdciU9Vh0XkduCngW6aGEZ3LKq8qF+EMROw1RaNMSZCWJeLMcZECAt0Y4yJEBboxhgTISzQjTEmQligG2NMhLBAN8aYCGGBbowxEeL/A0EEFFmFGW1yAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "p = sns.distplot(df_scores['true'])\n",
    "p.set_xlabel('true score')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the simulated true scores exhibit a normal distribution with the expected mean and standard deviation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Relationship between error & inter-rater correlations\n",
    "\n",
    "Another thing that would be interesting to look at is the effect of the error term (as sampled from $\\mathcal{N}(0, \\sigma$) on simulater inter-rater correlations as we vary $\\sigma$. This is handled by the private method `_find_best_error_sd_for_rho()` of the dataset instance which basically varies $\\sigma$ along a line to find the one that simulates the rater scores with average inter-rater correlation closest to the desired value of the given $\\rho$. However, this method also returns the values of the inter-rater correlations and $\\sigma$ computed as part of the search. So, we can simply plot these two values to see the relationship between them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de5hddX3v8fdnJheG3EsGxExiIo0I9oCSERSs4gUFFSmWKiKX0h7TFJFqn3qwF6ut9ilUjqcoYBJzEKlVrEeiUaOIWMAKSIJyDQ1NQWCIQLgYYhIIk/meP9aasLKzZ89vJrP27Mvn9Tz7yV5r/fba3ySz57t/d0UEZmbWvjrGOwAzMxtfTgRmZm3OicDMrM05EZiZtTknAjOzNjdhvAMYqdmzZ8f8+fPHOwwzs6Zy2223PRER3dWuNV0imD9/PmvXrh3vMMzMmoqkB4e65qYhM7M250RgZtbmnAjMzNpcaYlA0uWSHpd09xDXJelzkjZIulPSEWXFYmZmQyuzRnAFcHyN6ycAC/PHYuALJcZiZmZDKC0RRMSNwFM1ipwEXBmZW4CZkg4sI5aBgWDTlud45OltbNryHAMDXmjPzGzQeA4fnQM8XDjuy8/9qrKgpMVktQbmzZs3ojcZGAjWP7aFD1y5lr6nt9Mzq4svntnLwQdMo6NDo4/ezKxFjGdncbXfwlW/qkfE8ojojYje7u6q8yGG9OTWHbuSAEDf09v5wJVreXLrjhEHbGbWisazRtAHzC0c9wAbx/pNdvTv3JUEXjV3JkuOPYiZXRPZ0b+TgYFwrcDM2t54JoJVwLmSrgKOAjZHxB7NQntr0oROemZ10T11Mn/xtoM5/5t3uonIzKygtEQg6WvAscBsSX3AJ4CJABGxFFgNvB3YAGwDzi4jjv2mTOKLZ/by6OZndyUBgO6pk3l087NMmdxJ18QJ7DdlkhOCmbWl0hJBRLxvmOsBfLCs9x/U0SEOPmAaUyZ37tZE5NqBmVmmLWYWd3SIrokT6JnVBcCSYw/alQReNXcmH3/noWx9rp9Hn3nWQ0vNrO20RSKAF5qIemZ1MbNr4q4k8BdvO5hPfXcdpyy9mfcsu5n1j21xMjCzttJ0y1CP1mAT0cpzjmFH/056ZnXtUTNYcuxBu2oGL5q+j5uJzKwttE0igCwZdE+bzMBA8MUze9n6XP9uNQP3GZhZO2qbpqGiwdrBi2d2Va0ZuM/AzNpJW9UIijo6xIum7+OagZm1vbasEQxyzcDMrI1rBINcMzCzdtfWNYJBtWoG4IXqzKy1tX2NYFC1mgF4oToza31OBAWDNYNHn3nWC9WZWdtwIqhQrBkUF6rzhDMza1VOBFVULlTnzmMza2XuLB5CcaE6Dys1s1bmGkENgwvVeVipmbWyUmsEko6XtF7SBkkfq3J9lqSVku6UdKuk3ykznpHyhDMzawdl7lDWCVwKHEe2P/EaSasiYl2h2F8Bt0fEyZJenpd/c1kxjYYnnJlZqyuzRnAksCEi7o+IHcBVwEkVZQ4FrgOIiP8E5ks6oMSYRsU1AzNrZWX2EcwBHi4c95FtUl90B/Bu4D8kHQm8BOgBHisWkrQYWAwwb968suKtyTUDM2tVZdYIqv0mrPy6fAEwS9LtwIeAXwD9e7woYnlE9EZEb3d399hHmsg1AzNrRWXWCPqAuYXjHmBjsUBEPAOcDSBJwAP5o2G5ZmBmrabMGsEaYKGkBZImAacCq4oFJM3MrwH8T+DGPDk0NNcMzKyVlFYjiIh+SecC1wCdwOURcY+kJfn1pcAhwJWSdgLrgD8uK56x5pqBmbWKUieURcRqYHXFuaWF5zcDC8uMoUyVi9RV1gy8NpGZNQPPLN5LrhmYWbPzWkNjwH0GZtbMXCMYI64ZmFmzco1gDLlmYGbNaNgagaR3AxcC+5NNEhMQETG95NiakmsGZtZsUpqG/gk4MSLuLTuYVuHRRGbWTFISwWNOAiPnmoGZNYuURLBW0teBbwHPDZ6MiKtLi6pFuGZgZs0gJRFMB7YBby2cC8CJIMFwNYPuqZM5780L2dE/wL6TO5k9ZbITgpnV1bCJICLOrkcgrWyomkH31MluKjKzcTfs8FFJPfl2ko9LekzSNyX11CO4VlKsGew3ZRJ9T2/frakIoHvqZB7d/Cx9v97Gpi3PeYipmdVFStPQl4CvAn+QH5+enzuurKBaVWXNYGbXxF1JwB3JZjZeUiaUdUfElyKiP39cAYzf7jBNrlgz2LZjJz2zugA8+czMxk1KInhC0umSOvPH6cCTZQfWygZrBofPncGy0xftVjsYrBl86rvrOGXpzbxn2c2sf2yLk4GZlSalaeiPgEuA/0M2Wuim/JzthY4O8VtTJjOzaxIrzzmGHf07PcTUzMZFyqihh4B31SGWttTRIbqnTWZgIDzE1MzGxZCJQNL/ioh/kvR59tx0nog4b7ibSzoeuJhsh7IVEXFBxfUZwFeAeXksF0XEl0b2V2gNHmJqZuOlVo1gcFmJtaO5saRO4FKy0UV9wBpJqyJiXaHYB4F1EXGipG5gvaR/jYgdo3nPZldt8tnH33mom4rMrFRDJoKI+E7+dFtEfKN4TdIfVHlJpSOBDRFxf/6aq4CTyPYm3vU2wDRJAqYCTwH96eG3nqGGmHp4qZmVJWXU0F8mnqs0B3i4cNyXnyu6hGwD+43AXcCfRcRA5Y0kLZa0VtLaTZs2Jbx1c6s2xNTDS82sLLX6CE4A3g7MkfS5wqXppH1rr/Y1tfI31tuA24E3AQcB10r6SUQ8s9uLIpYDywF6e3vb4rfeYM3ggOmTWXb6IrY/v9M1AzMrRa0+go1k/QPvAm4rnN8CfCTh3n3A3MJxT37PorOBCyIigA2SHgBeDtyacP+WVxxi6hVMzawstfoI7gDukPTViHh+FPdeAyyUtAB4BDgVOK2izEPAm4GfSDoAOBi4fxTv1dK8gqmZlSllQtl8Sf8IHArsM3gyIl5a60UR0S/pXOAasuGjl0fEPZKW5NeXAp8CrpB0F1lT0vkR8cTo/iqtzcNLzawsqYvOfYJsZvEbyZpzkn7DRMRqYHXFuaWF5xvZfZ8Dq8HDS82sDCmJoCsirpOkiHgQ+KSkn5AlB6szDy81s7GWkgieldQB/Ffe1PMIsH+5YVktxZrBo5vdiWxmeyclEXwY2Bc4j6xN/03AWWUGZcNLGV7qTmQzS5Gy6Nya/OlvyPoHrEHUGl7qTmQzS1VrQtl3qLLY3KCI8IqkDcKdyGa2N2rVCC6qWxS219yJbGajVWtC2Q2DzyV1AfMiYn1dorJRcSeymY3GsIvOSTqRbD2gH+THr5S0quzAbHQqt8Hcb8qkPbbA/Ifv3cv6R7fw0FPbeHyLF60za3cpo4Y+Sbak9PUAEXG7pPmlRWR7zZ3IZjYSKctQ90fE5tIjsTFXbCoarBl4OWszq5RSI7hb0mlAp6SFZPMJbio3LBsr7kQ2s+Gk1Ag+BLwCeA74KrCZbJKZNQlvdGNmtdSsEeT7Dq+KiLcAf12fkKwMnolsZkOpmQgiYqekbZJmuJ+g+bkT2cyqSVp0DrhL0rXA1sGTEXFeaVFZqYabiQzQPXUyj25+limTO+maOIH9pkxyQjBrUSmJ4Hv5Y8QkHQ9cTLYxzYqIuKDi+keB9xdiOQTojoinRvN+lm6oTmTAHclmbSalj+CMvI9gRPLXXgocR7Z/8RpJqyJi3WCZiPgM8Jm8/InAR5wE6qfaTORqQ0w9G9mstdUcNRQRO4FtkmaM4t5HAhsi4v6I2AFcBZxUo/z7gK+N4n1sL1TORK42xPRT313HKUtv5j3Lbmb9Y1s8qsisxZTZRzAHeLhw3AccVa2gpH2B44Fzh7i+GFgMMG/evISQbSSKncgrzzmGHf3Vh5i6ZmDWmsrsI6j2W2Kor5InAj8dqlkoIpYDywF6e3v9dbQkHR2ie9pkBgZit45kDzE1a20pG9N8WdIk4GX5qfUR8XzCvfuAuYXjHmDjEGVPxc1CDaOyI9lDTM1aW8rqo8cC/0XW8XsZcJ+k1yfcew2wUNKCPJGcCuyxamne//AG4NsjiNtK5nWKzNpHStPQ/wbeOrgXgaSXkX17X1TrRRHRn292fw3Z8NHLI+IeSUvy60vzoicDP4yIrUPcysaJ1ykyaw8piWBicUOaiLhP0sSUm0fEamB1xbmlFcdXAFek3M/qz5vdmLW+lESwVtL/Bf4lP34/cFt5IVmj8TpFZq0tJRH8KfBBsuWnBdxI1ldgbcTrFJm1rpREMAG4OCI+C7tmDE8uNSprWMOtU+SmIrPmk5IIrgPeAvwmP+4CfggcXVZQ1tjciWzWWlI2ptknIgaTAPnzfcsLyZqBN7sxax0pNYKtko6IiJ8DSFoEbC83LGsGKZ3IrhmYNb6URPBh4BuSBmcFHwi8t7yQrJnU6kR2n4FZc0hZYmKNpJcDB5ONGvrPxCUmrI1U60T28FKz5pBSIyD/xX93ybFYk/MaRWbNKaWz2CyZ1ygyaz7D7VAmoCciHq5VzqzIw0vNmstwO5QF8K06xWItxMNLzZpHSh/BLZJeHRFrSo/GWoqHl5o1h5RE8EZgiaRfkm1VKbLKwmFlBmatwcNLzRpfSiI4ofQorOV5eKlZ40qZR/CgpNcBCyPiS5K6ganlh2atxsNLzRpTylaVnwDOB/4yPzUR+ErKzSUdL2m9pA2SPjZEmWMl3S7pHkk3pAZuzcnDS80aT0rT0MnAq4CfA0TERknThntRvlz1pcBxZBvZr5G0KiLWFcrMJNvb4PiIeEjS/qP4O1iT8fBSs8aSMqFsRz6MNAAkTUm895HAhoi4PyJ2AFcBJ1WUOQ24OiIeAoiIxxPvbU3Ow0vNGkdKjeDfJC0DZkr6APBHwIqE180BihPR+oCjKsq8DJgo6XpgGtkGOFdW3kjSYmAxwLx58xLe2pqBh5eaNYaUzuKLJB0HPEO28NzfRsS1Cfeu9omt/Fo3AVgEvJlsw5ubJd0SEfdVxLAcWA7Q29vrr4YtxMNLzcbfsIlA0oURcT5wbZVztfQBcwvHPcDGKmWeiIitZPse3AgcDtyHtZXhhpf2Pb2dtx66P3/zjkPp7BCTJnSy35RJTgpmYyClj+C4KudS5hasARZKWiBpEnAqsKqizLeB35U0QdK+ZE1H9ybc21rQYFPRi2d2Va0ZnHX0Ak5b8TOOufDfOfmyn7L+sS3uOzAbA0PWCCT9KXAO8FJJdxYuTQN+OtyNI6Jf0rnANUAncHlE3CNpSX59aUTcK+kHwJ3AALAiIrzcdRurVjMA3FxkVqJaTUNfBb4P/CNQnAOwJSKeSrl5RKwGVlecW1px/BngM0nRWluoHF7a9/R2DzE1K9GQTUMRsTkifhkR74uIB8n2KQ5gqiQP3bFSFWsGPbO6+PX25z3E1KwkKZ3FJwKfBV4MPA68hKwd/xXlhmbtbrBmsPKcYxgYGGDZGYvYvmPPIaZep8hs76TMI/g08BrgRxHxKklvBN5XblhmmY4O0T1tMgDd0/bxOkVmJUgZNfR8RDwJdEjqiIh/B15Zclxme/A6RWblSKkR/FrSVOBG4F8lPQ70lxuWWXVep8hs7KXUCE4CtgEfAX4A/DdwYplBmdXidYrMxtZwm9d3At+OiLeQjfP/cl2iMhtGyjpF7kQ2S1MzEUTETknbJM2IiM31CsosRa11ityJbJYupY/gWeAuSdeS7VkMQEScV1pUZiNQbTbyx995qGcimyVKSQTfyx9mDcudyGajl7IM9a5+AUlHRMTPyw3JbHSKNYNHN3tJa7NUKTWCohXAEWUEYjYW3IlsNnIjTQT+tFjDcyey2cjUnEcgqVPSVwqn/q7keMzGjGcim6VJGT7aLWlSROyIiG/VKzCzsZDSieymImt3KU1DvwR+KmkVuw8f/exwL5R0PHAx2cY0KyLigorrx5LtUvZAfurqiPj7pMjNEtXqRHZTkVlaItiYPzrIdidLks9KvpRsq8s+YI2kVRGxrqLoTyLinan3NRuNoTqRPd/ALG346N8BSJqSbzKf6khgQ0Tcn7/+KrJ1iyoTgVldVOtEdlORWcKic5JeK2kd+abykg6XdFnCvecADxeO+/JzlV4r6Q5J35dUdbMbSYslrZW0dtOmTQlvbTa0WovWDTYVffzbd3PsRdfz7stuYv1jW9yRbC0tZfXRfwbeBjwJEBF3AK9PeF21r1CVn6afAy+JiMOBzwNVO6MjYnlE9EZEb3d3d8Jbm9U22FR0+NwZLDt9kUcVWVtLmkcQEQ9Lu/1e35nwsj5gbuG4h6yvoXjfZwrPV0u6TNLsiHgiJS6zveGmIrNMSiJ4WNLRQEiaBJxH3kw0jDXAQkkLgEeAU4HTigUkvQh4LCJC0pFkNZQnR/IXMNtbHlVk7S4lESwhGwI6h+xb/g+Bc4Z7UUT0SzoXuIZs+OjlEXGPpCX59aXAKcCfSuoHtgOnRoTr31Z3HlVk7SwlERwcEe8vnpB0DPDT4V4YEauB1RXnlhaeXwJckhaqWbncVGTtKiURfJ49F5qrds6sJbipyNrNkIlA0muBo4FuSX9euDSdrKnHrGWlNBUBdE+dzKObn2XK5E66Jk5gvymTnBCs6dSqEUwCpuZlijOKnyFr2zdrabWaioCqzUULZk9xc5E1nSETQUTcANwg6YqIeLCOMZk1lGpNRcU5B24usmaXMqFssqTlkn4o6ceDj9IjM2sglRPQirUDT0KzZpfSWfwNYCnZ7mQpE8nMWlKxqWjlOcewo3+nRxZZS0hJBP0R8YXSIzFrEh0donvaZAYGYtiRRe47sGaQkgi+I+kcYCXw3ODJiHiqtKjMmsBwI4vcd2DNIiURnJX/+dHCuQBeOvbhmDWXWiOLPCvZmkXKfgQL6hGIWTOrNrLIfQfWLGpNKHtTRPxY0rurXY+Iq8sLy6z5VDYVPb7lOc9KtqZQq0bwBuDHwIlVrgXgRGBWodhUdODMHV7AzppCrQlln8j/PLt+4Zi1Bi9gZ80kaWMaMxud1AXsPMzUxpMTgVnJPMzUGp0TgVkdeJipNbKkRJBvVTm/WD4irkx43fFku5t1Aisi4oIhyr0auAV4b0T8v5SYzJqRh5laIxo2EUj6F+Ag4HZeWGsogJqJQFIncClwHNkWl2skrYqIdVXKXUi2paVZyxvJMFP3HVg9pNQIeoFDR7GX8JHAhoi4H0DSVcBJwLqKch8Cvgm8eoT3N2taKcNMnRCsXlKWob4beNEo7j0HeLhw3Jef20XSHOBkstVNhyRpsaS1ktZu2rRpFKGYNabBhHDIgdN58cyuqstbDyaEj3/7bj7y9du555FneOipbTy+xctc29hIqRHMBtZJupXdF5171zCvq/Z1pfKn9p+B8yNipzT0t5uIWA4sB+jt7fVPvrWcWn0HHl1kZUtJBJ8c5b37gLmF4x5gY0WZXuCqPAnMBt4uqT8ivjXK9zRrWkP1HXh0kZUtZdG5G0Z57zXAQkkLgEeAU4HTKu69a0E7SVcA33USsHZWre+gMiF4dJGNtZRRQ68BPg8cQrahfSewNSKm13pdRPRLOpdsNFAncHlE3CNpSX69Zr+AWTurlRCGWsTurYfuz9+841A6O8SkCZ3sN2WSk4IlSWkauoTs2/w3yJpyzgQWptw8IlYDqyvOVU0AEfGHKfc0aycjWcTurKMXcNqKn3mEkY1Y0oSyiNggqTMidgJfknRTyXGZWUGtmcmA5yDYXklJBNskTQJul/RPwK+AKeWGZWbVVBtd1Pf09pojjJwQbDgp8wjOyMudC2wlGwn0+2UGZWZDGxxddPjcGSw7fRE9s7r49fbnh52DcOxF1/Puy25i/WNbPP/AdpMyauhBSV3AgRHxd3WIycyGUWwqWnnOMQwMDLDsjEU8/oyHnNrIpYwaOhG4iGzE0AJJrwT+PmFCmZmVrKNDdE+bDED3tH04cIaHnNrIpU4oOxK4HiAibpc0v7SIzGxURjrk1H0HNiglEfRHxOZaS0CYWePwgnY2UimJ4G5JpwGdkhYC5wEePmrW4FI2w3FCMEgbNfQh4BVkC859DXgG+HCZQZnZ2CkOOd22Y6dXOLU9pIwa2gb8df4wsyaUuqCdawjtKWXUUC/wV+y5VeVh5YVlZmMtZUE7J4T2lNJH8K/AR4G7gIFywzGzsjkhWKWURLApIlaVHomZ1ZUTgg1KSQSfkLQCuI7ddyi7urSozKxunBAsJRGcDbwcmMgLTUMBOBGYtRAnhPaVkggOj4j/MZqbSzoeuJhsY5oVEXFBxfWTgE+RJZh+4MMR8R+jeS8zGxtOCO0nJRHcIunQiFg3khtL6gQuBY4j2794jaRVFfe5DlgVESHpMODfyGofZjbOnBDaR0oieB1wlqQHyPoIBETC8NEjgQ0RcT+ApKuAk4BdiSAiflMoP4WsycnMGshoE0Jx+8zJEzrYGRAR3kazAaUkguNHee85wMOF4z7gqMpCkk4G/hHYH3hHtRtJWgwsBpg3b94owzGzvTGShFDcPvPT31vHWUcvcG2hgSXtRzDKe1f7393jG39ErARWSno9WX/BW6qUWQ4sB+jt7XWtwWwcpSQEeGH7TDcfNb6kPYtHqY9sN7NBPcDGoQpHxI2SDpI0OyKeKDEuMxsDtRJCcftM9yc0vpRF50ZrDbBQ0oJ8z+NTgd0mpkn6beXrW0s6gmzzmydLjMnMxthgQjjkwOlVt89M2UbTC92Nr9JqBBHRL+lc4Bqy4aOXR8Q9kpbk15eS7X18pqTnge3AeyPCPwFmTWio7TMv/tF9XPj7h+2x8qlrCI2jzKYhImI1sLri3NLC8wuBC8uMwczqq3L7zH84+TAGBgaQNOIhqC87YCqSPNqoZKUmAjNrb8WkADB76uTkIahfvukBjzaqEycCM6ubkQxBddNR/TgRmFndpSQE9yXUjxOBmY2bWgmhcrSR+xLKU+bwUTOzJNWGoH7ztoerjjaqHH76tVsf5JdPbuM9y27m3K/+grsf2exhqCPkGoGZNYxiDSF1tJFrCnvPicDMGs5IRht51NHecyIws4Y3mr6E4VZE7eyQawk5JwIzaxrVEsLF11WfuVxrRdTTVvzMzUcFTgRm1nRS+hIqawrAHh3Nbj7KOBGYWdOq1ZdQWVOoVktwR3PGw0fNrGUUh6H+w8mH8dvdU3jFi6fvsSJqZfNRypDUX23ezsZfb+eRp7exactzLTU01YnAzFrOYE3hgBld7D99Hw45cDorzzmGV/bMYNkZi3bVEoZaIvv3F81tq7kKbhoys5ZXuSLqgTNG1tHc6k1ITgRm1lZG09E8krkKzZgY3DRkZm2pWvPRUMtbtHoTUqmJQNLxktZL2iDpY1Wuv1/SnfnjJkmHlxmPmdlQanU0D7Xu0XCJobgNZyN3NpfWNCSpE7gUOI5sI/s1klZFxLpCsQeAN0TE05JOAJYDR5UVk5nZcKoNSW31JqQy+wiOBDZExP0Akq4CTgJ2JYKIuKlQ/hagp8R4zMxGbCRzFVKXuygmhuKyF5MndLAzqHtyKLNpaA7wcOG4Lz83lD8Gvl/tgqTFktZKWrtp06YxDNHMbGTGsgmpuOzFp7+3jg2bto7L3IUyawTV0ljVv4WkN5IlgtdVux4Ry8majejt7W2chjUza1tj0YQELyx7MZ6rp5aZCPqAuYXjHmBjZSFJhwErgBMi4skS4zEzK81ompCKSSF19dSeWV188cxeDj5g2pglgzKbhtYACyUtkDQJOBVYVSwgaR5wNXBGRNxXYixmZnWV0oRUTArDLX8xWIPoe3o7H7hyLU9u3TFmsZZWI4iIfknnAtcAncDlEXGPpCX59aXA3wL7AZdJAuiPiN6yYjIzq7daTUj/9ievpVOw7IxFXPyj2h3PRX1Pb2dH/84xi1ERzdXk3tvbG2vXrh3vMMzMxszAQPDk1h27+hc2bXmOi6+7j7OOXsCO/gE+/u27d0sGPbO6WHnOMbslmOFIum2oL9pOBGZmDaZaYviTr9y2V30EtRKB1xoyM2sw1ZqTVp5zDDv6d5Yyv8CJwMyswVUmhjG/f2l3NjOzpuBEYGbW5pwIzMzanBOBmVmbcyIwM2tzTTePQNIm4MERvmw28EQJ4YylZogRmiNOxzg2HOPYaJQYXxIR3dUuNF0iGA1Jaxt96YpmiBGaI07HODYc49hohhjdNGRm1uacCMzM2ly7JILl4x1AgmaIEZojTsc4Nhzj2Gj4GNuij8DMzIbWLjUCMzMbghOBmVmba6lEIOl4SeslbZD0sSrXJelz+fU7JR3RgDG+P4/tTkk3STq80WIslHu1pJ2STqlnfPl7DxujpGMl3S7pHkk31DvGPIbh/r9nSPqOpDvyOM+uc3yXS3pc0t1DXG+Ez8xwMTbCZ6ZmjIVy4/aZqSkiWuJBth3mfwMvBSYBdwCHVpR5O/B9QMBrgJ81YIxHA7Py5yc0YoyFcj8GVgOnNFqMwExgHTAvP96/QX8m/wq4MH/eDTwFTKpjjK8HjgDuHuL6uH5mEmMc189MSoyFn4dx+cwM92ilGsGRwIaIuD8idgBXASdVlDkJuDIytwAzJR3YSDFGxE0R8XR+eAvQU8f4kmLMfQj4JvB4PYPLpcR4GnB1RDwEEBGNGmcA05Rt2j2VLBH01yvAiLgxf8+hjPdnZtgYG+Azk/LvCOP7mamplRLBHODhwnFffm6kZco00vf/Y7JvY/U0bIyS5gAnA0vrGFdRyr/jy4BZkq6XdJukM+sW3QtS4rwEOATYCNwF/FlEDNQnvCTj/ZkZqfH4zAyrAT4zNbXSDmXV9m2rHBubUqZMye8v6Y1kP9SvKzWiKm9d5VxljP8MnB8RO7MvsnWXEuMEYBHwZqALuFnSLRFxX9nBFaTE+TbgduBNwEHAtZJ+EhHPlB1covH+zCQbx89MivH+zNTUSomgD5hbOO4h+5Y10jJlSnp/SYcBK4ATIuLJOsU2KCXGXuCq/Ad6NvB2Sf0R8a36hJj8f/1ERGwFtkq6ETgcqGciSInzbOCCyBqRN0h6AHg5cGt9QhzWeH9mkozzZybFeH9mahvvToqxepAltfuBBbzQMfeKijLvYPeOr1sbMMZ5wAbg6Eb9d6wofwX17yxO+Xc8BLguL7svcDfwO+F7YTIAAAOQSURBVA0Y5xeAT+bPDwAeAWbXOc75DN0RO66fmcQYx/UzkxJjRbm6f2aGe7RMjSAi+iWdC1xD1jt/eUTcI2lJfn0pWW/928l+aLaRfRtrtBj/FtgPuCz/9tAfdVy5MDHGcZUSY0TcK+kHwJ3AALAiImoO7RuPOIFPAVdIuovsl+35EVG3JYslfQ04FpgtqQ/4BDCxEN+4fmYSYxzXz0xijA3NS0yYmbW5Vho1ZGZmo+BEYGbW5pwIzMzanBOBmVmbcyIwM2tzTgRmY0DS70k6dIhr84dblbLKa/5Q0iVjE51ZbU4E1jYkddY6HuI1kpTyOfk9oGoiMGt0TgTWEiSdLunWfP+BZYO/5CX9RtLfS/oZ8Noqx38u6e788eH8NfMl3SvpMuDn7L7EApIukLQuX//+IklHA+8CPpO//0GSFuV7DNwMfDDx73C2pPvyvROOKZzvlvRNSWvyxzGSOiT9UtLMQrkNkg7Yu39Ja0vjPbXZDz/29kG2nMR3gIn58WXAmfnzAN5TKLvrmGxRuruAKWRLQN8DvIpsqYAB4DVV3uu3gPW8MBlzZv7nFRSWDSCb0fyG/PlnGGbpAeBA4CGyPQkmAT8FLsmvfRV4Xf58HnBv/vxi4Oz8+VHAj8b7/8KP5ny4RmCt4M1kv9TXSLo9P35pfm0n2RrwVDl+HbAyIrZGxG+Aq4Hfza89GNn6+5WeAZ4FVkh6N9myC7uRNIMsQQzuivYvCX+Ho4DrI2JTZHsXfL1w7S3AJfnfbRUwXdK0vMx78zKnVrzGLFnLrDVkbU3AlyPiL6tcezYidg5xXGs94K3VTka2ftCRZMnmVOBcsiWkK+MZzdotQ72mA3htRGzf7U2yZqffltRN1kfx6VG8p5lrBNYSrgNOkbQ/gKTfkvSShNfdCPyepH0lTSHbOOQntV4gaSowIyJWAx8GXplf2gJMA4iIXwObJQ2ui//+hFh+BhwraT9JE4E/KFz7IVnCGYzhlfn7BLAS+CxZc1EjLr9sTcCJwJpeRKwD/gb4oaQ7gWvJ2tyHe93Pydr2byX7RbwiIn4xzMumAd/N3+cG4CP5+auAj0r6haSDyFbpvDT/1r7rm7ykF0taXSWWXwGfBG4GfkTWST3oPKA375xeBywpXPs6cHr+55D3N6vFq4+ambU51wjMzNqcE4GZWZtzIjAza3NOBGZmbc6JwMyszTkRmJm1OScCM7M29/8BewJyQhsWGP8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# call the private method - note that the actual value of rho doesn' matter since we are just\n",
    "# just interested in the error values searched the average inter-rater correlations for the\n",
    "# corresponding simulated rater scores\n",
    "error_sds, mean_inter_rater_correlations, _ = dataset._find_best_error_sd_for_rho(0.4, 100)\n",
    "p = sns.scatterplot(x=error_sds, y=mean_inter_rater_correlations)\n",
    "p.set_xlabel(r'error std. dev.')\n",
    "p.set_ylabel('mean inter-rater correlation')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the average inter-rater correlation monotonically decreases as larger error terms are added to the true scores to simulate the rater scores."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inter-rater correlations in each rater category\n",
    "\n",
    "The next thing we want to check is whether the mean average inter-rater correlations in the 4 simulated human rater categories are actually what we expect them to be. Recall that we wanted four rater categories -  \"low\" ($\\rho=0.4$), \"moderate\" ($\\rho=0.55$), \"average\" ($\\rho=0.65$), and \"high\" ($\\rho=0.8$), each with 50 simulated raters. So, we want to check whether our simulated rhos match these desired rhos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>low</th>\n",
       "      <th>moderate</th>\n",
       "      <th>average</th>\n",
       "      <th>high</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>expected</th>\n",
       "      <td>0.400000</td>\n",
       "      <td>0.550000</td>\n",
       "      <td>0.650000</td>\n",
       "      <td>0.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>simulated</th>\n",
       "      <td>0.401045</td>\n",
       "      <td>0.551284</td>\n",
       "      <td>0.648225</td>\n",
       "      <td>0.801706</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                low  moderate   average      high\n",
       "expected   0.400000  0.550000  0.650000  0.800000\n",
       "simulated  0.401045  0.551284  0.648225  0.801706"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the first thing we will do is to compute the correlations between all raters, all systems, and the true scores\n",
    "# by doing this, we will save a lot of work below by not having to compute the correlations again and again\n",
    "df_all_correlations = df_scores.corr()\n",
    "\n",
    "# first we extract all of the columns containing scores from human raters\n",
    "# these columns all have names of the form `h_X`\n",
    "rater_columns = [column for column in df_scores.columns if column.startswith('h_')]\n",
    "\n",
    "# get the correlations between every rater and every other rater and re-organize\n",
    "# the data frame such that each row corresponds to a pair of raters\n",
    "df_all_rater_correlations = df_all_correlations.loc[rater_columns, rater_columns]\n",
    "df_pairwise_rater_correlations = df_all_rater_correlations.unstack().reset_index()\n",
    "df_pairwise_rater_correlations.columns = ['rater1', 'rater2', 'correlation']\n",
    "\n",
    "# exclude correlations between the scores from the same rater\n",
    "different_raters = df_pairwise_rater_correlations['rater1'] != df_pairwise_rater_correlations['rater2']\n",
    "df_pairwise_rater_correlations = df_pairwise_rater_correlations[different_raters]\n",
    "\n",
    "# now merge with the categories from the rater metadata data frame; \n",
    "# we need to do this twice once for each rater of the pair\n",
    "df_pairwise_rater_correlations_with_categories = pd.merge(df_pairwise_rater_correlations,\n",
    "                                                          df_rater_metadata,\n",
    "                                                          left_on ='rater1',\n",
    "                                                          right_on='rater_id')\n",
    "df_pairwise_rater_correlations_with_categories = pd.merge(df_pairwise_rater_correlations_with_categories,\n",
    "                                                          df_rater_metadata,\n",
    "                                                          left_on='rater2',\n",
    "                                                          right_on='rater_id',\n",
    "                                                          suffixes=['_1', '_2'])\n",
    "\n",
    "# now trim away any rows where the raters are not in the same cateogry\n",
    "# since we want to compute within-category mean inter-rater correlations\n",
    "same_category = df_pairwise_rater_correlations_with_categories['rater_category_1'] == df_pairwise_rater_correlations_with_categories['rater_category_2']\n",
    "df_pairwise_rater_correlations_same_category = df_pairwise_rater_correlations_with_categories[same_category]\n",
    "\n",
    "# now group by the category and compute the average correlations\n",
    "simulated_mean_correlations = df_pairwise_rater_correlations_same_category.groupby('rater_category_1')['correlation'].mean()\n",
    "\n",
    "# now print out a data frame comparing our simulated values to the values we expected\n",
    "expected_mean_correlations = pd.Series(dict(zip(dataset.rater_categories, dataset.rater_rho_per_category)))\n",
    "pd.DataFrame([expected_mean_correlations, simulated_mean_correlations], index=['expected', 'simulated'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that our the average within-category inter-rater correlations for our simulated raters are quite close to what we expect. We can also visually compare the distributions of all within-category inter-rater correlations to those of all across-category inter-rater correlations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEWCAYAAACKSkfIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deZwU1b338c+XRSBRcQEUQQGNRgVkhAFNNAaXgDGuUa4oETTGEVFjvOrjksUt3JjojU9M1OtoFEWMoOISo3lU3JVEBh1EJASUiYx6BTEgREQYfs8fXTNpx56Znq0HqO/79epXV586VfU73fCb06eqTykiMDOzdGjX1gGYmVnhOOmbmaWIk76ZWYo46ZuZpYiTvplZijjpm5mliJO+FZSk/5H007aOI60kPSvpB03cdhdJqyW1b+m4rHCc9A0ASRWSDqtVdqqkF+vZ5gpJd+coD0lfybVNRIyPiKvzjGmSpJ/nU7e1NPQebM5q/5uIiHciYsuIqGrLuKx5nPRtsyWpQ3PWFyKGZu67fa3Xrd4e2/Q56VtBZffeJQ2XVCnpAklLJb0v6bRkXQkwBvg/yZDCH5PynSQ9IGmZpMWSfpi17ysk3S/pbkkfA6fmOH6FpIslvQ78S1IHSZdIekvSKklvSjouqbsX8D/A15IYViTlnSRdJ+kdSR8kQ1ZdarXpYkn/C9xRx/twhqT5WcccXH3MZAhmhaR5ko6u9d7dLOkxSf8CDq6jPftLejnZxxxJw+uIYTdJT0taLulDSVMkbZOsmwzsAvwxafv/kdQ3+RbXIeuzeETSR5IWSTqj1mcxTdJdSRvnSSpu4J+HFYCTvrW1HYGuQC/gdOBGSdtGRCkwBfhVMqRwlKR2wB+BOUn9Q4EfSRqZtb9jgPuBbZLtczkJ+A6wTUSsB94CvpHEcSVwt6SeETEfGA/MTGLYJtn+l8AeQBHwlSSWn9Vq03ZAH6Ck9sEljQKuAMYCWwNHA8sldUza9wTQAzgXmCLpq1mbnwxMBLYCqoedatoD7AD8Cfh5EsOFwAOSuud4HwT8AtgJ2AvYOYmLiDgFeAc4Kmn7r3Js/wegMtn+BOC/JB2atf5o4N4krkeA3+XYhxWYk75leyjpHa5IerU35bHNf2RvU90bboR1wFURsS4iHgNWA1+to+5QoHtEXBURn0XE28CtwOisOjMj4qGI2BARa+rYzw0RsaR6fUTcFxHvJdtMBRYCw3JtKEnAGcD5EfFRRKwC/qtWDBuAyyNibR0x/IDMH7NZkbEoIv4B7A9sCVyTtO9p4FEySb3awxHxUhLrpzna8z3gsYh4LKnzJFAGHFE7iOS4TyZxLgN+DXyzjves9vuwM3AgcHFEfBoR5cBtwClZ1V5M4qgCJgOD8tm3tS4nfct2bERsU/0AJlSvkDQm+Zq/WtLjWdtMy94mqzecr+VJb7vaJ2QSXy59gJ1q/YG5jEzvttqSrJgfz4p5TK46Sb2xksqz9jkA6FZHDN2BLwGzs+r/OSmvtiwrIeeyM5lvF7XtBCyJiA1ZZf8g800iZ+w5yvoAo2q9RwcCPWtvJKmHpHslvZsMh91N3e3OFWv1H726Yv3frOVPgM7yeYc25w/A8hIRU6h7uKTVDlvr9RJgcUTsns82EfHthupI6kPm28KhZL4lVEkqJzP0kSuGD4E1QP+IeDfPuGtbAuyWo/w9YGdJ7bIS/y7A3xvYd3bZEmByRJyRo15tv0i23Scilks6ls8PwdTXjveA7SRtlZX4dwHqek9sI+Gevm3MPgB2zXr9CvBxcuKyi6T2kgZIGtqMY3yZTHJbBqDMieQBtWLoLWkLgCQZ3wpcL6lHsk2vWucVGnIbcKGkIcr4SvLH56/Av8icvO6YnIA9isy4eL7uBo6SNDJ5fzonJ5d756i7FZnhtBWSegEX1Vpf+/2vERFLgJeBXyTH2IfMOZlCdwyskZz0bWP2e2DvZJjioWRs+CgyJ1AXk+l130bmBGyTRMSbwH8DM8kkuYHAS1lVngbmAf8r6cOk7GJgEfCXZFjkKeo+D5HrmPeRORl7D7AKeAjYLiI+I3Py89tJ224CxkbE3xqx7yVkTmZfRuYP2RIyyTzX//UrgcHASjInf6fXWv8L4CfJ+39hju1PAvqS6fU/SOY8xpP5xmptQ76JiplZerinb2aWIk76ZmYp4qRvZpYiTvpmZimy0V+n361bt+jbt29bh2FmtkmZPXv2hxHxhek3Nvqk37dvX8rKyto6DDOzTYqkf+Qq9/COmVmKOOmbmaWIk76ZWYps9GP6ZtawdevWUVlZyaef1je5p22OOnfuTO/evenYsWNe9Z30zTYDlZWVbLXVVvTt25fMlP+WBhHB8uXLqayspF+/fnlt0+DwTjKD3ivJbdfmSboyKb8imYe7PHkckbXNpcnt0xZkzz6YzCo4N1l3g/yv06xFfPrpp2y//fZO+Ckjie23375R3/Dy6emvBQ6JiNXJ7dxezLqJxvURcV2tIPYmcxeh/mRutPCUpD2SGRJvJnP7uL8AjwGHA9k35DCzJnLCT6fGfu4N9vST27mtTl52TB71Tc15DHBvcgu2xWSmoB0mqSewdUTMjMzUnncBxzYqWjMza5a8xvQltQdmk7kJ9I0R8VdJ3wbOkTSWzD04L4iIf5K5XdpfsjavTMrWJcu1y3Mdr4TkhtK77LJLoxpkZlA6u7RF91cy5Av3d/+CiRMncs8999C+fXvatWvHLbfcwn777deicbSEiooKXn75ZU4++eS2DqVN5JX0k6GZIknbAA9KGkBmqOZqMr3+q8nciOL7/Ps2c5/bRT3luY5XCpQCFBcXe8J/23iU5kimJQ0nxM3dzJkzefTRR3n11Vfp1KkTH374IZ999llbh5VTRUUF99xzT6sn/aqqKtq3b9+qx2iKRl2nHxErgGeBwyPig4ioyrp93LCkWiWZGz9X603mzjqVyXLtcjPbxL3//vt069aNTp06AdCtWzd22mknAK666iqGDh3KgAEDKCkpofrGTcOHD+f888/noIMOYq+99mLWrFl897vfZffdd+cnP/lJzb7vvvtuhg0bRlFREWeeeSZVVVVfOP6sWbP4+te/zqBBgxg2bBirVq2ioqKCb3zjGwwePJjBgwfz8ssvA3DJJZfwwgsvUFRUxPXXX09VVRUXXXQRQ4cOZZ999uGWW24BYMOGDUyYMIH+/ftz5JFHcsQRR3D//fcDMGPGDPbdd18GDhzI97//fdauXQtkpo256qqrOPDAA7nmmmsYPHhwTYwLFy5kyJAhLf3WN1o+V+90T3r4SOoCHAb8LRmjr3Yc8Eay/AgwWlInSf2A3YFXIuJ9YJWk/ZOrdsYCD7dgW8ysjYwYMYIlS5awxx57MGHCBJ577rmadeeccw6zZs3ijTfeYM2aNTz66KM167bYYguef/55xo8fzzHHHMONN97IG2+8waRJk1i+fDnz589n6tSpvPTSS5SXl9O+fXumTPn8bXg/++wzTjzxRH7zm98wZ84cnnrqKbp06UKPHj148sknefXVV5k6dSo//OEPAbjmmmv4xje+QXl5Oeeffz6///3v6dq1K7NmzWLWrFnceuutLF68mOnTp1NRUcHcuXO57bbbmDlzJpC5UurUU09l6tSpzJ07l/Xr13PzzTfXxNO5c2defPFFfvzjH9O1a1fKy8sBuOOOOzj11FNb6yPIWz49/Z7AM5JeB2YBT0bEo8CvkssvXwcOBs4HiIh5wDTgTeDPwNnJ8BDAWWTuaboIeAtfuWO2Wdhyyy2ZPXs2paWldO/enRNPPJFJkyYB8Mwzz7DffvsxcOBAnn76aebNm1ez3dFHHw3AwIED6d+/Pz179qRTp07suuuuLFmyhBkzZjB79myGDh1KUVERM2bM4O233/7csRcsWEDPnj0ZOnQoAFtvvTUdOnRg3bp1nHHGGQwcOJBRo0bx5ptv5oz9iSee4K677qKoqIj99tuP5cuXs3DhQl588UVGjRpFu3bt2HHHHTn44INrjtevXz/22GMPAMaNG8fzzz9fs78TTzyxZvkHP/gBd9xxB1VVVUydOnWjOI/Q4Jh+RLwO7Juj/JR6tplI5sbPtcvLgAGNjNHMNgHt27dn+PDhDB8+nIEDB3LnnXcyevRoJkyYQFlZGTvvvDNXXHHF564prx4OateuXc1y9ev169cTEYwbN45f/OIXdR43InJetnj99dezww47MGfOHDZs2EDnzp3r3P63v/0tI0eO/Fz5n/70pzrr1+fLX/5yzfLxxx/PlVdeySGHHMKQIUPYfvvt6922EDz3jpk124IFC1i4cGHN6/Lycvr06VOT4Lt168bq1atrxsTzdeihh3L//fezdOlSAD766CP+8Y/Pzxi855578t577zFr1iwAVq1axfr161m5ciU9e/akXbt2TJ48ueZcwFZbbcWqVatqth85ciQ333wz69atA+Dvf/87//rXvzjwwAN54IEH2LBhAx988AHPPvtszfEqKipYtGgRAJMnT+ab3/xmzvg7d+7MyJEjOeusszjttNMa1fbW4mkYzDZD+Vxi2ZJWr17Nueeey4oVK+jQoQNf+cpXKC0tZZtttqkZYunbt2/NEEy+9t57b37+858zYsQINmzYQMeOHbnxxhvp06dPTZ0tttiCqVOncu6557JmzRq6dOnCU089xYQJEzj++OO57777OPjgg2t64Pvssw8dOnRg0KBBnHrqqZx33nlUVFQwePBgIoLu3bvz0EMPcfzxxzNjxgwGDBjAHnvswX777UfXrl3p3Lkzd9xxB6NGjWL9+vUMHTqU8ePH19mGMWPGMH36dEaMGNG0N7eFqaGvKm2tuLg4fBMV22hspJdszp8/n7322qutw9jsrF69mi233JLly5czbNgwXnrpJXbcccdG7eO6665j5cqVXH311a0UZe7PX9LsiCiuXdc9fTOzOhx55JGsWLGCzz77jJ/+9KeNTvjHHXccb731Fk8//XQrRdh4TvpmZnWoHsdvqgcffLBlAmlBPpFrZpYiTvpmZinipG9mliJO+mZmKeITuWabo1yXljZHHpeltm/fnoEDB7Ju3To6dOjAuHHj+NGPfkS7du0oKyvjrrvu4oYbbmDt2rV85zvf4cMPP+TSSy9lp512Yvz48XTs2JGZM2fSpUuXlo09MWnSJEaMGFEzEVxTrVixgnvuuYcJEya0UGSF5aRvtinZSH8nANClS5eaycWWLl3KySefzMqVK7nyyispLi6muDhzyfhrr73GunXrauqOHz+eCy+8MO9frEYEEUG7do0bqJg0aRIDBgxokaR/0003tXrSX79+PR06tHyK9vCOmbW4Hj16UFpayu9+9zsigmeffZYjjzySpUuX8r3vfY/y8nKKioq45ZZbmDZtGldddRVjxowB4Nprr62Z5vjyyy8HMnPg77XXXkyYMIHBgwezZMmSeuudccYZ9O/fnxEjRrBmzRruv/9+ysrKGDNmDEVFRaxZs+Zz8S5atIjDDjuMQYMGMXjwYN566y1Wr17NoYceyuDBgxk4cCAPP5yZFPiSSy7hrbfeoqioiIsuuqjOmAGuvvpq9txzT771rW9x0kkncd11mbvLlpeXs//++7PPPvtw3HHH8c9//hPITDd92WWX8c1vfpOJEyfSr1+/mukhPv74Y/r27Vvzuqmc9M2sVey6665s2LChZt4cyPwxuO2222qmNj7zzDM5+uijufbaa5kyZQpPPPEECxcu5JVXXqG8vJzZs2fXzGC5YMECxo4dy2uvvVYz10+uegsXLuTss89m3rx5bLPNNjzwwAOccMIJFBcXM2XKFMrLy78whDRmzBjOPvts5syZw8svv0zPnj3p3LkzDz74IK+++irPPPMMF1xwARHBNddcw2677UZ5eTnXXnttnTGXlZXxwAMP8NprrzF9+nSyZxYYO3Ysv/zlL3n99dcZOHAgV155Zc26FStW8Nxzz3H55ZczfPjwmonf7r33Xo4//ng6duzYrM/Fwztm1moaO83LE088wRNPPMG++2Ym9l29ejULFy5kl112oU+fPuy///4N1uvXrx9FRUUADBkyhIqKinqPuWrVKt59912OO+44gJrZONetW8dll13G888/T7t27Xj33Xf54IMP8o551apVHHPMMTV/YI466igAVq5cyYoVK2omaRs3bhyjRo2q2V/tqZl/9atfceyxx3LHHXdw6623NuLdzM1J38xaxdtvv0379u3p0aMH8+fPz2ubiODSSy/lzDPP/Fx5RUXF56Ysrq9e9hTN7du3/8JQTq5j5jJlyhSWLVvG7Nmz6dixI3379v3ctNANxXL99dfXe9y6ZLfzgAMOoKKigueee46qqioGDGj+zPQe3jGzFrds2TLGjx/POeeck3Ou+7qMHDmS22+/ndWrVwPw7rvvfm54qLH1stWeUrna1ltvTe/evXnooYcAWLt2LZ988gkrV66kR48edOzYkWeeeaZmSudcUzPniuXAAw/kj3/8I59++imrV6+uGabp2rUr2267LS+88AJQ/9TMkBkKOumkk1psamb39M02R21wRc+aNWsoKiqquWTzlFNO4T//8z8btY8RI0Ywf/58vva1rwGZO3LdfffdX7jBeL71sp166qmMHz+eLl26fOHS0MmTJ3PmmWfys5/9jI4dO3LfffcxZswYjjrqKIqLiykqKmLPPfcEYPvtt+eAAw5gwIABfPvb3+baa6/NGcvQoUM5+uijGTRoEH369KG4uJiuXbsCcOeddzJ+/Hg++eQTdt11V+6444464x4zZgw/+clPOOmkkxr1XtbFUyubNUZbXzJZx/E9tfLGqXpq5k8++YSDDjqI0tLSz90sPR/3338/Dz/8MJMnT66zjqdWNjPbCJSUlPDmm2/y6aefMm7cuEYn/HPPPZfHH3+cxx57rMVictI3M2sl99xzT7O2/+1vf9tCkfybT+SabSY29qFaax2N/dwbTPqSOkt6RdIcSfMkXZmUbyfpSUkLk+dts7a5VNIiSQskjcwqHyJpbrLuBjXmtL6Z1alz584sX77ciT9lIoLly5fX/LYgH/kM76wFDomI1ZI6Ai9Kehz4LjAjIq6RdAlwCXCxpL2B0UB/YCfgKUl7REQVcDNQAvwFeAw4HHg8/yaaWS69e/emsrKSZcuWtXUoVmCdO3emd+/eeddvMOlHpuuwOnnZMXkEcAwwPCm/E3gWuDgpvzci1gKLJS0ChkmqALaOiJkAku4CjsVJ36zZOnbsSL9+/do6DNsE5DWmL6m9pHJgKfBkRPwV2CEi3gdInnsk1XsBS7I2r0zKeiXLtctzHa9EUpmkMvdczMxaTl5JPyKqIqII6E2m117fb4FzjdNHPeW5jlcaEcURUdy9e/d8QjQzszw06uqdiFhBZhjncOADST0Bkufq30BXAjtnbdYbeC8p752j3MzMCiSfq3e6S9omWe4CHAb8DXgEGJdUGwc8nCw/AoyW1ElSP2B34JVkCGiVpP2Tq3bGZm1jZmYFkM/VOz2BOyW1J/NHYlpEPCppJjBN0unAO8AogIiYJ2ka8CawHjg7uXIH4CxgEtCFzAlcn8Q1MyugfK7eeR3YN0f5cuDQOraZCEzMUV4GNH9uUDMzaxL/ItfMLEWc9M3MUsRJ38wsRZz0zcxSxEnfzCxFnPTNzFLESd/MLEWc9M3MUsRJ38wsRZz0zcxSxDdGt01LaekXy0pKCh+H2SbKPX0zsxRxT99aR64eObhXbtbG3NM3M0sRJ30zsxTx8E6aeMjFLPXc0zczSxEnfTOzFHHSNzNLESd9M7MUcdI3M0uRBpO+pJ0lPSNpvqR5ks5Lyq+Q9K6k8uRxRNY2l0paJGmBpJFZ5UMkzU3W3SBJrdMsMzPLJZ9LNtcDF0TEq5K2AmZLejJZd31EXJddWdLewGigP7AT8JSkPSKiCrgZKAH+AjwGHA483jJNMTOzhjSY9CPifeD9ZHmVpPlAr3o2OQa4NyLWAoslLQKGSaoAto6ImQCS7gKOxUk/N08sZmatoFFj+pL6AvsCf02KzpH0uqTbJW2blPUClmRtVpmU9UqWa5fnOk6JpDJJZcuWLWtMiGZmVo+8k76kLYEHgB9FxMdkhmp2A4rIfBP47+qqOTaPesq/WBhRGhHFEVHcvXv3fEM0M7MG5JX0JXUkk/CnRMR0gIj4ICKqImIDcCswLKleCeyctXlv4L2kvHeOcjMzK5B8rt4R8HtgfkT8Oqu8Z1a144A3kuVHgNGSOknqB+wOvJKcG1glaf9kn2OBh1uoHWZmlod8rt45ADgFmCupPCm7DDhJUhGZIZoK4EyAiJgnaRrwJpkrf85OrtwBOAuYBHQhcwLXJ3HNNkOls+uY3C9LyRBfmNAW8rl650Vyj8c/Vs82E4GJOcrLgAGNCdDMzFqOf5FrZpYiTvpmZinipG9mliJO+mZmKeKkb2aWIk76ZmYp4qRvZpYiTvpmZinipG9mliJO+mZmKZLP3Dtmm7185oqBzG3fzDZl7umbmaWIk76ZWYo46ZuZpYiTvplZijjpm5mliJO+mVmKOOmbmaWIk76ZWYo46ZuZpUiDSV/SzpKekTRf0jxJ5yXl20l6UtLC5HnbrG0ulbRI0gJJI7PKh0iam6y7QVKuG66bmVkryaenvx64ICL2AvYHzpa0N3AJMCMidgdmJK9J1o0G+gOHAzdJap/s62Yyv2TfPXkc3oJtMTOzBjSY9CPi/Yh4NVleBcwHegHHAHcm1e4Ejk2WjwHujYi1EbEYWAQMk9QT2DoiZkZEAHdlbWNmZgXQqDF9SX2BfYG/AjtExPuQ+cMA9Eiq9QKWZG1WmZT1SpZrl+c6TomkMklly5Yta0yIZmZWj7yTvqQtgQeAH0XEx/VVzVEW9ZR/sTCiNCKKI6K4e/fu+YZoZmYNyCvpS+pIJuFPiYjpSfEHyZANyfPSpLwS2Dlr897Ae0l57xzlZmZWIPlcvSPg98D8iPh11qpHgHHJ8jjg4azy0ZI6SepH5oTtK8kQ0CpJ+yf7HJu1jZmZFUA+N1E5ADgFmCupPCm7DLgGmCbpdOAdYBRARMyTNA14k8yVP2dHRFWy3VnAJKAL8HjyMDOzAmkw6UfEi+Qejwc4tI5tJgITc5SXAQMaE6CZmbUc/yLXzCxFnPTNzFLESd/MLEWc9M3MUsRJ38wsRZz0zcxSxEnfzCxFnPTNzFLESd/MLEWc9M3MUiSfuXfMCq+0tK0jMNsspT7pl87OL7mUDClp5UjMzFqfh3fMzFLESd/MLEWc9M3MUiT1Y/pWYLlO0Jb4fIlZobinb2aWIk76ZmYp4qRvZpYiTvpmZinipG9mliINJn1Jt0taKumNrLIrJL0rqTx5HJG17lJJiyQtkDQyq3yIpLnJuhskqeWbY2Zm9cnnks1JwO+Au2qVXx8R12UXSNobGA30B3YCnpK0R0RUATcDJcBfgMeAw4HHmxW92ebM8w9ZK2iwpx8RzwMf5bm/Y4B7I2JtRCwGFgHDJPUEto6ImRERZP6AHNvUoM3MrGmaM6Z/jqTXk+GfbZOyXsCSrDqVSVmvZLl2eU6SSiSVSSpbtmxZM0I0M7NsTU36NwO7AUXA+8B/J+W5xumjnvKcIqI0Ioojorh79+5NDNHMzGprUtKPiA8ioioiNgC3AsOSVZXAzllVewPvJeW9c5SbmVkBNSnpJ2P01Y4Dqq/seQQYLamTpH7A7sArEfE+sErS/slVO2OBh5sRt5mZNUGDV+9I+gMwHOgmqRK4HBguqYjMEE0FcCZARMyTNA14E1gPnJ1cuQNwFpkrgbqQuWrHV+6YmRVYg0k/Ik7KUfz7eupPBCbmKC8DBjQqOjMza1H+Ra6ZWYo46ZuZpYhvorK58q85zSwH9/TNzFLESd/MLEWc9M3MUsRj+hsDj7+bWYG4p29mliLu6VveSmc3/I2kZEhJASIxs6ZyT9/MLEWc9M3MUsRJ38wsRZz0zcxSxCdyrUVVn+zd8x/P11nnoD4HFSocM6vFPX0zsxRx0jczSxEnfTOzFHHSNzNLESd9M7MUcdI3M0uRBpO+pNslLZX0RlbZdpKelLQwed42a92lkhZJWiBpZFb5EElzk3U3SFLLN8fMzOqTT09/EnB4rbJLgBkRsTswI3mNpL2B0UD/ZJubJLVPtrkZKAF2Tx6192lmZq2swaQfEc8DH9UqPga4M1m+Ezg2q/zeiFgbEYuBRcAwST2BrSNiZkQEcFfWNmZmViBNHdPfISLeB0ieeyTlvYAlWfUqk7JeyXLtcjMzK6CWPpGba5w+6inPvROpRFKZpLJly5a1WHBmZmnX1KT/QTJkQ/K8NCmvBHbOqtcbeC8p752jPKeIKI2I4ogo7t69exNDNDOz2pqa9B8BxiXL44CHs8pHS+okqR+ZE7avJENAqyTtn1y1MzZrGzMzK5AGZ9mU9AdgONBNUiVwOXANME3S6cA7wCiAiJgnaRrwJrAeODsiqpJdnUXmSqAuwOPJwxqjrhuol/gWhWaWnwaTfkScVMeqQ+uoPxGYmKO8DBjQqOjMzKxF+Re5ZmYp4qRvZpYivnOWWXPlOtfi8yy2kXJP38wsRZz0zcxSxEnfzCxFnPTNzFLEJ3LNJyLNUsQ9fTOzFHHSNzNLESd9M7MUcdI3M0sRJ30zsxRx0jczSxEnfTOzFHHSNzNLESd9M7MUcdI3M0uRzXoahtLZddxT1swspdzTNzNLESd9M7MUaVbSl1Qhaa6kckllSdl2kp6UtDB53jar/qWSFklaIGlkc4M3M7PGaYme/sERURQRxcnrS4AZEbE7MCN5jaS9gdFAf+Bw4CZJ7Vvg+GZmlqfWOJF7DDA8Wb4TeBa4OCm/NyLWAoslLQKGATNbIQazZtlz+vO5V/Q5qLCBmLWw5vb0A3hC0mxJ1Xfd2CEi3gdInnsk5b2AJVnbViZlXyCpRFKZpLJly5Y1M0QzM6vW3J7+ARHxnqQewJOS/lZPXeUoi1wVI6IUKAUoLi7OWcfMzBqvWT39iHgveV4KPEhmuOYDST0BkuelSfVKYOeszXsD7zXn+GZm1jhNTvqSvixpq+plYATwBvAIMC6pNg54OFl+BBgtqZOkfsDuwCtNPb6ZmTVec+dclSkAAAfBSURBVIZ3dgAelFS9n3si4s+SZgHTJJ0OvAOMAoiIeZKmAW8C64GzI6KqWdGbmVmjNDnpR8TbwKAc5cuBQ+vYZiIwsanHNDOz5tms596xwqjz8kYz2+h4GgYzsxRx0jczSxEP79imrzTHFNolJV8sMzP39M3M0sRJ38wsRZz0zcxSxEnfzCxFnPTNzFLESd/MLEV8yaa1vVyXXJpZq3BP38wsRZz0zcxSxEnfzCxFPKZv1gjP/6PhGUUP8s3TbSPmnr6ZWYq4p7858NUv6VbX5+9J5ywHJ/08lc5uOLGWDPF/MjPbuDnpF5J75GbN01rfavL9v9la354KOD24x/TNzFLEPX3b7OUzNGfW4jbScy0FT/qSDgd+A7QHbouIawodg5nloa2HPKxVFDTpS2oP3Ah8C6gEZkl6JCLeLGQclodc/+GHFD4Ms81OG5/bK3RPfxiwKCLeBpB0L3AM4KSfIv6BUy2b2An+2p/f32a3USDWJIqIwh1MOgE4PCJ+kLw+BdgvIs6pVa8EqP7O+FVgQQHC6wZ8WIDjFJrbtWnZHNu1ObYJNv529YmI7rULC93TV46yL/zViYhSoKDdH0llEVFcyGMWgtu1adkc27U5tgk23XYV+pLNSmDnrNe9gfcKHIOZWWoVOunPAnaX1E/SFsBo4JECx2BmlloFHd6JiPWSzgH+H5lLNm+PiHmFjKEem9bZtPy5XZuWzbFdm2ObYBNtV0FP5JqZWdvyNAxmZinipG9mliKpS/qSDpe0QNIiSZfUU2+opKrktwUbvYbaJWm4pJWSypPHz9oizsbI57NK2lUuaZ6k5wodY1Pk8VldlPU5vZH8O9yuLWJtjDza1VXSHyXNST6v09oizsbKo13bSnpQ0uuSXpE0oC3izFtEpOZB5uTxW8CuwBbAHGDvOuo9DTwGnNDWcbdEu4DhwKNtHWsLt2kbMr/m3iV53aOt426JdtWqfxTwdFvH3UKf12XAL5Pl7sBHwBZtHXsLtOta4PJkeU9gRlvHXd8jbT39mmkgIuIzoHoaiNrOBR4AlhYyuGbIt12bknzadDIwPSLeAYiITeHzauxndRLwh4JE1jz5tCuArSQJ2JJM0l9f2DAbLZ927Q3MAIiIvwF9Je1Q2DDzl7ak3wtYkvW6MimrIakXcBzwPwWMq7kabFfia8lX68cl9S9MaE2WT5v2ALaV9Kyk2ZLGFiy6psv3s0LSl4DDyXRANnb5tOt3wF5kfpA5FzgvIjYUJrwmy6ddc4DvAkgaBvQh88PTjVLa5tPPZxqI/wtcHBFVmQ7JJiGfdr1KZi6O1ZKOAB4Cdm/1yJounzZ1IDP356FAF2CmpL9ExN9bO7hmyGsqksRRwEsR8VErxtNS8mnXSKAcOATYDXhS0gsR8XFrB9cM+bTrGuA3ksrJ/DF7jY34G0zakn4+00AUA/cmCb8bcISk9RHxUGFCbJIG25X9HysiHpN0k6RuEbGxThiVz2dVCXwYEf8C/iXpeWAQsDEn/cZMRTKaTWNoB/Jr12nANZEZ/F4kaTGZMfBXChNik+T7f+s0gGToanHy2Di19UmFQj7I/JF7G+jHv0/K9K+n/iQ2jRO5DbYL2JF//xhvGPBO9euN8ZFnm/YiM5baAfgS8AYwoK1jb267knpdyYx5f7mtY27Bz+tm4IpkeQfgXaBbW8feAu3ahuSENHAGcFdbx13fI1U9/ahjGghJ45P1m9I4fo0823UCcJak9cAaYHQk/0o3Rvm0KSLmS/oz8Dqwgcyd2N5ou6gb1oh/g8cBT0TmW8xGL892XQ1MkjSXzLDJxbHxftME8m7XXsBdkqrIXE12epsFnAdPw2BmliJpu3rHzCzVnPTNzFLESd/MLEWc9M3MUsRJ38wsRZz0zZpA0hWSLmygzrGS9s56fZWkw1o/OrO6OembJSR1qO91ExxLZjIuACLiZxHxVDP3adYsTvq2WZI0NpnffI6kyZL6SJqRlM2QtEtSb5KkX0t6Bvhljte7SfpzMqHbC5L2zHGsMyTNSo71gKQvSfo6cDRwbTIv/m7Jvk9ItjlU0muS5kq6XVKnpLxC0pWSXk3WfeF4Zs3hpG+bnWQG0R8Dh0TEIOA8MjM83hUR+wBTgBuyNtkDOCwiLsjxuhQ4NyKGABcCN+U45PSIGJocaz5wekS8DDwCXBQRRRHxVlZ8nclM8XFiRAwk81P/s7L292FEDCYzbUG9Q0hmjeWkb5ujQ4D7q3/iH5lZKr8G3JOsnwwcmFX/voioqv1a0pbA14H7khkUbwF65jjegORbwFxgDNDQtNVfBRbHv2cDvRM4KGv99OR5NtC3gX2ZNUqq5t6x1BB1T1dcLXt97fltql+3A1ZERFED+5oEHBsRcySdSuYuZQ3FV5+1yXMV/j9qLcw9fdsczQD+Q9L2AMn9ZV8mM1UxZHrjLza0k8hMmbtY0qhkP5I0KEfVrYD3JXVM9l1tVbKutuq7K30leX0KsEnc39c2fU76ttmJiHnAROA5SXOAXwM/BE6T9DqZJHtenrsbA5ye7GceuW9t+FPgr8CTZBJ6tXuBi5ITtrtlxfcpmfnX70uGhDawad2pzTZhnmXTzCxF3NM3M0sRJ30zsxRx0jczSxEnfTOzFHHSNzNLESd9M7MUcdI3M0uR/w/qxGbRMp1/1gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# we already have the within-category inter-rater correlations\n",
    "# we just need to get the rows that are the opposite - where the two raters are in different categories\n",
    "different_category = df_pairwise_rater_correlations_with_categories['rater_category_1'] != df_pairwise_rater_correlations_with_categories['rater_category_2']\n",
    "df_pairwise_rater_correlations_different_category = df_pairwise_rater_correlations_with_categories[different_category]\n",
    "\n",
    "# plot the distributions of both types of inter-rater correlations\n",
    "sns.distplot(df_pairwise_rater_correlations_same_category['correlation'], kde=False, color='green', label='Same category')\n",
    "sns.distplot(df_pairwise_rater_correlations_different_category['correlation'], kde=False, color='red', label='Different category')\n",
    "plt.title(\"H-H inter-rater correlation\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>N raters</th>\n",
       "      <th>correlation</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>low</th>\n",
       "      <td>50</td>\n",
       "      <td>0.401045</td>\n",
       "      <td>3.826802</td>\n",
       "      <td>1.136638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>moderate</th>\n",
       "      <td>50</td>\n",
       "      <td>0.551284</td>\n",
       "      <td>3.832834</td>\n",
       "      <td>0.988477</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>average</th>\n",
       "      <td>50</td>\n",
       "      <td>0.648225</td>\n",
       "      <td>3.833874</td>\n",
       "      <td>0.914957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>high</th>\n",
       "      <td>50</td>\n",
       "      <td>0.801706</td>\n",
       "      <td>3.833890</td>\n",
       "      <td>0.828453</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          N raters  correlation      mean       std\n",
       "low             50     0.401045  3.826802  1.136638\n",
       "moderate        50     0.551284  3.832834  0.988477\n",
       "average         50     0.648225  3.833874  0.914957\n",
       "high            50     0.801706  3.833890  0.828453"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's also create a table summarizing the information about rater categories\n",
    "\n",
    "# select only the human rater columns for the scores data frame and create a longer version\n",
    "df_human_long = df_scores[rater_columns].melt(var_name='rater_id', value_name='score')\n",
    "\n",
    "# merge in the rater metadata\n",
    "df_human_with_metadata = pd.merge(df_human_long, df_rater_metadata)\n",
    "\n",
    "# get some descriptive statistics for the simulated rater score by rater category\n",
    "grouper = df_human_with_metadata.groupby('rater_category')\n",
    "df_human_score_descriptives = grouper['score'].describe()\n",
    "\n",
    "# also merge in the mean correlations for the table and add the number of raters per category\n",
    "df_descriptives_with_correlation = pd.concat([df_human_score_descriptives, simulated_mean_correlations], axis=1)\n",
    "df_descriptives_with_correlation['N raters'] = dataset.num_raters_per_category\n",
    "\n",
    "# now print out the table in the order we want\n",
    "human_rater_description_table = df_descriptives_with_correlation.loc[dataset.rater_categories,\n",
    "                                                               ['N raters', 'correlation', 'mean', 'std']]\n",
    "human_rater_description_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlation between simulated system performance and true scores\n",
    "\n",
    "The next thing we want to look at is the correlation between the various automated systems we have simulated in each category and the correlation of these systems' scores with the true scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAd3UlEQVR4nO3de5hdVZnn8e8vCYEiRIKkoBmKECRprm07dIm2OrZgA4mKNHgDHWGCkmYEguI44KW11ekRWxklwnQMGsELxsEmTWwJILRcxhupQO6ClOF2DI+kSGMgCQmpevuPvcscil1V6xS1zzk5+X2e5zx19l57nfPuTai31tprr6WIwMzMbKAxjQ7AzMyakxOEmZkVcoIwM7NCThBmZlbICcLMzAqNa3QAo2ny5MkxderURodhZrbLWLZsWU9EtBeVtVSCmDp1Kl1dXY0Ow8xslyHp0cHK3MVkZmaFnCDMzKxQaQlC0gJJT0paPUi5JM2V1C1ppaTjqspmSHowL7usrBjNzGxwZbYgrgVmDFE+E5iev2YD/wQgaSxwdV5+NHCWpKNLjNPMzAqUliAi4m5g4xCHnAZ8OzK/BCZJOgg4HuiOiHURsR1YmB9rZtYyenp6uOiii3jqqacaHcqgGnkP4mDg8artSr5vsP2FJM2W1CWpa8OGDaUEamY22q677jpWrlzJdddd1+hQBtXIBKGCfTHE/kIRMT8iOiOis729cCivmVlT6enpYcmSJUQES5YsadpWRCOfg6gAh1RtdwDrgfGD7DczK9XcuXNZsmRJzfW2bNnCSJdO2LZtG6effnry8ZLYe++9a/6emTNnMmfOnJrqNLIFsRg4Ox/N9FrgDxHxBLAUmC7pMEnjgTPzY83MrI5U1oJBkr4PvAmYDPwe+AywB0BEzJMk4CqykU5bgFkR0ZXXfQvwVWAssCAi/iHlOzs7O8NPUptZs7viiiu4+eabef7559ljjz1461vfyiWXXNKQWCQti4jOwrJWWlHOCcLMdgU9PT2ceeaZbN++nT333JOFCxey//77NySWoRKEn6Q2M6uzyZMnM3PmTCQxc+bMhiWH4bTUZH1mZruKc845h0ceeYRzzjmn0aEMygnCzKwBJk+ezNe+9rVGhzEkdzGZmVkhJwgzMyvkBGFmZoWcIMzMrJAThJmZFXKCMDOzQk4QZmZWyAnCzMwKOUGYmVkhJwgzMyvkBGFmZoWcIMzMrJAThJmZFXKCMDOzQk4QZmZWyAnCzMwKOUGYmVkhJwgzMyvkBGFmZoWcIMzMrJAThJmZFXKCMDOzQqUmCEkzJD0oqVvSZQXl+0laJGmlpHslHVtV9oikVZKWS+oqM04zM3uxcWV9sKSxwNXASUAFWCppcUSsrTrsE8DyiDhd0pH58W+uKj8hInrKitHMzAZXZgvieKA7ItZFxHZgIXDagGOOBu4AiIgHgKmSDiwxJjMzS1RmgjgYeLxqu5Lvq7YCOANA0vHAoUBHXhbAbZKWSZo92JdImi2pS1LXhg0bRi14M7PdXZkJQgX7YsD25cB+kpYDFwH3AzvystdHxHHATOACSW8s+pKImB8RnRHR2d7ePkqhm5lZafcgyFoMh1RtdwDrqw+IiE3ALABJAh7OX0TE+vznk5IWkXVZ3V1ivGZmViWpBSHpUEl/nb9vkzQxodpSYLqkwySNB84EFg/43El5GcAHgbsjYpOkCf3fIWkCcDKwOu2UzMxsNAzbgpB0HjAbeDlwOFlLYB4vHG30IhGxQ9KFwK3AWGBBRKyRdH5ePg84Cvi2pF5gLfCBvPqBwKKsUcE44PqIuKX20zMzs5FK6WK6gKx751cAEfGQpANSPjwibgZuHrBvXtX7XwDTC+qtA/485TvMzKwcKV1M2/JhqgBIGseLbzabmVmLSUkQd0n6BNAm6STgBuBH5YZlZmaNlpIgLgU2AKuAvyXrMvpUmUGZmVnjDXkPQtIYYGVEHAtcU5+QzMysGQzZgoiIPmCFpCl1isfMzJpEyiimg4A1ku4FNvfvjIi3lxaVmZk1XEqC+GzpUZiZWdMZNkFExF35DKuvznfdGxFPlhuWmZk12rCjmCS9G7gXeBfwbuBXkt5ZdmBmZtZYKV1MnwRe3d9qkNQO3A78sMzAzMyssVKegxgzoEvpqcR6Zma2C0tpQdwi6Vbg+/n2e4Al5YVkZmbNIOUm9ccknQG8gWwRoPkRsaj0yMzMrKFSpvs+DLg5Im7Mt9skTY2IR8oOzszMGiflXsINQF/Vdm++z8zMWlhKghhXPd13/n78EMebmVkLSEkQGyT9cVoNSacBPeWFZGZmzSBlFNP5wPckXUV2k/px4OxSozKzupk7dy7d3d0116tUKmzdurWEiF6sra2Njo6OmutNmzaNOXPmlBDR7iFlFNNvgddK2gdQRDxTflhmVi933nknPT3N3SmwefPmEcVYqVScIF6ClFFMFwPfAp4BrpF0HHBZRNxWdnBmVr5JkyaNqCWwbds2+vr6hj9wFIwZM4Y999yz5nqTJk0qIZrdR0oX07kRcaWkU4ADgFlkCcMJwqwFLFiwoNEhWJNKuUmt/OdbgG9FxIqqfWZm1qJSEsQySbeRJYhbJU3khc9FmJlZC0rpYvoA8CpgXURskbQ/WTeTmZm1sJRRTH3AfVXbT5HN6GpmZi2s1Gm7Jc2Q9KCkbkmXFZTvJ2mRpJWS7pV0bGpdMzMrV2kJQtJY4GpgJnA0cJakowcc9glgeUS8kuzhuytrqGtmZiVKShCS3iBpVv6+PZ/hdTjHA90RsS6fv2khcNqAY44G7gCIiAeAqfn61yl1zcysRClrUn8GuBT4eL5rD+C7CZ99MNm0HP0q+b5qK4Az8u85HjgU6Eis2x/fbEldkro2bNiQEJaZmaVIaUGcDrwd2AwQEeuBiQn1ip6ViAHblwP7SVoOXATcD+xIrEsez/yI6IyIzvb29oSwzMwsRcow1+0REZICQNKExM+uAIdUbXcA66sPiIhN5ENmJQl4OH/tPVxdMzMrV0oL4v9J+jowSdJ5wO3ANQn1lgLTJR0maTxwJrC4+gBJk/IygA8Cd+dJY9i6ZmZWrpTnIL4s6SRgE3AE8OmI+ElCvR2SLgRuBcYCCyJijaTz8/J5wFHAtyX1AmvJHsobtO6IztDMzEZEEYVd+7ukzs7O6OrqanQYZma7DEnLIqKzqCxluu9n2HmDeDzZKKbNEfGy0QvRzMyaTUoX0wtGLEn6G7LnFMzMrIXV/CR1RPwLcGIJsZiZWRNJ6WI6o2pzDNDJIM8kmJlZ60h5DuLUqvc7gEfwtBdmZi1vyASRT5q3MiK+Uqd4zMysSQx5DyIiesmm2TAzs91MShfTzyVdBfyAfD4mgIi4b/AqZma2q0tJEK/Lf36ual/gkUxmZi0taU3qiFhXvUPSK0qKx8zMmkTKcxA/LNh3w2gHYmZmzWXQFoSkI4FjgH0HPAvxMmCvsgMzM7PGGqqL6QjgbcAkXvgsxDPAeWUGZWZmjTdogoiIm4CbJP1lRPyijjGZmVkTGPYehJODmdnuqebJ+szMbPfgBGFmZoVSZnPdE3gHMLX6+Ij43GB1zMxs15fyoNxNwB+AZcC2csMxq4+5c+fS3d1dc71KpcLWrVtLiOjF2tra6OjoqLnetGnTmDNnTgkR2e4mJUF0RMSM0iMxq6Pu7m7uX3N/Noi7FpvJJr2vg2f7nmXD7zbUVunpcmKx3VPqZH1/FhGrSo/GrE4qlUr2i77WX6i91G+5rF5qT0Y78nMzGwVDPUm9iux/hXHALEnryLqYBEREvLI+IZqNvkmTJo2oq2jbtm309fWVENGLjRkzhj3H71lbpfHZuZmNhqFaEG+rWxRmdbZgwYJGh2DW9IZ6kvpRAEnfiYj3V5dJ+g7w/sKKZmbWElKegzimeiNfhvQvygnHzMyaxaAJQtLHJT0DvFLSpvz1DPAk2dDXYUmaIelBSd2SLiso31fSjyStkLRG0qyqskckrZK0XFLXCM7NzMxegqG6mL4AfEHSFyLi47V+cN7SuBo4CagASyUtjoi1VYddAKyNiFMltQMPSvpeRGzPy0+IiJ5av9vMzF66IdeDiIgHgBskHTewPGFN6uOB7v7V6CQtBE4DqhNEABMlCdgH2EjdRpmbmdlQhhrFdAkwG7iioCxlTeqDgcertivAawYccxWwGFgPTATeExH9YwgDuE1SAF+PiPlFXyJpdh4nU6ZMGSYkMzNLNVQX0+z85wkj/GwVfeyA7VOA5WTJ5nDgJ5LuiYhNwOsjYr2kA/L9D0TE3QVxzgfmA3R2dtbrESYzs5Y37CgmSfdI+of8hvPEGj67AhxStd1B1lKoNgu4MTLdwMPAkQARsT7/+SSwiKzLyszM6iRlmOs5wINkM7r+XFKXpK8k1FsKTJd0mKTxwJlk3UnVHgPeDCDpQLJlTtdJmtCfjCRNAE4GVqeckJmZjY5h52KKiHWStgLb89cJwFEJ9XZIuhC4FRgLLIiINZLOz8vnAZ8Hrs2n9RBwaUT0SHoFsCi7d8044PqIuGVEZ2hmZiOiiKG77SX9FugBrgfuAZZX3UhuKp2dndHV5UcmzMxSSVoWEZ1FZSldTHPJuoLOAuYA50g6fBTjMzOzJjRsgoiIKyPiXcBfky0a9PfAb0qOy8zMGixlydErgDeQPcj2C+DTZF1NZmbWwlIWDPol8I8R8fuygzEzs+aRMorphnoEYmZmzSXlJrWZme2GnCDMzKxQyj2I/qm7D6w+PiIeKysoMzNrvJRRTBcBnwF+D1TPtPrKEuMyM7MGS2lBXAwcERFPlR2MmZk1j5R7EI8Dfyg7EDMzay4pLYh1wJ2Sfgxs698ZEf+ntKjMzKzhUhLEY/lrfP4yM7PdQMqDcp8FyNdniIh4tvSozMys4VJWlDtW0v1kC/askbRM0jHlh2ZmZo2UcpN6PnBJRBwaEYcCHwWuKTcsMzNrtJQEMSEiftq/ERF3AhNKi8jMzJpC0igmSX8HfCff/q/Aw+WFZGZmzSClBXEu0A7cCCzK388qMygzM2u8lFFM/0621KiZme1GBk0Qkr4aER+W9COyuZdeICLeXmpkZmbWUEO1IPrvOXy5HoGYmVlzGTRBRMSy/O2rIuLK6jJJFwN3lRmYmZk1VspN6nMK9v23UY7DzMyazFD3IM4C3gscJmlxVdFEwFN/m5m1uKHuQfwceAKYDFxRtf8ZYGXKh0uaAVwJjAW+ERGXDyjfF/guMCWP5csR8a2UumZmVq6h7kE8CjwK/OVIPjhfpvRq4CSgAiyVtDgi1lYddgGwNiJOldQOPCjpe0BvQl0zMytRymR9r5W0VNKzkrZL6pW0KeGzjwe6I2JdRGwHFgKnDTgmgImSBOwDbAR2JNY1M7MSpdykvgo4C3gIaAM+CHwtod7BZKvR9avk+wZ+9lHAemAVcHFE9CXWBUDSbEldkro2bNiQEJaZmaVISRBERDcwNiJ683sEJyRUU9FHDdg+BVgO/CfgVcBVkl6WWLc/tvkR0RkRne3t7QlhmZlZipTJ+rZIGg8sl/SPZDeuU2ZzrQCHVG13kLUUqs0CLo+IALolPQwcmVjXzMxKlJIg3k82kuhC4CNkv7jfkVBvKTBd0mHA74AzyYbNVnsMeDNwj6QDgSPI1sB+OqGu1Wju3Ll0d3fXXK9SqbB169YSInqxtrY2Ojo6aq43bdo05szxlGFmoyllsr5H87dbgc+mfnBE7JB0IXArWYJZEBFrJJ2fl88DPg9cK2kVWbfSpRHRA1BUN/20rEh3dze/WX0fU/bprale75Yx9PUW9fqNvt7nN/HcI0/UVOexZ8eWFI3Z7m2oB+VWMUi/P0BEvHK4D4+Im4GbB+ybV/V+PXByal17aSqVCjHof9HBHbh33+gHM4oisnMzs9E1VAvibXWLwupmW6949Jna/uJ+vk/0jSCxjMQYwR5javuybb3yEodmJRjuQTkAJB0KTI+I2yW1DVXPmteb3vSmlr4HYWaja9hf9JLOA2YDLwcOJxtRNI/s5rLtQnwT18xqkfIcxAXA64FNABHxEHBAmUGZmVnjpSSIbfl0FwBIGscQN6/NzKw1pCSIuyR9AmiTdBJwA/CjcsMyM7NGS0kQlwIbyOZK+luyoaefKjMoMzNrvCFvUksaA6yMiGOBa+oTkpmZNYMhWxD5zKorJE2pUzxmZtYkUp5nOAhYI+leYHP/zoh4e2lRmZlZw6UkiOT5l8zMrHWk3IO4Or8HYWZmuxHfgzAzs0K+B2FmZoV8D8LMzAqlLBh0V77a26vzXfdGxJPlhmVmZo027JPUkt4N3Au8C3g38CtJ7yw7MDMza6yULqZPAq/ubzVIagduB35YZmBmZtZYKXMxjRnQpfRUYj0zM9uFpbQgbpF0K/D9fPs9wJLyQjIzs2aQcpP6Y5LOAN4ACJgfEYtKj8zMzBpq0AQhaRpwYET8LCJuBG7M979R0uER8dt6BWlmZvU31L2ErwLPFOzfkpeZmVkLGypBTI2IlQN3RkQXMLW0iMzMrCkMlSD2GqKsbbQDMTOz5jJUglgq6byBOyV9AFiW8uGSZkh6UFK3pMsKyj8maXn+Wi2pV9LL87JHJK3Ky7pST8jMzEbHUKOYPgwskvQ+diaETmA8cPpwHyxpLHA1cBJQIUs4iyNibf8xEfEl4Ev58acCH4mIjVUfc0JE9NRwPmZmNkoGTRAR8XvgdZJOAPrXg/hxRPxb4mcfD3RHxDoASQuB04C1gxx/FjuftTAzswZLeQ7ip8BPR/DZBwOPV21XgNcUHShpb2AGcGH1VwO3SQrg6xExf5C6s4HZAFOmeNkKM7PRUuaUGSrYF4MceyrwswHdS6+PiOOAmcAFkt5YVDEi5kdEZ0R0tre3v7SIzczsj8pMEBXgkKrtDmD9IMeeyYDupYhYn/98ElhE1mVlZmZ1UmaCWApMl3SYpPFkSWDxwIMk7Qv8FXBT1b4Jkib2vwdOBlaXGKuZmQ2QMlnfiETEDkkXArcCY4EFEbFG0vl5+bz80NOB2yJic1X1A8lGUPXHeH1E3FJWrGZm9mKKGOy2wK6ns7Mzurr8yISZWSpJyyKis6jM6zqYmVkhJwgzMyvkBGFmZoWcIMzMrJAThJmZFXKCMDOzQk4QZmZWyAnCzMwKOUGYmVkhJwgzMyvkBGFmZoWcIMzMrJAThJmZFXKCMDOzQk4QZmZWyAnCzMwKOUGYmVkhJwgzMyvkBGFmZoXGNTqAZjV37ly6u7trqlOpVADo6Oioqd60adOYM2dOTXXMzMrW8gni3HPP5Yknnqi53rZt2+jr66upTv/xGzdurKne6tWrWbJkSU11AA466CAWLFhQcz0zsxQtnyCefvppnt28BcbWeqoCja25CkBvjfV6A55/bntt39W7g6effrq2OmZmNWj5BNHR0cHvt43juaPf1uhQRtVea/+Vjo4/aXQYZtbCWj5BAIzZspG91v5r6d+j5zYBEHu9rPTvGrNlI+AEYWblafkEMW3atBHVq1QqbN26taY6W3ufA6CtTzXVa2trq/nGNvzJiM/NzCxFqQlC0gzgSmAs8I2IuHxA+ceA91XFchTQHhEbh6ubaqSjgzyKycx2d4qIcj5YGgv8BjgJqABLgbMiYu0gx58KfCQiTqy1br/Ozs7o6uoaxbMwM2ttkpZFRGdRWZkPyh0PdEfEuojYDiwEThvi+LOA74+wrpmZjbIyE8TBwONV25V834tI2huYAfzzCOrOltQlqWvDhg0vOWgzM8uUmSCK7tQO1p91KvCziOh/wiy5bkTMj4jOiOhsb28fQZhmZlakzARRAQ6p2u4A1g9y7Jns7F6qta6ZmZWgzASxFJgu6TBJ48mSwOKBB0naF/gr4KZa65qZWXlKG+YaETskXQjcSjZUdUFErJF0fl4+Lz/0dOC2iNg8XN2yYjUzsxcrbZhrI3iYq5lZbRo1zNXMzHZhLdWCkLQBeLTBYUwGehocQ7PwtdjJ12InX4udmuFaHBoRhUNAWypBNANJXYM113Y3vhY7+Vrs5GuxU7NfC3cxmZlZIScIMzMr5AQx+uY3OoAm4muxk6/FTr4WOzX1tfA9CDMzK+QWhJmZFXKCMDOzQk4QIyRphqQHJXVLuqygXJLm5uUrJR3XiDjrYbhrUXXcqyX1SnpnPeMrU8q5S3qTpOWS1ki6q2r/I5JW5WUtNwVAwv8j+0r6kaQV+bWZ1Yg4y5Bw7vtJWpT/brhX0rFVZRdLWp1fkw/XN/IBIsKvGl9k80P9FngFMB5YARw94Ji3AEvIpi5/LfCrRsfdqGtRddy/ATcD72x03HX8dzAJWAtMybcPqCp7BJjc6PNo4LX5BPDF/H07sBEY3+jY63TuXwI+k78/Ergjf38ssBrYm2yuvNuB6Y06F7cgRiZlxbvTgG9H5pfAJEkH1TvQOkhd/e8isgWhnqxncCVLOff3AjdGxGMAEdFK5z+UlGsTwERJAvYhSxA76htmKVLO/WjgDoCIeACYKulA4CjglxGxJSJ2AHeRTWjaEE4QI5Oy4l3yqni7uGHPU9LBZP/I59FaUv4b/ymwn6Q7JS2TdHZVWQC35ftnlxxrvaVcm6vIfiGuB1YBF0dEX33CK1XKua8AzgCQdDxwKNm6N6uBN0raP19p8y28cG2cuiptuu8Wl7LiXS0r6u3KUs7zq8ClEdGb/bHYMlLOfRzwF8CbgTbgF5J+GRG/AV4fEeslHQD8RNIDEXF3uSHXTcq1OQVYDpwIHE52De6JiE1lB1eylHO/HLhS0nKy5Hg/sCMifi3pi8BPgGfJEknDWlVOECOTsuLd7rIqXsp5dgIL8+QwGXiLpB0R8S/1CbE0qf8OeiJb72SzpLuBPwd+ExHrIet2krSIrGuiVRJEyrWZBVweWed7t6SHyfrj761PiKUZ9tzzJDgLsgEtwMP5i4j4JvDNvOx/55/XEO5iGpmUFe8WA2fno5leC/whIp6od6B1MOy1iIjDImJqREwFfgh8qAWSA6T9O7gJ+C+SxuVdBq8Bfi1pgqSJAJImACeTdS+0ipRr8xhZy4q8//0IYF1doyzHsOcuaVJeBvBB4O7+llPeokTSFLJuqOrlmOvKLYgRiLTV8m4m6z/sBraQ/7XQahKvRUtKOfe8y+AWYCXQB3wjIlZLegWwKG9VjQOuj4hbGnMmoy/x38XngWslrSLrlrk0Iho99fVLlnjuRwHfltRLNsrtA1Uf8c+S9geeBy6IiH+v7xns5Kk2zMyskLuYzMyskBOEmZkVcoIwM7NCThBmZlbICcLMzAo5QVjLymeOXV71GnSm2VH8zkmSPjSCen8v6X8Msv93efxrJZ1VcMx7JW2X9KkB+0/Kp/FYlf88sda4bPfm5yCslW2NiFfV+TsnAR8C/u8ofuZXIuLLkqYDyyT9MCKeB8h/6f9PssnfvimpEhHX5vV6gFPz6TyOJRuX34rzgVlJ3IKw3Uq+BsGDko7It78v6bz8/bOSrpB0n6Q7JLXn+w+XdEv+V/g9ko7M9x+Yz+m/In+9jmyOncPzv/i/lB/3MUlLlc39/9mqWD6Zx3I72VPEQ4qIh8geutwvr/9nwP8CTomIbrIHM98r6ZT8+Pv7p/MA1gB7SdrzJV9E2224BWGtrC2fDK3fFyLiB/lTrtdKuhLYLyKuycsnAPdFxEclfRr4DHAh2cLy50fEQ5JeQ9Y6OBGYC9wVEadLGks2ZfVlwLH9LRdJJwPTyeZZErBY0huBzWRTMPxnsv8P7wOWDXUyyhadeqh/yvCIWAW8rr88n+/p5EGqvwO4PyK2DXPNzP7ICcJaWWEXU0T8RNK7gKvJJs7r1wf8IH//XeBGSfuQ/RK+QTtnou3/K/xE4Oz8M3uBP0jab8DXnZy/7s+39yFLGBOBRRGxBUDSwHmKqn0kb+W8Apgx5BkXkHQM8EUGTx5mhZwgbLcjaQzZXDhbgZcz+GyZQdYN+/RLuJchspbL1wfE8GHSp3/vvwdxBtn8PYdHxHNJXy51AIuAsyPit7UEbuZ7ELY7+gjwa+AsYIGkPfL9Y4D+9bLfC/z/fIbNh/MWR/9a4/2tjjuA/57vHyvpZcAzZK2DfrcC5+YtESQdnM/WeTdwuqS2fFbXU4cLOiJuBLqAc1JOUtIk4MfAxyPiZyl1zKo5QVgraxswzPVySX9KNr3yRyPiHrJf1P3DQzcDx0haRtZ99Ll8//uAD0haQXazt3/5yIuBE/LZSJcBx0TEU8DPlC06/6WIuA24nmyhoFVk051PjIj7yLqzlpMtxXpP4jl9DrgkbwUN50JgGvB3VdfggMTvMfNsrmb9JD0bEfs0Og6zZuEWhJmZFXILwszMCrkFYWZmhZwgzMyskBOEmZkVcoIwM7NCThBmZlboPwCNTdbo3i1ueQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# first we extract all of the columns containing scores from human raters\n",
    "# these columns all have names of the form `sys_X`; note that we also need\n",
    "# the true score column\n",
    "system_columns = [column for column in df_scores.columns if column.startswith('sys_')]\n",
    "\n",
    "# get the correlations we are interested in\n",
    "df_system_true_score_correlations = df_all_correlations[system_columns + ['true']]\n",
    "\n",
    "# extract only the correlations with the true score\n",
    "df_true_score_correlations = df_system_true_score_correlations['true'].reset_index(name='true_score_correlation')\n",
    "\n",
    "# now merge these with the system metadata so we can easily make the boxplot we need\n",
    "df_system_metadata_with_true_score_correlations = pd.merge(df_system_metadata,\n",
    "                                                           df_true_score_correlations,\n",
    "                                                           left_on='system_id',\n",
    "                                                           right_on='index')\n",
    "\n",
    "p = sns.boxplot(x='expected_r2_true', y='true_score_correlation', data=df_system_metadata_with_true_score_correlations)\n",
    "p.set_xlabel('Expected R^2')\n",
    "p.set_ylabel('Correlation with true score')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the systems with higher performance are more correlated with the underlying true score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RMSE and $R^2$ between system scores and true scores\n",
    "\n",
    "In a similar vein to the previous analysis, we can also compute the averege $R^2$ and RMSE values between the simulated system scores in each system category and the true scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAFgCAYAAABtzdQiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de7hcdX3v8feHhHAJ4VJJrAIKtGAOiEXdeqwiTY+oaFuojx4B22O1tegpXntotT2tpWgvHlsvKBYpRWprpdbqKd5QwAIVpRKRW0RCDoikUIyKgNxy+54/1grZ7Oyd7CT7tyd79vv1PPPMrN+sWfOd2bPnM791+a1UFZIktbTToAuQJA0/w0aS1JxhI0lqzrCRJDVn2EiSmps76AK21rHHHlsXXnjhoMuQpMnIoAvYUcy4ns33v//9QZcgSdpKMy5sJEkzj2EjSWrOsJEkNWfYSJKaM2wkSc0ZNpKk5pqFTZJzk3wvyQ0T3J8kZyRZkeS6JE9rVYskabBa9mzOA47dzP0vAg7pLycDf9WwFknSADULm6q6HPjhZmY5Hvhoda4E9k7yuFb1SJIGZ5DD1ewH3D5qemXfdudgypGkRzv2vZfy7bvuf2R68WPnc+FblgyuoBlskDsIjDdm0LinDU1ycpKlSZauWrWqcVmStGnQAHz7rvs59r2XDqagGW6QYbMSOGDU9P7AHePNWFVnV9VIVY0sXLhwWoqTNLuNDZottWvzBhk2FwCv7PdKexZwT1W5Ck2ShlCzbTZJPg4sAfZNshL4I2BngKo6C/g88GJgBfAA8OpWtUiSBqtZ2FTVSVu4v4BTWj2/JG2PxY+dP+4qs8WPnT+AamY+RxCQpHFc+JYlmwSLe6Ntuxl3pk5Jmi4Gy9SxZyNJas6wkSQ1Z9hIkpozbCRJzRk2kqTmDBtJUnOGjSSpOcNGktScYSNJas6wkSQ1Z9hIkpozbCRJzRk2kqTmDBtJUnOGjSSpOcNGktScYSNJas6wkSQ1Z9hIkpozbCRJzRk2kqTmDBtJUnOGjSSpOcNGktScYSNJas6wkSQ1Z9hIkpozbCRJzRk2kqTmDBtJUnOGjSSpOcNGktScYSNJas6wkSQ1Z9hIkpozbCRJzRk2kqTmDBtJUnOGjSSpOcNGktScYSNJas6wkSQ1Z9hIkpozbCRJzc0ddAGSdhxnXLycc75yK/evXsf8eXN4zVEH8cZjDh10WRoCTXs2SY5NclOSFUneNs79eyX5TJJrkyxL8uqW9Uia2BkXL+c9F9/MvQ+tZd364t6H1vKei2/mjIuXD7o0DYFmYZNkDnAm8CLgMOCkJIeNme0U4FtV9TPAEuAvk8xrVZOkiX3wyyu2ql3aGi17Ns8EVlTVLVW1GjgfOH7MPAUsSBJgD+CHwNqGNUmawOr1tVXt0tZoGTb7AbePml7Zt432QeC/AHcA1wNvqqr1YxeU5OQkS5MsXbVqVat6JUmNtAybjNM29ifSC4FrgMcDRwIfTLLnJg+qOruqRqpqZOHChVNfqSSpqZZhsxI4YNT0/nQ9mNFeDXyqOiuAW4HFDWuSNIH999plq9qlrdEybK4CDklyUL/R/0TggjHzfBd4HkCSxwJPAm5pWJOkCbzzJU9hwS5z2KlfJ7FTYMEuc3jnS54y2MI0FJodZ1NVa5O8HvgiMAc4t6qWJXldf/9ZwDuA85JcT7fa7a1V9f1WNUma2JLFi/jASU/jw5ffwsq7H2D/fXbntUcfzJLFiwZdmoZAqmbWniYjIyO1dOnSQZchSZMx3rbrWcnhaiRJzRk2kqTmDBtJUnOGjSSpOcNGktScYSNJas6wkSQ1N/wnT1t+EXz1/fCj22DvJ8Kz3wSHPn/QVUnSrDLcPZvlF8EXToX77oJd9+muv3Bq1y5JmjbDHTZffT+sWQ333QGrbuyu16zu2iVJ02a4V6N970Z44Ac8cmaDdath9f2wfs1Ay5Kk2Wa4ezYP3cOmp9ApeOhHg6hGkmat4Q6biXow9mwkaVoNd9hIknYIQx42E43u7ajfkjSdhjtsjnj51rVLkpoY7r3RXnp2d73sk7B+Hew0Bw5/2cZ2SdK08EydktSO6+x7w70aTZK0QzBsJEnNGTaSpOYMG0lSc4aNJKk5w0aS1JxhI0lqzrCRJDVn2EiSmjNsJEnNGTaSpOYMG0lSc4aNJKm54T7FgLQFZ1y8nHO+civ3r17H/HlzeM1RB/HGYw4ddFnS0DFsNGudcfFy3nPxzY9M3/vQ2kemDRxparkaTbPWB7+8YqvaJW07w0az1ur14584cKJ2SdvOsJEkNWfYaNaa6Hy9nsdXmnqGjWat+buM//HfY4J2SdvO/yrNWkfstw977zaXnfquzE6BvXeby5P322ewhUlDyLDRrPXaow9mz93mcdC+83ny4/fkoH3ns+du83jt0QcPujRp6Bg2mrWWLF7E6ccdzqIFu3LPg2tYtGBXTj/ucJYsXjTo0qSh40GdmtWWLF5kuEjTwJ6NJKk5w0aS1JxhI0lqzrCRJDVn2EiSmmsaNkmOTXJTkhVJ3jbBPEuSXJNkWZLLWtYjSRqMZrs+J5kDnAk8H1gJXJXkgqr61qh59gY+BBxbVd9N4j6okjSEWvZsngmsqKpbqmo1cD5w/Jh5XgF8qqq+C1BV32tYjyRpQFqGzX7A7aOmV/Ztox0K7JPk0iTfSPLK8RaU5OQkS5MsXbVqVaNyJUmttAyb8UZqH3tWqrnA04FfAF4I/GGSTc7HW1VnV9VIVY0sXLhw6iuVJDXVcrialcABo6b3B+4YZ57vV9X9wP1JLgd+BljesC5J0jRr2bO5CjgkyUFJ5gEnAheMmedfgOcmmZtkd+C/Ajc2rEnLL4LzfhHed0R3vfyiQVckaRZoFjZVtRZ4PfBFugD5RFUtS/K6JK/r57kRuBC4Dvg6cE5V3dCqpllv+UXwhVPhvrtg13266y+cauBIai5VYzej7NhGRkZq6dKlgy5jZjrvF+GuZfDQPVDrIHNg173gsYfDqz476OqkYeRZxnueYmA2ufM6ePiejdO1Dh78YdcuSQ05XM1sMjpoJtMuSVPEsJEkNWfYSJKaM2xmkzm7bF27JE0Rw2Y2ee7/YtOdY9K3S1I77o02myx5a3d95Znw8I9hlz3gWadsbJekRjzORpLa8TibnqvRJEnNGTaSpOYMG0lSc4aNJKk5w0aS1JxhI0lqbothk+SxSf4myRf66cOS/Eb70iRJw2IyPZvz6E6A9vh+ejnw5lYFSZKGz2TCZt+q+gSwHh45A+e6plVJkobKZMLm/iSPAQogybMAT4AiSZq0yYyN9tvABcBPJbkCWAi8rGlVkqShssWwqaqrk/wc8CS6cX5uqqo1zSuTJA2NLYZNkleOaXpaEqrqo41qkiQNmcmsRnvGqNu7As8DrgYMG0nagSU5Enh8VX1+gvtHgFdW1Rtb1zKZ1WhvGD2dZC/g75pVJEmaKkcCI8AmYZNkblUtBablnC3bMoLAA8AhU12IJGlTSQ5M8u0k5yS5IcnHkhyT5IokNyd5ZpL5Sc5NclWSbyY5Psk84HTghCTXJDkhyWlJzk7yJeCjSZYk+Wz/PHsk+UiS65Ncl+SlU/k6JrPN5jP0uz3ThdNhwCemsghJ0mb9NPDfgZOBq4BXAEcBxwG/D3wL+HJV/XqSvYGvAxcDbwdGqur1AElOA54OHFVVDyZZMuo5/hC4p6qO6OfdZypfwGS22fzFqNtrgduqauVUFiFJ2qxbq+p6gCTLgEuqqpJcDxwI7A8cl+TUfv5dgSdMsKwLqurBcdqPAU7cMFFVd09V8TC5bTaXTeUTSpK22sOjbq8fNb2e7nt8HfDSqrpp9IOS/NdxlnX/BM8RNq7FmnITbrNJcl+Se8e53Jfk3lYFSZK22heBNyQJQJKn9u33AQsmuYwvAa/fMDHVq9EmDJuqWlBVe45zWVBVe05lEZKk7fIOYGfguiQ39NMA/woctmEHgS0s453APv1OCNcCPz+VBaZqcr2mJIvo1gMCUFXfncpCJmtkZKSWLp2WPfUkaXtl0AXsKCZzPpvjktwM3ApcBnwH+ELjuiRJQ2Qyx9m8A3gWsLyqDqIbQeCKplVJkobKZMJmTVX9ANgpyU5V9a90R6VKkjQpkznO5kdJ9gD+DfhYku/RHW8jSdKkTKZnczmwN/Am4ELg/wG/1LIoSdJwmUzPJnT7cP8QOB/4x361mjTzLb8Ivvp++NFtsPcT4dlvgkOfP+iqpKGzxZ5NVf1xVR0OnAI8HrgsycXNK5NaW34R/MtvwcqlcO+d3fW//FbXLg2pJLsn+Vw/uOeyJH8+Hc+7NaM+fw/4T+AHwKI25UjT6OI/gvtXwdoHYf2a7vr+VV27NLwCvKeqFgNPBZ6T5EWtn3Qyoz7/T+AEYCHwSeA3q+pbrQuTmlt1I5sOBVV9uzQ4B77tc8cCvwMcRHeM47u/8+e/cOG2Li/JgXTHR/4r8LPALwNU1eokV9MN5NnUZHo2TwTeXFWHV9UfGTQaGrV+69qladAHzZnA4+i2lT8OOLNv3x5PAj5aVU+tqtsA+tMR/BJwyXYue4sms83mbVV1TetCJElA16N5mO5ElfTXD/ft2+O2qrpyw0SSucDHgTOq6pbtXPYWbcuZOqUhMdGwVQ5npYE6iI1Bs8EDffv2GHtqgbOBm6vqfdu53EkxbDR7HfHyrWuXpsetwO5j2nbv26dEkncCewFvnqplbolho9nrpWfDESfATnO66Z3mdNMvPXuwdWm2ezewCxsDZ/d++t1TsfAk+wP/GzgMuLo//cBrpmLZm33eyZ5iYEfhKQa2z1vOv5oLrvtP1q0v5uwUjnvKT/LeE5826LKkYbVN62Snem+0HYFhM4u85fyr+fQ1d27S/pIjH2fgSG24AbDXdDVakmOT3JRkRZK3bWa+ZyRZl+RlLeuZ7f7l2k2DZnPtkjRVmoVNkjl0+4q/iG7d4ElJDptgvnfRjb+mhtZP0ImdqF2SpkrLns0zgRVVdUtVraYbxPP4ceZ7A/DPdMPhSJKGUMuw2Q+4fdT0yr7tEUn2A14CnLW5BSU5OcnSJEtXrVo15YXOFo/ZffzRiSZql6Sp0jJsxtswNnaFzfuAt1bVus0tqKrOrqqRqhpZuHDhlBU42/zas8c/JmyidkmaKi1/0q4EDhg1vT9wx5h5RoDzkwDsC7w4ydqq+r8N65q1vnbLD3ncXrtw74NrWb1uPfPm7MSeu83la7f8kDcOujhJQ61l2FwFHJLkIOA/gBOBV4yeoaoe+Umd5DzgswZNO7ff/QCPmb8L++6x6yNtVcXKu8eOjCFpWKX7dZ+q6R1xtlnYVNXaJK+n28tsDnBuVS1L8rr+/s1up9HUO2Cf3fnefQ+x+7yNf/YH16xj/33GjowhaaBO22uTgzo57Z6pOsXAScCPklwCPAu4FvgI8Md05yr7lar6epKfA97fL6KAo6vqviS/A7ycblSDT1fVpE4A1fQ4m6r6fFUdWlU/VVV/0redNV7QVNWrquqTLeuZ7V579MGsWVc8sHotVd31mnXFa48+eNClSdqgC5pNTjHQt2+PJwEfpTth2gF0QfIUYDHdWqejgFOB3+/nPxU4paqOBJ4LPJjkBcAhdHsbHwk8PcnRk3lyx0abRZYsXsTpxx3OogW7cs+Da1i0YFdOP+5wliz2xKvSDmQ6TjFwa1Vd369KWwZcUt1wMtcDB/bzXAG8J8kbgb2rai3wgv7yTeBquqA6ZDJP7j6vs8ySxYsMF2nHdhBdj2a0qT7FwMOjbq8fNb2ePheq6s+TfA54MXBlkmPo9jL+s6r68NY+uT0bSdqxND/FwGQk+am+9/MuYCldL+aLwK8n2aOfZ78kk/r1as9GknYs76bbZgNdj2ZKTzGwFd6c5OeBdcC3gC9U1cNJ/gvwtf6QlR8Dv8okRoBx1GdJamfbRn2e4r3RdgT2bCRpR9MFy4wOl7HcZiNJas6wkSQ1Z9hIkpozbCRJzbmDgKSNll8EX30//Og22PuJ8Ow3waHPH3RVGgKGjaTO8ovg/FfA+tXd9I++C9/9dzjxHwwcbTdXo0nqfPLXNwbNButXd+3SdjJsJHVW37t17dJWMGwkSc0ZNpKk5gwbSZ0nTnAOrInapa1g2EjqvPozmwbLE4/u2qXt5K7PkjYyWNSIPRtJUnOGjSSpOcNGktScYSNJas6wkSQ1Z9hIkpozbCRJzRk2kqTmDBtJUnOGjSSpOcNGktScYSNJas6wkSQ1Z9hIkpozbCRJzRk2kqTmDBtJUnOGjSSpOcNGktScYSNJas6wkSQ1Z9hIkpozbCRJzRk2kqTmDBtJUnOGjSSpOcNGktScYSNJaq5p2CQ5NslNSVYkeds49/9Kkuv6y1eT/EzLeiRJg9EsbJLMAc4EXgQcBpyU5LAxs90K/FxVPQV4B3B2q3okSYPTsmfzTGBFVd1SVauB84HjR89QVV+tqrv7ySuB/RvWI0kakJZhsx9w+6jplX3bRH4D+MJ4dyQ5OcnSJEtXrVo1hSVKkqZDy7DJOG017ozJz9OFzVvHu7+qzq6qkaoaWbhw4RSWKEmaDnMbLnslcMCo6f2BO8bOlOQpwDnAi6rqBw3rkSQNSMuezVXAIUkOSjIPOBG4YPQMSZ4AfAr4H1W1vGEtkqQBatazqaq1SV4PfBGYA5xbVcuSvK6//yzg7cBjgA8lAVhbVSOtapIkDUaqxt2MssMaGRmppUuXDroMSZqM8bZdz0qOICBJas6wkSQ1Z9hIkpozbCRJzRk2kqTmDBtJUnOGjSSpOcNGktScYSNJas6wkSQ1Z9hIkpozbCRJzRk2kqTmDBtJUnOGjSSpOcNGktScYSNJas6wkSQ1Z9hIkpozbCRJzRk2kqTmDBtJUnOGjSSpOcNGktScYSNJas6wkSQ1N3fQBUjSDuvM58CqGzZOL3wynHLF4OqZwezZSNJ4xgYNdNNnPmcw9cxwho0kjWds0GypXZtl2EiSmjNsJEnNGTaSNJ6FT966dm2WYSNJ4znlik2Dxb3Rtpm7PkvSRAyWKWPPRpLUnGEjSWrOsJEkNWfYSJKaM2wkSc0ZNpKk5gwbSVJzho0kqbmhP6jzjIuXc85XbuX+1euYP28OrznqIN54zKGDLkuSZpWhDpszLl7O+y65mfXVTd/70Fred8nNAAaOJE2joV6NdtZlt7C+IGy8rK+uXZI0fYa6Z/PAmnUA1ATtkqTp0bRnk+TYJDclWZHkbePcnyRn9Pdfl+RpLeuRJA1Gs7BJMgc4E3gRcBhwUpLDxsz2IuCQ/nIy8FdTWcMuc8d/eRO1S5LaaPmt+0xgRVXdUlWrgfOB48fMczzw0epcCeyd5HFTVcCCXeeSMW3p2yVJ06dl2OwH3D5qemXftrXzkOTkJEuTLF21atWkCzhk0QJ+cq9dmD9vDjvPCfPnzeEn99qFQxYtmPyrkCRtt5ZhM7ZTAZtuq5/MPFTV2VU1UlUjCxcunHQBrz36YNauK9atL6q667XritceffCklyFJ2n4tw2YlcMCo6f2BO7Zhnu1SAIEkkHGSTJLUXMuwuQo4JMlBSeYBJwIXjJnnAuCV/V5pzwLuqao7p6qAD19+C3vttjOHLFrA4p/ck0MWLWCv3Xbmw5d7nI0kTadmW8qram2S1wNfBOYA51bVsiSv6+8/C/g88GJgBfAA8OqprOH2ux9g7912flTbbjvPYeXdD0zl00iStqDpbllV9Xm6QBnddtao2wWc0ur5D9hnd75330PsPm/jy3xwzTr232f3Vk8pSRrHUB9w8tqjD2bNuuKB1Wup6q7XuIOAJE27oQ6bJYsXcfpxh7Nowa7c8+AaFi3YldOPO5wlixcNujRJmlWG/ujGJYsXGS6SNGBD3bORJO0YDBtJUnOGjSSpOcNGktScYSNJas6wkSQ1Z9hIkpozbCRJzRk2kqTm0o2FOXMkWQXctg0P3Rf4/hSXM5P5fjya78ej+X482ra+H9+vqmOnupiZaMaFzbZKsrSqRgZdx47C9+PRfD8ezffj0Xw/tp+r0SRJzRk2kqTmZlPYnD3oAnYwvh+P5vvxaL4fj+b7sZ1mzTYbSdLgzKaejSRpQAwbSVJzhs2QSvLjQdcwkyS5NMlIf/vzSfYedE0tJDkwyQ3jtJ+e5JgtPPa0JKe2q27HlWRhkn9P8s0kz93Kxx6Z5MWtapsphv600NsiyZyqWjfoOjQYVTXrvhiq6u2DrmFHlWQu8Dzg21X1a9uwiCOBEeDzU1rYDDPjezb9L7VvJ/nbJNcl+WSS3ZM8r/8Vcn2Sc5Ps0s8/Uft3krw9yVeA/z7QFzWF0nl3khv613xC3/6hJMf1tz+d5Nz+9m8keecga56sUX/7c/rX97EkxyS5IsnNSZ6ZZH7/d76q/7sf3z92tyTn95+ZfwR2G7Xc7yTZd2wvIMmpSU7rb1+a5L1JLk9yY5JnJPlU/7w7+vs3J8lfJ1mW5Ev9e3FekpcBJHlx/75+JckZST476rGH9a/9liRvHFD9W20z3xNPT3JZkm8k+WKSx/XzX5rkT5NcBrwJ+D/Ai5Nc079fL0jytSRXJ/mnJHv0j3tGkq8muTbJ15PsBZwOnNA/9oSBvQmDVlUz+gIcCBTwnH76XOAPgNuBQ/u2jwJvBnYdr72//R3gdwf9eqbwfflxf/1S4CJgDvBY4LvA44ATgXf383wduLK//RHghYOufyv+9muBI+h+OH2j//sHOB74v8CfAr/az783sByYD/w2cG7f/pR+OSOjPgv79su/YdTznQqc1t++FHhXf/tNwB39+7oLsBJ4zKDfny28Z0f2058AfhU4D3jZqP+Rg/r7Pw58tr99GvDV/jXuC/wA2HnQr2krXvfY74nf6V/Pwr7thFGfiUuBD416/KuAD/a39wUuB+b3028F3g7MA24BntG370m39uiRx87my4zv2fRur6or+tt/T9flvbWqlvdtfwscDTxpgvYN/nE6ip1mRwEfr6p1VXUXcBnwDODfgOcmOQz4FnBX/6vuZ+n+AWeKW6vq+qpaDywDLqnuP/16ui+YFwBvS3IN3RfIrsAT6P7ufw9QVdcB123Dc1/QX18PLKuqO6vqYbovnAO2+RW1d2tVXdPf/gbd+7TBYuCWqrq1n/74mMd+rqoerqrvA9+j+wEzU4z9nngh8GTgov7z8QfA/qPmn+j74FnAYcAV/eN+DXgi3ffLnVV1FUBV3VtVa6f+ZcxMw7LNZrIHC2UL99+/vYXsgMZ9zVX1H0n2AY6l+5X2E8DL6XpE901jfdvr4VG314+aXk/3+V4HvLSqbhr9oCSw5c/NWh69qnnXCZ579POOfu4d1eha1zFqFSJb/h8Z+9gd+XWONfbvfR/dj4SfnWD+ib4PAlxUVSc9qjF5yjjPod6w9GyekGTDB+Yk4GLgwCQ/3bf9D7pf9N+eoH2YXU63vnhOkoV0v+i/3t/3NbrVi5fT9XRO7a+HyReBN6RPlyRP7dsvB36lb3sy3aq0se4CFiV5TL9t7xenod5B+zZwcJID++lh2sYw9nviSmDhhrYkOyc5fBLLuRJ4zobvkX7bz6F0793jkzyjb1+QbueC+4AFU/xaZpxhCZsbgV9Lch3dL/T3Aq8G/inJ9XS/NM+qqofGax9QzdPl03SriK4Fvky3Xeo/+/v+DZhbVSuAq+neu2ELm3cAOwPX9Rv739G3/xWwR/+Z+V02BvAjqmoN3cbdfwc+S/dlMtSq6kHgt4AL+51l7gLuGWxVU2bs98QH6LZTvSvJtcA1wLO3tJCqWkW3Hebj/bKuBBZX1Wq6cP5Av7yL6HrD/0q3Y8Ws3kFgxg9X0/8C+2xVPXnApUhDIckeVfXjvjd4JnBzVb130HVtD78nBm9YejaSps5v9hu+lwF7AR8ecD0aAjO+ZyNJ2vHZs5EkNWfYSJKaM2wkSc0ZNtqhJXlVkscPuo4NkixJssXdYyU9mmGjHd2rgB0mbIAlTOJYjO2Rjv+bGip+oNVUP+ry5/pRcG9IckKST4+6//n9aMlz+pGHN4xO/ZZ+FOIR4GOjRtvd3Ci92zQKc5JX9iMBX5vk7/q2X8rG85dcnOSx/bEarwPe0tfz3HTnOfnndKNKX5XkOf3jFya5qB8V+MNJbkuyb3/fb/ev84Ykb+7bDuzr/hDdAbZ/mOS9o2r8zSTvmbq/jDTNBj0SqJfhvtCNOv3Xo6b3ojsSf8NIu/8A/BLwdLrxpjbMt3d/fSkbR2Pemc2P0rvVozADhwM3Afv20z/RX+/DxkMDXgP8ZX/7NODUUY//B+Co/vYTgBv72x8Efq+/fSzdmFn79q/zerqRp/egO5blqXSDYa4HntU/Zj7w/+hHVe5f9xGD/nt68bKtl5k0iJ5mpuuBv0jyLrojuP+t7z38apKP0I0y/Uq6saMOTvIB4HPAl8ZZ1pPYOEovdKdNuHPU/ZuMwgyQZMMozD8YZ5n/DfhkdaMYU1U/7Nv3B/6x7znNA24d57EAx9ANRbJhes8kC+hG235Jv8wLk9zd338U8Omqur+v7VPAc/vab6uqK/vH3J/ky8AvJrmRLnSun6AGaYdn2Kipqlqe5OnAi4E/S/Il4BzgM8BDwD9VNwz73Ul+hm7Y91PoRqD+9TGLC5sfpXdbRmEO44/U+wHgPVV1QZIldD2a8ewE/Gx1Y4ptXOio9Bnn+SYydpThc4Dfp+sJfmQzj5N2eG6zUVP9nmQPVNXfA38BPK2q7qBbzfUHdCftot+esVNV/TPwh8DT+kWMHjH3JrZtlN7NuQR4eZLH9Mv8ib59L+A/+tujTwU8dgTfLwGvH/V6j+xvfoUuMEnyArrVctCNNv3L/UjB8+l6P+MOflpV/07XI3sFm55XRppR7NmotSOAdydZD6wB/mff/jG6bS/f6qf3Az4yai+s3+uvzwPOSvIg3Sq3lwFnpDvd7lzgfXTbPbZJVS1L8ifAZUnWAd+k2wPuNLrRwf+DblTfg/qHfAb4ZLrTS78BeCNwZj/671y6MHkd8Md0owKfQHcaizuB+6rq6iTnsXGU6XOq6pvZOKT/WJ+gO6vm3RPcL80Ijo2mgUjyQeCbVfU3g66lhXTnv1lXVWv7nthfVdWRW3rcOMv5LPDeqlSXxsQAAABkSURBVLpkyouUppE9G027JN+g2z7xvwZdS0NPAD7R99RWA7+5NQ9Osjdd7+dag0bDwJ6NZoV+m8x4X9rPq6rx9lKTNIUMG0lSc+6NJklqzrCRJDVn2EiSmjNsJEnNGTaSpOb+P0XXfrmUnRjdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 421x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# first let's compute the RMSE between the system scores and the true scores, per system category\n",
    "\n",
    "# first we extract the scores for each simulated system and the true scores\n",
    "df_system_and_true_scores = df_scores[system_columns + ['true']]\n",
    "\n",
    "# next we merge in the system categories \n",
    "df_system_and_true_scores = df_system_and_true_scores.melt(id_vars=['true'],\n",
    "                                                           var_name='system_id',\n",
    "                                                           value_name='score')\n",
    "\n",
    "# next we compute the squared error between each simulated system's score for each response and the corresponding true score\n",
    "df_system_and_true_scores['se'] = (df_system_and_true_scores['true'] - df_system_and_true_scores['score']) ** 2\n",
    "\n",
    "# finally we compute the RMSE for each system\n",
    "df_rmse_by_system = df_system_and_true_scores.groupby('system_id')['se'].apply(lambda se: np.sqrt(np.mean(se))).reset_index(name='rmse')\n",
    "\n",
    "# now let's merge in the categories and plot the RMSE value\n",
    "df_rmse_by_system_with_categories = df_rmse_by_system.merge(df_system_metadata, left_on='system_id', right_on='system_id')\n",
    "\n",
    "# next let's compute the R^2 values for each system\n",
    "df_r2_by_system = df_system_and_true_scores.groupby('system_id').apply(lambda row: r2_score(row['true'], row['score'])).reset_index(name='r2')\n",
    "\n",
    "# now merge in the r2 and rmse data frames and re-orient it so that we can do a single plot with two metrics\n",
    "df_rmse_r2_with_categories = df_rmse_by_system_with_categories.merge(df_r2_by_system)\n",
    "df_rmse_r2_with_categories = df_rmse_r2_with_categories[['system_id', 'r2', 'rmse', 'system_category']]\n",
    "\n",
    "# create a longer version of this dataframe that is more amenable to plotting\n",
    "df_rmse_r2_with_categories_long = df_rmse_r2_with_categories.melt(id_vars=['system_id', 'system_category'], var_name='metric')\n",
    "\n",
    "# now plot both the metric values in a single plot\n",
    "sns.lmplot(x='system_category',\n",
    "           y='value',\n",
    "           hue='metric',\n",
    "           fit_reg=False,\n",
    "           data=df_rmse_r2_with_categories_long)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlations between system scores and rater scores\n",
    "\n",
    "Next, we want to examine the average correlations between the simulated scores for the systems in each system category and the simulated scores for the raters in each rater category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>system_category</th>\n",
       "      <th>poor</th>\n",
       "      <th>low</th>\n",
       "      <th>medium</th>\n",
       "      <th>high</th>\n",
       "      <th>perfect</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rater_category</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>low</th>\n",
       "      <td>0.448462</td>\n",
       "      <td>0.501075</td>\n",
       "      <td>0.545229</td>\n",
       "      <td>0.578107</td>\n",
       "      <td>0.629919</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>moderate</th>\n",
       "      <td>0.526030</td>\n",
       "      <td>0.587710</td>\n",
       "      <td>0.639470</td>\n",
       "      <td>0.678007</td>\n",
       "      <td>0.738710</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>average</th>\n",
       "      <td>0.570367</td>\n",
       "      <td>0.637270</td>\n",
       "      <td>0.693415</td>\n",
       "      <td>0.735222</td>\n",
       "      <td>0.801093</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>high</th>\n",
       "      <td>0.631518</td>\n",
       "      <td>0.705556</td>\n",
       "      <td>0.767683</td>\n",
       "      <td>0.813938</td>\n",
       "      <td>0.886787</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "system_category      poor       low    medium      high   perfect\n",
       "rater_category                                                   \n",
       "low              0.448462  0.501075  0.545229  0.578107  0.629919\n",
       "moderate         0.526030  0.587710  0.639470  0.678007  0.738710\n",
       "average          0.570367  0.637270  0.693415  0.735222  0.801093\n",
       "high             0.631518  0.705556  0.767683  0.813938  0.886787"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get the correlations between the simulated human rater scores and the simualted system scores\n",
    "df_system_rater_correlations = df_all_correlations.loc[rater_columns, system_columns]\n",
    "\n",
    "# now re-orient the data frame so we have pairwise correlations on each row\n",
    "df_pairwise_system_rater_correlations = df_system_rater_correlations.unstack().reset_index()\n",
    "df_pairwise_system_rater_correlations.columns = ['system_id', 'rater_id', 'correlation']\n",
    "\n",
    "# now merge in the rater category and system category labels from the two metadata frames\n",
    "df_pairwise_system_rater_correlations_with_categories = pd.merge(df_pairwise_system_rater_correlations,\n",
    "                                                                 df_rater_metadata,\n",
    "                                                                 left_on='rater_id',\n",
    "                                                                 right_on='rater_id').merge(df_system_metadata,\n",
    "                                                                                            right_on='system_id',\n",
    "                                                                                            left_on='system_id')\n",
    "\n",
    "\n",
    "# now group the raters and systems by categories and compute \n",
    "# the mean correlations for each rater category x each system cateogry\n",
    "grouper = df_pairwise_system_rater_correlations_with_categories.groupby(['rater_category', 'system_category'])['correlation']\n",
    "df_mean_system_rater_correlations = grouper.mean().unstack()\n",
    "\n",
    "# reorder the columns in order of agreement/performance instead of alphabetical\n",
    "df_mean_system_rater_correlations.loc[dataset.rater_categories, dataset.system_categories]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, systems that have been simulated to have poor performance actually vary quite a lot in performance depending on the inter-rater correlations of the human raters against which they are being evaluated. This means that correlation is not a very stable metric. We can also see this visually via a boxplot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEHCAYAAACjh0HiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXiU1dn48e+dhZAACiEUEJCgEIlAANlESwAliKLwgijyShtRKlIFxaq1biz6at9aWgWtCxaBahVba6G+pIILBSz+ZJV9STVARJEJi0BCkknO7495ZpwkM5NJmMls9+e6cjHPOmcekrmfc55z7iPGGJRSSsWuuFAXQCmlVGhpIFBKqRingUAppWKcBgKllIpxGgiUUirGaSBQSqkYlxDqAtRVWlqaSU9PD3UxlFIqomzatMlmjGnlaVvEBYL09HQ2btwY6mIopVREEZED3rYFtWlIREaIyF4RyReRhz1sbyEi74nINhH5XES6B7M8SimlagpaIBCReOBF4FrgUmCCiFxabbdHgK3GmCzgp8DzwSqPUkopz4JZI+gP5BtjvjTGlAFvA6Or7XMp8BGAMWYPkC4irYNYJqWUUtUE8xlBO+CQ23IhMKDaPl8AY4F1ItIf6Ai0B47U5Y3Ky8spLCzk7Nmz51Bc5a/GjRvTvn17EhMTQ10UpVQABDMQiId11TPc/Rp4XkS2AtuBLYC9xolE7gTuBLjwwgtrnLSwsJBmzZqRnp6OiKe3VYFijKGoqIjCwkI6deoU6uIopQIgmE1DhUAHt+X2wGH3HYwx3xtjJhljeuF4RtAK+Kr6iYwxrxpj+hpj+rZqVbP309mzZ2nZsqUGgQYgIrRs2VJrX0pFkWAGgg1AFxHpJCKNgFuA5e47iEhzaxvAZGCNMeb7+ryZBoGGo9daqegStKYhY4xdRO4BPgDigYXGmJ0icpe1/WUgE1giIhXALuCOYJWnrhYtWsTw4cO54IILQl0UAFavXk2jRo244oorQl0UpVQDmTdvHvn5+YCjCRygffv2AHTu3Jnp06cH5H2COqDMGLMCWFFt3ctur9cDXYJZhvpatGgR3bt3D6tA0LRp06AGAmMMxhji4jTziFKhcPvtt/PNN9+4lktLS6msrARw/Xvs2DEAduzYQV5eHgBt27Zl4cKF9X7fqP2LP3PmDCNHjqRnz550796dpUuXMmbMGNf2VatWMXbsWCoqKrjtttvo3r07PXr04Pe//z1//etf2bhxI7feeiu9evWipKSETZs2MXjwYPr06cM111zj+s8aMmQIM2bMIDs7m8zMTDZs2MDYsWPp0qULjz32mM8yLlmyhKysLHr27MlPfvITAP7xj38wYMAAevfuzbBhwzhy5AgFBQW8/PLL/P73v6dXr16sXbuWo0ePcuONN9KvXz/69evHp59+CsDRo0fJycnhsssuY8qUKXTs2BGbzQbA7373O7p370737t157rnnACgoKCAzM5Of//znXHbZZTz55JPMmDHDVcYFCxZw//33B+4/Rinl1YkTJzhz5ozrx263U1lZ6QoCgGvZbre79jtx4sS5vbHzLjBSfvr06WOq27VrV411f/3rX83kyZNdyydOnDCXXHKJ+e6774wxxkyYMMEsX77cbNy40QwbNsy13/Hjx40xxgwePNhs2LDBGGNMWVmZGThwoOvYt99+20yaNMm130MPPWSMMea5554zbdu2NYcPHzZnz5417dq1MzabrUbZjDFmx44dJiMjwxw9etQYY0xRUZExxphjx46ZyspKY4wxCxYsMPfff78xxpiZM2eaZ5991nX8hAkTzNq1a40xxhw4cMB07drVGGPM3XffbZ5++mljjDF5eXkGMEePHjUbN2403bt3N6dPnzanTp0yl156qdm8ebP56quvjIiY9evXG2OMOX36tLnoootMWVmZMcaYgQMHmm3btvl1zZVS5+b5558306ZNc/2MGTPGjBgxwowYMcIMHjzYDB482LU8ZswY137PP/98recGNhov36sRl2vIXz169OCBBx7gl7/8Jddffz2DBg3iJz/5CW+88QaTJk1i/fr1LFmyhFOnTvHll18ybdo0Ro4cyfDhw2uca+/evezYsYOcnBwAKioqaNu2rWv7qFGjXO/ZrVs317aLLrqIQ4cO0bJlyxrn/Pjjjxk3bhxpaWkApKamAo52wPHjx/PNN99QVlbmtYvmhx9+yK5du1zL33//PadOnWLdunW89957AIwYMYIWLVoAsG7dOsaMGUOTJk0AGDt2LGvXrmXUqFF07NiRyy+/HIAmTZpw1VVX8f7775OZmUl5eTk9evTw97Irpc5B9Tb/qHhGEEoZGRls2rSJFStW8Ktf/Yrhw4czefJkbrjhBho3bsxNN91EQkICLVq04IsvvuCDDz7gxRdf5J133qnR1maMoVu3bqxfv97jeyUlJQEQFxfneu1ctttrDItwndNT75tp06Zx//33M2rUKFavXs2sWbM8Hl9ZWcn69etJTk6ucV5v7+eNMzg4TZ48maeffpquXbsyadIkr8cppYIrUF/0tYnaZwSHDx8mJSWFiRMn8sADD7B582YuuOACLrjgAp566iluu+02AGw2G5WVldx44408+eSTbN68GYBmzZpx6tQpAC655BKOHj3qCgTl5eXs3LnznMp39dVX884771BUVAT88ADo5MmTtGvXDoDFixe79ncvD8Dw4cN54YUXXMtbt24F4Mc//jHvvPMOACtXruT48eMAZGdn8/e//53i4mLOnDnDe++9x6BBgzyWbcCAARw6dIg///nPTJgw4Zw+p1Iq/EVtjWD79u08+OCDxMXFkZiYyEsvvQTArbfeytGjR7n0Ukf+u6+//ppJkya5HsY888wzANx2223cddddJCcns379ev76178yffp0Tp48id1u57777qNbt271Ll+3bt149NFHGTx4MPHx8fTu3ZtFixYxa9YsbrrpJtq1a8fll1/OV185xtfdcMMNjBs3jmXLljF//nzmzZvH3XffTVZWFna7nezsbF5++WVmzpzJhAkTWLp0KYMHD6Zt27Y0a9aMyy67jNtuu43+/fsDjrv+3r17U1BQ4LF8N998M1u3bnU1LSmlopf4ajIIR3379jXV5yPYvXs3mZmZfh1/zz330Lt3b+64I2yGLARUaWkp8fHxJCQksH79eqZOneqqLdTF9ddfz4wZM7j66qs9bq/LNVdKhZ6IbDLG9PW0LWprBJ706dOHJk2aMHfu3FAXJWgOHjzIzTffTGVlJY0aNWLBggV1Ov7EiRP079+fnj17eg0CSqnoElOBYNOmTQ3+nkVFRR6/UD/66COPvYnOVZcuXdiyZUu9j2/evDn79u0LYImUUuEupgJBKLRs2bJeTTNKKdVQorbXkFJKKf9oIFBKqRingUAppWKcBgKllIpxGgiUUirGxWSvobvve4AjtmMBO1/rtFRefO63ATufvyoqKoiPj2/w91VKRZeYDARHbMf4qu2QwJ3wm9W17lJQUMCIESMYMGAAW7ZsISMjgyVLlrB+/XoeeOAB7HY7/fr146WXXiIpKYmPPvrI4/r09HRuv/12Vq5cyT333MMtt9wSuM+hlIpJ2jTUgPbu3cudd97Jtm3bOO+88/jd737HbbfdxtKlS9m+fTt2u52XXnqJs2fPelzv1LhxY9atW6dBQCkVEBoIGlCHDh248sorAZg4cSIfffQRnTp1IiMjA4Dc3FzWrFnD3r17Pa53Gj9+fMMXXikVtTQQNCBP8w94UlsiwOrzByil1LnQQNCADh486JrT4K233mLYsGEUFBS4ZiD605/+xODBg+natavH9UopFQwaCBpQZmYmixcvJisri2PHjjFjxgxef/11brrpJnr06EFcXBx33XUXjRs39rheKaWCISZ7DbVOS/Wrp0+dzueHuLg4Xn755Srrrr76ao/ZQr2t9zaRjFJK1VdMBoJQ9PlXSqlwpU1DDSQ9PZ0dO3aEuhhKqTBgs9mYNm2aa87yUNNAoJRSDWzx4sVs27aNxYsXh7oogAYCpZRqUDabjby8PIwx5OXlhUWtQAOBUko1oMWLF7vGClVWVoZFrSAmHxYrpVRDmjdvnmtc0LZt26isrASgvLyc5cuXk5CQwPTp00NWPq0RNJCmTZuGughKqTDQokULn8uhEJM1gl/NuJuTRd8G7Hznt2zDM79/MWDnUyoa7Nu3j3vvvZf58+fTuXPnUBenwbnXAty1adPG9VxARGjTpg35+fmuGkHnzp0bvHYQk4HgZNG3PNx5X8DO9+ua/9deGWN46KGHyMvLQ0R47LHHGD9+PD//+c8ZMWIEo0aNYsyYMbRo0YKFCxfyxz/+ka+++oqnnnoqYOVVqiE89dRTnDlzhjlz5rBkyZJQF6fBrV69GpvN5nMfYww7d+6ssq6wsFADQbT729/+xtatW/niiy+w2Wz069eP7OxssrOzWbt2LaNGjeLrr7/mm2++AdB00yoi7du3zzUK3pk3K9ZqBc2bN6ekpMTjtpKSEiorK0lJSamRjLJ58+YNUbwqNBA0sHXr1jFhwgTi4+Np3bo1gwcPZsOGDQwaNIjnnnuOXbt2cemll3L8+HG++eYb1q9fz7x580JdbKX84mwO2b17d5X1U6dOJTMzMyTNHqGycOFC1+vqzUT79+8HoEuXLkBomoPcBfVhsYiMEJG9IpIvIg972H6+iPxDRL4QkZ0iMimY5QkH3lJMt2vXjuPHj/PPf/6T7OxsBg0axDvvvEPTpk1p1qxZA5dSqXNTWlrqcznWJScnk5ycHOpiuAStRiAi8cCLQA5QCGwQkeXGmF1uu90N7DLG3CAirYC9IvKmMaYsWOUKtezsbF555RVyc3M5duwYa9as4dlnnwVg4MCBPPfcc3z88ccUFRUxbtw4xo0bF+ISK+Wbp4eiSUlJVb78k5KSAEL+UDRUwv1zBrNpqD+Qb4z5EkBE3gZGA+6BwADNxNFI1hQ4BtiDWKaQGzNmDOvXr6dnz56ICL/5zW9o06YNAIMGDWLlypV07tyZjh07cuzYMQYNGhTiEivl2+rVqymyFZEQ38i1rtJUVtmnorySndt/aC6yV5SF5KGo8iyYgaAdcMhtuRAYUG2fF4DlwGGgGTDemGq/QUFwfss2derp48/5anP69GnA0V3s2WefddUC3N1xxx3ccccdACQmJnLmzJnAFVKpBhQncVS4LYvokKVwFsxA4GlexuoN5NcAW4GrgIuBVSKy1hjzfZUTidwJ3Alw4YUXnnPBtM+/UoEzZMgQj/3l9+zZw9mzZ8nIyCAlJaXG9ljrRRTOghkICoEObsvtcdz5u5sE/No4nqDmi8hXQFfgc/edjDGvAq8C9O3b1/eEvkqpBuXevOP+vCA+Pp4mTZpUCQKx9FwgkgSzvrYB6CIinUSkEXALjmYgdweBqwFEpDVwCfBlEMuklGog4dYzRnkXtBqBMcYuIvcAHwDxwEJjzE4Rucva/jLwJLBIRLbjaEr6pTHG91A8pVTY0rv9yBTUAWXGmBXAimrrXnZ7fRgYHswyKKWU8k0f5SulVIzTQBCBhgwZwsaNGwG47rrrOHHiRIhLpFRsCre5h+srJnMN3fOLezhSdCRg52vdsjUvzH0hYOerixUrVtS+k1IqKNznHr7//vtDXZx6i8lAcKToCIf7VO/Jeg421b5LQUEBI0aM4Mc//jGfffYZPXv2ZNKkScycOZPvvvuON998k27dujFt2jS2b9+O3W5n1qxZjB49mpKSEiZNmsSuXbvIzMysktEwPT2djRs3cvr0aa6//np27NgBwG9/+1tOnz7NrFmzGDJkCL1792bTpk0cPXqUJUuW8Mwzz7B9+3bGjx+vKa6Vqofqcw/n5ubSsmXLUBerXmIyEIRKfn4+f/nLX3j11Vfp168ff/7zn1m3bh3Lly/n6aef5tJLL+Wqq65i4cKFnDhxgv79+zNs2DBeeeUVUlJS2LZtG9u2beOyyy6r83s3atSINWvW8PzzzzN69Gg2bdpEamoqF198MTNmzIjYX2ClQsXT3MORWivQQNCAOnXqRI8ePQDo1q0bV199NSJCjx49KCgooLCwkOXLl/Pb3/4WgLNnz3Lw4EHWrFnj6paXlZVFVlZWnd971KhRAPTo0YNu3brRtm1bAC666CIOHTqkgUApPzkHzYXj3MP1pYGgATkzMALExcW5luPi4rDb7cTHx/Puu+9yySWX1Di2+uQV1SUkJLh+KcERRDy9t/v7ur+3UqpuWrRoUeUhcTjMPVxfGgjCyDXXXMP8+fOZP38+IsKWLVvo3bs32dnZvPnmmwwdOpQdO3awbdu2Gse2bt2a7777jqKiIpo2bcr777/PiBEjQvAplIo+t99+u2vWQKfqc4uUlJSQl5dHXl6ea13btm2rTFATrjQQhJHHH3+c++67j6ysLIwxpKen8/777zN16lQmTZpEVlYWvXr1on///jWOTUxM5IknnmDAgAF06tSJrl27huATKBWdTpw4UWs24OLiYo/HRQLxNmNWuOrbt69x9qF32r17N5mZmX6fI5q6j4ZKXa+5UpHM0+Q7ANu3b6eiooIWLVrQsWPHGtvDKcmeiGwyxvT1tC0mawSx9qWtlDo3njKslpeXU1HhmHXhxIkTXHDBBWRmZobNF39d6MhipZSqh2+//db12hhTZTnSxGSNQCml6st5x1+9M0ZJSUlE1gZAawRKKT9ES06dQMrJySExMRFwdNYYPjxyEylrIFBK1co9p45yyM3NdY3viYuLIzc3N8Qlqj8NBEopn6rn1NFagUNaWhrXXnstIsK1114b0aPz9RlBAykoKKiSFM7piSeeIDs7m2HDhnk9dtasWTRt2pQHHngg2MWMOdW7BRYWFgLQvn17ILy6/4VKNOXUCbTc3FwKCgoiujYAMRoIHrrnHk4c+S5g52ve+kf85oX6dUmdM2dOwMqhzp17ZtdYF405dQItLS2N+fPnh7oY5ywmA8GJI99x65HADSh708/9Kioq+NnPfsa///1v2rVrx7Jly5g6dSrXX38948aNY8WKFdx///2kpaVx2WWX8eWXX/L+++8DsGvXLoYMGcLBgwe577779I8wQKpfR+fyvHnzQlGcsBRNOXWUZzEZCEJl//79vPXWWyxYsICbb76Zd99917Xt7NmzTJkyhTVr1tCpUycmTJhQ5dg9e/bwySefcOrUKS655BKmTp3q6rGg6sbbKFFw/B+B50nYY62ZyPlZbTYbN954I8YYkpKSeO211yK6PVzVpIGgAXXq1IlevXoB0KdPHwoKClzb9uzZw0UXXUSnTp0AmDBhAq+++qpr+8iRI0lKSiIpKYkf/ehHHDlyxNWOrepm9erVFNmKSIhvVGNbRWU5ADu3766y3l5RRmFhYVQHAl8BMj4+HrvdTtOmTZk9e3aVbbEWIKORBoIG5J7+OT4+vkp7dG05n6ofq6mjVaA5AqSNmuERnAnOTxUVsdutmagMoj5AxgINBGGia9eufPnllxQUFJCens7SpUtDXaSoNWTIkFqbhrp06VJjW+fOnYNarnAmQLz1r4o+GgjCRHJyMn/4wx8YMWIEaWlpHlNNq8Dwdfcayw+LfQXIvXv3UlpaSufMzBrPpmI5QEaLmAwEzVv/yO+ePv6erzbp6elVxhB4GhMwdOhQ9uzZgzGGu+++m759HRljZ82aVWW/6mMRVP1Vbxev/rA4ltq/fX3OMWPGUFJSQpcuXXQMQRSKyUBQ3z7/wbZgwQIWL15MWVkZvXv3ZsqUKaEuUsxJTk4OdRHChnu6ZWf30WXLlrF///6ITbesPIvJQBCuZsyYwYwZM0JdjJiiX2a185RuWSclii4aCJRSHkVjumXlmSadU0r5FE3plpVnGgiUUj5FU7pl5Zk2DSkVQ2w2G7Nnz2bWrFl+p4lwpltevnx5xKdbdqeZZ3/gV41ARDJEZIGIrBSRj50/wS6c+sHRo0cZMGAAvXv3Zu3atXU6duvWraxYsSJIJVORpL4TzOTm5pKVlRXVtYGSkpKYzT7rb43gL8DLwAKgInjFaRi/uO9BimzHA3a+lmktmPvcswE7X3V2u52PPvqIrl271muGqK1bt7Jx40auu+66IJRORYrqE8zk5ubWqVYQDemW3Wnm2R/4GwjsxpiXglqSBlRkO07f1qMDdr6NR5bVuk9BQQEjRoxgwIABbNmyhYyMDJYsWcLu3bu5//77OX36NGlpaSxatIi2bdsyZMgQrrjiCj799FNGjRrF888/T0lJCb169WL9+vWsXbuWmTNnUlpaysUXX8zrr79O06ZN2bBhA/feey9nzpwhKSmJVatW8cQTT1BSUsK6dev41a9+xfjx4wP22VXk0AlmNPOsN/4Ggn+IyM+B94BS50pjzDFfB4nICOB5HGlKXjPG/Lra9geBW93Kkgm0qu28kWrv3r388Y9/5Morr+T222/nxRdf5L333mPZsmW0atWKpUuX8uijj7Jw4UIATpw4wb/+9S8AWrZsycaNG3nhhRew2Ww89dRTfPjhhzRp0oT//d//5Xe/+x0PP/ww48ePZ+nSpfTr14/vv/+elJQU5syZ4zpWxa5Vq1ZRXu7IrlpeXs7KlStjLhCsXr2aY7ajJMXXTPJYXul4IL53++Yq60srJOoT6/kbCJwNgw+6rTPARd4OEJF44EUgBygENojIcmPMLtcJjHkWeNba/wZgRrQGAYAOHTpw5ZVXAjBx4kSefvppduzYQU5ODuCYuKZt27au/b3duX/22Wfs2rXLda6ysjIGDhzI3r17adu2Lf369QPgvPPOC+bHURHCeRecnJxMcXGxa31ycjLTp0+P+rtdfxnjaPeONxAXY9n1/AoExphO9Th3fyDfGPMlgIi8DYwGdnnZfwLwVj3eJ2I4u+A5NWvWjG7durF+/XqP+zdp0sTjemMMOTk5vPVW1cu1bdu2Gu8Rq+rTOybatWnTxpUqQkRo06ZNiEvU8Hwl1vviiy8c0SCxMZdUGzkd7Yn1/AoEIpIITAWyrVWrgVeMMeU+DmsHHHJbLgQGeDl/CjACuMfL9juBOwEuvPBCf4oclg4ePMj69esZOHAgb731FpdffjkLFixwrSsvL2ffvn1069bN53kuv/xy7r77bvLz8+ncuTPFxcUUFhbStWtXDh8+zIYNG+jXrx+nTp0iOTmZZs2acerUqQb6lOHBvXdMrDV/VOd+tz9mzBiKiooYPXp0TF4X92vh/ryguLjY9fyktLSU4uJisrKyYqam5G/T0EtAIvAHa/kn1rrJPo7xdGvqbfaVG4BPvTULGWNeBV4F6Nu3r+8ZXMJYZmYmixcvZsqUKXTp0oVp06ZxzTXXMH36dE6ePIndbue+++6rNRC0atWKRYsWMWHCBEpLHY9snnrqKTIyMli6dCnTpk2jpKSE5ORkPvzwQ4YOHcqvf/1revXqFRMPi8+ld0y08PZQtLi4mLi4OPbv3x+TD0W9OXDgQI3lrKysEJWm4UltM2MBiMgXxpieta2rtn0gMMsYc421/CsAY8wzHvZ9D/iLMebPtZWlb9++ZuPGjVXW7d69u05JsELRfbSgoIDrr78+alJI1/WaN6S5c+eyYsUKysvLSUxMZOTIkTF39zt27FiOFh2teavn7Pwd7+EgO7Rq2Yq//e1vQS5d+MnOzq6xbs2aNSEoSfCIyCZjTF9P2/ytEVSIyMXGmP9YJ7yI2scTbAC6iEgn4GvgFuC/PRTufGAwMNHPspyzYPb5V6GnvWNUXaWnp1eZQzw9PT1kZQkFfwPBg8AnIvIljiafjsAkXwcYY+wicg/wAY77j4XGmJ0icpe1/WVr1zHASmPMmfp8gEhRfWIaFXi+esfMmzcvppo8vD0Udc40lplRc6YxiP6Hot489thjTJ78Q0v3E088EcLSNDx/ew19JCJdgEtwBII9xpjSWg7DGLMCWFFt3cvVlhcBi/wsr1K10t4x3udZ0JnGPMvIyHDVCtLT02MuIPoMBCJylTHmYxEZW23TxSKCMSZsGhONMdp1soH481wpFLR3jG82m41jxxz9MWL1Ibovjz32GPfee2/M1Qag9hrBYOBjHL16qjNAWASCxo0bU1RURMuWLTUYBJkxhqKiIho3bhzqovjUpk0bzp49G9VJ0vzlbDI7dOhQlS6SkydPpkOHDjHbU6i6jIwM8vLyQl2MkPAZCIwxM62Xc4wxX7lvsx4Ch4X27dtTWFjI0aNHQ12UmNC4cWNXqt5Q89ZN0vngb/bs2R6Pi8Uvv+PHj9dY7tChQ4hKo8KJv91HNxtjLqu2bpMxpk/QSuaFp+6jKnaNHTuWIpuNRtXW261/Pd3plAEt09JirpukdquNbfXuPioiXYFuwPnVnhOcB4R324BSqorc3FxX04fONKbc1faM4BLgeqA5VZ8TnAJ+FqxCKeUvb90kd+/eTWlpKR0vushj8r1Y6xUC0TvTmDp3tT0jWAYsE5GBxhjPmdGUCiFv7fxDhgwB4MiRIyxatKjhChTmcnNzKSgo0NqAqsLfAWVbRORuHM1EriYhY8ztQSmVUufg888/p7KyEoAzZ86wadMm+vRp8MdZYSkaZxpT586vOYuBPwFtgGuAfwHtcTQPKRV0NpuNadOmuQaJeeMcPfzQQw9VWf+LX/wiJqcfVMpf/gaCzsaYx4EzxpjFwEigR/CKpdQP6jrhurM24G1ZKVWVv4HAOe/ACRHpDpwPpAelREq5qZ5S2letYPr06V7v/KNxzIC/NSWlauNvIHhVRFoAjwPLccwy9puglUopi6cJ12szcODAKstXXHFFUMoWanWtKSnljV+BwBjzmjHmuDHmX8aYi4wxP6qePE6pYPCUUro2Dz74oM/laGCz2VixYgXGGFasWKG1AnVOfAYCEbnf109DFVLFrpycHFe65MTERIYPH17rMWlpaa5awRVXXBGV/eUXL16M3e4YP11eXq61AnVOaqsRNKvlR4VYNLYTO3v/TJ8+nfz8fFeNwG63s3//fr96AD344IP07NkzKmsDACtXrnQ1mRlj+OCDD0JcIhXJahtQ5jljlwob0T5Je2JiIgkJCdjtdlJTUz1OpuJJtPeXb926dZUZtVq3bh26wqiI52/SudTa5JkAABuASURBVAwck9W3NsZ0F5EsYJQx5qlgF7A6TTr3A5vNxi233EJZWRlJSUm8/fbbUdkMMnXqVAoKCnjjjTei8vPVhTPb6rZt26p0i42Li2Ps2LFR2TtKBYavpHP+9hpaAPwKqxupMWYbjjmIVQjVp0dNJEpMTKRLly4xHwTctWjRwueyUnXhb4qJFGPM59UmfbF721kFl6e7wvLycpYvX05CQkJE3hV6m1cAYP/+/YDnsQCxNq+A87PabDZuvPFGjDE0atSI1157TQOlqjd/A4FNRC7GMSsZIjIO+CZopVJ+adGiRZWHxJF8V5ifn8+WnVsceW6rs1pAtny9per6E0EvVsj5CpDx8fHY7XaaNWvmcQKeWAqS7tepsLAQoMrkSbF0LerD30BwN/Aq0FVEvga+Am4NWqmUT57uCpOSkiL6rtD5x+tR03oeFwVWr17NMdtRkuJrPsszlY4a+pkTR9l7oursfKUVQmFhYUx++ZWUlIS6CBGn1kAgIvHAVGPMMBFpAsQZYzThXBhIS0sjNTWVoqKi6Mgvb8fzXX6F9W+8h/1jQFK8oWOzitp3dHPgVPWLFd3cA57ztSYa9F+tgcAYUyEifazXZ4JfJFWdr+aB4uJi4uLi2L9/f427v0iqDnubYAZg165dlJWVcWH7C0lNTa2yLdonmGnfvj37TnzncduRYkdfj9YpNZPqiRA280oHi7e/C1/PlCCy/i4aSl3mI1gO/AVwBQNjTGxN+hoiq1evpshWREJ89Zl5oaLSMdhq356qfxD2irKIahrwVc7s7GwADh8+zBtvvNFQRQoLvgJdyZ49nD17llbpmaSkpFTZllHLsdFg9erVHLUVQXy1r7FKR+1py/adNQ+qsEfU30VD8TcQpAJFwFVu6wyggaCBJMQ3onnKj/ze/0Sx57vISPPhhx+6Xtvtdj755BOGDh0awhI1LF9fWDk5OQCUlZXx2muvNVSRVBTy9xmBzRgTnWP1I0D79u05eXy3x22nzx4HoGnjmj2GIrVpwL3Kv3Xr1irbZs6cyfbt22P2js55bYqLiyktLQWgoKCAyZMnk5WVFVPXxVtz4h6rppRxcecaNSWI/ppSffj7jOCyhiiM8szXL+7+/ccAaHdx1QfF7Wipv/BR7MCBAzWWs7KyQlSa0PAW9LSmVHf+ppiYC3QhDJ4RaIqJqqK9h8RVV13lyrIJkJCQwMcffxzCEoUH53MTd2vWrAlBScKDe01p3759rvUZGRmkpKToA2ICk2LC/RnBDdbP9YEpnlLePfLII1WWH3/88RCVJLykp6f7XI5VnmpKqnZ+1QjCidYIqrahO7vKdenSBYjOrnHOWoHWBn6wb98+Jk+e7FpeuHChNgWiNSVfzrlGICLtReQ9EflORI6IyLsiEplPIqNMcnIyycnJoS6GX+o7d4KzVqC1gR9kZGS4agHp6ekaBCxaU6off58RrAL+DPzJWjURuNUYkxPEsnmkNYLINXfuXJYvX87o0aOjcu6EhrZv3z7uvfde5s+fr4HAojUl7wLxjKCVMeZ1Y4zd+lkEtApYCVXUs9ls5OXlYYwhLy8vqmZUC5WMjAzy8vL0i86N1pTqx99AYBORiSISb/1MxPHwWAVINE456S5W5k5QoffYY4/RpEkTnnjiiVAXJWL4GwhuB24GvsWRfnocMKm2g0RkhIjsFZF8EXnYyz5DRGSriOwUkX/5W/Bo4z7lZDRatWqVa+7h8vJyVq5cGeISqWilNaW68zcQPAnkGmNaGWN+hCMwzPJ1gDUi+UXgWuBSYIKIXFptn+bAH3BMe9kNuKluxY8OsdBskpOT45pvODExkeHDh4e4REopJ38DQZYx5rhzwRhzDOhdyzH9gXxjzJfGmDLgbWB0tX3+G/ibMeagdd7oSJBTR4sXL3YNmiovL4+qWsG8efOYPn06+fn5rhqB3W53ZUuN1oFwSkUSfwNBnIi4ktmISCq1p6doBxxyWy601rnLAFqIyGoR2SQiP/V0IhG5U0Q2isjGo0ePetoloq1atco15WRlZWVUNpskJiaSkOD4lUlNTXXVDpRSoedv9tG5wL9F5K84so7eDPxPLceIh3XV+6omAH2Aq4FkYL2IfGaM2VflIGNexTFDGn379o2sEXB+yMzMZNOmTa7lbt26hbA0geU+uG3q1KkUFBRE9ExqwWKz2Zg9ezazZs3Sa6ManF81AmPMEuBG4AhwFBhrjPmT76MoBDq4LbcHDnvY55/GmDPGGBuwBujpT5miSfUMm5s3bw5RSYIrMTGRLl266BedB9HeWUCFt6ClmBCRBGAfjrv9r4ENwH8bY3a67ZMJvABcAzQCPgduMcbs8HbeaBpQ5kwVUT0QAPTq1Sti00XUNnOUMx1GdZH6ec+VzWZj/PjxlJeX06hRI5YuXarBUgWcrwFl/jYN1Zkxxi4i9wAf4JhtdqExZqeI3GVtf9kYs1tE/glsAyqB13wFARUZ8vPz2bdjMxc2rTrPbqNyRwX0bMGGGsccPB1bc+y689RZQEdeq4akSefCwIcffsicOXNcy7Nnz47oWbjGjh3LmePf1WnC9QOn4mnS4kf87W+xN+ndNddcQ0lJiWs5OTmZDz74IIQlUtEoECkmVBANGzbM9TohISGig4BTaYVw4FR8lZ/8kwnkn0yosf7AqXhKKzz1LYgNrVu39rmsVLAFrWlIeeat/bxRo0aUlZVxwQUXeGwnj6T2c29TCPrzjCBWuP8eHDx4sMq2gwcPusZfKNUQNBA0sPz8fPZs3UqbauuTrJ+4gwc5Ue2L4duGKlyAePsCmzhxIgcPHmTMmDFRUesJlBYtWlQZTd6iRc35p5UKJg0EQeCrT3hhYWGNwRQAvvqIGOu4SOe8833yySdjPhC4B0ubzcaNN96IMYZGjRrpOAvV4DQQBIF7n3BPvT/KcGTuc+ecldfTf0hZgMvXkJxNIMeOHXOts9vtTJw4kdTU1Ihq8gqWtLQ0UlNTKSoq4rrrrtMgoBqcBoIAq55ALjc3t8ofdqy2n3tqB09NTQ1RaULD2/MhgOLiYuLi4lw5mNxpsFTBpt1HA2zu3LmsWLGC8vJyEhMTGTlypF99wp3pF954442ovCPUuWQd3WqP2oog3sP9V6XV1Tau2niKCjut0lrGZLdaFVjafbQB1Tfv/uHDhzlz5gyvvPJKMIsXMs6Ec96WlVKho3+NAZaTk8OyZctcy/7k3bfZbBw/7sjyvXLlSqZMmRJ1tYJHHnmkyqC5WJyI3luzIMDevXspLS0ls2tGjcyskd4sqMKfBoIAGzRoUJVAMHjwYK/7OtuMDxw44FpXWVnJ7bffTseOHaOqbXjYsGE8/fTT2O32qBk0V1e+/i9HjhxJSUkJzZs355lnnmnAUimlTUMB4xwA9Oijj1ZZ//DDD9c6+YqzNuBtOVo88sgjQGzWBnyx2WycOnUKgE8//TQqZ6hT4U1rBAFWWlrqc9md8w5x6NChVFT8kJcnPj4+KmfuGjZsWJV0GrHMvQfRf/7znyrbfvrTn3LNNddETW1QhT+tEQSIc9rF9PT0KuvT09Nr/YOu/uWYk5MT6OIFnM1mY9q0aXr3GgDO2oC3ZaWCTbuPBti+ffuYPHmya3nhwoW1Puyz2WyMGzeOyspK4uLiePfdd8P+YfHcuXNZvnw5o0eP1pTJ50i71qqGoN1HG1BGRgZJSUmAozbgT4+PtLQ0Vy1g+PDhYR8Eqg+a01rBuUlJSfG5rFSw6TOCevI1SlTEkVK5UaNGfmcSnTJlCt9++y1TpkwJfGEDbPHixThrkpWVlTqRyjmaM2cODzzwgGv5f/6ntunAlQosDQT1lJ+fz5adW6C5h40CJMPe43uhegegE57Pl5aWxvz58wNcyuDwNGhOA0H99e/fn5SUFIqLi0lJSaFPnz6hLpKKMdo0VE8+s4E2tX7qc2wEyMnJcQ16SkxM9GvQnPJtzpw5xMXFaW1AhYTWCM6FHc93+M6eoJ6m4bV7WBchnM1h5eXlrhqB3W5n//79OpHKOerfvz+rV68OdTFUjNJAUE++0gVEeybRxMREEhISsNvtpKam1kiJoJSKLNp9NAiiPZMoxMZnVCqaaPfRc1CfgVOFhYWcOXMmKkcHOyUmJtKlSxcNAkpFAQ0EtXCfbcwfNpuNkydPAvDJJ59oH3ulVNjTpiEfbDYbt9xyC2VlZSQlJfH22297vQN2Pkj96quvXIEA4PzzzycnJyciH6T6Givhz3OQSPzMSkUrX01D+rDYh/oMnHIPAp6WI0l+fj5btu+iMqXmlJJS5rgum/7zbY1tccXHaqxTSoUvbRryoS6zjTmTznnbFokc4x081xhN4/Mwjc/zcqSJ+LESSsUSDQQ+5OTkuKZUTEhI8GvgVIcOHXwuK6VUuNFA4ENubi6VlZWAo2koNze31mNmzpxZZXn27NlBKVtDaN++PY58GTXJ2e+Rs997OVKsY5VSkUCfEQRYRkYGHTp04NChQ3To0CGiB4/5Kvv+/Y6c+V0ubuNha5uI/txKxRrtNVSNe0+ZQ4cOVen+2bJlS4YOHVprm/++ffu49957mT9/ftR+IeqAMqUiiw4oq6f6ziWckZFBXl5e1AYBgG+//ZYzZ874Pb5CKRW+tGmoGve7/blz57Js2TLAMZJ25MiREdsDKJBsNhvHjjm6iObl5ZGbm6u1AqUimAYCH3Jzc1m+fDnGGOLi4vx6WBzNnM1mhw4dco2vKC0tZfLkyX41mSmlwlNQm4ZEZISI7BWRfBF52MP2ISJyUkS2Wj9PBLM8dZWWlkZqqmMw1bXXXqt3vZb6NpkppcJT0B4Wi0g8sA/IAQqBDcAEY8wut32GAA8YY67397zBeFjsK5XC3r17KS0tJTMzs0a65VhNozB37lxWrFhBeXm5q8lMZyhTKryFKsVEfyDfGPOlVYi3gdHALp9HhUB+fj77dmzmwqYVNbY1roijcQJUfL0V960HT3uadSY25ObmkpeXB6BNZkpFgWA2DbUDDrktF1rrqhsoIl+ISJ6IdAtiebwqLCzEW8WodUolrVMqa6w3JvKnnKyvtLQ0rr32WkREm8yUigLBrBF4GpJa/et2M9DRGHNaRK4D/g7USGcpIncCdwJceOGFgS6nqofc3FwKCgq0NqBUFAhmjaAQcE+00x447L6DMeZ7Y8xp6/UKIFFE0qqfyBjzqjGmrzGmb6tWrQJe0Pbt2yOeMylwpDiOI8U1L5MIMZ1GIS0tjfnz52ttQKkoEMwawQagi4h0Ar4GbgH+230HEWkDHDHGGBHpjyMwNfhMLr4GfpVZefcbp1etqGTUcpxSSkWKoAUCY4xdRO4BPgDigYXGmJ0icpe1/WVgHDBVROxACXCLCUHOC189fyZOnMjBgwcZM2YMQ4cObcBSKaVUw9BcQ7XIzs4GHGmoP/744wZ7X6WUCiTNNVRPH374oeu13W7nk08+CWFplFIqOGImxYTNZmP27NnMmjXL5wNO98FlW7durbJt5syZbN++PSYHkSmlolfM1AgWL17Mtm3bNFumUkpVExPPCGw2G7fccgtlZWUkJSXx9ttv+9Xt8aqrrsJut7uW9TmBUipSxfwzgsWLF7uyZVZWVvpdK3jkkUeqLD/++OMBL5tSSoVaTASCVatWUV5eDkB5eTkrV67067hhw4ZVmbxeu48qpaJRTASCnJwcV+bQxMREhg8f7vexzlqB1gaUUtFKnxEopVQMiPlnBJotUymlvIvqcQTuYwIOHjxIfHw8+/fvd40DiNWJZZRSyl1M1AjAMbduUlJSjVnGlFIq1kV1jcD9bt/5et68eaEqjlJKhaWYqREopZTyTAOBUkrFuKhrGnJ/QOxuvzXBjLeHw/rgWCkVq6IuEOTn57Nl+y4qU1KrrJcyx3iJTf/5tsYxccXHGqRsSikVjqIuEBQWFgI1B8mZxuf5OMpYxymlVOzRZwRKKRXjoi4QtG/fHpAa6+Xs98jZ770cJdZxSikVe6Kuaahz584e1+/ffwqALhe38bC1jdfjlFIq2kVdIPDW80cHlCmllGdR1zTkzbFjx9i6datOQK+UUtXETCA4ePAgAHPmzAlxSZRSKrxEXdOQO+fgsmPHfhgnUFFRwcSJE0lNTdVBZEopRYzUCJy1AW/LSikVy2JihrLs7Owa69asWROoIimlVNiL+RnKlFJKeRcTgaBPnz5Vlvv16xeikiilVPiJiUDw6KOPVll+5JFHQlQSpZQKPzERCNLS0ly1gn79+unk9Uop5SYmAgE4agU9e/bU2oBSSlUT1eMI3KWlpTF//vxQF0MppcJOzNQIlFJKeaaBQCmlYpwGAqWUinERN7JYRI4CB0JdDj+kAbZQFyKK6PUMHL2WgRUp17OjMaaVpw0RFwgihYhs9DacW9WdXs/A0WsZWNFwPbVpSCmlYpwGAqWUinEaCILn1VAXIMro9QwcvZaBFfHXU58RKKVUjNMagVJKxTgNBEopFeM0EJwDETkd6jJEIxEpEJG0AJ3rNhG5IBDnUrFFRNJFZIeH9XNEZFgtx84SkQeCV7rAipmkcyp6iUi8MabCy+bbgB3A4YYrUWSo5bopL4wxT4S6DIGmNYIAEIdnRWSHiGwXkfHW+j+IyCjr9XsistB6fYeIPBXKMgeadfe0R0Res67DmyIyTEQ+FZH9ItJfRFJF5O8isk1EPhORLOvYliKyUkS2iMgrgLidd6KIfC4iW0XkFRGJt9aftu7M/h8wUESeEJEN1nu/av2fjAP6Am9axyeLSB8R+ZeIbBKRD0SkbSiuV31Y126TiOwUkTtFZKqI/MZt+20iMt96Xe/rZu3Xz/p/Wu/83bbWx1vLG6ztU0JwKRpSvIgssK75Sut3aJH1u4WIXGf93q8TkXki8r7bsZeKyGoR+VJEpoeo/P4xxuhPPX+A09a/NwKrgHigNXAQaAvcAjxr7fM58Jn1+nXgmlCXP8DXIh2wAz1w3GBsAhbi+FIfDfwdmA/MtPa/CthqvZ4HPGG9HgkYHMP2M4F/AInWtj8AP7VeG+Bmt/dPdXv9J+AG6/VqoK/1OhH4N9DKWh4PLAz1tavDNU61/k3GUctpDeS7bc8Dfhyg67YDuMJ6/Wtgh/X6TuAx63USsBHoFOprE+Tf6V7W8jvARGARMA5oDBxyfn7gLeB96/Us63ctyfpdLnL+f4TjjzYNBcaPgbeMo5p9RET+BfQD1gL3icilwC6ghXUHOhAI7zuE+vnKGLMdQER2Ah8ZY4yIbMfxR9URR9DEGPOxVRM4H8gGxlrr/09EjlvnuxroA2ywblSTge+sbRXAu27vPVREHgJSgFRgJ44vQ3eXAN2BVdb54oFvAvPRG8R0ERljve4AdAK+FJHLgf04Pt+nwN2cw3UTkbVAM2PMv619/gxcb70eDmQ574iB84EuwFeB/KBh5CtjzFbr9SYcv8dOXYEvjTHOz/4WjkDp9H/GmFKgVES+wxG4C4Nc3nrRQBAY4mmlMeZrEWkBjADW4PhDuxlHTeJUA5avoZS6va50W67E8btm93CMqfavOwEWG2N+5WHbWSvwIiKNcdz19jXGHBKRWTju1jydb6cxZmBtHyTciMgQYBgw0BhTLCKrcXzGpTh+p/YA71mB91yvm8ffZ2dRgGnGmA8C88nCnvvvdAWOoOrk6zp5OjZsv2/1GUFgrAHGW+2nrXDc4X5ubVsP3GftsxZ4wPo3Fq0BbgXXF5vNGPN9tfXXAi2s/T8CxonIj6xtqSLS0cN5nV/6NhFpiqPa7nQKaGa93gu0EpGB1vkSRaRbgD5bsJ0PHLeCQFfgcmv934D/AibgCApwjtfNGHMcOGXVNMDRxOn0ATBVRBKtc2eISJOAfMLIswe4SETSreXxoSvKuQnbCBVh3sPR3PMFjjvbh4wx31rb1gLDjTH5InIAR60gVgPBLOB1EdkGFAO51vrZwFsishn4F45nLBhjdonIY8BKEYkDynE0e1RJQ26MOSEiC4DtQAGwwW3zIuBlESnB8X80DphnNUklAM/haEYKd/8E7rKu3V7gM3B8aYvILuBSY8zn1rpAXLc7gAUicgbHc5aT1vrXcDSPbLZqHkdxBKKYY4wpEZGfA/8UERs/3PxFHE0xoZSqQUSaGmNOW68fBtoaY+4NcbHCjvM6WUHxRWC/Meb3oS5XXWnTkFLKk5FW19MdwCAgqro7B9DPRGQrjlrl+cArIS5PvWiNQCmlYpzWCJRSKsZpIFBKqRingUDFBBG5T0RSQvC+za2eJUqFLQ0EKmqIg7ff6ftwjJ6ty/niz71UNAeCHghERLuCq3rTQKAimjiS3e0WkT8Am4E/ishGK0nYbGuf6cAFwCci8om1briVUG2ziPzFGlDlTIH9hIisA27y8p6dReRDEfnCOv5iEWkqIh9Zy9tFZLS1+6+Bi60eOM9axz8oPyRtm+123setBGarROQtsdIYi0gvcSTp2yaO5IUtrPWrReRpcaQ0eVREvnIb6HWe9VkSA33NVRQKdbIj/dGfc/nBMbipErjcWnYmZovHMRAqy1ouANKs12k4RjM3sZZ/yQ9J7wpwDAj09Z7/DxhjvW6Mo6aRAJzndv58HCkI0rEStlnbhuOY41Zw3Ii9j2Mkel9gK44UBs1w5A56wDpmGzDYej0HeM56vRr4g9u5Xwf+y3p9JzA31P8/+hMZP1qdVNHggDHmM+v1zSJyJ44v5rbApTi+SN1dbq3/1ErK1ghHKhCnpXghIs2AdsaY9wCMMWet9YnA0yKSjSMwtcORZKy64dbPFmu5KY6kbc2AZcaYEut8/7D+PR9oboz5l7X/YuAvXsr6GvAQjkyvk4CfefscSrnTQKCiwRkAEemEI5dTP+NIvbAI78nnVhljJvg6nxfeEo3dCrQC+hhjykWkwMd7P2OMqTLwSERm+HhPX1xlNcZ8ajWVDQbijTE1ZtdSyhN9RqCiyXk4vhhPikhr4Fq3be7J5z4DrhSRzgAikiIiGf68gXEkySsUkf+yjk2yeiOdD3xnBYGhOFJuV39fcCRtu93tmUQ7KzncOuAGEWlsbRtpvd9J4LiIDLKO/wmOfEzeLMGRDvl1fz6PUqA1AhVFjDFfiMgWHMP9v8SRm9/pVSBPRL4xxgwVkdtwJLpLsrY/Buzz861+ArwiInNwJHS7CXgT+IeIbMTR1r/HKlOROGZp2wHkGWMeFJFMYL3VLHUamGiM2SAiy3EkLjyAY8IXZ6K3XByJ81KszzXJR9nexJEO4i0/P4tSmmJCqXDhlsAsBcfD7DuNMZvreI5xwGhjzE+CUkgVlbRGoFT4eFUcs9k1xjGxTF2DwHwczWHXBaNwKnppjUApL0TkReDKaqufN8Zo+7uKKhoIlFIqxmmvIaWUinEaCJRSKsZpIFBKqRingUAppWKcBgKllIpxGgiUUirG/X/Qn+ATRUHS/AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.boxplot(x='rater_category',\n",
    "            y='correlation',\n",
    "            hue='system_category',\n",
    "            data=df_pairwise_system_rater_correlations_with_categories)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>N systems</th>\n",
       "      <th>R2 (GS)</th>\n",
       "      <th>r (GS)</th>\n",
       "      <th>r (`average`)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>system_category</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>poor</th>\n",
       "      <td>5</td>\n",
       "      <td>0.006296</td>\n",
       "      <td>0.709008</td>\n",
       "      <td>0.570367</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>low</th>\n",
       "      <td>5</td>\n",
       "      <td>0.403778</td>\n",
       "      <td>0.792025</td>\n",
       "      <td>0.637270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>medium</th>\n",
       "      <td>5</td>\n",
       "      <td>0.652204</td>\n",
       "      <td>0.861673</td>\n",
       "      <td>0.693415</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>high</th>\n",
       "      <td>5</td>\n",
       "      <td>0.801259</td>\n",
       "      <td>0.913514</td>\n",
       "      <td>0.735222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perfect</th>\n",
       "      <td>5</td>\n",
       "      <td>0.990063</td>\n",
       "      <td>0.995071</td>\n",
       "      <td>0.801093</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 N systems   R2 (GS)    r (GS)  r (`average`)\n",
       "system_category                                              \n",
       "poor                     5  0.006296  0.709008       0.570367\n",
       "low                      5  0.403778  0.792025       0.637270\n",
       "medium                   5  0.652204  0.861673       0.693415\n",
       "high                     5  0.801259  0.913514       0.735222\n",
       "perfect                  5  0.990063  0.995071       0.801093"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's also make a table that summarizes some metrics by system category\n",
    "# to do this, we need to merge a bunch of the dataframes we have already computed\n",
    "\n",
    "# for the first frame, keep only the R2 column along with the system metadata\n",
    "df_rmse_r2_table = df_rmse_r2_with_categories[['system_id',\n",
    "                                               'system_category',\n",
    "                                               'r2']]\n",
    "\n",
    "# for the second, keep only the true score correlation along with metadata\n",
    "df_true_score_correlations_table = df_system_metadata_with_true_score_correlations[['system_id',\n",
    "                                                                                    'system_category',\n",
    "                                                                                    'true_score_correlation']]\n",
    "\n",
    "# for the third, keep only the correlations and only for the average rater, along with metadata\n",
    "rater_category_is_average = df_pairwise_system_rater_correlations_with_categories['rater_category'] == 'average'\n",
    "df_average_rater_correlations_table = df_pairwise_system_rater_correlations_with_categories[rater_category_is_average]\n",
    "df_average_rater_correlations_table = df_average_rater_correlations_table[['system_id',\n",
    "                                                                           'system_category',\n",
    "                                                                           'correlation']]\n",
    "\n",
    "\n",
    "# now merge all three frames together\n",
    "df_metrics_table = (df_rmse_r2_table.merge(df_true_score_correlations_table)\n",
    "                                    .merge(df_average_rater_correlations_table))\n",
    "\n",
    "# add in the number of systems per category to include in the table\n",
    "df_metrics_table['N systems'] = dataset.num_systems_per_category\n",
    "\n",
    "# rename the column names to be a bit more specific to this table\n",
    "df_metrics_table.rename(columns={'r2': 'R2 (GS)',\n",
    "                                 'true_score_correlation': 'r (GS)',\n",
    "                                 'correlation': 'r (`average`)'}, inplace=True)\n",
    "\n",
    "# now group by the system category, compute the mean, and re-order the index and columns\n",
    "system_description_table = df_metrics_table.groupby('system_category').mean().loc[dataset.system_categories, \n",
    "                                                                                  ['N systems', 'R2 (GS)', 'r (GS)', 'r (`average`)']]\n",
    "system_description_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save dataset and frames to disk\n",
    "\n",
    "Now let's save this dataset instance to disk so that we can reuse it for additional analyses in other notebooks. \n",
    "If you want to save your dataset in a standard tabular format, you can use `dataset.save_frames(output_dir)`.\n",
    "\n",
    "**Note**: The dataset frames are only saved for convenience. Any changes made to these CSV files will _not_ affect any of the analyses in the other notebook since they only use the `.dataset` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first save the full dataset instance\n",
    "dataset.save('../data/default.dataset')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We only keep our data in .csv so that the user always has it for reference even if they play with the settings\n",
    "dataset.save_frames('../data')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "prmse",
   "language": "python",
   "name": "prmse"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}