7.2 AB test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d6110f2f-1109-4a3b-b9d6-b9374be33f5c",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Задание 2.\n",
    "Пришло время проанализировать результаты эксперимента, который мы провели вместе с командой дата сайентистов. Эксперимент проходил  с 2022-01-28 по 2022-02-03 включительно. Для эксперимента были задействованы 0 и 1 группы.  \n",
    "В 1 группе был использован новый алгоритм рекомендации постов, 0 группа использовалась в качестве контроля. \n",
    "Основная гипотеза заключается в том, что новый алгоритм в 1 группе приведет к увеличению CTR. \n",
    "Ваша задача — проанализировать данные АB-теста.\n",
    "\n",
    "### Задачи:\n",
    "1. Выбрать метод анализа и сравнить CTR в двух группах.\n",
    "2. Провести контрольную проверку, что новый алгоритм не привел к тому, что сократилось число просмотров у пользователей. Типичная ситуация в АB-тестах — вырастили число лайков, но уменьшили число просмотров.\n",
    "3. Написать рекомендацию, будем ли мы раскатывать новый алгоритм на всех новых пользователей."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ad73a3f-8b6c-45cb-88b5-33b5f44c60e0",
   "metadata": {},
   "source": [
    "### 1. Подготовка к работе"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d86ef028-b2a1-42d8-ba27-d97dd746dc35",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Подгружаем библиотеки\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import pandahouse\n",
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0834b915-658d-4f28-b918-473d98ff89ab",
   "metadata": {},
   "source": [
    "### 2. Загрузка данных"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4a5d0ffe-1c5b-40b5-ad25-cc9e979e390f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Подключение к БД\n",
    "connection = {\n",
    "    'host': 'https://clickhouse.lab.karpov.courses',\n",
    "    'password': 'dpo_python_2020',\n",
    "    'user': 'student',\n",
    "    'database': 'simulator_20220120'\n",
    "}\n",
    "\n",
    "# Запрос к БД\n",
    "query = '''\n",
    "SELECT user_id,\n",
    "       exp_group,\n",
    "       countIf(user_id, action = 'like' ) as like,\n",
    "       countIf(user_id, action = 'view' ) as view,\n",
    "       like/view as ctr\n",
    "FROM simulator_20220120.feed_actions\n",
    "WHERE (toDate(time) >= '2022-01-28' AND toDate(time) <= '2022-02-03' \n",
    "AND exp_group in [0,1])\n",
    "GROUP BY user_id, exp_group\n",
    "'''\n",
    "\n",
    "# Создание датафрейма\n",
    "df = pandahouse.read_clickhouse(query, connection=connection)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cdd44dbd-d88f-4c8e-a06c-4225227b6e19",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>exp_group</th>\n",
       "      <th>like</th>\n",
       "      <th>view</th>\n",
       "      <th>ctr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>26117</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>21</td>\n",
       "      <td>0.238095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>138232</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>29</td>\n",
       "      <td>0.172414</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>49204</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>62</td>\n",
       "      <td>0.161290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49286</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>19</td>\n",
       "      <td>0.368421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>145957</td>\n",
       "      <td>1</td>\n",
       "      <td>54</td>\n",
       "      <td>129</td>\n",
       "      <td>0.418605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25994</th>\n",
       "      <td>3681</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>28</td>\n",
       "      <td>0.214286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25995</th>\n",
       "      <td>110956</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>63</td>\n",
       "      <td>0.190476</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25996</th>\n",
       "      <td>132567</td>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "      <td>66</td>\n",
       "      <td>0.272727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25997</th>\n",
       "      <td>30872</td>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>82</td>\n",
       "      <td>0.304878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25998</th>\n",
       "      <td>11718</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>37</td>\n",
       "      <td>0.189189</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>25999 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       user_id  exp_group  like  view       ctr\n",
       "0        26117          1     5    21  0.238095\n",
       "1       138232          1     5    29  0.172414\n",
       "2        49204          1    10    62  0.161290\n",
       "3        49286          1     7    19  0.368421\n",
       "4       145957          1    54   129  0.418605\n",
       "...        ...        ...   ...   ...       ...\n",
       "25994     3681          0     6    28  0.214286\n",
       "25995   110956          1    12    63  0.190476\n",
       "25996   132567          1    18    66  0.272727\n",
       "25997    30872          1    25    82  0.304878\n",
       "25998    11718          0     7    37  0.189189\n",
       "\n",
       "[25999 rows x 5 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Проверка df\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dbe4373-5188-43cb-84fd-5c9db81bcf72",
   "metadata": {},
   "source": [
    "### 3. Анализ данных"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b57733eb-cf7b-49d9-9960-4b7c71ffc0a6",
   "metadata": {},
   "source": [
    "Визуализируем распределение CTR в группах 0 и 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f069a656-9fc2-4c9b-8e44-21382a0e0b4b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean_ctr: 0.22195492956578466\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAFgCAYAAACFYaNMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbPklEQVR4nO3de5CcV3nn8e9jOb6NL5IlWbZk+bZShE02gBlYc9kU4CXlS4KcBRxTCRaUYXYThYRNNotZUhXvLlvAXkJg1+WUgrPIFDfjBSxix1lHGFKhYkBgYbCNcDNrlSVfJNlGYInLavzsH31Gbg8jqUeat09P9/dT1TXve/r0zFM9o5/ePu95zxuZiSSp946qXYAkDSsDWJIqMYAlqRIDWJIqMYAlqZKjaxdwJC655JK84447apchSZ2i245z+gh4165dtUuQpMM2pwNYkuYyA1iSKjGAJakSA1iSKjGAJakSA1iSKjGAJakSA1iSKjGAJakSA1iSKjGAJakSA1iSKpnTq6HpyExMTNBqtfbvr1ixgnnz5lWsSBouBvAQa7VajF1/GyOLlrJn1yOsW3s5q1atql2WNDQM4CE3smgpJ59+du0ypKHkGLAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IljQVwRKyKiM0djx9GxDsj4tSIuDMiHixfF5T+EREfjohWRNwbERc2VZsk9YPGAjgzt2TmCzPzhcCLgb3A54BrgY2ZuRLYWPYBLgVWlscYcENTtUlSP+jVEMTFwPczcyuwGlhf2tcDV5Tt1cBN2XY3MD8izuhRfZLUc70K4KuAT5btJZn5aNl+DFhStpcBD3e8Zltpe46IGIuITRGxaefOnU3VK0mNazyAI+IY4HXAZ6Y+l5kJ5Ey+X2auy8zRzBxdvHjxLFUpSb3XiyPgS4FvZubjZf/xyaGF8nVHad8OLO943ZmlTZIGUi8C+E08O/wAsAFYU7bXALd2tF9dZkNcBOzuGKqQpIHT6F2RI2IEeC3wrzqa3w/cHBHXAFuBK0v77cBlQIv2jIm3NlmbJNXWaABn5h5g4ZS2J2jPipjaN4G1TdYjSf3EK+EkqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqZJG1wNWf5iYmKDVau3fX7FiBfPmzatYkSQwgIdCq9Vi7PrbGFm0lD27HmHd2stZtWpV7bKkoWcAD4mRRUs5+fSza5chqYNjwJJUiQEsSZUYwJJUiQEsSZV4Em7I5DPPMD4+DsD4+DiZlQuShpgBPGT2PPkY131+KwuX7Wbng5s5afn5tUuShpZDEENoZGF7StrxC06rXYo01AxgSarEAJakSgxgSarEAJakSgxgSaqk0QCOiPkRcUtEfDciHoiIl0XEqRFxZ0Q8WL4uKH0jIj4cEa2IuDciLmyyNkmqrekj4A8Bd2Tm84AXAA8A1wIbM3MlsLHsA1wKrCyPMeCGhmuTpKoaC+CIOAX4FeBGgMz8WWb+AFgNrC/d1gNXlO3VwE3ZdjcwPyLOaKo+SaqtySPgc4GdwP+KiHsi4iMRMQIsycxHS5/HgCVlexnwcMfrt5W254iIsYjYFBGbdu7c2WD5ktSsJgP4aOBC4IbMfBGwh2eHGwDIzARmtBpBZq7LzNHMHF28ePGsFStJvdZkAG8DtmXmV8v+LbQD+fHJoYXydUd5fjuwvOP1Z5Y2SRpIjQVwZj4GPBwRkzcfuxi4H9gArClta4Bby/YG4OoyG+IiYHfHUIW6MDExwZYtW/Y/JiYmapck6SCaXg3tHcDHI+IYYBx4K+3QvzkirgG2AleWvrcDlwEtYG/pqxmYrZtvehdlqTcaDeDM3AyMTvPUxdP0TWBtk/UMg9m4+aZ3UZZ6w/WA57jOo9XZXGDduyhLzTOA57jOo9XOBdZneucL75Qh9Z4BPAAmj1af3vXI/raZ3vnCO2VIvediPANspne+8E4ZUm8ZwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUcXbsA9bd85hnGx8f3769YsYJ58+ZVrEgaHAawDmrPk49x3ee3snDZbvbseoR1ay9n1apVtcuSBkKjQxAR8VBEfDsiNkfEptJ2akTcGREPlq8LSntExIcjohUR90bEhU3Wpu6NLFzKyaefzQmnns74+DhbtmxhYmKidlnSnNeLMeBXZ+YLM3O07F8LbMzMlcDGsg9wKbCyPMaAG3pQm2agfTS8mbHrb6PVatUuR5rzapyEWw2sL9vrgSs62m/KtruB+RFxRoX6dBAjC5cysmhp7TKkgdB0ACfwfyLiGxExVtqWZOajZfsxYEnZXgY83PHabaXtOSJiLCI2RcSmnTt3NlW3JDWu6ZNwr8zM7RFxGnBnRHy388nMzIjImXzDzFwHrAMYHR2d0WslqZ80egScmdvL1x3A54CXAo9PDi2UrztK9+3A8o6Xn1naJGkgNRbAETESESdNbgO/CnwH2ACsKd3WALeW7Q3A1WU2xEXA7o6hCkkaOE0OQSwBPhcRkz/nE5l5R0R8Hbg5Iq4BtgJXlv63A5cBLWAv8NYGa5Ok6hoL4MwcB14wTfsTwMXTtCewtql6JKnfuBaEJFViAEtSJQawJFViAEtSJQawJFViAEtSJQawJFViAEtSJQawJFViAEtSJQawJFXiTTl1RCYmJp5zeyLvmix1zwDWEWm1Woxdfxsji5Z612RphgxgHbGRRe27JkuaGQNYM5bPPMP4+DgA4+PjpDeGkg6LAawZa9+efisLl+1m54ObOWn5+bVLkuYkZ0HosIwsbA87HL/gtNqlSHOWASxJlRjAklSJASxJlRjAklSJASxJlRjAklSJ84DnCNdckAZPVwEcEa/IzK8cqk3Ncc0FafB0OwTxP7psU4Mm11wYWbS0dimSZsFBj4Aj4mXAy4HFEfGHHU+dDPj5V5KOwKGGII4BTiz9Tupo/yHwhqaKkqRhcNAAzswvA1+OiI9m5tYe1SRJQ6HbWRDHRsQ64JzO12Tma5ooSpKGQbcB/BngL4CPABMz+QERMQ/YBGzPzF+LiHOBTwELgW8Ab87Mn0XEscBNwIuBJ4DfzMyHZvKzJGku6XYWxL7MvCEzv5aZ35h8dPnaPwAe6Nj/APDBzFwBPAVcU9qvAZ4q7R8s/SRpYHUbwF+IiN+NiDMi4tTJx6FeFBFnApfTPnImIgJ4DXBL6bIeuKJsry77lOcvLv0laSB1OwSxpnz94462BM47xOv+HPh3PDuDYiHwg8zcV/a3AcvK9jLgYYDM3BcRu0v/XZ3fMCLGgDGAs846q8vyJan/dBXAmXnuTL9xRPwasCMzvxERr5rp6w9SyzpgHcDo6Kh3I5M0Z3V7KfLV07Vn5k0HedkrgNdFxGXAcbQv3vgQMD8iji5HwWcC20v/7cByYFtEHA2cQvtknCQNpG7HgF/S8fjnwHXA6w72gsx8d2aemZnnAFcBX8zM3wLu4tmLONYAt5btDTw71PGG0t8jXEkDq9shiHd07kfEfNpTyQ7Hu4BPRcR7gXuAG0v7jcDHIqIFPEk7tCVpYB3ucpR7gK7HhTPzS8CXyvY48NJp+vwEeONh1jNU8plnGB8fB2B8fJx+/Jzg8pnSoXU7BvwF2rMeoL0Iz/nAzU0VpYPb8+RjXPf5rSxctpudD27mpOXn1y7p57h8pnRo3R4B/7eO7X3A1szc1kA96tLIwvbSlE/veqR2KQc0uXympOl1dRKuLMrzXdrzeRcAP2uyKEkaBl0FcERcCXyN9hjtlcBXI8LlKCXpCHQ7BPEe4CWZuQMgIhYDf8ezlxRLkmao23nAR02Gb/HEDF4rSZpGt0fAd0TE3wKfLPu/CdzeTEmSNBwOdU+4FcCSzPzjiPiXwCvLU/8IfLzp4iRpkB3qCPjPgXcDZOZngc8CRMQ/Lc/9eoO1SdJAO9Q47pLM/PbUxtJ2TiMVSdKQOFQAzz/Ic8fPYh2SNHQOFcCbIuLtUxsj4m207+cmSTpMhxoDfifwuYj4LZ4N3FHgGOA3GqxLkgbeQQM4Mx8HXh4RrwZ+qTTflplfbLwyzTlzYZU2qZ90ux7wXbQXUpcOaC6s0ib1E69m06yaXKXt+AWn1S5F6nsGsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiXd3hVZFUxMTNBqtQCXd5QGkQHcx1qtFmPX38bIoqUu7ygNoMaGICLiuIj4WkR8KyLui4j/UNrPjYivRkQrIj4dEceU9mPLfqs8f05Ttc0lI4tc3lEaVE2OAf8UeE1mvgB4IXBJRFwEfAD4YGauAJ4Crin9rwGeKu0fLP0kaWA1FsDZ9nTZ/YXySOA1wC2lfT1wRdleXfYpz18cEdFUfZJUW6NjwBExj/bNPFcA1wPfB36QmftKl23AsrK9DHgYIDP3RcRuYCGwq8ka1VudJxYBVqxYwbx58ypWJNXTaABn5gTwwoiYD3wOeN6Rfs+IGAPGAM4666wj/Xbqgak363zf7Q9w4uKl7Nn1COvWXs6qVasqVyjV0ZNZEJn5g4i4C3gZMD8iji5HwWcC20u37cByYFtEHA2cAjwxzfdaB6wDGB0ddWLWHDDdzTpPPv3s2mVJ1TU5C2JxOfIlIo4HXgs8QPvuym8o3dYAt5btDWWf8vwXM535Oii8Waf085o8Aj4DWF/GgY8Cbs7Mv46I+4FPRcR7gXuAG0v/G4GPRUQLeBK4qsHaJKm6xgI4M+8FXjRN+zjw0mnafwK8sal6JKnfuBaEJFViAEtSJQawJFViAEtSJQawJFViAEtSJa4H3GdchF0aHgZwn3ERdml4GMB9aHIR9qd3PVK7lCpcMU3DwgBW3+n8FOCKaRpkBrD60uSnAGmQOQtCkioxgCWpEgNYkipxDFjVdN6qCJztoOFjAKuazlsVOdtBw8gAVlWTtyqShpFjwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUifOA+8TkGrjeBUMaHgZwn5hcA3fvUzu8C4Y0JAzgPjKyaCke/ErDwwBWX+hcmMdhGA0LA1h9oXNhnm5uRup94zQIDGD1jcmFebq5Gan3jdMgaGwaWkQsj4i7IuL+iLgvIv6gtJ8aEXdGxIPl64LSHhHx4YhoRcS9EXFhU7VpMEzeN25k0dLapUiHpcl5wPuAP8rMC4CLgLURcQFwLbAxM1cCG8s+wKXAyvIYA25osDZJqq6xAM7MRzPzm2X7R8ADwDJgNbC+dFsPXFG2VwM3ZdvdwPyIOKOp+iSptp5cCRcR5wAvAr4KLMnMR8tTjwFLyvYy4OGOl20rbZI0kBoP4Ig4EfjfwDsz84edz2VmwsymvkbEWERsiohNO3funMVKJam3Gg3giPgF2uH78cz8bGl+fHJooXzdUdq3A8s7Xn5maXuOzFyXmaOZObp48eLmipekhjU5CyKAG4EHMvPPOp7aAKwp22uAWzvary6zIS4CdncMVUjSwGlyHvArgDcD346IzaXt3wPvB26OiGuArcCV5bnbgcuAFrAXeGuDtUlSdY0FcGb+AxAHePriafonsLapeiSp37gesCRVYgBLUiWuBaG+1rlKGrjojgaLAay+1rlKmovuaNAYwOp7k6ukSYPGMWBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsS1IDRndK6MNj4+Ts7odq5S/zGAe2xiYoJWq7V/3+UVu9e5MtrOBzdz0vLza5ckHREDuMdarRZj19/GyKKlLq94GCZXRnt61yO1S5GOmAFcwcgil1eU5Ek4SarGI+CKPKkkDTcDuCJPKknDzSGIyiZPKh2/4LTapUjqMQNYkioxgCWpEseANed1nswEL27R3GEAa87rPJnpxS2aSwxgDYTJk5nSXOIYsCRV4hGwBp4LIKlfNXYEHBF/FRE7IuI7HW2nRsSdEfFg+bqgtEdEfDgiWhFxb0Rc2FRdGj6TCyD9m0/fw9j1tz0njKWamhyC+ChwyZS2a4GNmbkS2Fj2AS4FVpbHGHBDg3X1xMTEBFu2bNn/mJiYqF3SUJtcAGlk0dLapUj7NTYEkZl/HxHnTGleDbyqbK8HvgS8q7TflJkJ3B0R8yPijMx8dDZr6uVHUZedrGO6KWlSv+r1GPCSjlB9DFhStpcBD3f021bafi6AI2KM9lEyZ5111ox+eK9D0WUne2+6KWlSv6p2Ei4zMyJmvP5XZq4D1gGMjo7O+PU1QtFVz3rrYFPSvGhD/aTXAfz45NBCRJwB7Cjt24HlHf3OLG0DwVXP+ocXbaif9Hoe8AZgTdleA9za0X51mQ1xEbB7tsd/a3PVs/4x+bvwhJxqa+wIOCI+SfuE26KI2Ab8KfB+4OaIuAbYClxZut8OXAa0gL3AW5uqS8Ojc7jBYR/1oyZnQbzpAE9dPE3fBNY2VYuG0+RwwzN7dzvso77klXAaaCMLl7Lv2GNrlyFNy7UgJKkSA1iSKnEIQkPLOcGqzQDW0DrQnGBXT1OvGMAaatNdNec6HuoVA1iahut4qBcMYAnX61AdBrCE63WoDqehSYXrdajXPAKWuuTsCM02A1jqkrMjNNsM4FnUeYTkiZzB5OwIzSYD+AhNDd333f4AJy5e6okcSYdkAB+hzo+lk6F78uln8/SuR2qXplng9DQ1yQCeBZMfSw3dwXOg6WkHWkfCE3WaCQP4MDjWO1wmp6d1/gd7oHUkZvtEnYE+2AzgwzDdsIOGz4HuvjybJ+qceTHYDODD5LCDesWZF4PLAJZ6zGEFTTKApSM005kSDitokgEsHaFuFvKZeuL2hAOMH2u4GMDSLJhupsTUI2Mv0tFUBrDUkOmOjD1xq04uRyk1yCUudTAeAUsVdQ5TTExMADBv3jxnRgwJA/ggOk+cdP7j8Oo3zZapwxRHnXAKxx177KzMjHC6W/8zgA9i6hVvR51wCguXnetJFM2qzhN4R5+4kOOOO3ZWvq/T3fqfAXwInVe8HX3iQk+iqK853W1uMYCncKEdzQUHGx6bbrrbTFdv69XwxbAPk/RVAEfEJcCHgHnARzLz/b2uwYV2NBccanjsYKu3Pb1jG+++/Pmcd955zwnsbtphdkOyxjBJP4V+3wRwRMwDrgdeC2wDvh4RGzLz/qZ/9oE+tjnUoBoOdGnz1PYTFs5seKxzrPm6z2+edn7yodonw/jss9vDGpNHy9NtQ3dH1LOx2NCBPhFM/Vlw4NCvEcx9E8DAS4FWZo4DRMSngNXArAbwnvIHumfXI4yPnwK0/5j/5GMbOWHBaTy59bucuGwlEfDjp3Zw1E9/yg+PO7Zn28/s3d3znzmINc7len+0dzfv+s6POWXJsuf8Pe76/r3Tth/WzznhlGf/TTzxSNftP969i3fdeAcTP/kR8447aX8t023vfWoH733zxfuPqCf/jU1tn+7f5ExN/Tc8XQ2dfae+dur32PvUDj7xp29v/Gg8sk8GOSPiDcAlmfm2sv9m4J9l5u9N6TcGjJXdVcCWGf6oRcCuIyx3tljLz+uXOsBaptMvdUD/1rIrMy/p5kX9dATclcxcB6w73NdHxKbMHJ3Fkg6btfRvHWAt/VwHDEYt/XQp8nZgecf+maVNkgZSPwXw14GVEXFuRBwDXAVsqFyTJDWmb4YgMnNfRPwe8Le0p6H9VWbe18CPOuzhiwZYy8/rlzrAWqbTL3XAANTSNyfhJGnY9NMQhCQNFQNYkioZ2ACOiEsiYktEtCLi2mmePzYiPl2e/2pEnFOxll+JiG9GxL4yH7pWHX8YEfdHxL0RsTEiGlvFpYta/nVEfDsiNkfEP0TEBbVq6ej3+ojIiGhs6lMX78tbImJneV82R8TbatRR+lxZ/l7ui4hPNFFHN7VExAc73o/vRcQPKtZyVkTcFRH3lH9Hlx30G2bmwD1on8T7PnAecAzwLeCCKX1+F/iLsn0V8OmKtZwD/DJwE/CGinW8GjihbP9O5ffk5I7t1wF31Kql9DsJ+HvgbmC04vvyFuB/NvHzZ1jHSuAeYEHZP63m76ej/zton8Cv9b6sA36nbF8APHSw7zmoR8D7L2vOzJ8Bk5c1d1oNrC/btwAXR0TUqCUzH8rMe4FnGvj5M6njrszcW3bvpj0Xu1YtP+zYHQGaOlvczd8KwH8CPgD8pKE6ZlJL07qp4+3A9Zn5FEBm7qhYS6c3AZ+sWEsCJ5ftU4CDLigzqAG8DHi4Y39baZu2T2buA3YDCyvV0gszreMa4G9q1hIRayPi+8B/AX6/Vi0RcSGwPDNva6iGrmspXl8+3t4SEcuneb4Xdfwi8IsR8ZWIuLusZNiErv9uy5DZucAXK9ZyHfDbEbENuJ32EfkBDWoA6whExG8Do8B/rVlHZl6fmf8EeBfwJzVqiIijgD8D/qjGz5/GF4BzMvOXgTt59lNcrx1NexjiVbSPOv8yIuZXqmXSVcAtmTlRsYY3AR/NzDOBy4CPlb+haQ1qAHdzWfP+PhFxNO2PC09UqqUXuqojIv4F8B7gdZn505q1dPgUcEWlWk4Cfgn4UkQ8BFwEbGjoRNwh35fMfKLj9/IR4MU16qB99LchM/9fZv5f4Hu0A7lGLZOuornhh25ruQa4GSAz/xE4jvZCPdNrcjC/1oP2/87jtD+OTA6WP39Kn7U89yTczbVq6ej7UZo7CdfNe/Ii2icZVvbB72dlx/avA5tq/35K/y/R3Em4bt6XMzq2fwO4u1IdlwDry/Yi2h/NF9b6/QDPAx6iXFxW8ffzN8Bbyvb5tMeAD1hTI4X2w4P24f/3SqC8p7T9R9pHdtD+n+kzQAv4GnBexVpeQvuIYg/to/D7KtXxd8DjwOby2FDxPfkQcF+p466DhWLTtUzp21gAd/m+vK+8L98q78vzKtURtIdm7ge+DVxV8/dDe+z1/U3VMIP35QLgK+X3sxn41YN9Py9FlqRKBnUMWJL6ngEsSZUYwJJUiQEsSZUYwJJUiQGsoRcRVzS52pp0IAaw1L7KbtoALldJSo1wHrCGTkRcDfxb2itX7aB9Oe/u8ng9cCPtSfSvBD6Zmf+9TqUadP7vrqESEc+nvbDPyzNzV0ScSvuKrr/OzFtKH4BjMrOxhdclcAhCw+c1wGcycxdAZj55gH6f7l1JGlYGsDS9PbUL0OAzgDVsvgi8MSIWApQhiB/RXnZS6inHgDVUMvO+iPjPwJcjYoL2fc3+kvaC4r8PNHZTVGkqZ0FIUiUOQUhSJQawJFViAEtSJQawJFViAEtSJQawJFViAEtSJf8fq7SZ7mmCZlUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Распределение экспериментальной группы 0\n",
    "sns.displot(df[df.exp_group == 0].ctr, kde = False);\n",
    "print('mean_ctr:', df[df.exp_group == 0].ctr.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c0f340f2-89a1-4056-9af4-9ebb0a81db6a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean_ctr: 0.23648646948093363\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAFgCAYAAACFYaNMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbH0lEQVR4nO3df5RfdX3n8edbUsRAICQMOdkEDOKoQbcqjoo/To/K2sMPa+iqFLdK9GDjttHWY9uFrnu2XWlPcX/U6sqhpOIaXCsiqyexIF2KYE89okbkxwLiZFgiyUIyEEBiqnbCe//4fibcjJPkO8nc+XznO8/HOXPm3s/3fr/fd75z55U7n/u5nxuZiSRp5j2rdgGSNFcZwJJUiQEsSZUYwJJUiQEsSZXMq13A4TjrrLPyxhtvrF2GJB1MTNY4q4+AH3300dolSNIhm9UBLEmzmQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZXM6tnQNHVjY2MMDw/vXR8cHGTePHcDqQZ/8+aY4eFh3n/59RwzsIxdo9u4cu25rFy5snZZ0pxkAM9Bxwws49ilK2qXIc159gFLUiUGsCRV0loAR8QLI+KOxtePI+JDEbEoIm6KiOHy/fiyfUTEJyNic0TcFRGnt1WbJPWC1gI4M+/PzJdl5suAVwC7ga8AlwA3Z+YgcHNZBzgbGCxfa4Ar2qpNknrBTHVBnAmMZOYWYBWwvrSvB84ry6uAq7PjNmBhRCydofokacbNVABfAHyhLC/JzIfL8iPAkrK8DHio8ZytpW0fEbEmIjZFxKbR0dG26pWk1rUewBFxJPBW4EsTH8vMBHIqr5eZ6zJzKDOHBgYGpqlKSZp5M3EEfDZwe2ZuL+vbx7sWyvcdpX0bcFLjectLmyT1pZkI4HfyTPcDwEZgdVleDWxotF9YRkOcATzZ6KqQpL7T6pVwEXE08Gbg/Y3my4BrI+IiYAtwfmm/ATgH2ExnxMR726xNkmprNYAz8yfA4gltj9EZFTFx2wTWtlmPJPUSr4STpEqcjKdPHe60k05bKbXP36g+dbjTTjptpdQ+A7iPHe60k05bKbXLPmBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsTZ0LRXcw7gkZEROjcpkdQWA1h7NecA3nH/7Sx47kqOq12U1MfsgtA+xucAnr9oSe1SpL5nAEtSJQawJFViH7AO6uk9exgZGdm77g06penhb5EOavfOR7h0wxYWL3/KG3RK08gAVlfmn7DUG3RK08w+YEmqxACWpEoMYEmqxACWpEoMYEmqxFEQc0BzHK+T7Ei9wwCeA5rjeJ1kR+odrXZBRMTCiLguIn4QEfdFxGsiYlFE3BQRw+X78WXbiIhPRsTmiLgrIk5vs7a5Znwcr5PsSL2j7T7gTwA3ZuaLgJcC9wGXADdn5iBwc1kHOBsYLF9rgCtark2SqmotgCPiOOBXgKsAMvPnmfkEsApYXzZbD5xXllcBV2fHbcDCiFjaVn2SVFubR8CnAKPA/4iI70fEpyPiaGBJZj5ctnkEGP+beBnwUOP5W0vbPiJiTURsiohNo6OjLZYvSe1qM4DnAacDV2Tmy4Gf8Ex3AwDZOR0/pVPymbkuM4cyc2hgYGDaipWkmdZmAG8Ftmbmt8v6dXQCeft410L5vqM8vg04qfH85aVNkvpSawGcmY8AD0XEC0vTmcC9wEZgdWlbDWwoyxuBC8toiDOAJxtdFZLUd9oeB/xB4PMRcSTwAPBeOqF/bURcBGwBzi/b3gCcA2wGdpdtJalvtRrAmXkHMDTJQ2dOsm0Ca9usR5J6iXNBSFIlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1Il3pKoj4yNjTE8PAx47zdpNjCA+8jw8DDvv/x6jhlY5r3fpFnALog+c8zAMu/9Js0SBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlzgWhQ9ac/AdgcHCQefPcpaRu+duiQ9ac/GfX6DauXHsuK1eurF2WNGsYwDos45P/SJo6+4AlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqZJWAzgiHoyIuyPijojYVNoWRcRNETFcvh9f2iMiPhkRmyPirog4vc3aJKm2mTgCfmNmviwzh8r6JcDNmTkI3FzWAc4GBsvXGuCKGahNkqqpcSXcKuANZXk9cCtwcWm/OjMTuC0iFkbE0sx8uEKNOgzOESF1p+3figT+d0QkcGVmrgOWNEL1EWBJWV4GPNR47tbStk8AR8QaOkfInHzyyS2WrkPlHBFSd9oO4Ndn5raIOBG4KSJ+0HwwM7OEc9dKiK8DGBoamtJzNXOcI0I6uFYDODO3le87IuIrwKuA7eNdCxGxFNhRNt8GnNR4+vLSppY8vWcPIyMje9dHRkbo9ABJmgmtBXBEHA08KzOfKsu/CnwU2AisBi4r3zeUp2wEPhAR1wCvBp60/7ddu3c+wqUbtrB4+VMA7Lj/dhY8dyXHVa5LmivaPAJeAnwlIsbf528y88aI+C5wbURcBGwBzi/b3wCcA2wGdgPvbbE2FfNPWLq3q2DXqH9wSDOptQDOzAeAl07S/hhw5iTtCaxtqx5J6jVeCSdJlRjAklSJASxJlXh5kqakOXTNYWvS4TGANSXNoWsOW5MOj10QmrLxoWvzFy05+MaS9ssAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKvBBjlmvef80r06TZxQCe5Zr3X/PKNGl2sQuiD4zff80r06TZxQCWpEoMYEmqxACWpEo8Cadp4TzB0tQZwJoWzhMsTZ1dEJo2zhMsTY0BLEmVGMCSVIkBLEmVGMCSVIkBLEmVGMCSVIkBLEmVGMCSVIkBLEmVeCnyLORdMKT+0FUAR8TrMvObB2vbz3OPADYB2zLzLRFxCnANsBj4HvDuzPx5RDwbuBp4BfAY8BuZ+eCU/jVzhHfBkPpDt10Q/73Ltsn8HnBfY/1jwMcz8/nA48BFpf0i4PHS/vGynfbDu2BIs98Bj4Aj4jXAa4GBiPhw46FjgSMO9uIRsRw4F/gz4MMREcCbgH9TNlkP/AlwBbCqLANcB3wqIiL9+1pSnzrYEfCRwDF0gnpB4+vHwNu7eP2/BP4d8HRZXww8kZljZX0rsKwsLwMeAiiPP1m230dErImITRGxaXR0tIsSJKk3HfAIODO/AXwjIj6bmVum8sIR8RZgR2Z+LyLecOgl/kJN64B1AENDQx4dS5q1uh0F8eyIWAesaD4nM990gOe8DnhrRJwDHEWn2+ITwMKImFeOcpcD28r224CTgK0RMQ84js7JOPWJ5ugNgMHBQebNcyCO5q5u9/4vAX8FfBrY080TMvOPgD8CKEfAf5CZvxkRX6LTfXENsBrYUJ6ysax/qzz+dft/+0tz9Mau0W1cufZcVq5cWbssqZpuA3gsM6+Ypve8GLgmIv4U+D5wVWm/CvhcRGwGdgIXTNP7qYeMj96Q1H0AfzUifgf4CvCz8cbM3NnNkzPzVuDWsvwA8KpJtvkp8I4u65GkWa/bAF5dvv9hoy2B501vOZormndRBvuDNTd1tcdn5iltF6K5pXkXZfuDNVd1eynyhZO1Z+bV01uO5pLxuyhLc1W3f/O9srF8FHAmcDuduRskSYeg2y6IDzbXI2IhnWFkkqRDdKjzAf8EsF9Ykg5Dt33AX6Uz6gE6k/CsBK5tqyhJmgu67QP+r43lMWBLZm5toR5JmjO66oIok/L8gM5MaMcDP2+zKEmaC7oK4Ig4H/gOnSvVzge+HRHdTEcpSdqPbrsgPgK8MjN3AETEAPD3dCZOlyQdgm5HQTxrPHyLx6bwXEnSJLo9Ar4xIv4O+EJZ/w3ghnZKkqS54WD3hHs+sCQz/zAi/jXw+vLQt4DPt12cJPWzgx0B/yVlUvXM/DLwZYCI+JflsV9rsTZJ6msHC+AlmXn3xMbMvDsiVrRTkvpJc9rJkZERvMmJ9IyDBfDCAzz2nGmsQ32qOe3kjvtvZ8FzV3Jc7aKkHnGwkQybIuK3JjZGxPuA77VTkvrN+LST8xctqV2K1FMOdgT8IeArEfGbPBO4Q8CRwK+3WJck9b0DBnBmbgdeGxFvBF5Smq/PzK+3Xpkk9blu5wO+Bbil5VokaU7xajZJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKWgvgiDgqIr4TEXdGxD0R8Z9K+ykR8e2I2BwRX4yII0v7s8v65vL4irZqk6Re0OYR8M+AN2XmS4GXAWdFxBnAx4CPZ+bzgceBi8r2FwGPl/aPl+0kqW+1FsDZsaus/lL5SuBNPHM7+/XAeWV5VVmnPH5mRERb9UlSba32AUfEERFxB7ADuAkYAZ7IzLGyyVZgWVleBjwEUB5/Elg8yWuuiYhNEbFpdHS0zfIlqVWtBnBm7snMlwHLgVcBL5qG11yXmUOZOTQwMHC4LydJ1XQ1H/DhyswnIuIW4DXAwoiYV45ylwPbymbbgJOArRExDzgOeGwm6psNxsbGGB4eBry5pdQvWgvgiBgA/rmE73OAN9M5sXYL8HbgGmA1sKE8ZWNZ/1Z5/Otpyuw1PDzM+y+/nmMGlnlzS6lPtHkEvBRYHxFH0OnquDYz/zYi7gWuiYg/Bb4PXFW2vwr4XERsBnYCF7RY26x0zMAyjl26gl2j2w6+saSe11oAZ+ZdwMsnaX+ATn/wxPafAu9oqx5J6jVeCSdJlczISTjpQJ7es4eRkZG964ODg8yb566p/uderup273yESzdsYfHyp3hq+4+4+JwXc+qpp+59fDyQmyNBmu3SbOXeq54w/4Sle08wXrrhThYvfwqAXaPbuHLtuaxcuXKfkSDNdmm2MoDVc8bDeDLjI0GkfuBJOEmqxACWpErsguhhXn4s9TcDuId5+bHU3+yC6HHjJ53mL1pSuxRJ08wAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqqS1AI6IkyLiloi4NyLuiYjfK+2LIuKmiBgu348v7RERn4yIzRFxV0Sc3lZtktQL2jwCHgN+PzNPA84A1kbEacAlwM2ZOQjcXNYBzgYGy9ca4IoWa5Ok6loL4Mx8ODNvL8tPAfcBy4BVwPqy2XrgvLK8Crg6O24DFkbE0rbqk6TaZqQPOCJWAC8Hvg0sycyHy0OPAEvK8jLgocbTtpa2ia+1JiI2RcSm0dHR9oqWpJa1HsARcQzwv4APZeaPm49lZgI5ldfLzHWZOZSZQwMDA9NYqSTNrFYDOCJ+iU74fj4zv1yat493LZTvO0r7NuCkxtOXlzZJ6kttjoII4Crgvsz8i8ZDG4HVZXk1sKHRfmEZDXEG8GSjq0KS+s68Fl/7dcC7gbsj4o7S9u+By4BrI+IiYAtwfnnsBuAcYDOwG3hvi7Vplnh6zx5GRkYAGBkZodNrJfWH1gI4M/8RiP08fOYk2yewtq16ZouxsTGGh4cBAwdg985HuHTDFhYvf4od99/Ogueu5LjaRUnTpM0jYB2C4eFh3n/59RwzsMzAKeafsJRjl65g16inBNRfvBS5Bx0zsIxjl65g/qIlB99Y0qxlAEtSJQawJFViH3AP8MSbNDcZwD3AE2/Tp/mfGcDg4CDz5rmbqze5Z/aI8RNvnuk/PM3/zHaNbuPKteeycuXK2mVJkzKA1XfG/zOTep0n4SSpEo+ANet5ElOzlQGsWc+TmJqt7IJQX/DqQc1GBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlXgmnWcm7JasfGMCalabrbsnOH6ya3NM0a03H3ZKdP1g1GcCa85w/WLV4Ek6SKjGAJakSA1iSKrEPuEWeYZd0IKZBizzDLulADOCWeYZd0v7YByxJlXgErDnB/nj1otb2wIj4DPAWYEdmvqS0LQK+CKwAHgTOz8zHIyKATwDnALuB92Tm7W3Vprlh4nwRH/vafSw4cbn98eoZbR4CfBb4FHB1o+0S4ObMvCwiLinrFwNnA4Pl69XAFeV732iGwTiPwto12XwR9serl7T225+Z/xARKyY0rwLeUJbXA7fSCeBVwNXZmdLqtohYGBFLM/Phtuqbac0wADwKmyHTMV+E1JaZPvxa0gjVR4AlZXkZ8FBju62l7RcCOCLWAGsATj755PYqbcF4GEgSVBwFUY52pzyJa2auy8yhzBwaGBhooTJJmhkzHcDbI2IpQPm+o7RvA05qbLe8tElS35rpAN4IrC7Lq4ENjfYLo+MM4Ml+6v9Vbxk/IXrfffd5Nw1V1eYwtC/QOeF2QkRsBf4YuAy4NiIuArYA55fNb6AzBG0znWFo722rLseDarrupiEdrjZHQbxzPw+dOcm2Caxtq5Ym52cQODpCvWFOHvo5P4OkXuBcEJJUyZw8Au4F3lZdkgFciSeCJNkFUdH4iaD5i5YcfGNJfccAlqRKDGBJqsQ+YGkSEy/YAS/a0fRzb5Im0bxgB/adPtSrKTVd3Guk/djfBTvNcH5q+4+4+JwXc+qppwKHHsaG+tzkT1jqwsRx20ef8C/2Xsp86YY7Wbz8qcO6tN1L5OcmA3gaNI9exsbGAJg3b54XWPSRA43bnq6J9r1Efu4xgKdB8+hlx/23c8TRC1m8/HleYNFnnMBH080AnibjRy+7RrdxxIJF/qLOQl4erplmAB+iZreDv6z9wcvDNdMM4EM0sdvBX9b+YDeDZpJXwh2G8W4H53KQdCg8ApamSbMPGZ4Zy+sYX+2Pe4E0TZp9yM0LNEZGRvjY1+5jwYnLHeOrfRjA0jRq9iGPX6Axfo7AMb6ayD5gqSXO96yDMYAlqRIDWJIqMYAlqRJPwkkzaH9D1TQ3+ZOXZlBzqJpD0mQASzNsuqav9AKP2c+fljRLOYn77GcAH4STratXTDzibd6ZQ7OTAXwQTrauXjHxRqFT3Qftsug9fvpdcLJ11TJx3unmEW9zH+xmIiDnpOg9BvAknGxdM6GbO3B0O+/0/kZXTPb8iV0WUz0yPtwjaY/En9FT/+qIOAv4BHAE8OnMvKxGHU62rpnQ7R04mn+BHcj46IoD3cF5Ms39vTmLG7RzJO3Jw2f0TABHxBHA5cCbga3AdyNiY2beOxPvv78/9exqUJvauANHN8F+oJAen8Vtf1NqNo+kJ3Z9NE9UN5dh3yPd8f9Ums/v5rmHcvTczXP2t03bR+s9E8DAq4DNmfkAQERcA6wCpj2Ax3f2XaPbGBlZAHR2wv/4P29m/sIT2fmjH7Bg2QuICHbv3M4RP/sZPz7qOdO2DLTyutbUH3VM3C/H99cp1XT0wr37++5HH/6FbR7dfCeX/J9/YuGJy35xfy/P/acnHuWSz9y4zzYTX7P5OgA7f/QDnnXUgr3PGV/e/cQOPvquM/eG+fi/aWId3Tx3/Pe02X4g3Txnf9tMbP/8H6+Z1qP16JX+zYh4O3BWZr6vrL8beHVmfmDCdmuANWX1hcD9h/B2JwCPHka506UX6uiFGsA6JrKOfc32Oh7NzLMmNvbSEXBXMnMdsO5wXiMiNmXm0DSVNKvr6IUarMM65modvTQb2jbgpMb68tImSX2plwL4u8BgRJwSEUcCFwAbK9ckSa3pmS6IzByLiA8Af0dnGNpnMvOelt7usLowplEv1NELNYB1TGQd++rLOnrmJJwkzTW91AUhSXOKASxJlfRtAEfEWRFxf0RsjohLJnn82RHxxfL4tyNiRaU6fiUibo+IsTIWuhVd1PHhiLg3Iu6KiJsj4rmV6vi3EXF3RNwREf8YEafVqKOx3dsiIiOilSFQXXwe74mI0fJ53BER76tRR9nm/LKP3BMRf1Ojjoj4eOOz+GFEPFGpjpMj4paI+H75nTnnkN4oM/vui85JvBHgecCRwJ3AaRO2+R3gr8ryBcAXK9WxAvhl4Grg7RU/jzcC88vyb1f8PI5tLL8VuLFGHWW7BcA/ALcBQ5U+j/cAn2pjv5hiHYPA94Hjy/qJtX4uje0/SOdkfY3PYx3w22X5NODBQ3mvfj0C3ntZc2b+HBi/rLlpFbC+LF8HnBkRMdN1ZOaDmXkX8PQ0v/dU67glM3eX1dvojMOuUcePG6tHA22cJe5m/wC4FPgY8NMWaphKHW3rpo7fAi7PzMcBMnNHpTqa3gl8oVIdCRxblo8D/t+hvFG/BvAy4KHG+tbSNuk2mTkGPAksrlDHTJhqHRcBX6tVR0SsjYgR4D8Dv1ujjog4HTgpM69v4f27rqN4W/kz97qIOGmSx2eijhcAL4iIb0bEbWXmwhp1AFC6yE4Bvl6pjj8B3hURW4Eb6ByNT1m/BrAOUUS8CxgC/kutGjLz8sw8FbgY+A8z/f4R8SzgL4Dfn+n3nsRXgRWZ+cvATTzzV9tMm0enG+INdI48/zoiFlaqBTrdhtdl5p5K7/9O4LOZuRw4B/hc2W+mpF8DuJvLmvduExHz6PwZ8ViFOmZCV3VExL8CPgK8NTN/VquOhmuA8yrUsQB4CXBrRDwInAFsbOFE3EE/j8x8rPGz+DTwimmuoas66BwFbszMf87M/wv8kE4gz3Qd4y6gne6Hbuu4CLgWIDO/BRxFZ6KeqWmrY7/mF53/rR+g8yfKeCf6iydss5Z9T8JdW6OOxrafpb2TcN18Hi+nc+JhsPLPZbCx/GvAppo/l7L9rbRzEq6bz2NpY/nXgdsq1XEWsL4sn0DnT/TFNX4uwIuABykXklX6PL4GvKcsr6TTBzzleqa9+F75ovNnwQ9LqHyktH2UztEddP7H+hKwGfgO8LxKdbySztHFT+gcgd9TqY6/B7YDd5SvjZXq+ARwT6nhlgMFY5t1TNi2lQDu8vP48/J53Fk+jxdVqiPodMvcC9wNXFDr50Kn//WyNt5/Cp/HacA3y8/lDuBXD+V9vBRZkirp1z5gSep5BrAkVWIAS1IlBrAkVWIAS1IlBrBURMR5bc2+Jk3GAJaecR6d8Z2/oFwtKU0rxwFrzoqIC4E/oDOz1Q46l/k+Wb7eBlxFZ5D964EvZOZ/q1Op+pX/q2tOiogX05no57WZ+WhELKJzpdffZuZ1ZRuAIzOzlcnYJbsgNFe9CfhSZj4KkJk797PdF2euJM01BrB0YD+pXYD6lwGsuerrwDsiYjFA6YJ4is5UlNKMsA9Yc1Jm3hMRfwZ8IyL20Lnf2V/TmWj8d4HWbpAqjXMUhCRVYheEJFViAEtSJQawJFViAEtSJQawJFViAEtSJQawJFXy/wFamKCljzLqjgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Распределение экспериментальной группы 1\n",
    "sns.displot(df[df.exp_group == 1].ctr, kde = False);\n",
    "print('mean_ctr:', df[df.exp_group == 1].ctr.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "978650ad-41a0-4523-b06f-af84d88492eb",
   "metadata": {},
   "source": [
    "При визуальном сравнении 2х распределений можно отметить общий схожий нормальный характер распределений, однако в диапазаоне 0.18-0.25 на вершине распределения видны некоторые видимые различия. В целом критичных визуальных различий в распределениях между экспериментальными группами не выявлено."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97c9940e-1e5b-4d8f-8911-70223ce251e6",
   "metadata": {},
   "source": [
    "**Задача 1. Выбрать метод анализа и сравнить CTR в двух группах.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca72e518-e629-433c-abf7-d3903b32d4b0",
   "metadata": {},
   "source": [
    "***Гипотеза:*** Выдвинем гипотезу, что при использовании нового алгоритма в экспериментальной группе 1, CTR увеличится. \\\n",
    "***Нулевая гипотеза:*** средние значения CTR в контрольной и экспериментальной группaх равны. \\\n",
    "***Альтернативная гипотеза:*** средние значения CTR в экспериментальной группе выше значений в контрольной. \\\n",
    "***Пороговое значение p-value:*** 0.05"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e4c982f-91bb-475d-b5c9-70f1502f9ab8",
   "metadata": {},
   "source": [
    "**Метод 1.** Использование пользовательского CTR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "548b1223-7654-4477-b765-e2253354fcb5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MannwhitneyuResult(statistic=75455843.5, pvalue=9.550851479424271e-51)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Сравнение непараметричиским U-критерием Манна — Уитни\n",
    "stats.mannwhitneyu(df[df.exp_group == 0].ctr, df[df.exp_group == 1].ctr, alternative='less')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "45cdca8b-bc91-4160-a59c-f0f85e0f7cd5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ttest_indResult(statistic=-13.161168998357258, pvalue=9.816547547340797e-40)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Сравнение параметричиским T-тестом\n",
    "stats.ttest_ind(df[df.exp_group == 0].ctr, df[df.exp_group == 1].ctr, equal_var = False, alternative='less')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4054cae7-34b1-4578-97b1-ed0bb160ff2f",
   "metadata": {},
   "source": [
    "Оба критерия однозначно показывают p-value значительно < 0.05, что позволяет нам отменить нулевую гипетезу и принять альтернативную гипетезу. \\\n",
    "**Вывод:** средние значения CTR в экпериментальной группе выше значений в контрольной"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa95f1fa-0d8a-4662-bf6e-1fea3a1552d5",
   "metadata": {},
   "source": [
    "**Метод 2.** Использование сглаженного CTR, метод Smoothed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "25f2dae2-05f1-41e8-855d-9c00a639aa85",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_smoothed_ctr(user_likes, user_views, global_ctr, alpha = 5):\n",
    "    \"\"\"\n",
    "    Функция принимает на вход ctr юзера, возвращает сглаженный ctr\n",
    "    --------------------\n",
    "    Параметры:\n",
    "    user_likes - лайки пользователя\n",
    "    user_views - просмотры пользователя\n",
    "    global_ctr - like/view для всех пользователей\n",
    "    alpha - коэффициент сглаживания\n",
    "\n",
    "    Функция возвращает: \n",
    "    smooth_ctr - значение сглаженного CTR\n",
    "    \"\"\"\n",
    "    smooth_ctr = (user_likes + alpha * global_ctr) / (user_views + alpha)\n",
    "    return smooth_ctr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "26f7d7d0-1466-4ef8-9454-2aeef95dcb3b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.22149774745073217"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "global_ctr = df.like.sum() / df.view.sum()\n",
    "global_ctr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7e763b83-39e8-48e0-aaf2-8c3801bad146",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>exp_group</th>\n",
       "      <th>like</th>\n",
       "      <th>view</th>\n",
       "      <th>ctr</th>\n",
       "      <th>smoothed_ctr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7486</th>\n",
       "      <td>138172</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.184581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23644</th>\n",
       "      <td>139520</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.184581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22598</th>\n",
       "      <td>130204</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.184581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24357</th>\n",
       "      <td>112433</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.184581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22718</th>\n",
       "      <td>138035</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.158213</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12221</th>\n",
       "      <td>38820</td>\n",
       "      <td>0</td>\n",
       "      <td>50</td>\n",
       "      <td>303</td>\n",
       "      <td>0.165017</td>\n",
       "      <td>0.165933</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22897</th>\n",
       "      <td>39976</td>\n",
       "      <td>0</td>\n",
       "      <td>41</td>\n",
       "      <td>312</td>\n",
       "      <td>0.131410</td>\n",
       "      <td>0.132831</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22216</th>\n",
       "      <td>51499</td>\n",
       "      <td>1</td>\n",
       "      <td>48</td>\n",
       "      <td>339</td>\n",
       "      <td>0.141593</td>\n",
       "      <td>0.142754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14883</th>\n",
       "      <td>145951</td>\n",
       "      <td>1</td>\n",
       "      <td>64</td>\n",
       "      <td>362</td>\n",
       "      <td>0.176796</td>\n",
       "      <td>0.177405</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21182</th>\n",
       "      <td>30247</td>\n",
       "      <td>0</td>\n",
       "      <td>57</td>\n",
       "      <td>366</td>\n",
       "      <td>0.155738</td>\n",
       "      <td>0.156624</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>25999 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       user_id  exp_group  like  view       ctr  smoothed_ctr\n",
       "7486    138172          1     0     1  0.000000      0.184581\n",
       "23644   139520          1     0     1  0.000000      0.184581\n",
       "22598   130204          1     0     1  0.000000      0.184581\n",
       "24357   112433          0     0     1  0.000000      0.184581\n",
       "22718   138035          1     0     2  0.000000      0.158213\n",
       "...        ...        ...   ...   ...       ...           ...\n",
       "12221    38820          0    50   303  0.165017      0.165933\n",
       "22897    39976          0    41   312  0.131410      0.132831\n",
       "22216    51499          1    48   339  0.141593      0.142754\n",
       "14883   145951          1    64   362  0.176796      0.177405\n",
       "21182    30247          0    57   366  0.155738      0.156624\n",
       "\n",
       "[25999 rows x 6 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Добавим колонку smoothed_ctr\n",
    "df['smoothed_ctr'] = df.apply(lambda x: get_smoothed_ctr(x['like'], x['view'], global_ctr, 5), axis = 1)\n",
    "df.sort_values('view')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d855be6b-2a28-47af-8f5d-0f3267d10767",
   "metadata": {},
   "source": [
    "Посмотрев на сглаженную метрику CTR можно увидеть как вносится штраф на пользователей с малым количеством просмотров"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "846d7a64-c126-4aa8-831c-96d7384497eb",
   "metadata": {},
   "source": [
    "Визуализируем распределения сглаженного CTR, полученные используя метод Smoothed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9f6fdf6d-ef13-4b47-9283-d73c5b7eb11d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean_ctr: 0.2209584855620722\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAFgCAYAAACFYaNMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAccklEQVR4nO3df5RdZX3v8fdXIoohEn7EkCYZQjFovE2r3KhYvV0o2gvcW7HVIuqtyMLm0sZera4WWl3tbetaYHsv/riy4KZShUpF9OoiKKIYwVaXiCiQSAaHgAKJJAQUiCDage/94zwTN+Nk5iSZPc85Z96vtc6avZ+9z5nvnJz5ZM+zn/3syEwkSTPvKbULkKTZygCWpEoMYEmqxACWpEoMYEmqZE7tAvbFCSeckFdffXXtMiRpKjFRY18fAd9///21S5CkvdbXASxJ/cwAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqqS1AI6I50TEzY3HwxHxjog4JCKuiYjby9eDy/4RER+KiM0RsSEijmmrNknqBa3NhpaZ3wOeDxAR+wFbgc8CZwPrM/PciDi7rJ8FnAgsL48XAxeUr6pgdHSU4eHhJ7WtWLGCOXP6egI9qafM1G/T8cAdmXlXRJwMHFfaLwauoxPAJwOXZOcuoddHxPyIWJSZ985QjWoYHh7mzPOvZN7CIQB2br+bC9fAypUrK1cmDY6ZCuBTgU+U5YWNUN0GLCzLi4F7Gs/ZUtqeFMARsRpYDTA0NNRWvQLmLRxi/uKjapchDazWT8JFxP7Aq4FPjd9WjnZzT14vM9dm5qrMXLVgwYJpqlKSZt5MjII4EfhOZm4v69sjYhFA+Xpfad8KLG08b0lpk6SBNBMB/AZ+0f0AsA44rSyfBlzRaH9zGQ1xLPCQ/b+SBlmrfcARMRd4FfDfG83nApdHxBnAXcAppf0q4CRgM/AocHqbtUlSba0GcGY+Ahw6ru0BOqMixu+bwJo265GkXuKVcJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiXMLapfmFJQjIyN0hmZLaosBrF2aU1Bu23QDBy1z6kmpTXZB6EnGpqCce+ii2qVIA88AlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsT5gGex5gTs4CTs0kwzgGeZ8Xe9OO9LtzHv8CMAnIRdmmEG8Cwz0V0v5i8+CoCd2++pXJ00u9gHPAt51wupNxjAklSJASxJlRjAklSJASxJlRjAklRJqwEcEfMj4tMRcVtEDEfESyLikIi4JiJuL18PLvtGRHwoIjZHxIaIOKbN2iSptraPgD8IXJ2ZzwV+AxgGzgbWZ+ZyYH1ZBzgRWF4eq4ELWq5NkqpqLYAj4iDgt4CLADLz55n5IHAycHHZ7WLgNWX5ZOCS7LgemB8RDlSVNLDaPAI+EtgBfDQiboqIj0TEXGBhZt5b9tkGLCzLi4HmpVhbStuTRMTqiLgxIm7csWNHi+VLUrvaDOA5wDHABZn5AuARftHdAEB2Zn7Zo9lfMnNtZq7KzFULFiyYtmIlaaa1GcBbgC2Z+c2y/mk6gbx9rGuhfL2vbN8KLG08f0lpk6SB1FoAZ+Y24J6IeE5pOh7YBKwDTittpwFXlOV1wJvLaIhjgYcaXRXaB6Ojo2zcuJGNGzc65aTUQ9qeDe1PgEsjYn/gTuB0OqF/eUScAdwFnFL2vQo4CdgMPFr21TSYaAY0SfW1GsCZeTOwaoJNx0+wbwJr2qxnNhubAc0pJ6Xe4ZVwklSJASxJlXhHDHXliSceZ2RkZNf6ihUrmDPHj4+0L/wNUlce2fFDzrnyMQ474qfs3H43F66BlSs9mSftCwNYXZu7YMmu+8dJ2nf2AUtSJQawJFViAEtSJQawJFViAEtSJQawJFXiMLQBMTo6yvDw8K51L5SQep+/oQOiOeOZF0pI/cEAHiBjM55J6g8G8AAaP2+Dk7BLvckAHkDNeRsAJ2GXepQBPKCa8zY4CbvUmwxg7ZPxoy/AERhSt/wt0T5pjr4AHIEh7QEDWPvM0RfS3vFKOEmqxACWpEoMYEmqxACWpEoMYEmqxFEQ2mPNS529zFnaewaw9ljzUmcvc5b2nl0Q2itjlzrPPXRR7VKkvmUAS1IlBrAkVWIAS1IlBrAkVdJqAEfEDyJiY0TcHBE3lrZDIuKaiLi9fD24tEdEfCgiNkfEhog4ps3aJKm2mTgCfnlmPj8zV5X1s4H1mbkcWF/WAU4ElpfHauCCGahNkqqp0QVxMnBxWb4YeE2j/ZLsuB6YHxGOcZI0sNq+ECOBL0VEAv83M9cCCzPz3rJ9G7CwLC8GmvfO2VLa7m20ERGr6RwhMzQ01GLp2lfeLUOaXNu/CS/LzK0R8Szgmoi4rbkxM7OEc9dKiK8FWLVqldfA9jDvliFNrtUAzsyt5et9EfFZ4EXA9ohYlJn3li6G+8ruW4GljacvKW3qY94tQ9q91vqAI2JuRMwbWwZ+G/gusA44rex2GnBFWV4HvLmMhjgWeKjRVSFJA6fNI+CFwGcjYuz7/EtmXh0R3wIuj4gzgLuAU8r+VwEnAZuBR4HTW6xNkqprLYAz807gNyZofwA4foL2BNa0VY8k9RqvhJOkSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakSrw7Yh9r3vRyZGSEzpTKkvqFAdzHmje93LbpBg5a5s0upX5iF0SfG7vp5dxDF9UuRdIeMoAlqRIDWJIqMYAlqRJPwmlaPfHE44yMjACOzJCmYgBrWj2y44ecc+VjHHbETx2ZIU3BLghNu7kLljgyQ+qCASxJlRjAklSJASxJlRjAklSJoyD6SHPyHXCYl9TvDOA+0px8B3CYl9TnDOA+Mzb5DsDO7fdUrkbSvrAPWJIqMYAlqZLWAzgi9ouImyLic2X9yIj4ZkRsjohPRsT+pf1pZX1z2b6s7dokqaaZOAJ+OzDcWH8f8P7MfDbwY+CM0n4G8OPS/v6ynyQNrFYDOCKWAP8F+EhZD+AVwKfLLhcDrynLJ5d1yvbjy/4aQKOjo2zcuHHXY3R0tHZJ0ozrahRERLw0M78+VdsEPgD8OTCvrB8KPJiZY79tW4DFZXkxcA9AZo5GxENl//vHfd/VwGqAoaGhbspXD2oOqdu5/W4uXAMrVzqkTrNLt0fA/6fLtl0i4r8C92Xmt/e4qklk5trMXJWZqxYsWDCdL60ZNjakbmxcszTbTHoEHBEvAX4TWBAR72xseiaw3xSv/VLg1RFxEvD08pwPAvMjYk45Cl4CbC37bwWWAlsiYg5wEPDAHv48ktQ3pjoC3h84kE5Qz2s8HgZeN9kTM/MvMnNJZi4DTgW+kplvAq5tPPc04IqyvK6sU7Z/Jb3OVtIAm/QIODO/Cnw1Ij6WmXdN0/c8C7gsIt4L3ARcVNovAv45IjYDP6IT2pI0sLq9FPlpEbEWWNZ8Tma+opsnZ+Z1wHVl+U7gRRPs8xjw+13WI0l9r9sA/hRwIZ3hZI+3V44kzR7dBvBoZl7QaiWSNMt0Owztyoj444hYFBGHjD1arUySBly3R8BjoxP+rNGWwK9ObzmSNHt0FcCZeWTbhUjSbNPtpchvnqg9My+Z3nI0yJ544nFGRkYAb6ckQfddEC9sLD8dOB74DmAAq2uP7Pgh51z5GIcd8VNvpyTRfRfEnzTXI2I+cFkbBWmwzV2whPmLj/J2ShJ7Px3lI4D9wpK0D7rtA76SzqgH6EzCswK4vK2iJGk26LYP+H81lkeBuzJzSwv1SNKs0VUXRJmU5zY6M6EdDPy8zaIkaTboKoAj4hTgBjqT5ZwCfDMiJp2OUpI0uW67IN4NvDAz7wOIiAXAl/nFvd0kSXuo21EQTxkL3+KBPXiuJGkC3R4BXx0RXwQ+UdZfD1zVTkmSNDtMdU+4ZwMLM/PPIuL3gJeVTd8ALm27OEkaZFMdAX8A+AuAzPwM8BmAiFhZtv1Oi7VJ0kCbqh93YWZuHN9Y2pa1UpEkzRJTBfD8SbYdMI11SNKsM1UA3xgRfzi+MSLeCny7nZIkaXaYqg/4HcBnI+JN/CJwVwH7A7/bYl2SNPAmDeDM3A78ZkS8HPi10vz5zPxK65VJ0oDrdj7ga4FrW65Fs1TzThljVqxYwZw53Q5Tl/qTn3BV17xTBsDO7Xdz4RpYudI7ZmiwGcDqCWN3ypBmE+dzkKRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqqS1AI6Ip0fEDRFxS0TcGhF/U9qPjIhvRsTmiPhkROxf2p9W1jeX7cvaqk2SekGb44B/BrwiM38SEU8FvhYRXwDeCbw/My+LiAuBM4ALytcfZ+azI+JU4H107rwxq42OjjI8PAzAyMgImVm5opnV/PnHeJWcBkVrn+LsJMVPyupTyyOBVwBvLO0XA/+TTgCfXJahc7PPD0dE5GxLnHGGh4c58/wrmbdwiG2bbuCgZYN/dVjz0uSRkRHO+9JtzDv8CMCr5DRYWj2MiIj96Myi9mzgfOAO4MHMHC27bAEWl+XFwD0AmTkaEQ8BhwL3j3vN1cBqgKGhoTbL7xnzFg4xf/FR7Nx+T+1SZkTz0uSx/3S8Sk6DqNWTcJn5eGY+H1gCvAh47jS85trMXJWZqxYsWLCvL6ceNXZp8txDF9UuRWrNjIyCyMwH6cym9hJgfkSMHXkvAbaW5a3AUoCy/SDggZmoT5JqaHMUxIKImF+WDwBeBQzTCeLXld1OA64oy+vKOmX7V2Z7/6+kwdZmH/Ai4OLSD/wU4PLM/FxEbAIui4j3AjcBF5X9LwL+OSI2Az8CTm2xNkmqrs1REBuAF0zQfied/uDx7Y8Bv99WPZLUa7wSTpIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqafOmnNoLo6OjDA8P71ofGRnBm0NLg8kA7jHDw8Ocef6VzFs4BMC2TTdw0LKVlauS1AYDuAfNWzjE/MVHAbBz+z2Vq5HUFvuAJakSj4DVt8b3l69YsYI5c/xIq3/4aVXfavaX79x+NxeugZUr7S9X/zCA1dea/eVSv7EPWJIq8Qi4BzT7Mh33O7knnnickZERwPdK/c8A7gHNvkzH/U7ukR0/5JwrH+OwI37qe6W+ZxdEjxjry5x76KLapfS8uQuW+F5pIBjAklSJASxJlRjAklRJawEcEUsj4tqI2BQRt0bE20v7IRFxTUTcXr4eXNojIj4UEZsjYkNEHNNWbZLUC9o8Ah4F3pWZzwOOBdZExPOAs4H1mbkcWF/WAU4ElpfHauCCFmuTpOpaG4aWmfcC95blnRExDCwGTgaOK7tdDFwHnFXaL8nOwM7rI2J+RCwqryNNqjk+eIxzQ6jXzcinMyKWAS8AvgksbITqNmBhWV4MNOde3FLanhTAEbGazhEyQ0ND7RWtvtIcHww4N4T6QusBHBEHAv8PeEdmPhwRu7ZlZkbEHl3KlJlrgbUAq1at8jIo7TI2PljqF60GcEQ8lU74XpqZnynN28e6FiJiEXBfad8KLG08fUlpk/bY+C4JuyPUi1r7REbnUPciYDgzz2tsWgecBpxbvl7RaH9bRFwGvBh4yP5f7a1ml4TdEepVbR4SvBT4A2BjRNxc2v6STvBeHhFnAHcBp5RtVwEnAZuBR4HTW6xNs8BYl4Qn6NSr2hwF8TUgdrP5+An2T2BNW/Vo9vIEnXqVhwCaFTxBp17kpciSVIkBLEmVGMCSVIl9wJV4GyJJBnAl3oZIkl0QFXkbIml2M4AlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIn49Gs1pyVDrxXnGaWnzTNOs2bdI6MjHDel25j3uFHeK84zTgDWLNO8yadY1OBer841WAfsGalsZt0OhWoajKAJakSA1iSKjGAJakSA1iSKjGAJakSA1iSKjGAJakSA1iSKjGAJakSA1iSKmktgCPinyLivoj4bqPtkIi4JiJuL18PLu0RER+KiM0RsSEijmmrLknqFW0eAX8MOGFc29nA+sxcDqwv6wAnAsvLYzVwQYt1zZjR0VE2bty46zE6Olq7JEk9pLXZ0DLzXyNi2bjmk4HjyvLFwHXAWaX9ksxM4PqImB8RizLz3rbqmwnDw8Ocef6VzFs45FSHkn7JTE9HubARqtuAhWV5MXBPY78tpe2XAjgiVtM5SmZoaKi9SqfJvIVDTnUoaULV5gPOzIyI3IvnrQXWAqxatWqPn19LcxJw6EwE3jngVz8Yf+cM8O4Z2ncz/enZPta1EBGLgPtK+1ZgaWO/JaVtYDQnAQd2TQSu/tDsTgLsUtK0mOkAXgecBpxbvl7RaH9bRFwGvBh4qN/7fycyNgk4wM7t90yxt2prHvWOjIxw4LOW2p2kadVaAEfEJ+iccDssIrYAf00neC+PiDOAu4BTyu5XAScBm4FHgdPbqkvqVvOo179Y1IY2R0G8YTebjp9g3wTWtFWLtLfGTqKO/4tlfJ++/cHaG35ipGJPTpQ2+/TtD9beMoClYk9PlDb79KW9YQBLDZ4o1UxyMh5JqsQAlqRKDGBJqsQAlqRKPAk3jcbPF+B8D5ImYwBPo/HzBXj1lKTJGMDTrDn9pMOYJE3GAJb2UfMKurG7njQvS/YyZe2OnwppHzWvoNu26QbmzD2Yw45YDjhtpSZnAEvTYOwKup3b72HOvMO8RFldcRiaJFXiEbA0Q8YPU7RvWP7rSzOkOUzx4Xu/z7v+8whHH330ru0G8uzjv7Y0g5oTvJ9z5YZdU196sm52MoClSpxPWJ6Ek6RKPALeC55MUbeaF2k4N4jGMzX2wu5OpvgLpvHGX6QxnXODjD8QAA8G+o3/UntpopMpTr6jiTQv0phO4yd/8kRe/zGAuzDVNJNt/YJJU2lO/qT+YwDvRjN0R0ZGOO9LtzHv8CMAp5nU9Gv2FYNdCbOF/8K70fzzbixwnWZSbWn2FY/vSvCk7+DyX3ESzX5eqW1jXVnjj4abf4FNdtLXo+j+47+O1GOaR8PAk/4Cm+yk72RH0epNBnDh/dzUS5pXyY3/C2yyk757enWdQ9nq8l0uvJ+bZiOHstVlADd4PzcNquaRbvO2SSMjIxz4rKUOZavEAJZmgfGjesZum+RfenUZwNIs0RzVM3bbJP/Sq2tWB/D4iy086aZBMdFQtm4+35MNZXM88vTrqXcvIk4APgjsB3wkM8+d7u+xuyvc/FNMg2R3Q9n25HnjT8hNdkePvQlqA72HAjgi9gPOB14FbAG+FRHrMnPTdH6f3V3h5p9iGjSTDWXr5nkTHUWPnbBrjkceH8a7u3BkzFjQTsctmvYm7JsnISdan8mj/p4JYOBFwObMvBMgIi4DTgamNYDHe2THFh58xgE88sC9zHnsMR58xgGd9sZ6t9v67TX6rd5Beo1+qPe+277Ne256lPmHfxeAB75/K89cuoKI6Ow392AAHv3xDt7z0S/80n7zJtj26I+28VdveuWuK/nGTLbfZEZGRvjbS7/MMw45fNLnNPd74Pu3st8Bz2T+4Ut31Tu2Pv41xr/+x//mj6d1iF70Sr9nRLwOOCEz31rW/wB4cWa+bdx+q4HVZfU5wPdmtNC9dxhwf+0i9kC/1Qv9V3O/1Qv9V3Ov1Ht/Zp4wvrGXjoC7kplrgbW169hTEXFjZq6qXUe3+q1e6L+a+61e6L+ae73eXron3FZgaWN9SWmTpIHUSwH8LWB5RBwZEfsDpwLrKtckSa3pmS6IzByNiLcBX6QzDO2fMvPWymVNp37rNum3eqH/au63eqH/au7penvmJJwkzTa91AUhSbOKASxJlRjA0ygiToiI70XE5og4e4LtvxUR34mI0TLuubouan5nRGyKiA0RsT4ijqhR57iapqr5zIjYGBE3R8TXIuJ5Neps1DNpvY39XhsRGRFVh0118f6+JSJ2lPf35oh4a406x9U05XscEaeUz/KtEfEvM13jhDLTxzQ86Jw4vAP4VWB/4BbgeeP2WQb8OnAJ8Lo+qfnlwDPK8h8Bn+yDmp/ZWH41cHUv11v2mwf8K3A9sKqX6wXeAny45udgL2peDtwEHFzWn1W77sz0CHga7bqUOjN/DoxdSr1LZv4gMzcAT9QocALd1HxtZj5aVq+nMz67pm5qfrixOheoeaZ5ynqLvwPeBzw2k8VNoNt6e0k3Nf8hcH5m/hggM++b4RonZABPn8VAc8aTLaWtl+1pzWcAX2i1oql1VXNErImIO4C/B/7HDNU2kSnrjYhjgKWZ+fmZLGw3uv1MvLZ0S306IpZOsH0mdVPz0cDREfH1iLi+zLxYnQGsrkTEfwNWAf9Qu5ZuZOb5mXkUcBbwntr17E5EPAU4D3hX7Vr2wJXAssz8deAa4OLK9XRjDp1uiOOANwD/GBHzaxYEBvB06sdLqbuqOSJeCbwbeHVm/myGatudPX2fLwNe02ZBU5iq3nnArwHXRcQPgGOBdRVPxE35/mbmA43PwUeA/zhDte1ON5+JLcC6zPz3zPw+MEInkOuq3Qk9KA86/8PeCRzJL04E/Ifd7PsxeuMk3JQ1Ay+gc4Jjee1696Dm5Y3l3wFu7OV6x+1/HXVPwnXz/i5qLP8ucH0ffCZOAC4uy4fR6bI4tGbdmWkAT/MH4SQ6/7PeAby7tP0tnSNHgBfS+Z/4EeAB4NY+qPnLwHbg5vJY1wc1fxC4tdR77WSB1wv1jtu3agB3+f6eU97fW8r7+9w++EwEna6eTcBG4NTaNWemlyJLUi32AUtSJQawJFViAEtSJQawJFViAEtSJQawJFViAGvWiYhlEfHGxvpbIuLD+/B6x0XE5/bieX+5t99Tg8EA1my0DHjjVDvNgAkDODr83ZwF/EdWz4iIuRHx+Yi4JSK+GxGvj4gfRMQ5ZeLvGyPimIj4YkTcERFnludFRPxDec7GiHj9ZO3AucB/Kq/5p6XtVyLi6oi4PSL+vlHTb0fEN8pE+p+KiANL+wkRcVtEfAf4vSl+rgMj4qOlhg1l4vVzgQNKDZeWo/LvRcQlwHd58twGGlS1L8Xz4WPsAbwW+MfG+kHAD4A/KuvvBzbQmcBmAbC98bxr6EzMvRC4G1g0SftxwOca3+ctdOYSOAh4OnAXnQA8jM4k6XPLfmcBf1X2uYfOZC4BXN58vQl+rvcBH2isj00K/pNG2zI680QfW/vfwcfMPXrmtvQSnWv0/3dEvI9OoP1bRACsa2w/MDN3Ajsj4mdlSsGXAZ/IzMeB7RHxVTrzbuyu/WF+2frMfAggIjYBRwDzgecBXy917A98A3gu8P3MvL3s/3Fg9SQ/1yuBU8dWskwKPoG7MvP6SV5HA8YAVs/IzJEyOflJwHsjYn3ZNDb14RON5bH16foMN1/38fK6AVyTmW9o7hgRz5+m7zneIy29rnqUfcDqGRHxK8CjmflxOhO/H9PlU/8NeH1E7BcRC4DfAm6YpH0nnW6MqVwPvDQinl3qmxsRRwO3Acsi4qiy3xt29wLFNcCaxs95cFn894h4alc/oQaSAaxeshK4ISJuBv4aeG+Xz/ssnb7hW4CvAH+emdsmad8APF5O9v3pbl6TzNxBp3/4ExGxgdL9kJmP0ely+Hw5CTfV/cXeCxxcTgbeQudGpwBrgQ0RcWmXP6cGjNNRSlIlHgFLUiWehJOmSUScDrx9XPPXM3PNRPtLdkFIUiV2QUhSJQawJFViAEtSJQawJFXy/wE73bxlS90/EgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Получившийся smoothed_ctr для экспериментальной группы 0\n",
    "sns.displot(df[df.exp_group == 0].smoothed_ctr, kde = False);\n",
    "print('mean_ctr:', df[df.exp_group == 0].smoothed_ctr.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "99f41374-f9fd-4a33-bd8c-2869df5f60fb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean_ctr: 0.234139241899461\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAFgCAYAAACFYaNMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAazElEQVR4nO3df7RdZX3n8fdXEvxBjAG5ZCKQBisijqmaiUqr41KxXUhbodXij04FBstqTR37Y6yos6YzU9cUK1OlisxQaQ3WX2h1EZXBQRTbukBNEbhC/BFoKMkKJCASkYW9ge/8cZ4rm8PNvefee/Z9zo/3a62z7t7P3ufc770kn2ye/eznicxEkrT0HlO7AEkaVwawJFViAEtSJQawJFViAEtSJctqF7AYJ510Ul5xxRW1y5CkucRMjUN9BXzXXXfVLkGSFmyoA1iShpkBLEmVGMCSVIkBLEmVGMCSVIkBLEmVGMCSVIkBLEmVGMCSVIkBLEmVGMCSVIkBLEmVDPVsaBocU1NTTE5OPqJt/fr1LF++vFJF0uAzgNUXk5OTvOmCLaxcsw6Afbt38MFNsGHDhrqFSQPMAFbfrFyzjsPWHle7DGlo2AcsSZUYwJJUiQEsSZXYBzzGHLkg1WUAjzFHLkh1GcBjzpELUj32AUtSJQawJFViF4QWrHkTb9u2bZBZuSJpuBjA+qmHHtzfCdKG2UZFNG/i7Z68hlVPffZSlCmNDANYP3Xfnp2cd/kDTGybAnobFTF9E2/f7h1LVKU0OgxgPcKKI9Y6KkJaIt6Ek6RKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKWn0SLiJWAR8CngUk8B+B7wKfBNYBO4DTMvOeiAjgfOBk4H7gjMy8rs36ND/dK2g4AY+0OG0/inw+cEVmvjoiDgaeALwDuCozz42Ic4BzgLcBrwCOLa8XABeWrxoQ3StoOAGPtDitdUFExJOAFwMXA2Tmv2bmD4FTgM3ltM3AqWX7FOCS7LgWWBURa9qqTwszPfnOYWuP45DD/c8jLUabfcDHAHuBv4mIb0XEhyLiEGB1Zu4u59wBrC7bRwK3N96/s7Q9QkScHRFbI2Lr3r17WyxfktrVZgAvAzYAF2bmc4Ef0+lu+KnMTDp9wz3LzIsyc2NmbpyYmOhbsZK01NoM4J3Azsz8etn/NJ1AvnO6a6F83VOO7wKObrz/qNImSSOptQDOzDuA2yNienLZE4GbgS3A6aXtdOCysr0FeEN0nADc2+iqkKSR0/YoiDcDHy0jIG4FzqQT+pdGxFnAbcBp5dzL6QxB205nGNqZLdcmSVW1GsCZeT2wcYZDJ85wbgKb2qxHkgaJT8JJUiUGsCRVYgBLUiUGsCRVYgBLUiVtD0PTEHvowf2dGc8KZz+T+ssA1gHdt2cn513+ABPbpoD5zX7WHd4A69evZ/ny5X2vUxpWBrBmteKItRy2tvMw477dO3p+X3d479u9gw9ugg0bNrRRpjSUDGC1phnekh7Nm3CSVIkBLEmVGMCSVIkBLEmVGMCSVIkBLEmVGMCSVIkBLEmVGMCSVIkBLEmVGMCSVIkBLEmVGMCSVImzoY2ZqakpJicnASdYl2ozgMfM5OQkb7pgCyvXrJvXBOuS+s8uiDG0cs06Dlt7HIccvqZ2KdJY8wpYS8IliqRHM4C1JFyiSHo0A1hLxiWKpEeyD1iSKjGAJakSA1iSKjGAJakSA1iSKjGAJakSA1iSKmk1gCNiR0RMRsT1EbG1tB0WEVdGxPfL10NLe0TEX0bE9oi4MSIcoS9ppC3FFfBLM/M5mbmx7J8DXJWZxwJXlX2AVwDHltfZwIVLUJskVVOjC+IUYHPZ3gyc2mi/JDuuBVZFhLPFSBpZbT+KnMD/i4gE/k9mXgSszszd5fgdwOqyfSRwe+O9O0vb7kYbEXE2nStk1q5d22Lpo6E5/y84B7A0SNoO4Bdl5q6IOAK4MiK+0zyYmVnCuWclxC8C2Lhxo0kyh+b8v4BzAEsDpNUAzsxd5eueiPgs8HzgzohYk5m7SxfDnnL6LuDoxtuPKm1apOn5f6EzC5mkwdBaH3BEHBIRT5zeBn4J+DawBTi9nHY6cFnZ3gK8oYyGOAG4t9FVIUkjp80r4NXAZyNi+vt8LDOviIhvApdGxFnAbcBp5fzLgZOB7cD9wJkt1iZJ1bUWwJl5K/CozsbMvBs4cYb2BDa1VY8kDRqfhJOkSlwRQ9V1D5UD14vTeDCAVV33UDnXi9O4MIA1EJpD5aRxYR+wJFViAEtSJQawJFViAEtSJQawJFViAEtSJQawJFViAEtSJQawJFXik3AjxiWIpOFhAI8YlyCShocBPIJcgkgaDvYBS1IlBrAkVWIAS1IlBrAkVeJNOFXx0IP7O0PkmHuonEsWaVQZwKrivj07Oe/yB5jYNjXnUDmXLNKoMoBVzYoj1nLY2uN6GirnkkUaRfYBS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlrQdwRBwUEd+KiM+X/WMi4usRsT0iPhkRB5f2x5b97eX4urZrk6SaluIK+C3Atsb+u4H3ZubTgHuAs0r7WcA9pf295TxJGlmtBnBEHAX8MvChsh/Ay4BPl1M2A6eW7VPKPuX4ieV8SRpJbV8Bvw/4Y+Chsv9k4IeZub/s7wSOLNtHArcDlOP3lvMfISLOjoitEbF17969LZYuSe1qLYAj4leAPZn5T/383My8KDM3ZubGiYmJfn60JC2pNpckeiHwyog4GXgcsBI4H1gVEcvKVe5RwK5y/i7gaGBnRCwDngTc3WJ9I6F7wcq5FrgcBs0FO2E0fiZpJq0FcGa+HXg7QES8BPjPmfmbEfEp4NXAJ4DTgcvKW7aU/WvK8S9n+rduLt0LVs61wOUwaC7YCaPxM0kzqbEo59uAT0TEu4BvAReX9ouBj0TEduAHwGsr1DaUmgtW9rLA5TCYXrATHv0zdV8hu0S9htWSBHBmXg1cXbZvBZ4/wzkPAL+xFPVouDWvkF2iXsPMZek1lJpXyNKw8lFkSarEAJakSgxgSarEAJakSnoK4Ih4YS9tkqTe9XoF/P4e2yRJPZp1GFpE/DzwC8BERPxh49BK4KA2C5OkUTfXOOCDgRXlvCc22vfReVxYkrRAswZwZn4V+GpEfDgzb1uimiRpLPT6JNxjI+IiYF3zPZn5sjaKkqRx0GsAfwr433RWtniwvXIkaXz0GsD7M/PCViuRpDHT6zC0z0XEmyJiTUQcNv1qtTJJGnG9XgGfXr6+tdGWwFP7W44kjY+eAjgzj2m7EEkaNz0FcES8Yab2zLykv+VI0vjotQvieY3txwEnAtcBBrAkLVCvXRBvbu5HxCo6i2pKkhZoodNR/hiwX1iSFqHXPuDP0Rn1AJ1JeI4HLm2rKEkaB732AZ/X2N4P3JaZO1uoR5qX7iXqwWXqNTx67QP+akSs5uGbcd9vrySpd80l6gGXqddQ6bUL4jTgPcDVQADvj4i3ZuanW6xN6olL1GtY9doF8U7geZm5ByAiJoAvAQawJC1Qr6MgHjMdvsXd83ivJGkGvV4BXxERXwQ+XvZfA1zeTkmSNB7mWhPuacDqzHxrRPw68KJy6Brgo20XJy3W1NQUk5OTj2hzlIQGxVxXwO8D3g6QmZ8BPgMQEevLsV9tsTZp0SYnJ3nTBVtYuWYd4CgJDZa5Anh1Zk52N2bmZESsa6ckqb9WrlnnKAkNpLlupK2a5djj+1iHJI2duQJ4a0T8dndjRLwR+Kd2SpKk8TBXF8TvA5+NiN/k4cDdCBwM/FqLdUnSyJs1gDPzTuAXIuKlwLNK8xcy88utVyZJI67XuSC+Anyl5Vokaay09jRbRDwuIr4RETdExE0R8d9L+zER8fWI2B4Rn4yIg0v7Y8v+9nJ8XVu1SdIgaPNx4p8AL8vMZwPPAU6KiBOAdwPvzcynAfcAZ5XzzwLuKe3vLedJ0shqLYCz476yu7y8EngZD0/isxk4tWyfUvYpx0+MiGirPkmqrdUJdSLioIi4HtgDXAncAvwwM/eXU3YCR5btI4HbAcrxe4Enz/CZZ0fE1ojYunfv3jbLl6RWtRrAmflgZj4HOAp4PvCMPnzmRZm5MTM3TkxMLPbjJKmaJZlSMjN/SGcUxc8DqyJievTFUcCusr0LOBqgHH8SnWkvJWkktTkKYqIsX09EPB74RWAbnSB+dTntdOCysr2l7FOOfzkzE0kaUb3OB7wQa4DNEXEQnaC/NDM/HxE3A5+IiHcB3wIuLudfDHwkIrYDPwBe22JtklRdawGcmTcCz52h/VY6/cHd7Q8Av9FWPZI0aFxWSJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqZI254JQS6amppicnARg27Zt4JxF0lAygIfQ5OQkb7pgCyvXrGP35DWseuqza5c0MB56cH/nH6XCf6A0yAzgIbVyzToOW3sc+3bvqF3KQLlvz07Ou/wBJrZNAfgPlAaaAayRs+KItRy29jgA/4HSQPMmnCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUOQ9NYaz5VCLB+/XqWL19esSKNEwNYY635VOG+3Tv44CbYsGFD7bI0Jgxgjb3ppwqlpWYfsCRVYgBLUiUGsCRVYgBLUiXehBsC3UOlnONWGg0G8BBoDpUC57iVRoUBPCSaQ6Wc41YaDQawxopLFmmQGMAaKy5ZpEFiAGvsuGSRBoXD0CSpEgNYkippLYAj4uiI+EpE3BwRN0XEW0r7YRFxZUR8v3w9tLRHRPxlRGyPiBsjwimpJI20Nq+A9wN/lJnPBE4ANkXEM4FzgKsy81jgqrIP8Arg2PI6G7iwxdokqbrWAjgzd2fmdWX7R8A24EjgFGBzOW0zcGrZPgW4JDuuBVZFxJq26pOk2pakDzgi1gHPBb4OrM7M3eXQHcDqsn0kcHvjbTtLW/dnnR0RWyNi6969e9srWpJa1noAR8QK4O+A38/Mfc1jmZnAvEbBZ+ZFmbkxMzdOTEz0sVJJWlqtBnBELKcTvh/NzM+U5junuxbK1z2lfRdwdOPtR5U2SRpJbY6CCOBiYFtm/kXj0Bbg9LJ9OnBZo/0NZTTECcC9ja4KSRo5bT4J90Lgt4DJiLi+tL0DOBe4NCLOAm4DTivHLgdOBrYD9wNntlibJFXXWgBn5j8CcYDDJ85wfgKb2qpHkgaNT8JJUiUGsCRVYgBLUiUGsCRV4nzA0gF0L4YKsH79epYvX16pIo0aA3gAuQryYOheDHXf7h18cBNs2OBEfeoPA3gAuQry4Gguhir1mwE8oFwFWRp93oSTpEoMYEmqxACWpErsA5aKhx7c3xlxUjj6RG0zgKXivj07Oe/yB5jYNgU4+kTtM4ClhhVHrHX0iZaMfcCSVIkBLEmVGMCSVIkBLEmVGMCSVImjIKQedY8TBqen1OIYwAPA6SeHQ/c4Yaen1GIZwAPA6SeHR3OcsLRYBvCAcPpJafwYwJU0ux3scpDGkwFcSbPbwS4HaTw5DK2i6W6HQw5fU7sUSRUYwJJUiQEsSZUYwJJUiQEsSZU4CkJaoO5Hk30sWfNlAEsL1Hw02ceStRAGsLQIPpqsxbAPWJIq8QpY6oPu/uCpqSkigmXLHv4rZh+xurUWwBHx18CvAHsy81ml7TDgk8A6YAdwWmbeExEBnA+cDNwPnJGZ17VVm9RvMy1pv2zFoUwc8wzAqSs1szavgD8MfAC4pNF2DnBVZp4bEeeU/bcBrwCOLa8XABeWr9LQ6F7SfvnKIw7YP9w9B7RXx+OptQDOzL+PiHVdzacALynbm4Gr6QTwKcAlmZnAtRGxKiLWZObutuqTampOxuTV8fha6j7g1Y1QvQNYXbaPBG5vnLeztBnAGlnNOaA1nqqNgihXu/OeBDcizo6IrRGxde/evS1UJklLY6kD+M6IWANQvu4p7buAoxvnHVXaHiUzL8rMjZm5cWJiotViJalNSx3AW4DTy/bpwGWN9jdExwnAvfb/Shp1bQ5D+zidG26HR8RO4E+Ac4FLI+Is4DbgtHL65XSGoG2nMwztzLbqkmroHifsMlSCdkdBvO4Ah06c4dwENrVVi1TbTOOEXYZKPgknLZHuccKSc0FIUiUGsCRVYhdES7ofNQUfN5X0SAZwS5qPmoKTsUh6NAO4Rc1HTR2GJKmbAbxEHIYkqZsBvIQchiSpyVEQklSJASxJldgFIQ0YhzCODwNYqmymETLvv+p7rHzKMYBDGEeZASxVdqARMgcawgheEY8KA1gaALONkOkOaK+IR4cBLA2BZkB7RTw6DGBpyHhFPDoM4D7pvnPto8ZqU/OKWMPLAO6T7sl3fNRY0lwM4D5qTr7jo8aS5uKTcJJUiVfAC2Sfr6TFMoAXyD5fSYtlAC+Cfb4aBM1xwVNTU0QEy5Y9/FfbMcKDywCWhlxzXPDuyWtYtuJQJo55BuAY4UFnAEsjYHpc8L7dO1i+8gifmhsSBrA0wnxqbrAZwNKI86m5wWUAz6J7qFnzBofDziQtlgE8i5mGmk3f4HDYmYaRfcKDxQCeQ/dQs+kbHA470zDq7hP+4a5beMvLt3H88ccDhvFSM4ClMdM9+ft5l08ysW3qUWEMCw9k17XrjQHc4OPFGkfNIWzTYQyPHjExn1Dt7r5z9MXMDOAGHy/WuJtt5Y3uxULn6r5odt9pZgZwFx8vljrmWiy0ecXsFe7CjHUA2+UgzW62xUK7j2v+xi6Am6Hb/b9UdjlICzNTd8V8Lmaafy/HaUKhgQrgiDgJOB84CPhQZp7b7+/R7Oed6X+pJM3fgborpnUHdHfINi+GxmlCoYEJ4Ig4CLgA+EVgJ/DNiNiSmTf3+3tN9/MauFL/zNZdMVNAN0O2eTE014RC3eHd3J/t2Ez78PDV9WxPvnaf2y8DE8DA84HtmXkrQER8AjgF6HsAT//h+PFdu1n2wAP84JAnzHt/Me/t52f5M/gzDEIdPZ274lBmc9+ef5nxs+68+Zv8t633sWrNtwG4+9abOOjxT2TVmrWP2p/t2Ez79999B+94/cs5/vjj2bZtG//zY1/iCU/+Nwc89yN/+ua+XolHDshNp4h4NXBSZr6x7P8W8ILM/L2u884Gzi67xwHfXdJCF+5w4K7aRSzCMNc/zLXDcNc/zLVD/+q/KzNP6m4cpCvgnmTmRcBFteuYr4jYmpkba9exUMNc/zDXDsNd/zDXDu3XP0irIu8Cjm7sH1XaJGkkDVIAfxM4NiKOiYiDgdcCWyrXJEmtGZguiMzcHxG/B3yRzjC0v87MmyqX1U9D123SZZjrH+baYbjrH+baoeX6B+YmnCSNm0HqgpCksWIAS1IlBnCfRcRJEfHdiNgeEefMcPzFEXFdROwvY58HRg+1/2FE3BwRN0bEVRHxMzXqPJAe6v+diJiMiOsj4h8j4pk16pzJXLU3zntVRGREDNTQrh5+92dExN7yu78+It5Yo86Z9PK7j4jTyp/9myLiY3375pnpq08vOjcPbwGeChwM3AA8s+ucdcDPAZcAr65d8zxrfynwhLL9u8Ana9c9z/pXNrZfCVxRu+5eay/nPRH4e+BaYGPtuuf5uz8D+EDtWhdY+7HAt4BDy/4R/fr+XgH3108fp87MfwWmH6f+qczckZk3Ag/VKHAWvdT+lcy8v+xeS2es9qDopf59jd1DgEG5Az1n7cWfAu8GHljK4nrQa/2DqJfafxu4IDPvAcjMPf365gZwfx0J3N7Y31nahsF8az8L+L+tVjQ/PdUfEZsi4hbgz4H/tES1zWXO2iNiA3B0Zn5hKQvrUa9/dl5Vuq8+HRFHz3C8hl5qfzrw9Ij4WkRcW2Zt7AsDWPMWEf8B2Ai8p3Yt85WZF2TmzwJvA/5L7Xp6ERGPAf4C+KPatSzC54B1mflzwJXA5sr1zMcyOt0QLwFeB/xVRKzqxwcbwP01zI9T91R7RLwceCfwysz8yRLV1ov5/u4/AZzaZkHzMFftTwSeBVwdETuAE4AtA3Qjbs7ffWbe3fjz8iHg3y1RbXPp5c/NTmBLZk5l5j8D36MTyItXuxN8lF50/qW8FTiGhzv0/+0Bzv0wg3UTbs7agefSuWFxbO16F1j/sY3tXwW21q57vn9uyvlXM1g34Xr53a9pbP8acG3tuudR+0nA5rJ9OJ0uiyf35fvX/gWM2gs4mc6/kLcA7yxt/4POFSPA8+j8i/pj4G7gpto1z6P2LwF3AteX15baNc+z/vOBm0rtX5kt5Aat9q5zByqAe/zd/1n53d9QfvfPqF3zPGoPOl1ANwOTwGv79b19FFmSKrEPWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWGMnItZFxOsb+2dExAcW8XkviYjPL+B971jo99RoMIA1jtYBr5/rpCUwYwBHh383x4D/kTUwIuKQiPhCRNwQEd+OiNdExI6I+LMyiffWiNgQEV+MiFsi4nfK+yIi3lPeMxkRr5mtHTgX+PflM/+gtD0lIq6IiO9HxJ83avqliLimTKL/qYhYUdpPiojvRMR1wK/P8XOtiIi/KTXcWCZVPxd4fKnho+Wq/LsRcQnwbR45P4FGVe3HAH35mn4BrwL+qrH/JGAH8Ltl/73AjXQmp5kA7my870o6k2uvBv4FWDNL+0uAzze+zxl05gN4EvA44DY6AXg4nQnQDynnvQ34r+Wc2+lMyBLApc3Pm+Hnejfwvsb+9MTe9zXa1tGZI/qE2v8dfC3da2CWpZfoPGf/vyLi3XQC7R8iAmBL4/iKzPwR8KOI+EmZFvBFwMcz80Hgzoj4Kp05Nw7Uvo9Huyoz7wWIiJuBnwFWAc8EvlbqOBi4BngG8M+Z+f1y/t8CZ8/yc70ceO30TpaJvWdwW2ZeO8vnaMQYwBoYmfm9MvH4ycC7IuKqcmh6GsOHGtvT+/36M9z83AfL5wZwZWa+rnliRDynT9+z249b+lwNKPuANTAi4inA/Zn5t3Qme9/Q41v/AXhNRBwUERPAi4FvzNL+IzrdGHO5FnhhRDyt1HdIRDwd+A6wLiJ+tpz3ugN9QHElsKnxcx5aNqciYnlPP6FGkgGsQbIe+EZEXA/8CfCuHt/3WTp9wzcAXwb+ODPvmKX9RuDBcrPvDw7wmWTmXjr9wx+PiBsp3Q+Z+QCdLocvlJtwc60R9i7g0HIz8AY6i5sCXATcGBEf7fHn1IhxOkpJqsQrYEmqxJtwUp9ExJnAW7qav5aZm2Y6X7ILQpIqsQtCkioxgCWpEgNYkioxgCWpkv8PTaAnLS2RXAMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Получившийся smoothed_ctr для экспериментальной группы 1\n",
    "sns.displot(df[df.exp_group == 1].smoothed_ctr, kde = False);\n",
    "print('mean_ctr:', df[df.exp_group == 1].smoothed_ctr.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0fc68be8-97ec-423f-a4bc-006c2a4913dd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MannwhitneyuResult(statistic=75190942.5, pvalue=1.227686809848908e-53)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Сравнение непараметричиским U-критерием Манна — Уитни\n",
    "stats.mannwhitneyu(df[df.exp_group == 0].smoothed_ctr, \n",
    "                   df[df.exp_group == 1].smoothed_ctr, alternative='less')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f58b6c1f-258f-4863-9c11-d7d9b29fe4c6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ttest_indResult(statistic=-14.27852163637724, pvalue=2.225628818760266e-46)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Сравнение параметричиским T-тестом\n",
    "stats.ttest_ind(df[df.exp_group == 0].smoothed_ctr, \n",
    "                df[df.exp_group == 1].smoothed_ctr, equal_var = False, alternative='less')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c71f263-ce26-454f-b693-f05ce2a6b1b8",
   "metadata": {},
   "source": [
    "Используя метод Smoothed для метрики CTR, мы внесли внесли штрафы для пользователей с малым количеством просмотров. Теперь CTR пользователей с малым и большим количеством просмотром стал более сбалансированный. \\\n",
    "Оба критерия однозначно показывают p-value значительно < 0.05, что позволяет нам отменить нулевую гипетезу и принять альтернативную гипетезу. \\\n",
    "**Вывод:** средние значения CTR в экпериментальной группе выше значений в контрольной"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db33bb7d-3398-45e3-b92b-9960fd6dd4ce",
   "metadata": {},
   "source": [
    "**Метод 3.** Использование общего CTR, метод bootstrap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e84bdd9b-08e7-445a-8a15-45e784895add",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_global_ctr(df, exp_group, count_tests = 1000):\n",
    "    \n",
    "    \"\"\"\n",
    "    функция get_global_ctr использует метод \n",
    "    bootstrap для нахождения глобальных CTR\n",
    "    --------------------\n",
    "    Параметры:\n",
    "    df - датафрейм\n",
    "    exp_group - экспериментальная группа в датафрейме\n",
    "    count_tests - число проведенных имитаций A/B тестов\n",
    "\n",
    "    Функция возвращает: \n",
    "    pd.Series(glob_ctr_list) - Series из общих CTR\n",
    "    \"\"\"\n",
    "    \n",
    "    glob_ctr_list = []\n",
    "    for _ in range(count_tests):\n",
    "        sample_data = df[df.exp_group == exp_group].sample(frac=0.5, replace=True)\n",
    "        glob_ctr = sample_data.like.sum() / sample_data.view.sum()\n",
    "        glob_ctr_list.append(glob_ctr)\n",
    "    return pd.Series(glob_ctr_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "98601180-0523-4a15-a2a4-920414627688",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Сохраним полученные bootstrap-распределения CTR в переменные\n",
    "glob_ctr_group_0 = get_global_ctr(df, 0)\n",
    "glob_ctr_group_1 = get_global_ctr(df, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3ce0f3a-46e1-4fe7-b8eb-d3350e21e609",
   "metadata": {},
   "source": [
    "Визуализируем распределения общего CTR, полученные используя метод bootstrap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "bdd97e8d-5697-43be-b443-88ef980fd212",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.21159720815380192 0.21312935433192917 0.21159720815380192\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAFgCAYAAACFYaNMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAASo0lEQVR4nO3de5CddX3H8fc3xHARbVC2GVw2JRZG29panJVq6AXBTlNahVokdKxGig2jxUvpKFhnyoxTZ3DKeB2rZACJLYNBigPecDDipY1Gw0VF8JIimCUBFhUvtZWufPvHedDTuGHPbs5zvufsvl8zZ/Y8l3POZw7LJ7/9ned5TmQmkqTBW1YdQJKWKgtYkopYwJJUxAKWpCIWsCQVWV4dYH+sW7cur7/++uoYkjSXmG3lSI+AH3jggeoIkrRgI13AkjTKLGBJKmIBS1IRC1iSiljAklTEApakIhawJBWxgCWpiAUsSUUsYEkqYgFLUhELWJKKWMCSVMQClqQiFrAWvfGJ1URET7fxidXVcbWEjPQF2aVe7J7axfqLt/W075az17acRvo5R8CSVMQClqQiFrAkFbGAJamIBSxJRSxgSSpiAUtSEQtYkopYwJJUxAKWpCIWsCQVsYAlqYgFLElFLGBJKmIBS1IRC1iSiljAklSktQKOiMsi4v6IuK1r3RMi4oaI+Gbz87BmfUTEOyJiZ0R8OSKe0VYuSRoWbY6ALwfW7bXufGBrZh4DbG2WAf4YOKa5bQTe3WIuSRoKrRVwZn4G+O5eq08BNjf3NwOndq1/X3Z8HlgZEUe0lU2ShsGg54BXZeae5v69wKrm/jiwq2u/qWbdL4iIjRGxIyJ2TE9Pt5dUklpW9iFcZiaQC3jcpsyczMzJsbGxFpJJ0mAMuoDve2Rqofl5f7P+HmCia78jm3WStGgNuoCvAzY09zcA13atf0lzNMSzgO93TVVIQ2t8YjUR0dNtfGJ1dVwNmeVtPXFEXAmcABweEVPABcCFwFURcRZwN3B6s/tHgZOBncCPgTPbyiX10+6pXay/eFtP+245e23LaTRqWivgzPyLfWw6aZZ9E/ibtrJI0jDyTDiNpPn86S8Nq9ZGwFKb/NNfi4EjYEkqYgFLUhELWJKKWMCSVMQP4aRuy5Z75IQGxgKWuj080/PRFeARFto/TkFIUhELWJKKWMCSVMQClqQiFrAkFbGAJamIBSxJRSxgSSpiAUtSEQtYkopYwJJUxAKWpCIWsCQVsYAlqYgFLElFLGBJKmIBS1IRC1iSiljAklTEApakIhawJBWxgCWpiAUsSUUsYEkqYgFLUhELWJKKWMCSVMQClqQiFrAkFbGAJamIBSxJRSxgSSpiAUtSEQtYkopYwJJUxAKWpCIWsCQVsYA1FMYnVhMRPd+kxWB5dQAJYPfULtZfvK3n/becvbbFNNJgOAKWpCIlBRwRfxsRX42I2yLiyog4KCLWRMT2iNgZEVsiYkVFNkkalIEXcESMA68CJjPzacABwBnAm4G3ZubRwPeAswadTZIGqWoKYjlwcEQsBw4B9gAnAlc32zcDp9ZEk6TBGHgBZ+Y9wEXAt+kU7/eBm4AHM3Om2W0KGJ/t8RGxMSJ2RMSO6enpQUSWpFZUTEEcBpwCrAGeBDwWWNfr4zNzU2ZOZubk2NhYSyklqX0VUxDPBb6VmdOZ+b/ANcDxwMpmSgLgSOCegmySNDAVBfxt4FkRcUh0jqg/CbgduBE4rdlnA3BtQTapPcuW93yiyfjE6uq0GoCBn4iRmdsj4mrgZmAGuAXYBHwEeH9E/GOz7tJBZ5Na9fBMzyebeKLJ0lByJlxmXgBcsNfqO4HjCuJIUgnPhJOkIhawJBWxgCWpiAWs1sznEpPSUuTlKNWa+Vxi0k/9tRQ5ApakIhawJBWxgCWpiAUsSUUsYEkqYgFLUhELWJKKWMCSVMQClqQiFrAkFbGAJamIBSxJRSxgSSpiAUtSEQtYkopYwJJUxAKWpCIWsCQVsYAlqYgFLElFLGBJKmIBS1IRC1iSiljAklTEApakIhawJBWxgDUv4xOriYiebpIe3fLqABotu6d2sf7ibT3tu+XstS2nkUabI2BJKmIBS1IRC1iSiljAklTEApakIhawJBWxgCWpiAUsSUUsYEkqYgFLUhELWJKKWMCSVMQClqQiFrAkFbGAJalITwUcEcf3sq5XEbEyIq6OiK9FxB0R8eyIeEJE3BAR32x+HrbQ55ekUdDrCPidPa7r1duB6zPzqcDTgTuA84GtmXkMsLVZlqRF61G/ESMing2sBcYi4tyuTY8HDljIC0bELwG/D7wUIDMfAh6KiFOAE5rdNgOfAs5byGtI0iiYawS8AjiUTlE/ruv2A+C0Bb7mGmAaeG9E3BIRl0TEY4FVmbmn2edeYNVsD46IjRGxIyJ2TE9PLzCCJNV71BFwZn4a+HREXJ6Zd/fxNZ8BvDIzt0fE29lruiEzMyJyH5k2AZsAJicnZ91HkkZBr1/KeWBEbAKO6n5MZp64gNecAqYyc3uzfDWdAr4vIo7IzD0RcQRw/wKeW5JGRq8F/AHgPcAlwE/35wUz896I2BURT8nMrwMnAbc3tw3Ahc3Pa/fndSRp2PVawDOZ+e4+vu4rgSsiYgVwJ3AmnfnoqyLiLOBu4PQ+vp4kDZ1eC/hDEfEK4IPATx5ZmZnfXciLZuatwOQsm05ayPNJ0ijqtYA3ND9f27UugSf3N44kLR09FXBmrmk7iCQtNT0VcES8ZLb1mfm+/saRpKWj1ymIZ3bdP4jOXO3NgAUsSQvU6xTEK7uXI2Il8P42AknSUrHQy1H+F51TiiVJC9TrHPCH6Bz1AJ2L8PwacFVboSRpKeh1DviirvszwN2ZOdVCHklaMnqagmguyvM1OldCOwx4qM1QkrQU9PqNGKcDXwBeSOcU4e0RsdDLUUqS6H0K4g3AMzPzfoCIGAM+QedKZpKkBej1KIhlj5Rv4zvzeKyk+Vq2nIjo+TY+sbo6sRag1xHw9RHxceDKZnk98NF2Ikni4RnWX7yt5923nL22xTBqy1zfCXc0na8Kem1EvAD43WbT54Ar2g4nSYvZXCPgtwGvB8jMa4BrACLiN5ttz2sxmyQtanPN467KzK/svbJZd1QriSRpiZirgFc+yraD+5hDkpacuQp4R0T89d4rI+JlwE3tRJKkpWGuOeDXAB+MiBfx88KdBFYAf9ZiLkla9B61gDPzPmBtRDwHeFqz+iOZ+cnWk0nSItfr9YBvBG5sOYskLSmezSZJRSxgSSpiAUtSEQtYkopYwJJUxAKWpCIWsCQVsYCXuPGJ1fO68Lek/un1guxapHZP7fLC31IRR8CSVMQClqQiFrAkFbGApcVgHt+i7DcoDw8/hJMWg3l8i7IfpA4PR8CSVMQClqQiFrAkFbGAJamIBSxJRSxgSSpiAUtSEQtYkopYwJJUxAKWpCIWsCQVsYAlqUhZAUfEARFxS0R8uFleExHbI2JnRGyJiBVV2SRpECpHwK8G7uhafjPw1sw8GvgecFZJKkkakJICjogjgT8BLmmWAzgRuLrZZTNwakU2SRqUqhHw24DXAQ83y08EHszMmWZ5Chif7YERsTEidkTEjunp6daDSlJbBl7AEfGnwP2ZedNCHp+ZmzJzMjMnx8bG+pxOkgan4hsxjgeeHxEnAwcBjwfeDqyMiOXNKPhI4J6CbJI0MAMfAWfm6zPzyMw8CjgD+GRmvgi4ETit2W0DcO2gs0nSIA3TccDnAedGxE46c8KXFueRpFaVfilnZn4K+FRz/07guMo8kjRIwzQClqQlxQKWpCIWsCQVsYAlqYgFLElFLGBJKmIBS1IRC1iSiljAklTEApakIhawJBWxgCWpiAUsSUUsYEkqYgFLUhELWJKKWMCSVMQClqQiFrAkFbGAJamIBSxJRSxgSSpiAUtSEQtYWmqWLScierqNT6yuTruoLa8OIGnAHp5h/cXbetp1y9lrWw6ztDkClqQiFrAkFbGAJamIBSxJRSxgSSpiAUtSEQtYkopYwJJUxAKWpCIW8CI0PrG651NNJdXxVORFaPfULk81lUaAI2BJKmIBS1IRC1iSiljAklTEApakIhawJBWxgCWpiAUsSUUsYEkqYgFLUhELWJKKWMCSVGTgBRwRExFxY0TcHhFfjYhXN+ufEBE3RMQ3m5+HDTrbMPMKZ9LiU3E1tBng7zLz5oh4HHBTRNwAvBTYmpkXRsT5wPnAeQX5hpJXOJMWn4GPgDNzT2be3Nz/IXAHMA6cAmxudtsMnDrobJI0SKVzwBFxFHAssB1YlZl7mk33Aqv28ZiNEbEjInZMT08PJqgktaCsgCPiUODfgNdk5g+6t2VmAjnb4zJzU2ZOZubk2NjYAJJKUjtKCjgiHkOnfK/IzGua1fdFxBHN9iOA+yuySdKgVBwFEcClwB2Z+ZauTdcBG5r7G4BrB51Nkgap4iiI44EXA1+JiFubdX8PXAhcFRFnAXcDpxdkk6SBGXgBZ+a/A/s6WPWkQWaRpEqeCSdp35Yt7/kEoPGJ1dVpR45fSy9p3x6e8QSgFjkClqQiFrAkFbGAJamIBSxJRSxgSSpiAUtSEQtYkopYwJJUxAKWpCIWsCQVsYAlqYgFLElFLGBJKmIBS1IRC1iSiljAkvpjHhdv9wLuHV6QXVJ/zOPi7eAF3MERsCSVsYAlqYgFLElFLGBJKmIBS1IRC1iSiljAkmrM47jhxXrMsMcBS6oxj+OGF+sxw46AJamIBdxH4xOrPRVTUs+cguij3VO7PBVTUs8cAUtSEQtYkopYwJJUxAKWpCIWcKV5HIguLWmL9KQNj4Ko5IHoUm8W6f8rjoAlqYgFPIf5nFwhSfPhFMQc5nNyxSj96SOpniNgSSpiAUtSEQtYkopYwJJUxAKWpCIWsCQVsYAlLS7zOG05Ili+4qCy05w9DljS4jKP05ahc/x+1bH+joAlqchQFXBErIuIr0fEzog4v63X8fRiScNgaKYgIuIA4F3AHwJTwBcj4rrMvL3fr+XpxZKGwTCNgI8DdmbmnZn5EPB+4JTiTJLUmsjM6gwARMRpwLrMfFmz/GLgdzLznL322whsbBafAny9uX848MCA4vbLKGaG0cxt5sEw8+weyMx1e68cmimIXmXmJmDT3usjYkdmThZEWrBRzAyjmdvMg2Hm+RmmKYh7gImu5SObdZK0KA1TAX8ROCYi1kTECuAM4LriTJLUmqGZgsjMmYg4B/g4cABwWWZ+dR5P8QvTEiNgFDPDaOY282CYeR6G5kM4SVpqhmkKQpKWFAtYkooMbQHPdVpyRJwbEbdHxJcjYmtE/ErXtusj4sGI+PBej1kTEdub59zSfNg37JnPaZ4vI+LwfuZtMfMVzXPeFhGXRcRjRiDzpRHxpeYxV0fEocOeuWv7OyLiR/3M22buiLg8Ir4VEbc2t98egcwREW+KiG9ExB0R8aq+hM3MobvR+RDuP4EnAyuALwG/vtc+zwEOae6/HNjSte0k4HnAh/d6zFXAGc399wAvH4HMxwJHAXcBh4/I+3wyEM3tyhF5nx/fdf8twPnDnrnZNgn8C/Cjfv5utPxeXw6c1u+8LWc+E3gfsKxZ/uV+5B3WEfCcpyVn5o2Z+eNm8fN0jht+ZNtW4Ifd+0dEACcCVzerNgOnDnPmZv0tmXlXH3MOIvNHswF8ofsxQ5z5B/Cz35ODgX5+Ot1K5uhcP+WfgNf1MWvruVvWVuaXA2/MzIeb/e7vR9hhLeBxYFfX8lSzbl/OAj42x3M+EXgwM2d6fM75aiNz21rN3Ew9vBi4fkHpZtda5oh4L3Av8FTgnQsNOIu2Mp8DXJeZe/Yj26Np8/fjTc0UwFsj4sCFBpxFW5l/FVgfETsi4mMRccx+ZPyZoTkOeKEi4i/p/Bn2B9VZerWEMv8z8JnM/Gw7qR7dfDNn5pnNqPKdwHrgvS3Gm1WvmSPiScALgRMGEGtO83yvX0/nH7oVdI7BPQ94Y3vpZjfPzAcC/5OZkxHxAuAy4Pf2N8OwjoB7Oi05Ip4LvAF4fmb+ZI7n/A6wMiIe+Uen36c6t5G5ba1ljogLgDHg3D7k7Nbq+5yZP6XzZ+uf72fObm1kPhY4GtgZEXcBh0TEzv7E/ZlW3uvM3NPMUP2Ezj9yx/UpL7T3+zEFXNPc/yDwW/uZs6ONifD9vdEZmd8JrOHnE+m/sdc+x9KZbD9mH89xAr84kf4B/v+HcK8Y9sxd2+6i/x/CtfU+vwzYBhw8Cr8bdD4sPLrr/kXARcOceZbtbXwI19bvxxFd7/XbgAtHIPOFwF91bf9iX/L2+z9aH9/Ik4FvNG/UG5p1b6TzLxbAJ4D7gFub23Vdj/0sMA38N51/uf6oWf9kOh8K7aRTxgeOQOZXNcszwG7gkhHIPNM83yOP+YdhzkznL8H/AL4C3AZcQddREcOYeZbn73sBt/j78cmu9/pfgUNHIPNK4CNN7s8BT+9HVk9FlqQiwzoHLEmLngUsSUUsYEkqYgFLUhELWJKKWMCSVMQClqQi/wdUMoAOIMTmhAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Распределение группы 0\n",
    "sns.displot(glob_ctr_group_0, kde = False);\n",
    "print(glob_ctr_group_0.quantile(0.05),\n",
    "      glob_ctr_group_0.mean(),\n",
    "      glob_ctr_group_0.quantile(0.05))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "434b73eb-7371-470e-b5d9-1fa1929c183d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.22823722554192843 0.22991403103033514 0.22823722554192843\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAFgCAYAAACFYaNMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAUp0lEQVR4nO3dfZBld13n8fdn0juByMYJ0k4NPT0SzOiKiA/bZrPDsgWJlmPUJGoqE0thhOAEBRTZQmH5A8vSKlgsedpdzSwgwxYVJhuCiRsIxjGIVkiggRDyAGGIxulMyHRWFF1K4zDf/eOe2VyHSfpOz5z7u939flXd6nPPw72fuVP96d8995xzU1VIksZvXesAkrRWWcCS1IgFLEmNWMCS1IgFLEmNTLUOcDK2b99eN910U+sYkrSUHG/mih4BP/LII60jSNKy9VbASd6d5FCSu4bmvTnJ55PcmeSDSTYMLXtdkv1JvpDkR/rKJUmTos8R8HuA7cfMuxl4dlU9B7gPeB1AkmcBlwPf3W3z35Oc1mM2SWqutwKuqo8Bf3PMvD+uqsPd3duAzd30xcD7q+qfquovgf3AuX1lk6RJ0HIf8EuAD3fTM8CBoWUL3bxvkGRXkvkk84uLiz1HlKT+NCngJK8HDgPvO9Ftq2p3Vc1V1dz09PSpDydJYzL2w9CS/Dzw48AF9diVgB4EZodW29zNk6RVa6wj4CTbgV8DLqqqrw0tugG4PMnpSc4GtgKfGGc2SRq33kbASa4Gng88LckC8AYGRz2cDtycBOC2qnpZVd2d5BrgHga7Jl5eVV/vK5skTYKs5OsBz83N1fz8fOsYkrSU1XcmnCStZBawJDViAUtSIxawJDViAUtSIxawNGRmdgtJRrrNzG5pHVcr3Iq+ILt0qh1cOMCOq24dad29V27rOY1WO0fAktSIBSxJjVjAktSIBSxJjVjAktSIBSxJjVjAktSIBSxJjVjAktSIBSxJjVjAktSIBSxJjVjAktSIBSxJjVjAktSIBSxJjVjAktSIBSxJjVjAktSIBSxJjVjAktSIBSwt17opv75eJ8WvpZeW68jhkb7C3q+v1+NxBCxJjVjAktSIBSxJjVjAktSIBSxJjVjAktSIBSxJjVjAktSIBSxJjVjAktSIBSxJjVjAktSIBSz1bcSrpnnltLXHq6FJfRvxqmngldPWmt5GwEneneRQkruG5j01yc1Jvtj9PKubnyRvT7I/yZ1JfqCvXJI0KfrcBfEeYPsx814L7KuqrcC+7j7AjwJbu9su4Pd6zCVJE6G3Aq6qjwF/c8zsi4E93fQe4JKh+e+tgduADUk29ZVNkibBuD+E21hVD3XTXwY2dtMzwIGh9Ra6ed8gya4k80nmFxcX+0sqST1rdhREVRVQy9hud1XNVdXc9PR0D8kkaTzGXcAPH9210P081M1/EJgdWm9zN0+SVq1xF/ANwM5ueidw/dD8F3VHQ5wH/N3QrgpJWpV6Ow44ydXA84GnJVkA3gC8EbgmyRXAA8Bl3eofAi4E9gNfA17cVy5JmhS9FXBV/czjLLrgOOsW8PK+skjSJPJUZElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxAKWpEYsYElqxALWmjAzu4UkS96kcZpqHUAah4MLB9hx1a1Lrrf3ym1jSCMNNBkBJ/nVJHcnuSvJ1UmelOTsJLcn2Z9kb5L1LbJJ0riMvYCTzAC/DMxV1bOB04DLgTcBb6mqc4CvAFeMO5skjVOrfcBTwJOTTAFnAA8B5wPXdsv3AJe0iSZJ4zH2Aq6qB4HfAf6aQfH+HfAp4G+r6nC32gIwM+5skjROLXZBnAVcDJwNPB34JmD7CWy/K8l8kvnFxcWeUkpS/1rsgvgh4C+rarGq/hm4DngusKHbJQGwGXjweBtX1e6qmququenp6fEklqQetCjgvwbOS3JGBgdeXgDcA9wCXNqtsxO4vkE2rTCr7vjedVMj/XtmZre0TqpTYOzHAVfV7UmuBT4NHAY+A+wGbgTen+S3unnvGnc2rTyr7vjeI4dX179HT6jJiRhV9QbgDcfMvh84t0EcSWrCU5ElqRELWJIasYAlqRELWJIasYAlqRELWJIasYAlqRELWJIasYAlqRELWJIasYAlqRELWJIasYAlqRELWJIasYAlqRELWJIasYAlqRELWJIasYAlqRELWJIasYAlqRELWJIasYAlqRELWJIasYCllWjdFElGus3MbmmdVo9jqnUASctw5DA7rrp1pFX3Xrmt5zBaLkfAktSIBSxJjVjAktSIBSxJjVjAktSIBSxJjVjAktTISAWc5LmjzJMkjW7UEfA7RpwnSRrRE54Jl+TfA9uA6SSvHlp0JnBan8EkabVb6lTk9cBTuvX+9dD8rwKX9hVKktaCJyzgqvoz4M+SvKeqHhhTJklaE0a9GM/pSXYDzxjepqrO7yOUJK0Foxbw/wJ+H3gn8PX+4kjS2jFqAR+uqt/rNYkkrTGjHob2R0l+KcmmJE89eus1mSStcqOOgHd2P18zNK+AZ57aOJK0doxUwFV1dt9BJGmtGamAk7zoePOr6r3LedIkGxh8oPdsBiPplwBfAPYyONLir4DLquory3l8SVoJRt0H/INDt+cBvwFcdBLP+zbgpqr6N8D3AvcCrwX2VdVWYF93X5JWrVF3Qbxy+H43gn3/cp4wyTcD/xH4+e6xHwUeTXIx8PxutT3AR4FfX85zSNJKsNzLUf5fYLn7hc8GFoE/SPKZJO9M8k3Axqp6qFvny8DG422cZFeS+STzi4uLy4ygFmZmt4z0NepT65808leuSyvZqPuA/4jBvloYXITnu4BrTuI5fwB4ZVXdnuRtHLO7oaoqSR1v46raDewGmJubO+46mkwHFw6M9FXqe6/c5leua00Y9TC03xmaPgw8UFULy3zOBWChqm7v7l/LoIAfTrKpqh5Ksgk4tMzHl6QVYaRdEN1FeT7P4IpoZwGPLvcJq+rLwIEk39nNugC4B7iBx4433glcv9znkKSVYNRdEJcBb2bwwViAdyR5TVVdu8znfSXwviTrgfuBFzP4Y3BNkiuAB4DLlvnYkrQijLoL4vXAD1bVIYAk08CfMNh9cMKq6g5g7jiLLljO40nSSjTqURDrjpZv5/+cwLaSpOMYdQR8U5KPAFd393cAH+onkiStDUt9J9w5DI7PfU2SnwL+Q7fo48D7+g4nSavZUiPgtwKvA6iq64DrAJJ8T7fsJ3rMJkmr2lL7cTdW1eeOndnNe0YviSRpjViqgDc8wbInn8IckrTmLFXA80l+4diZSV4KfKqfSJK0Niy1D/hVwAeT/CyPFe4csB74yR5zSdKq94QFXFUPA9uSvIDBxdMBbqyqP+09mSStcqNeD/gW4Jaes0jSmuLZbJLUiAUsSY1YwJLUiAUsSY1YwJLUiAUsSY1YwJLUiAUsSY1YwJLUiAUsSY1YwJLUiAUsSY1YwJLUiAUsSY1YwJLUiAUsSY1YwJLUiAUsSY1YwJLUiAUsSY1YwJLUiAUsSY1YwJLUiAUsSY1YwNJqt26KJEveZma3tE665ky1DiCpZ0cOs+OqW5dcbe+V28YQRsMcAUtSIxawJDViAUtSIxawJDViAUtSIxawJDViAeukzMxuGekY0ySto0oTx+OAdVIOLhwY6RhT8DhT6VjNRsBJTkvymST/u7t/dpLbk+xPsjfJ+lbZJGkcWu6C+BXg3qH7bwLeUlXnAF8BrmiSSpLGpEkBJ9kM/Bjwzu5+gPOBa7tV9gCXtMgmSePSagT8VuDXgCPd/W8B/raqDnf3F4CZ422YZFeS+STzi4uLvQeVpL6MvYCT/DhwqKo+tZztq2p3Vc1V1dz09PQpTidJ49PiKIjnAhcluRB4EnAm8DZgQ5KpbhS8GXiwQTZJGpuxj4Cr6nVVtbmqngFcDvxpVf0scAtwabfaTuD6cWeTpHGapBMxfh14dZL9DPYJv6txHknqVdMTMarqo8BHu+n7gXNb5pGkcZqkEbAkrSkWsCQ1YgFLUiMWsCQ1YgFLUiMWsCQ1YgFLUiMWsCQ1YgFLUiMWsCQ1YgFLUiMWsCQ1YgFLUiMWsKSBdVMkWfI2M7ulddJVo+nlKCVNkCOH2XHVrUuutvfKbWMIszY4ApakRixgSWrEApakRixgSWrEApakRixgSWrEApakRixgSWrEApZ0YkY8Y86z5pbmmXCSTsyIZ8yBZ80txRGwJDViAUtSIxawJDViAUtSIxawJDViAUtSIxawJDViAUtSIxawJDViAeu4Zma3jHSqqaTl81RkHdfBhQN+QaPUM0fAktSIBSxJjVjAktSIBbyGjPrBmh+uSePhh3BryKgfrIEfrknj4AhYkhqxgCWpEQtYkhoZewEnmU1yS5J7ktyd5Fe6+U9NcnOSL3Y/zxp3NkkapxYj4MPAf6qqZwHnAS9P8izgtcC+qtoK7OvuS9KqNfYCrqqHqurT3fTfA/cCM8DFwJ5utT3AJePOJknj1HQfcJJnAN8P3A5srKqHukVfBjY+zja7kswnmV9cXBxPUEnqQbMCTvIU4APAq6rqq8PLqqqAOt52VbW7quaqam56enoMSSWpH00KOMm/YlC+76uq67rZDyfZ1C3fBBxqkU2SxqXFURAB3gXcW1W/O7ToBmBnN70TuH7c2SRpnFqcivxc4IXA55Lc0c37z8AbgWuSXAE8AFzWIJskjc3YC7iq/gJ4vKu9XDDOLJLUkmfCSVIjFrAkNWIBS+rPuqmRrj89M7ulddImvB6wpP4cOeyXuz4BR8CS1IgFLEmNWMCS1IgFLEmNWMCS1IgFLEmNWMCS1IgFLEmNWMCS1IgFLEmNWMCS1IgFLEmNWMCS1IgFLEmNWMCS1IgFLEmNWMCS1IgFLEmNWMCS1IgFLEmNWMCS1IgFLEmNWMCS1IgFLEmNWMCS2ls3RZKRbjOzW1qnPWWmWgeQJI4cZsdVt4606t4rt/UcZnwcAUtSIxbwKjAzu2Wkt26SJou7IFaBgwsHRnr7tpreukmrgSNgSWrEAh6zUXcXrKZPeqVTasQjJlbC75C7IMbM3QXSSRrxiImV8DvkCFiSGrGAJakRC1iSGnEf8KTqPmiQtHpZwJNqjZ6aKa0l7oKQpEYsYEmr0wlcYW1q/ZOaHFvsLghJq9MJ7sZrcWzxxI2Ak2xP8oUk+5O8to/nGPVstFH/Kq6Us24kTZaJGgEnOQ34b8APAwvAJ5PcUFX3nMrnOZGz0fwgTFJfJm0EfC6wv6rur6pHgfcDFzfOJEm9SFW1zvD/JbkU2F5VL+3uvxD4d1X1iqF1dgG7urvfCXyhm34a8MgY454o8y3fJGcD852stZDvkarafuzMidoFMYqq2g3sPnZ+kvmqmmsQaSTmW75JzgbmO1lrOd+k7YJ4EJgdur+5mydJq86kFfAnga1Jzk6yHrgcuKFxJknqxUTtgqiqw0leAXwEOA14d1XdPeLm37BbYsKYb/kmORuY72St2XwT9SGcJK0lk7YLQpLWDAtYkhqZyAJe6nTkJK9Ock+SO5PsS/Jt3fzvS/LxJHd3y3YMbZMkv53kviT3JvnlCcv350nu6G4Hk/zhhOW7IMmnu3x/keScCct3fpfvriR7kiz7842TyPdtQ6/R3UleNrTNv03yue4x354s/2LPPeX77SQHkvzDcnP1lS/JGUluTPL5bv4bJyVbt+ymJJ/t5v9+Bmf0jqaqJurG4MO3LwHPBNYDnwWedcw6LwDO6KZ/EdjbTX8HsLWbfjrwELChu/9i4L3Auu7+t05SvmO2/wDwoknKB9wHfFc3/UvAeyYlH4OBxAHgO7plvwlc0SDfeuD0bvopwF8BT+/ufwI4DwjwYeBHJyzfecAm4B8a/v4eNx9wBvCCoXX+fDmvX4+v3ZndzzD43b181EyTOAJe8nTkqrqlqr7W3b2NwfHCVNV9VfXFbvogcAiY7tb7ReA3q+pIt/zQhOUDIMmZwPnAH05YvgLO7Ka/GTg4Qfm+BXi0qu7rtrkZ+OkG+R6tqn/q5p9O9w4zySYGv6S31eA39b3AJZOSr1t2W1U9tMxMvearqq9V1S1H1wE+fXSb1tm6ZV/tJqcYFPXIRzZMYgHPMBjNHLXQzXs8VzAYUfwLSc5l8GJ8qZv17cCOJPNJPpxk64TlO+oSYN/Qf+qk5Hsp8KEkC8ALgeW+Dewj3yPAVJKjZytdyr88oWds+ZLMJrmze4w3dX8oZrrHGfUxx53vVOo1X5INwE8A+yYpW5KPMBgQ/D1w7aiBJrGAR5bk54A54M3HzN8E/E/gxUdHvAz+av1jDU4p/B/Auycs31E/A1zdd7Zl5PtV4MKq2gz8AfC7k5KvG1VeDrwlyScY/BJ8vUW+qjpQVc8BzgF2JtnYd461kq/br3818Paqun+SslXVjzDYhXM6g3ewI5nEAh7pdOQkPwS8Hrho6K3B0bfwNwKvr6rbhjZZAK7rpj8IPGfC8pHkaQzeJt24zGy95EsyDXxvVd3erbYXWO71N3t5/arq41X1vKo6F/gYg33WY883lOcgcBfwvG774bfMJ3OKfR/5TqU+8+0GvlhVb53AbFTVPwLXcyJXcBx1Z/G4bgz2o9wPnM1jO8q/+5h1vp/BW8+tx8xfz+CtyauO87hvBF7STT8f+OQk5euWvwzYM2mvX/eYj/DYh1xXAB+YlHzdsm/tfp7erXN+g3ybgSd302cx+CPwPd39Yz+Eu3CS8g2tc7IfwvX1+v0Wgw+41k1SNgYfyG0aevy9wCtGznQyL3ZfN+DC7h/4JQYjHRh8sn1RN/0nwMPAHd3thm7+zwH/PDT/DuD7umUbGIycPgd8nMGIbmLydcs/yuBynJP4+v1k99p9tsv5zAnL92bgXgaXJ31Vo9fvh4E7u9foTmDX0GPOMRg1fQn4r3RnoU5Qvv/C4F3ike7nb0xKPgblV93/79FtXjoh2TYyuIbNnd3/7zuAqVHzeCqyJDUyifuAJWlNsIAlqRELWJIasYAlqRELWJIasYAlqRELWJIa+X+es9oBbRQCcwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Распределение группы 1\n",
    "sns.displot(glob_ctr_group_1, kde = False);\n",
    "print(glob_ctr_group_1.quantile(0.05),\n",
    "      glob_ctr_group_1.mean(),\n",
    "      glob_ctr_group_1.quantile(0.05))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "93550b48-5cc7-4a45-b060-b406ce26249f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MannwhitneyuResult(statistic=0.0, pvalue=0.0)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Сравнение непараметричиским U-критерием Манна — Уитни\n",
    "stats.mannwhitneyu(glob_ctr_group_0, glob_ctr_group_1, alternative='less')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "55207854-0dec-447e-83bc-5d1f2fe58b84",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ttest_indResult(statistic=-380.182540370158, pvalue=0.0)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Сравнение параметричиским T-тестом\n",
    "stats.ttest_ind(glob_ctr_group_0, glob_ctr_group_1, equal_var = False, alternative='less')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "fe4e3524-982b-4a2c-b8e4-6123fd3a0269",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.21023123518042522 0.21623526887652675\n",
      "0.22627592004018882 0.23307510347403432\n"
     ]
    }
   ],
   "source": [
    "# Выборки не пересекаются\n",
    "print(glob_ctr_group_0.min(), glob_ctr_group_0.max())\n",
    "print(glob_ctr_group_1.min(), glob_ctr_group_1.max())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "109f15d8-e0b2-459b-9aa4-8f40928108c1",
   "metadata": {},
   "source": [
    "Используя метод bootstrap мы получили распределения общего CTR для групп 0 и 1. Еще на этапе визуализации можно было бы с достаточной уверенностью утверждать, что средние CTR в группе 1 выше значений в группе 0. p-value равное 0 в обоих критериях только подтверждает это, что позволяет нам отменить нулевую гипетезу и принять альтернативную гипетезу..\\\n",
    "**Вывод:** средние значения CTR в экпериментальной группе выше значений в контрольной"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbf05392-8890-4ec9-a172-45ba814eb6c4",
   "metadata": {},
   "source": [
    "**Задача 2. Провести контрольную проверку, что новый алгоритм не привел к тому, что сократилось число просмотров у пользователей. Типичная ситуация в АB-тестах — вырастили число лайков, но уменьшили число просмотров.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "7f4232ff-3b7f-4b31-beec-6445807005b7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "61.297914903439256\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAFgCAYAAACFYaNMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAYV0lEQVR4nO3df5BlZX3n8fc3tCAOLcOPSUPmB4MISVuZjVKjkWjcKsmPkWQz7BZhqUpF4pKwxNHVNcmK61aiW5tSdxMJSVGws6JCQglKSMFsGYwKGjdZQQTkx1xtZ0WZGWeGAQEnINGG7/5xnx7ONN09d3r63Od23/er6lafX33vt0/f++nTz3nOcyIzkST134/VLkCShpUBLEmVGMCSVIkBLEmVGMCSVMlI7QIOx4YNG/LWW2+tXYYkHUzMtHBRHwE/+uijtUuQpHlb1AEsSYuZASxJlRjAklSJASxJlRjAklSJASxJlRjAklSJASxJlRjAklSJASxJlRjAklSJASxJlSzq0dDaNDk5SafT2T8/Pj7OyIi7S9LCMVFm0el0uOSKLYyOrWHfnoe5ahOsW7eudlmSlhADeA6jY2tYvvK02mVIWqJsA5akSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakShwNbQFNH0MYHEdY0uxMhgXUHEMYcBxhSXNqtQkiIv5jRDwYEQ9ExCci4sURcWpE3BER2yLihog4smx7VJnfVtavbbO2tkyNIbx85Wn7g1iSZtJaAEfESuA/AOsz86eBI4ALgA8Bl2Xmy4HHgYvKt1wEPF6WX1a2k6Qlq+2TcCPA0RExArwE2AW8EbixrL8GOLdMbyzzlPVnR0S0XJ8kVdNaAGfmTuBPgIfpBu+TwFeBJzJzsmy2A1hZplcC28v3TpbtT5j+vBFxcUTcFRF37d27t63yJal1bTZBHEf3qPZU4CeAZcCGw33ezNycmeszc/2KFSsO9+kkqZo2myB+AXgoM/dm5o+Am4DXActLkwTAKmBnmd4JrAYo648FHmuxPkmqqs0Afhh4bUS8pLTlng1sBW4HzivbXAjcXKZvKfOU9bdlZrZYnyRV1WYb8B10T6bdDdxfXmsz8G7gXRGxjW4b79XlW64GTijL3wVc2lZtkjQIWr0QIzP/CPijaYu/Bbxmhm2fAX69zXokaZA4FoQkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlrQ5HuVRNTk7S6XT2z4+PjzMy4q6UdGhMjXnodDpccsUWRsfWsG/Pw1y1CdatW1e7LEmLjAE8T6Nja1i+8rTaZUhaxGwDlqRKDGBJqsQAlqRKDGBJqsQAlqRK7AVRTO/bOzExQWZWrEjSUmcAF82+vQC7t97JsWvt2yupPQZwQ7Nv77492ytXI2mpsw1YkirxCLhFzz33LBMTE/vnHTNCUpNp0KKn9n6XD2x5hhNP+cELxoxwQB9JfuJbtmzFqhnHjHBAH0kGcEUO6CMNNwP4MDXbee07LOlQGMCHqdnOa99hSYfCbmgLYKqdd9kJJ9cuRdIiYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUGsCRV0moAR8TyiLgxIr4eEZ2IOCsijo+Iz0bEN8vX48q2ERF/HhHbIuK+iDizzdokqba2j4AvB27NzJ8CfgboAJcCn8/M04HPl3mANwGnl8fFwJUt1yZJVbUWwBFxLPAG4GqAzPxhZj4BbASuKZtdA5xbpjcC12bXl4HlEXFyW/VJUm1tHgGfCuwFPhYR90TERyJiGTCWmbvKNruBsTK9Etje+P4dZdkBIuLiiLgrIu7au3dvi+VLUrvaDOAR4Ezgysx8FfAUzzc3AJCZCeShPGlmbs7M9Zm5fsWKFQtWrCT1W5sBvAPYkZl3lPkb6QbynqmmhfL1kbJ+J7C68f2ryjJJWpJaC+DM3A1sj4ifLIvOBrYCtwAXlmUXAjeX6VuAN5feEK8Fnmw0VUjSkjPS8vO/HbguIo4EvgW8hW7ofzIiLgK+A5xftv00cA6wDXi6bCtJS1arAZyZ9wLrZ1h19gzbJrCpzXokaZB4JZwkVdJ2E4QO0eTkJJ1O54Bl4+PjjIz4q5KWGj/VA6bT6XDJFVsYHVsDwL49D3PVJli3bl3lyiQtNAN4AI2OrWH5ytNqlyGpZbYBS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1Il9gPuk+eee5aJiYn98xMTE3SHv5A0rAzgPnlq73f5wJZnOPGUHwCwe+udHLvWq9ukYWYA99GyFav2X+G2b8/2g2wtaamzDViSKjGAJakSA1iSKjGAJakST8ItIg7WLi0tfnIXEQdrl5YWA3iRcbB2aemwDViSKjGAJakSA1iSKjGAJakST8L1wJHMJLXBAO6BI5lJaoMB3CNHMpO00GwDlqRKDGBJqsQAlqRKDGBJqsQAlqRKeuoFERGvy8x/ONgyzU+zn7F9jKXh0Ws3tL8Azuxhmeah2c/YPsbS8JgzgCPiLODngBUR8a7GqpcCR7RZ2LCZ6mdsH2NpeBzsCPhI4Jiy3Whj+feB89oqSpKGwZwBnJlfBL4YER/PzO/0qSZJGgq9tgEfFRGbgbXN78nMN7ZRlCQNg14D+FPAVcBHgGfbK0eShkevATyZmVe2WokkDZleL8TYEhFvjYiTI+L4qUerlUnSEtfrEfCF5esfNJYl8LKFLUeShkdPAZyZp7ZdiCQNm14vRX7zTMsz89qFLUeShkevTRCvbky/GDgbuBswgCVpnnptgnh7cz4ilgPXt1GQ5mdycpJOp7N/fnx8nJER7zglDbL5fkKfAmwXHiCdTodLrtjC6Nga9u15mKs2wbp1DuojDbJe24C30O31AN1BeMaBT7ZVlOZndGzN/huHShp8vR4B/0ljehL4TmbuaKEeSRoaPV2IUQbl+TrdEdGOA37YZlGSNAx6CuCIOB+4E/h14HzgjohwOEpJOgy9NkG8F3h1Zj4CEBErgM8BN7ZVmCQtdb0G8I9NhW/xGN7Qc2A17zE3xW5p0uDp9RN5a0R8BvhEmf+3wKfbKUmHq3mPOcBuadKAOtg94V4OjGXmH0TEvwFeX1b9X+C6tovT/E3dY07S4DrYEfCfAe8ByMybgJsAImJdWfevWqxNkpa0g7XjjmXm/dMXlmVrW6lIkobEwQJ4+Rzrjl7AOiRp6BwsgO+KiN+ZvjAifhv4ai8vEBFHRMQ9EfG/y/ypEXFHRGyLiBsi4siy/Kgyv62sX3uIP4skLSoHC+B3Am+JiC9ExJ+WxxeBi4B39Pga7wA6jfkPAZdl5suBx8tzUb4+XpZfVraTpCVrzgDOzD2Z+XPA+4Fvl8f7M/OszNx9sCePiFXAr9C9mzIREcAbef4CjmuAc8v0xjJPWX922V6SlqRexwO+Hbh9Hs//Z8B/ojuGBMAJwBOZOVnmdwAry/RKYHt5vcmIeLJs/2jzCSPiYuBigDVr1syjJEkaDK1dzRYRvwo8kpk9tRX3KjM3Z+b6zFy/YsWKhXxqSeqrNq9NfR3waxFxDt3bGL0UuBxYHhEj5Sh4FbCzbL8TWA3siIgR4Fi6lzxL0pLU2hFwZr4nM1dl5lrgAuC2zPwNuk0ZUyOpXQjcXKZvKfOU9bdlZjLkpsZ1uP/++5mYmMBdIi0dNUZneTdwfUT8N+Ae4Oqy/GrgLyNiG/A9uqE99JrjOuzeeifHrnU8B2mp6EsAZ+YXgC+U6W8Br5lhm2fojjesaabGddi3Z/sBy5ujnnl0LC0+jk+4iHl0LC1ujum7yE0dHS874eTapUg6RAawJFViAEtSJQawJFViAEtSJQawJFViAEtSJQawJFViAEtSJQawJFViAEtSJQawJFXiYDxDZnJykk7n+Xukjo+PMzLi20CqwU/ekOl0OlxyxRZGx9awb8/DXLUJ1q1zFDWpBgN4CI2OrWH5ytNqlyENPduAJakSA1iSKjGAJakSA1iSKvEk3BDw5p3SYDKAh4A375QGk00QQ8Kbd0qDxwCWpEoMYEmqxACWpEoMYEmqxACWpEoMYEmqxACWpEoMYEmqxACWpEoMYEmqxACWpEocjEf7ecNOqb/8dGk/b9gp9ZcBrAN4w06pf2wDlqRKhvoIuNnm6Z0iJPXbUAdws83TO0VI6rehb4KYavP0ThGS+m3oA1iSajGAJakSA1iSKjGAJakSA1iSKhnqbmjqneNESAvPT5B64jgR0sIzgNUzx4mQFpZtwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZXYDU2H7LnnnmViYmL/vBdlSPPjp0aH7Km93+UDW57hxFN+4EUZ0mEwgDUvy1as8qIM6TDZBixJlbR2BBwRq4FrgTEggc2ZeXlEHA/cAKwFvg2cn5mPR0QAlwPnAE8Dv5WZd7dVn17YluuNSaX+arMJYhL4vcy8OyJGga9GxGeB3wI+n5kfjIhLgUuBdwNvAk4vj58Frixf1ZJmWy7gjUmlPmutCSIzd00dwWbmPqADrAQ2AteUza4Bzi3TG4Frs+vLwPKI8E6ZLZtqy/XGpFL/9aUNOCLWAq8C7gDGMnNXWbWbbhMFdMN5e+PbdpRlkrQktR7AEXEM8NfAOzPz+8112W1wPKRGx4i4OCLuioi79u7du4CVSlJ/tdoNLSJeRDd8r8vMm8riPRFxcmbuKk0Mj5TlO4HVjW9fVZYdIDM3A5sB1q9f7xmjlniCTmpfm70gArga6GTmhxurbgEuBD5Yvt7cWP62iLie7sm3JxtNFeozT9BJ7WvzCPh1wG8C90fEvWXZf6YbvJ+MiIuA7wDnl3WfptsFbRvdbmhvabE29aB5scW+PdsPsrWkQ9VaAGfm/wFiltVnz7B9ApvaqkeSBo1XwklSJQawJFViAEtSJQawJFViAEtSJQawJFViAEtSJQawJFViAEtSJQawJFViAEtSJd4VWX0zOTlJp9PZPz8+Ps7IiG9BDS/f/eqbTqfDJVdsYXRsDfv2PMxVm2DdOoe41PAygHVYpg/cDnMf2Y6Ordk/xKU07AxgHZbpA7d7ZCv1zgDWYWsO3C6pd/aCkKRKPALWgpreJmxPB2l2fjK0oJptwrYHS3MzgLXgbBOWemMbsCRVYgBLUiUGsCRVYhuwWjO9R8TExASZWbEiabAYwGrN9Kvkdm+9k2PXvrBHxPRBesDuaxoOvsPVqmaPiH17ts+4TXOQnu52dl/TcDCANRAcpEfDyJNwklSJASxJlRjAklSJASxJlXgSTgPNLmpaynwXq4rmRRpzXaBhFzUtZQawqmhepDHbBRpT7KKmpco2YFUzdZHGshNOrl2KVIUBLEmVGMCSVIkBLEmVGMCSVIkBLEmV2A1NA6fXPsLSYmcAa+AcSh9haTGzCUIDyT7CGgYeAWtJcMwILUa+O7WoTL/R51TIzjVmhOGsQeU7UItKs314+sA8s40Z4YA+GlQGsBad5o0+ZzO9J8UxP77aAX00cAxgLVpzdVezJ4UWAwNYi9bBQnbqSHnfnu2zPkezfXhychLggLZh24rVJt9ZWtR6Cdm5NNuHd2+9k5Flx3HiKacDthWrfQawhs5s7cP79mxnZPRE24rVNwawho7twxoUXgmnoeSVdhoEHgFLs2g2VXiCTm3w3SPNYnpThSfotNAMYGkOzV4WnqDTQrMNWJIq8QhYmofZBgWabvpAQLYbq2no3gnND4R3W9B8NduHv7/rIX7vlyc444wzgANDtnmhx1ztxgb1cBq63/D0K5/sA6r5arYPf2DLfTOGcfNCj7mOmnsNai0tAxXAEbEBuBw4AvhIZn6wjdeZGrZwvpevStPNFMbAAX/k5zpqniuop3eBO9yj47aPtj2a793A7JWIOAK4AvhFYAfwlYi4JTO31q1MOjTN4TKn/5E/1KCeWjfVBa4Z3NODuTk/17qJiQk+/HdfZ/SkU17wh6DX54Dng3V64Pb6/HM9R3NdTW3/MRmYAAZeA2zLzG8BRMT1wEZgwQN4356HAXjqsV2MPPMMT7zk6AOm57tusT3HYqt3KT3HU4/tYmTZcQe8L5/au2PWdVOefnwv/+Vjf8vykx7gsYce5IijX8ryk1YDHDB/sHUvXT3O6LTnO5TnePp7u/nD3/gFzjjjDCYmJviv132Olxx/0v7tenn+uZ6jua6mZl1Pf283f/X+ty5o01AMykmoiDgP2JCZv13mfxP42cx827TtLgYuLrM/CXzjEF/qRODRwyx3oVlTbwaxJhjMuqypN/2q6dHM3DB94SAdAfckMzcDm+f7/RFxV2auX8CSDps19WYQa4LBrMuaelO7pkG6EGMnsLoxv6osk6QlaZAC+CvA6RFxakQcCVwA3FK5JklqzcA0QWTmZES8DfgM3W5oH83MB1t4qXk3X7TImnoziDXBYNZlTb2pWtPAnISTpGEzSE0QkjRUDGBJqmRoAjgiNkTENyJiW0RcWrmWb0fE/RFxb0TcVZYdHxGfjYhvlq8z98RfuBo+GhGPRMQDjWUz1hBdf1723X0RcWYfa3pfROws++reiDinse49paZvRMQvt1TT6oi4PSK2RsSDEfGOsrzavpqjpmr7KiJeHBF3RsTXSk3vL8tPjYg7ymvfUE6wExFHlfltZf3aPtb08Yh4qLGfXlmW9+V9foDMXPIPuif1/h/wMuBI4GvAKyrW823gxGnL/jtwaZm+FPhQyzW8ATgTeOBgNQDnAH8LBPBa4I4+1vQ+4Pdn2PYV5fd4FHBq+f0e0UJNJwNnlulRYKK8drV9NUdN1fZV+XmPKdMvAu4oP/8ngQvK8quA3y3TbwWuKtMXADe0sJ9mq+njwHkzbN+X93nzMSxHwPsvc87MHwJTlzkPko3ANWX6GuDcNl8sM/8e+F6PNWwErs2uLwPLI2LB72Y5S02z2Qhcn5n/nJkPAdvo/p4XuqZdmXl3md4HdICVVNxXc9Q0m9b3Vfl5/6nMvqg8EngjcGNZPn0/Te2/G4GzIyL6VNNs+vI+bxqWAF4JNEdF2cHcb9i2JfB3EfHVcmk1wFhm7irTu4GxCnXNVkPt/fe28i/hRxtNM32vqfyb/Cq6R1IDsa+m1QQV91VEHBER9wKPAJ+le6T9RGZOzvC6+2sq658ETmi7psyc2k9/XPbTZRFx1PSaZqi3FcMSwIPm9Zl5JvAmYFNEvKG5Mrv/D1XtHzgINRRXAqcBrwR2AX9ao4iIOAb4a+Cdmfn95rpa+2qGmqruq8x8NjNfSfcq1tcAP9XP15/J9Joi4qeB99Ct7dXA8cC7a9U3LAE8UJc5Z+bO8vUR4G/ovln3TP27U74+UqG02Wqotv8yc0/5ED0H/C+e/9e5bzVFxIvoBt11mXlTWVx1X81U0yDsq1LHE8DtwFl0/42fuuCr+br7ayrrjwUe60NNG0oTTmbmPwMfo9J+guEJ4IG5zDkilkXE6NQ08EvAA6WeC8tmFwI3VyhvthpuAd5czhK/Fniy8e93q6a1wf1ruvtqqqYLytn0U4HTgTtbeP0ArgY6mfnhxqpq+2q2mmruq4hYERHLy/TRdMf17tANvfPKZtP309T+Ow+4rfwn0XZNX2/84Qy6bdLN/dTf93nbZ/kG5UH3DOcE3Xap91as42V0z0h/DXhwqha67V+fB74JfA44vuU6PkH339Qf0W3rumi2GuieFb6i7Lv7gfV9rOkvy2veR/cDcnJj+/eWmr4BvKmlml5Pt3nhPuDe8jin5r6ao6Zq+wr4F8A95bUfAP6w8X6/k+6Jv08BR5XlLy7z28r6l/WxptvKfnoA+Cue7ynRl/d58+GlyJJUybA0QUjSwDGAJakSA1iSKjGAJakSA1iSKjGANZQi4ici4saDbym1x25oklSJR8Ba8iLigxGxqTH/voj4/SjjDpcBW/5HRHylDNDy78vyKyLi18r030TER8v0v4uIP67xs2hpMYA1DG4Azm/Mn8/zo4dB94q7JzPz1XQHaPmdcsnul4CfL9uspDuuLmXZ37dasYaCAawlLzPvAX68tPv+DPA4Bw47+Et0xwC4l24wn0B3vIQvAT8fEa8AtvL8ADxnAf/Yxx9BS9TA3JZeatmn6A76chLdI+KmAN6emZ+Z/k1lMJcNdI94j6d79PxP2R0IXTosBrCGxQ10h2g8EfiXdG/PM+UzwO9GxG2Z+aOIOAPYmZlPAV8G3kn3zg4n0L17g70ntCBsgtBQyMwH6d4/bWe+cIjBj9BtYri7nJj7nzx/cPIlYCQztwF30z0K/lJ/qtZSZzc0SarEI2BJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJquT/Aw4nW8vuXDMEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Распределение просмотров в группе 0\n",
    "print(df[df.exp_group == 0].view.mean())\n",
    "sns.displot(df[df.exp_group == 0].view);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "7497971a-0249-41c1-a116-f36b4cc59d59",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "61.2005076142132\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAFgCAYAAACFYaNMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAcsklEQVR4nO3df5TddZ3f8edbIggEHAJJzM4QQUhNNa1IR4qra11SbaCtYXuQsme7pDa7WXaj1VqtWM+pek49R7t2WdnDgabCGlarICuHLGVx2YA/elpQQH4EGHdGkE2yIQMIyI/jj9F3/7ifSW6GSXInzPd+7sx9Ps6ZM9/v5/u9977znXtf+d7P9/v9fCMzkSR138tqFyBJ/coAlqRKDGBJqsQAlqRKDGBJqmRB7QJeijVr1uTNN99cuwxJOpiYrnFO7wE/8cQTtUuQpEM2pwNYkuYyA1iSKjGAJakSA1iSKjGAJakSA1iSKjGAJakSA1iSKjGAJakSA1iSKjGAJakSA1iSKpnTo6HNpomJCUZGRvZpW7lyJQsWHHgTTX1cJ4+RJDCA9xgZGeGiy25k4ZIhAJ4b38EVG2HVqlUdP67Tx0gSGMD7WLhkiIHBU7r2OEn9zT5gSarEAJakSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakShoN4Ij4DxHxQERsi4gvR8QrIuLkiLgjIsYi4pqIOLyse0SZHyvLT2qyNkmqrbEAjohB4N8Dw5m5CjgMuAD4DHBJZp4KPAWsLw9ZDzxV2i8p60nSvNV0F8QC4MiIWAAcBewCzgKuK8s3A+eW6bVlnrJ8dUREw/VJUjWNBXBm7gQ+C/wtreB9BrgLeDozJ8pqO4DBMj0IbC+PnSjrH99UfZJUW5NdEMfR2qs9GfgV4GhgzSw874aIuDMi7nz88cdf6tNJUjVNdkH8U+CRzHw8M38OfA14CzBQuiQAhoCdZXoncCJAWf5K4MmpT5qZmzJzODOHFy9e3GD5ktSsJgP4b4EzI+Ko0pe7GngQuA04r6yzDrihTG8p85Tlt2ZmNlifJFXVZB/wHbQOpt0N3F9eaxPwEeCDETFGq4/3yvKQK4HjS/sHgYubqk2SekGjtyTKzI8DH5/S/DBwxjTr/gR4d5P1SFIv8Uo4SarEAJakSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakSgxgSarEAJakShq9JVE/m5iYYGRkZJ+2lStXsmCBm1xSi2nQkJGRES667EYWLhkC4LnxHVyxEVatWlW5Mkm9wgBu0MIlQwwMnlK7DEk9yj5gSarEAJakSgxgSarEAJakSgxgSaqksQCOiNdGxD1tPz+OiA9ExKKIuCUiRsvv48r6ERGXRsRYRNwXEac3VZsk9YLGAjgzv5+Zp2XmacA/Al4ArgcuBrZm5gpga5kHOBtYUX42AJc3VZsk9YJudUGsBn6QmY8Ca4HNpX0zcG6ZXgtcnS23AwMRsaxL9UlS13UrgC8Avlyml2bmrjL9GLC0TA8C29ses6O07SMiNkTEnRFx5+OPP95UvZLUuMavhIuIw4F3AR+duiwzMyJyJs+XmZuATQDDw8MzeuxM/PKXv2B0dHTPvOM4SJpt3UiUs4G7M3N3md8dEcsyc1fpYhgv7TuBE9seN1Taqnj+ib/jU1se4YTlzzuOg6RGdKML4jfZ2/0AsAVYV6bXATe0tV9YzoY4E3imrauiiqMXDzIweMqeAXUkaTY1ugccEUcD7wB+r63508C1EbEeeBQ4v7TfBJwDjNE6Y+I9TdYmSbU1GsCZ+Txw/JS2J2mdFTF13QQ2NlmPJPUSjypVMHWwdg/wSf3JT30F7YO1e4BP6l8GcCUO1i7JwXgkqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRIDWJIqMYAlqRLHguiAtyeS1ARTpAPenkhSEwzgDk3enkiSZot9wJJUiQEsSZUYwJJUiQEsSZV4EG4WtZ+uNjo6Clm5IEk9zQCeRe2nq+0euZNjX/16BmoXJaln2QUxyyZPVztq0dLapUjqcQawJFViAEtSJY0GcEQMRMR1ETESEQ9FxJsjYlFE3BIRo+X3cWXdiIhLI2IsIu6LiNObrE2Samt6D/hzwM2ZuRJ4A/AQcDGwNTNXAFvLPMDZwIryswG4vOHaJKmqxs6CiIhXAm8D/i1AZv4M+FlErAXeXlbbDHwD+AiwFrg6MxO4vew9L8vMXU3V2E2eoiZpqiZPQzsZeBz404h4A3AX8H5gaVuoPgZMni4wCGxve/yO0rZPAEfEBlp7yCxfvryx4mebp6hJmqrJLogFwOnA5Zn5RuB59nY3AFD2dme0L5iZmzJzODOHFy9ePGvFdoOnqElq12QA7wB2ZOYdZf46WoG8OyKWAZTf42X5TuDEtscPlTZJmpcaC+DMfAzYHhGvLU2rgQeBLcC60rYOuKFMbwEuLGdDnAk8M1/6fyVpOk1fivw+4EsRcTjwMPAeWqF/bUSsBx4Fzi/r3gScA4wBL5R1JWneajSAM/MeYHiaRaunWTeBjU3WI0m9xCvhJKkSA1iSKjGAJakSA1iSKnFA9hlqv6QYvKxY0qEzgGeo/ZJiwMuKJR0yA/gQTF5SDPDs+PaDrC1J07MPWJIqMYAlqRIDWJIqMYAlqRIDWJIq8SyIyqaeVwywcuVKFizwTyPNd37KK5t6XvFz4zu4YiOsWrWqcmWSmmYA94D284ol9Q/7gCWpEveAe9jExAQjIyP7tNk/LM0ffpJ72MjICBdddiMLlwwB9g9L840B3OMWLhmyf1iap+wDlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqqTRAI6IH0bE/RFxT0TcWdoWRcQtETFafh9X2iMiLo2IsYi4LyJOb7I2SaqtG3vAv56Zp2XmcJm/GNiamSuArWUe4GxgRfnZAFzehdokqZoaXRBrgc1lejNwblv71dlyOzAQEcsq1CdJXdF0ACfwVxFxV0RsKG1LM3NXmX4MWFqmB4H2WwzvKG2SNC81fSnyWzNzZ0QsAW6JiH1GlsnMjIicyROWIN8AsHz58tmrVJK6rNE94MzcWX6PA9cDZwC7J7sWyu/xsvpO4MS2hw+VtqnPuSkzhzNzePHixU2WL0mNaiyAI+LoiDhmchp4J7AN2AKsK6utA24o01uAC8vZEGcCz7R1VfSNyVsUbdu2rXWrohl9P5A0lzTZBbEUuD4iJl/nf2XmzRHxXeDaiFgPPAqcX9a/CTgHGANeAN7TYG09q/0WRbtH7uTYV7+egdpFSWpEYwGcmQ8Db5im/Ulg9TTtCWxsqp65ZPIWRc+Obz/4ypLmLK+Ek6RKDGBJqqSjAI6It3TSJknqXKd7wH/SYZskqUMHPAgXEW8GfhVYHBEfbFt0LHBYk4VJ0nx3sLMgDgcWlvWOaWv/MXBeU0VJUj84YABn5jeBb0bEFzLz0S7VJEl9odPzgI+IiE3ASe2PycyzmihKkvpBpwH8VeAK4PPAL5orR5L6R6cBPJGZDpAuSbOo09PQ/iIi/iAilpVbCi2KiEWNViZJ81yne8CTo5d9uK0tgdfMbjmS1D86CuDMPLnpQiSp33QUwBFx4XTtmXn17JYjSf2j0y6IN7VNv4LWcJJ3AwawJB2iTrsg3tc+HxEDwFeaKEiS+sWhDkf5PGC/sCS9BJ32Af8Fe+9Odhjw94FrmypKBzcxMcHIyN6bTK9cuZIFC5q+ybWk2dTpJ/azbdMTwKOZuaOBetShkZERLrrsRhYuGeK58R1csRFWrVpVuyxJM9BpH/A3I2Ipew/GjTZXkjq1cMkQA4On1C5D0iHqtAvifOAPgW8AAfxJRHw4M69rsDZNMXnLesBb1kvzQKddEB8D3pSZ4wARsRj4a8AA7iJvWS/NL52eBfGyyfAtnpzBYzWLJm9Zf9SipbVLkfQSdboHfHNEfB34cpn/18BNzZQkSf3hYPeEOxVYmpkfjoh/Bby1LPp/wJeaLk6S5rOD7QH/MfBRgMz8GvA1gIj4B2XZv2ywNkma1w7Wj7s0M++f2ljaTurkBSLisIj4XkTcWOZPjog7ImIsIq6JiMNL+xFlfqws7+j5JWmuOlgADxxg2ZEdvsb7gYfa5j8DXJKZpwJPAetL+3rgqdJ+SVlPkuatgwXwnRHxu1MbI+J3gLsO9uQRMQT8c1r3kiMiAjiLvaevbQbOLdNryzxl+eqyviTNSwfrA/4AcH1E/BZ7A3cYOBz4jQ6e/4+B/wQcU+aPB57OzIkyvwMYLNODwHaAzJyIiGfK+k+0P2FEbAA2ACxfvryDEvqP40RIc8MBP5WZuRv41Yj4dWByoIH/nZm3HuyJI+JfAOOZeVdEvP2lFtpW0yZgE8Dw8LDXgk3DcSKkuaHTsSBuA26b4XO/BXhXRJxDaxD3Y4HPAQMRsaDsBQ8BO8v6O4ETgR0RsQB4Ja0LPnQIHCdC6n2NXc2WmR/NzKHMPAm4ALg1M3+LVpCfV1ZbB9xQprew9+af55X13cOVNG/VuJz4I8AHI2KMVh/vlaX9SuD40v5B4OIKtUlS13TlyExmfoPWSGpk5sPAGdOs8xPg3d2oR5J6gYfG54H2YSrBoSqlucIAngfah6kEHKpSmiMM4HlicphKgGfHt1euRlInHNNXkioxgCWpEgNYkioxgCWpEgNYkioxgCWpEgNYkioxgCWpEgNYkioxgCWpEi9F7iNTb1UE3q5IqslPXh9pv1UR4O2KpMoM4D7jrYqk3mEfsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiUGsCRVYgBLUiWNBXBEvCIivhMR90bEAxHxydJ+ckTcERFjEXFNRBxe2o8o82Nl+UlN1SZJvaDJPeCfAmdl5huA04A1EXEm8Bngksw8FXgKWF/WXw88VdovKetJ0rzVWABny3Nl9uXlJ4GzgOtK+2bg3DK9tsxTlq+OiGiqPu1rYmKCbdu27fmZmJioXZI07zU6GlpEHAbcBZwKXAb8AHg6Myc/3TuAwTI9CGwHyMyJiHgGOB54YspzbgA2ACxfvrzJ8vtK+1CVDlMpdUejB+Ey8xeZeRowBJwBrJyF59yUmcOZObx48eKX+nRqMzlU5eR4wZKa1ZWzIDLzaeA24M3AQERM7nkPATvL9E7gRICy/JXAk92oT5JqaPIsiMURMVCmjwTeATxEK4jPK6utA24o01vKPGX5rZmZTdUnSbU12Qe8DNhc+oFfBlybmTdGxIPAVyLivwLfA64s618J/FlEjAE/Ai5osDZJqq6xAM7M+4A3TtP+MK3+4KntPwHe3VQ9ktRrvCfcPPfLX/6C0dFRgNZvO3WknmEAz3PPP/F3fGrLI5yw/Hl2j9zJsa9+PQO1i5IEOBZEXzh68SADg6dw1KKltUuR1MYAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsTR0PqYQ1VKdRnAfcyhKqW67ILocw5VKdVjAEtSJXZBaEYmJiYYGRnZM79y5UoWLPBtJB0KPzmakZGRES667EYWLhniufEdXLERVq1aVbssaU4ygDVjC5cMMTB4Su0ypDnPPmBJqsQ9YL1I+/nBk+zrlWafnyi9SPv5wYB9vVJDGuuCiIgTI+K2iHgwIh6IiPeX9kURcUtEjJbfx5X2iIhLI2IsIu6LiNObqk0HN3l+8MDgKSxcMlS7HGlearIPeAL4j5n5OuBMYGNEvA64GNiamSuArWUe4GxgRfnZAFzeYG2SVF1jAZyZuzLz7jL9LPAQMAisBTaX1TYD55bptcDV2XI7MBARy5qqT5Jq68pZEBFxEvBG4A5gaWbuKoseAyavgR0Etrc9bEdpk6R5qfGDcBGxEPhz4AOZ+eOI2LMsMzMiZjQGV0RsoNVFwfLly2ezVL1EXiUnzUyjn46IeDmt8P1SZn6tNO+OiGWZuat0MYyX9p3AiW0PHypt+8jMTcAmgOHhYQdQ7CFeJSfNTJNnQQRwJfBQZv5R26ItwLoyvQ64oa39wnI2xJnAM21dFZojJq+S88wJ6eCa3AN+C/DbwP0RcU9p+8/Ap4FrI2I98Chwfll2E3AOMAa8ALynwdo0Aw7cLjWjsQDOzP8DxH4Wr55m/QQ2NlWPDp0Dt0vNcCwIdcSB26XZZwBLUiUGsCRVYgBLUiV9fZZ8+4UDHt2X1G19HcDtFw54dF9St/V1AMPeCweeHd9+8JW1j6kDt/stQpqZvg9gHbqpA7f7LUKaGQNYL8nk+cGA3yKkGTKA1bipo6SBI6VJYACrC9oPdoL3mJMmGcDqismDnZL28kIMSarEAJakSgxgSarEPmA1wkHcpYMzgNUIB3GXDs4uCDXGQdylA3MPWFV5kYb6me9yVeVFGupnBrCq8yIN9Sv7gCWpEgNYkioxgCWpEgNYkioxgCWpksYCOCKuiojxiNjW1rYoIm6JiNHy+7jSHhFxaUSMRcR9EXF6U3VJUq9ocg/4C8CaKW0XA1szcwWwtcwDnA2sKD8bgMsbrEuSekJj5wFn5rci4qQpzWuBt5fpzcA3gI+U9qszM4HbI2IgIpZl5q6m6lM9DtQjtXT7QoylbaH6GDA5SMAg0H5Hxx2l7UUBHBEbaO0ls3z58uYqVWMcqEdqqXYQruztznjfJzM3ZeZwZg4vXry4gcrUDQ7UI3V/D3j3ZNdCRCwDxkv7TuDEtvWGSpv6THv3BDgwj+a3bu8BbwHWlel1wA1t7ReWsyHOBJ6x/7c/tbon7uVDX72Hiy678UUjpUnzSWO7FhHxZVoH3E6IiB3Ax4FPA9dGxHrgUeD8svpNwDnAGPAC8J6m6lLvm+yecG9Y812TZ0H85n4WrZ5m3QQ2NlWL5qb2g3UOU6n5yN0J9bTJvWFpPvJSZEmqxACWpEoMYEmqxD5gzQlTz4gAz4rQ3Oe7V3NC+xkR4M07NT8YwJoz2s+I8BxhzQe+YzUneY6w5gMDWHOW5whrrvMsCEmqxD1gzXn2B2uu8l2qOc/+YM1VBrDmBfuDNRcZwJpXvGBDc4nvSs0rnV6wMTEx8aLB3g1qdZvvNs07+7tgY2JiAoAFCxYwOjrKZ78+wjFLW3fCsu9YNRjAmtem3oH5sKMGOGH5qXvvxmy/sSoygDXvTe4RPzu+nQXHnLBnen+mdk/YNaGm+K6SphgZGeGiy25k4ZIhuybUKANYYt++4tHRURYuHpq2e2J/e8ce1NOh8N0h8eK+4mNf/XoGpllvf3vH7e3gQT11xgCWiva+4gNZuGT6veP9tUv7YwBLBzD1wo7R0VHIFy9rb5c61XcB3N5X54dGBzP1wo727olOuy3a2Vesdn33V2/vq+v0Q6P+1n5hx9TuiU67LSZN7St+9rFH+dCaUVasWAEYxv2mL//Sk311nX5opJk60BV47WdYPDu+nU9tudeR3PpUTwVwRKwBPgccBnw+Mz9duSTpkBz0Cry2dSf3ojsdSGhqN0Z7wO/vMepNPfNXiojDgMuAdwA7gO9GxJbMfLBuZdKhmekVeFP7m9u7Jw40jkV7wO/vMbBvUO9velInId7+H0Ev/icwG1c0Nn1VZM8EMHAGMJaZDwNExFeAtcCsB/Bz4zsAeOFHuznsJz/l6SOP3Gf6QMtqrdeLNflvbGC9owb2vE9fePpxPnbVXzLwqvt58pGHOOzIYxh41RBPPvIQx5z4Wo6Z5r29v8cAL3qO6aZbNY3z8X+zek+/9P6Mjo7yyS9u5ahFS/Z5jk4f37T2+g61pqnP8cVPXjSrXUSR2RunAUTEecCazPydMv/bwD/OzPdOWW8DsKHMvhb4/gxf6gTgiZdY7mzrxZrAumaiF2uC3qyrF2uCZut6IjPXTG3spT3gjmTmJmDToT4+Iu7MzOFZLOkl68WawLpmohdrgt6sqxdrgjp19dJdkXcCJ7bND5U2SZqXeimAvwusiIiTI+Jw4AJgS+WaJKkxPdMFkZkTEfFe4Ou0TkO7KjMfaOClDrn7okG9WBNY10z0Yk3Qm3X1Yk1Qoa6eOQgnSf2ml7ogJKmvGMCSVEnfBHBErImI70fEWERcXLmWH0bE/RFxT0TcWdoWRcQtETFafh/XhTquiojxiNjW1jZtHdFyadl+90XE6V2s6RMRsbNsr3si4py2ZR8tNX0/Iv5ZEzWV1zkxIm6LiAcj4oGIeH9pr7a9DlBT1e0VEa+IiO9ExL2lrk+W9pMj4o7y+teUg+1ExBFlfqwsP6mLNX0hIh5p21anlfauvN/JzHn/Q+ug3g+A1wCHA/cCr6tYzw+BE6a0/Tfg4jJ9MfCZLtTxNuB0YNvB6gDOAf4SCOBM4I4u1vQJ4EPTrPu68rc8Aji5/I0Pa6iuZcDpZfoY4G/K61fbXgeoqer2Kv/mhWX65cAdZRtcC1xQ2q8Afr9M/wFwRZm+ALimizV9AThvmvW78n7vlz3gPZc5Z+bPgMnLnHvJWmBzmd4MnNv0C2bmt4AfdVjHWuDqbLkdGIiIZV2qaX/WAl/JzJ9m5iPAGK2/9azLzF2ZeXeZfhZ4CBik4vY6QE3705XtVf7Nz5XZl5efBM4CrivtU7fV5Da8DlgdEdGlmvanK+/3fgngQaB9FJQdHPiN2rQE/ioi7iqXVgMszcxdZfoxYGmd0vZbR+1t+N7yVfCqtu6ZKjWVr8hvpLUX1RPba0pNUHl7RcRhEXEPMA7cQmtv++nMnJjmtffUVZY/AxzfdE2ZObmtPlW21SURccTUmqapd9b0SwD3mrdm5unA2cDGiHhb+8JsfQeqfn5gr9QBXA6cApwG7AL+e61CImIh8OfABzLzx+3Lam2vaWqqvr0y8xeZeRqtK1rPAFZ2u4apptYUEauAj9Kq7U3AIuAj3aypXwK4py5zzsyd5fc4cD2tN+juya845fd4pfL2V0e1bZiZu8uH55fA/2Tv1+au1hQRL6cVdF/KzK+V5qrba7qaemV7lVqeBm4D3kzra/zkxV/tr72nrrL8lcCTXahpTenGycz8KfCndHlb9UsA98xlzhFxdEQcMzkNvBPYVupZV1ZbB9xQo74D1LEFuLAcHT4TeKbtq3ejpvS9/Qat7TVZ0wXlKPrJwArgOw3VEMCVwEOZ+Udti6ptr/3VVHt7RcTiiBgo00fSGuP7IVqhd15Zbeq2mtyG5wG3lm8TTdc00vafZ9Dqk27fVs2/35s4steLP7SOav4Nrb6oj1Ws4zW0jkTfCzwwWQutPq+twCjw18CiLtTyZVpfUX9Oq49r/f7qoHU0+LKy/e4HhrtY05+V17yvfDCWta3/sVLT94GzG9xWb6XVvXAfcE/5Oafm9jpATVW3F/APge+V198G/Je29/53aB38+ypwRGl/RZkfK8tf08Wabi3bahvwRfaeKdGV97uXIktSJf3SBSFJPccAlqRKDGBJqsQAlqRKDGBJqsQAVl+KiF+JiOsOvqbUHE9Dk6RK3APWvBcRn46IjW3zn4iID0UZc7gM0vKHEfHdMijL75X2yyLiXWX6+oi4qkz/u4j4VI1/i+YXA1j94Brg/Lb589k7ahi0rrZ7JjPfRGtQlt8tl+p+G/i1ss4grfF0KW3farRi9QUDWPNeZn4PWFL6fd8APMW+Qw2+k9Z1//fQCubjaY2T8G3g1yLidcCD7B14583A/+3iP0HzVM/cll5q2FdpDfTyKlp7xO0CeF9mfn3qg8oALmto7fEuorX3/Fy2BkCXXhIDWP3iGlpDM54A/BNat+WZ9HXg9yPi1sz8eUT8PWBnZj4P3A58gNbdHI6ndccGz57QrLALQn0hMx+gdd+0nfniYQU/T6uL4e5yYO5/sHfn5NvAgswcA+6mtRf87e5UrfnO09AkqRL3gCWpEgNYkioxgCWpEgNYkioxgCWpEgNYkioxgCWpkv8PlT+qLuCfMNAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Распределение просмотров в группе 1\n",
    "print(df[df.exp_group == 1].view.mean())\n",
    "sns.displot(df[df.exp_group == 1].view);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca1ced13-f3ba-4aa5-a5a8-9f515475bfc6",
   "metadata": {},
   "source": [
    "Распределение далеко от нормального, поэтому сравнение выборок пока возможно провести только непараметрическими методами. Но попробуем как-то поработать с ним."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "ecb8c618-9461-4f15-ae2e-96ece63f4a99",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAFgCAYAAACFYaNMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZVklEQVR4nO3df9Bc1X3f8ffXgMDFsgRYeaof/BCx6hS7Y0EeKyY4LkFjDxDHIh1HwZMamZIo0wqPGTcJOO2M26k7g6edYBwzJCo4kVLCD2MzUlyKQ8GO7TYQC3gKfgzGigYhCZAEBpkfQ2Whb//YI7FaCbSSnrtnf7xfMzt777l3V1+B/eHuueecG5mJJKn33lK7AEkaVQawJFViAEtSJQawJFViAEtSJUfXLuBInH/++XnXXXfVLkOSDiYO1DjQV8DPPvts7RIk6bANdABL0iAzgCWpEgNYkioxgCWpEgNYkioxgCWpEgNYkioxgCWpEgNYkioxgCWpksYCOCLeFRETba+fRsQVEXFiRNwdET8u7yeU8yMivhQR6yPi4Yg4q6naJKkfNBbAmfmjzFyYmQuBXwReAe4ArgLuycwFwD1lH+ACYEF5LQeub6o2SeoHvVoNbTHwD5m5MSKWAOeW9lXAt4ErgSXA6mw9pO6+iJgZEbMz8+ke1Sj1vZ07dzIxMbFf+8KFC5k2bVrvC9IR6VUAXwzcXLbH2kL1GWCsbM8FNrV9ZnNp2yeAI2I5rStkTjnllKbqlfrSxMQEl1+3hhlzTt/btuOpDXx5BSxatKhiZTocjQdwREwDPgp8tvNYZmZEHNJjmTNzJbASYHx83Ec6a+TMmHM6J80/o3YZmgK9GAVxAfBgZm4t+1sjYjZAed9W2rcAJ7d9bl5pk6Sh1IsA/jivdz8ArAWWle1lwJq29kvKaIj3Azvs/5U0zBrtgoiI44EPAb/X1nw1cFtEXAZsBJaW9juBC4H1tEZMXNpkbdKwOtCNOm/S9adGAzgzXwZO6mh7jtaoiM5zE1jRZD3SKOi8UedNuv410A/llHRg3qgbDE5FlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsRhaNKIcaJG/zCApRHjRI3+YQBLI8iJGv3BPmBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKDGBJqsQAlqRKXIxHGnG7X9vF5OTkPm0uT9kbBrA04l7c+iTXbHyVscd3Ay5P2UsGsCSmj53q8pQV2AcsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUSaMBHBEzI+L2iHgsIh6NiLMj4sSIuDsiflzeTyjnRkR8KSLWR8TDEXFWk7VJUm1NXwFfC9yVmb8AvBd4FLgKuCczFwD3lH2AC4AF5bUcuL7h2iSpqsYCOCJmAB8EbgTIzJ2Z+QKwBFhVTlsFXFS2lwCrs+U+YGZEzG6qPkmqrckr4PnAduDPI+KhiLghIo4HxjLz6XLOM8BY2Z4LbGr7/ObSto+IWB4R6yJi3fbt2xssX5Ka1WQAHw2cBVyfmWcCL/N6dwMAmZlAHsqXZubKzBzPzPFZs2ZNWbGS1GtNBvBmYHNm3l/2b6cVyFv3dC2U923l+Bbg5LbPzyttkjSUGgvgzHwG2BQR7ypNi4EfAmuBZaVtGbCmbK8FLimjId4P7GjrqpCkodP0esCfAm6KiGnABuBSWqF/W0RcBmwElpZz7wQuBNYDr5RzJWloNRrAmTkBjB/g0OIDnJvAiibrkaR+4kw4SarEAJakSnwmnFTRzp07mZiY2KfNJxKPDgNYqmhiYoLLr1vDjDmnAz6ReNQYwFJlM+ac7hOJR5R9wJJUiQEsSZUYwJJUiQEsSZUYwJJUiaMgJO1j92u7mJyc3KfNscnNMIAl7ePFrU9yzcZXGXt8N+DY5CYZwJL2M33sVMcm94B9wJJUiQEsSZXYBSH1sc7FeiYnJ2ktna1hYABLfaxzsZ4tD3+PE37+zMpVaaoYwFKfa1+sZ8dTGypXo6lkH7AkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlDkOTGuRTj/VmDGCpQf3w1OPO5SWdTdc/DGCpYbWfety5vKSz6fqHASyNgPblJZ1N1z+8CSdJlRjAklSJASxJlRjAklRJowEcEU9ExCMRMRER60rbiRFxd0T8uLyfUNojIr4UEesj4uGIOKvJ2iSptl5cAf9qZi7MzPGyfxVwT2YuAO4p+wAXAAvKazlwfQ9qk6RqagxDWwKcW7ZXAd8Griztq7M1Qvy+iJgZEbMz8+kKNUqNcFKE2jUdwAn8TUQk8GeZuRIYawvVZ4Cxsj0X2NT22c2lzQDW0HBShNo1HcAfyMwtEfFzwN0R8Vj7wczMEs5di4jltLooOOWUU6auUqlHBm1SROdVO7iexVRpNIAzc0t53xYRdwCLgK17uhYiYjawrZy+BTi57ePzSlvnd64EVgKMj4/7201qWOdVe431LIZVYzfhIuL4iJi+Zxv4MPADYC2wrJy2DFhTttcCl5TREO8Hdtj/K/WHPVftJ80/Y+/CQjpyTV4BjwF3RMSeP+evMvOuiPg+cFtEXAZsBJaW8+8ELgTWA68AlzZYmyRV11gAZ+YG4L0HaH8OWHyA9gRWNFWPJPUbZ8JJUiUuRykNOMcWDy4DWBpwvR5b7LC0qWMAS0Ogl2OLHZY2dQxgSYesPfB1+AxgSUfkQF0SYLdENwxgSUeks0sC7JbolgEs6YjZJXF4HAcsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUictRSn3EB2yOFgNY6iO9fsCm6jKApT7Tywdsqi77gCWpEgNYkioxgCWpEgNYkioxgCWpEgNYkioxgCWpEgNYkippPIAj4qiIeCgivlH250fE/RGxPiJujYhppf3Ysr++HD+t6dokqaZeXAF/Gni0bf8LwDWZ+U7geeCy0n4Z8Hxpv6acJ0lDq9EAjoh5wK8BN5T9AM4Dbi+nrAIuKttLyj7l+OJyviQNpaavgL8I/CGwu+yfBLyQmbvK/mZgbtmeC2wCKMd3lPMlaSg1FsAR8RFgW2Y+MMXfuzwi1kXEuu3bt0/lV0tST3W1GlpEnJOZ//tgbR3OAT4aERcCxwFvB64FZkbE0eUqdx6wpZy/BTgZ2BwRRwMzgOc6vzQzVwIrAcbHx10oVT21c+dOJiYm9mlbuHAh06ZNq1OQBlq3y1H+CXBWF217ZeZngc8CRMS5wO9n5m9HxFeBjwG3AMuANeUja8v+35Xj96YrUavPTExMcPl1a5gx53SgtVzkl1fAokWLKlemQfSmARwRZwO/DMyKiM+0HXo7cNRh/plXArdExOeBh4AbS/uNwF9GxHrgJ8DFh/n9UqNmzDl973q90pE42BXwNOBt5bzpbe0/pXWV2pXM/Dbw7bK9AdjvciEzXwV+s9vvlKRB96YBnJl/C/xtRPxFZm7sUU2SNBK67QM+NiJWAqe1fyYzz2uiKEkaBd0G8FeBP6U1oeK15sqRpNHRbQDvyszrG61EkkZMtxMx/joi/k1EzI6IE/e8Gq1MkoZct1fAy8r7H7S1JXD61JYjSaOjqwDOzPlNFyJJo6bbqciXHKg9M1dPbTmSNDq67YJ4X9v2ccBi4EHAANbAcB0H9ZtuuyA+1b4fETNpreUgDQzXcVC/6fYKuNPLgP3CGjiu46B+0m0f8F/TGvUArUV4/ilwW1NFSdIo6PYK+L+2be8CNmbm5gbqkaSR0dVEjLIoz2O0VkQ7AdjZZFGSNAq6CuCIWAr8Pa3lIpcC90dE18tRSpL2120XxL8D3peZ2wAiYhbwv3j96caSpEPU7VoQb9kTvsVzh/BZSdIBdHsFfFdEfBO4uez/FnBnMyVJ0mg42DPh3gmMZeYfRMS/AD5QDv0dcFPTxUnSMDvYFfAXKU82zsyvA18HiIh/Vo79eoO1SRpQu1/bxeTk5D5tTvve38ECeCwzH+lszMxHIuK0ZkqSNOhe3Pok12x8lbHHdwNO+34jBwvgmW9y7K1TWIekITN97FSnfR/EwUYyrIuI3+1sjIjfAR5opiRJGg0HuwK+ArgjIn6b1wN3HJgG/EaDdUnS0HvTAM7MrcAvR8SvAu8pzf8jM+9tvDJJGnLdrgf8LeBbDdciSSPlcNcDlnQAnU/dmJycJDPf+AMaaQawNIU6n7qx5eHvccLPn1m5KvUrA1g6Ap0TDiYnJ3n77Pl7h1/teGpDrdI0AAxg6Qh0TjjwileHwgCWjlD7hAOveHUoXFJSkioxgCWpksa6ICLiOOA7wLHlz7k9Mz8XEfOBW4CTaM2u+0Rm7oyIY4HVwC/SWvD9tzLziabqk9Q7ro52YE32Af8/4LzMfCkijgG+FxH/E/gMcE1m3hIRfwpcBlxf3p/PzHdGxMXAF2gt/C5pwLk62oE11gWRLS+V3WPKK4HzeP1ZcquAi8r2krJPOb44IqKp+iT11p6blSfNP2PvOOlR12gfcEQcFRETwDbgbuAfgBcyc1c5ZTMwt2zPBTYBlOM7aHVTdH7n8ohYFxHrtm/f3mT5ktSoRgM4M1/LzIXAPGAR8AtT8J0rM3M8M8dnzZp1pF8nSdX0ZBREZr5AazGfs4GZEbGn73kesKVsbwFOBijHZ9C6GSdJQ6mxAI6IWRExs2y/FfgQ8CitIP5YOW0ZsKZsry37lOP3pquYSBpiTY6CmA2sioijaAX9bZn5jYj4IXBLRHweeAi4sZx/I/CXEbEe+AlwcYO1SVJ1jQVwZj4M7DcpPjM30OoP7mx/FfjNpuqRpH7jTDhJqsQAlqRKDGBJqsQAlqRKDGBJqsQF2TWyXKFLtRnAGlmu0KXaDGCNtPbHCUm9ZgBLb2Dnzp1MTEzs0zY5OYkz5DVVDGDpDUxMTHD5dWv2WbvWpx5rKhnA0puYMef0fboofOqxppLD0CSpEgNYkioxgCWpEgNYkirxJpyknnMWYosBLKnnnIXYYgBLqsJZiPYBS1I1BrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlBrAkVWIAS1IlzoTT0Op8pJCPE1K/MYA1tDofKeTjhNRvDGANtfZHCvk4IfUb+4AlqRIDWJIqMYAlqZLG+oAj4mRgNTAGJLAyM6+NiBOBW4HTgCeApZn5fEQEcC1wIfAK8MnMfLCp+jR8HPWgQdPkTbhdwL/NzAcjYjrwQETcDXwSuCczr46Iq4CrgCuBC4AF5fVLwPXlXeqKox40aBoL4Mx8Gni6bL8YEY8Cc4ElwLnltFXAt2kF8BJgdbYuWe6LiJkRMbt8j9QVRz1okPSkDzgiTgPOBO4HxtpC9RlaXRTQCudNbR/bXNo6v2t5RKyLiHXbt29vrmhJaljjARwRbwO+BlyRmT9tP1audg+pky4zV2bmeGaOz5o1aworlaTeanQiRkQcQyt8b8rMr5fmrXu6FiJiNrCttG8BTm77+LzSJh2QN9006JocBRHAjcCjmfnHbYfWAsuAq8v7mrb2yyPiFlo333bY/6s34003Dbomr4DPAT4BPBIRE6Xtj2gF720RcRmwEVhajt1JawjaelrD0C5tsDYNCW+6aZA1OQrie0C8weHFBzg/gRVN1SNJ/caZcJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZX4SCINDGe+adgYwBoYznwbXrtf28Xk5OQ+bQsXLmTatGmVKuoNA1gDxZlvw+nFrU9yzcZXGXt8N9D6d/vlFbBo0aLKlTXLAJbUF6aPnbr3P66jwptwklSJASxJlRjAklSJASxJlXgTTlLfGZVhaQaw+pYTL0bXqAxLM4DVt5x4MdpGYViaAay+5sQLDTNvwklSJQawJFViAEtSJQawJFViAEtSJQawJFViAEtSJQawJFViAEtSJc6Ek9T3hnVxHgNYUt8b1sV5DGBJA2EYF+cxgKWi82euy1+qaQawVHT+zHX5SzXNAFZf6Fx8Hepcgbb/zHX5SzWtsQCOiK8AHwG2ZeZ7StuJwK3AacATwNLMfD4iArgWuBB4BfhkZj7YVG3qP52Lr4NXoBp+TY4D/gvg/I62q4B7MnMBcE/ZB7gAWFBey4HrG6xLfWrP4ut7Xm97x5zaJUmNaiyAM/M7wE86mpcAq8r2KuCitvbV2XIfMDMiZjdVmyT1g17PhBvLzKfL9jPAWNmeC2xqO29zadtPRCyPiHURsW779u3NVSpJDas2FTlbd1cO+Q5LZq7MzPHMHJ81a1YDlUlSb/Q6gLfu6Voo79tK+xbg5Lbz5pU2SRpavQ7gtcCysr0MWNPWfkm0vB/Y0dZVIUlDqclhaDcD5wLviIjNwOeAq4HbIuIyYCOwtJx+J60haOtpDUO7tKm6JKlfNBbAmfnxNzi0+ADnJrCiqVokqR+5HrAkVWIAS1IlrgWhKjrXfnDlMY0iA1hVdK794LoPGkUGsKrZs/YDuPKYRpN9wJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZU4EUPSwNn92i4mJyf3aVu4cCHTpk2rVNHhMYAlDZwXtz7JNRtfZezx3UBrJuWXV8CiRYsqV3ZoDGBJA2n62Kl7p7IPKvuAJakSA1iSKrELQj3h+r/S/gxg9YTr/0r7M4DVM67/K+3LAJY08AZ1XLABLGngDeq4YANY0lAYxHHBBrCmROcoBxiMn4BSTQawpkTnKIdB+Qko1WQAa8q0j3KQdHDOhJOkSrwC1mFxZpt05AxgHRZntqmfHWhcMPTfjWEDWIfNmW3qV53jggGe3/RjViye5N3vfvfettqBbABLGkqd44J3PLWBa+76wd5Q7odANoAFOI5Xo6E9lDsDucbQyb4K4Ig4H7gWOAq4ITOvrlzSyHAcr0ZR7dlzfRPAEXEUcB3wIWAz8P2IWJuZP5zKP2cYrvQO9e/Q7fmO49Uo67xx97Of/QyAY445Zm/bVGdF3wQwsAhYn5kbACLiFmAJMKUBPDExwSX//oscf9JsAF5+7mmu/PiH9ukH6neTk5N84ea7u/47dHP+5OTkPjfSdjy1gcnJNx4m3nn+S88+xdGvvspzxx9/wM8f7PzO/W7O6fV+P9RgTc3V9PTkffyn+1/khDk/AODZDT/gqOOmc8KcU4HW/29Wf/6KKf1VGP0ydjMiPgacn5m/U/Y/AfxSZl7ecd5yYHnZfRfwo8P4494BPHsE5dZi3b1l3b01zHU/m5nndzb20xVwVzJzJbDySL4jItZl5vgUldQz1t1b1t1bo1h3P01F3gKc3LY/r7RJ0lDqpwD+PrAgIuZHxDTgYmBt5ZokqTF90wWRmbsi4nLgm7SGoX0lM/efSzg1jqgLoyLr7i3r7q2Rq7tvbsJJ0qjppy4ISRopBrAkVTJSARwR50fEjyJifURcVbuebkXEVyJiW0T8oHYt3YqIkyPiWxHxw4iYjIhP166pGxFxXET8fUT831L3f6xd06GIiKMi4qGI+EbtWroVEU9ExCMRMRER62rX062ImBkRt0fEYxHxaEScfcjfMSp9wGWq8+O0TXUGPj7VU52bEBEfBF4CVmfme2rX042ImA3MzswHI2I68ABwUb//846IAI7PzJci4hjge8CnM/O+yqV1JSI+A4wDb8/Mj9SupxsR8QQwnpkDNQkjIlYB383MG8rIrX+UmS8cyneM0hXw3qnOmbkT2DPVue9l5neAn9Su41Bk5tOZ+WDZfhF4FJhbt6qDy5aXyu4x5TUQVykRMQ/4NeCG2rUMu4iYAXwQuBEgM3ceavjCaAXwXGBT2/5mBiAQhkFEnAacCdxfuZSulJ/xE8A24O7MHIi6gS8CfwjsPsh5/SaBv4mIB8pSA4NgPrAd+PPS5XNDRBx/sA91GqUAVgUR8Tbga8AVmfnT2vV0IzNfy8yFtGZjLoqIvu/2iYiPANsy84HatRyGD2TmWcAFwIrS5dbvjgbOAq7PzDOBl4FDvq80SgHsVOceK32oXwNuysyv167nUJWflN8C9ltEpQ+dA3y09KfeApwXEf+9bkndycwt5X0bcAet7sJ+txnY3Pbr6HZagXxIRimAnercQ+Vm1o3Ao5n5x7Xr6VZEzIqImWX7rbRu2j5WtaguZOZnM3NeZp5G63/b92bmv6xc1kFFxPHlJi3lJ/yHgb4f7ZOZzwCbIuJdpWkxh7F0bt9MRW5aj6c6T6mIuBk4F3hHRGwGPpeZN9at6qDOAT4BPFL6UwH+KDPvrFdSV2YDq8qombcAt2XmwAzpGkBjwB2t/15zNPBXmXlX3ZK69ingpnJBtwG49FC/YGSGoUlSvxmlLghJ6isGsCRVYgBLUiUGsCRVYgBLUiUGsEZSRMyJiNtr16HR5jA0SarEK2ANvYi4OiJWtO3/h4j4/T3rK5fFd/5LRHw/Ih6OiN8r7ddFxEfL9h0R8ZWy/a8i4j/X+LtouBjAGgW3Akvb9pey78pslwE7MvN9wPuA342I+cB3gV8p58wFzijbvwJ8p9GKNRIMYA29zHwI+LnS7/te4Hn2XZr0w8AlZcr0/cBJwAJKAEfEGbTm+W8tC82fDfyfHv4VNKRGZi0IjbyvAh8D/jGtK+J2AXwqM7/Z+aGyMM/5tK54T6R19fxSWWReOiIGsEbFrcB/A94B/HPg2LZj3wT+dUTcm5k/i4h/AmzJzJeB+4ArgPNoXRnfXl7SEbMLQiOhrHw3nVawPt1x+AZaXQwPlhtzf8brFyffBY7OzPXAg7Sugr/bm6o17ByGJkmVeAUsSZUYwJJUiQEsSZUYwJJUiQEsSZUYwJJUiQEsSZX8f2YC4lJHa2pVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.displot(np.log(df[df.exp_group == 0].view));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "48214ebb-b351-4fdf-8493-84cbac2e8357",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAFgCAYAAACFYaNMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAYH0lEQVR4nO3df7RlZX3f8fdHYPwFzgBOJiODDlVqzEoR6fVXNNbAiguNFdtl0DRVYkmmqx2tLttETP8wXW3WIqs1/mhZJBRMoDUiQVlMLAtDQKNZFuKA9GIuWKcskJkOzOCPUfzRcfDbP84eOHMdZs6dufs895zzfq111t372fuc8x1n/LDvs5/n2akqJEnj96TWBUjSrDKAJakRA1iSGjGAJakRA1iSGjm2dQFH49xzz60bb7yxdRmSdDg5WONEXwE//PDDrUuQpCM20QEsSZPMAJakRgxgSWrEAJakRgxgSWrEAJakRgxgSWrEAJakRgxgSWrEAJakRgxgSWrEAJakRiZ6NTRJsHfvXubn5w9oO+OMM1i1alWjijQqA1iacPPz82y+ZAur128EYM/O+7hkM8zNzbUtTIdlAEtTYPX6jZy08QWty9AS2QcsSY0YwJLUiAEsSY0YwJLUiAEsSY0YwJLUiAEsSY0YwJLUiAEsSY0YwJLUiAEsSY0YwJLUiAEsSY0YwJLUiAEsSY0YwJLUiAEsSY0YwJLUiAEsSY0YwJLUiAEsSY30GsBJ1iS5Nsk9Se5O8vIkJyW5KcnXup8nducmyUeSbEsyn+SsPmuTpNb6vgL+MHBjVf0M8ELgbuAi4OaqOh24udsHeC1wevfaBFzac22S1FRvAZxkNfAq4AqAqtpbVd8GzgOu7E67Enhjt30ecFUN3AqsSbK+r/okqbU+r4BPA3YDf5zky0kuT/J0YF1V7ezOeRBY122fAjww9P7tXdsBkmxKsjXJ1t27d/dYviT1q88APhY4C7i0ql4EfI/HuxsAqKoCaikfWlWXVdVcVc2tXbt22YqVpHE7tsfP3g5sr6rbuv1rGQTwQ0nWV9XOrothV3d8B3Dq0Ps3dG3STNu7dy/z8/MHtJ1xxhmsWrWqUUVaLr0FcFU9mOSBJM+vqq8C5wAL3esC4OLu5/XdW7YA70hyNfBSYM9QV4U0s+bn59l8yRZWr98IwJ6d93HJZpibm2tbmI5an1fAAO8EPpZkFXAv8HYG3R7XJLkQuB84vzv3BuB1wDbg+925koDV6zdy0sYXtC5Dy6zXAK6qO4GD/Wf6nIOcW8DmPuuRpJXEmXCS1EjfXRCSxuzHj+5jYWHhsX1v2K1cBrA0Zb67azsf+PoPWHfPPm/YrXAGsDSFjl/3HG/aTQD7gCWpEQNYkhoxgCWpEQNYkhoxgCWpEQNYkhoxgCWpEQNYkhoxgCWpEWfCSVNs8boQ4NoQK4kBLE2x4XUhwMXcVxoDWJpyw+tCeEW8shjA0gw53BWxz58bLwNYmjGHWinN58+NlwEs6QA+f258HIYmSY0YwJLUiAEsSY0YwJLUiAEsSY04CkKaMIsnUywsLFBVzT9LS2cASxNm8WSKHXd9kTXPPZOTG3+Wls4AlibQ8GSKPTvvWzGfpaWxD1iSGjGAJakRA1iSGjGAJakRb8JJekKuH9wvA1jSE/KJGv0ygCUd0qHWD9bRsQ9YkhrpNYCT3JfkriR3JtnatZ2U5KYkX+t+nti1J8lHkmxLMp/krD5rk6TWxnEF/ItVdWZV7e80ugi4uapOB27u9gFeC5zevTYBl46hNklqpkUf8HnAq7vtK4HPAe/t2q+qwUogtyZZk2R9Ve1sUKO0bHzQpZ5I3wFcwF8kKeCPquoyYN1QqD4IrOu2TwEeGHrv9q7tgABOsonBFTLPfvazeyxdOjKLA3dhYYFLbvkaq591GuBIAj2u7wB+ZVXtSPJTwE1J7hk+WFXVhfPIuhC/DGBubs5187TiLH6y8P4VxhxJoMV6DeCq2tH93JXkOuAlwEP7uxaSrAd2dafvAE4devuGrk2aOMNPFnaFMT2R3m7CJXl6khP2bwOvAb4CbAEu6E67ALi+294CvK0bDfEyYI/9v5KmWZ9XwOuA65Ls/54/raobk3wJuCbJhcD9wPnd+TcArwO2Ad8H3t5jbZLUXG8BXFX3Ai88SPs3gHMO0l7A5r7qkaSVxplwktSIASxJjRjAktSIq6FJGpnrAy8vA1jSyFwfeHkZwJKWxPWBl499wJLUiFfA0gpzsMV8BsPkNW0MYGnMDncj64kW8zl53IWqdwawNGaj3MhyMZ/ZYABLDXgjS+BNOElqxgCWpEYMYElqxACWpEYMYElqxACWpEYMYElqxACWpEaciCHpiLk+8NExgCUdMdcHPjoGsKSj4rTqI2cfsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ14jhgqbHFs8l8CvLsMIClxhbPJvMpyLPDAJZWgOHZZD4FeXbYByxJjRjAktSIASxJjRjAktRI7wGc5JgkX07y6W7/tCS3JdmW5BNJVnXtT+72t3XHN/ZdmyS1NI5REO8C7gae0e3/PvDBqro6yR8CFwKXdj+/VVXPS/KW7rw3j6E+ScvEJ2QsTa8BnGQD8MvA7wHvSRLgbOCfdKdcCfwugwA+r9sGuBb4L0lSjkiXJoZPyFiavq+APwT8NnBCt38y8O2q2tftbwdO6bZPAR4AqKp9SfZ05z88/IFJNgGbAJ797Gf3WbukI+ATMkbXWx9wktcDu6rq9uX83Kq6rKrmqmpu7dq1y/nRkjRWfV4BvwJ4Q5LXAU9h0Af8YWBNkmO7q+ANwI7u/B3AqcD2JMcCq4Fv9FifJDXV2xVwVb2vqjZU1UbgLcAtVfVrwGeBN3WnXQBc321v6fbpjt9i/6+kadZiHPB7GdyQ28agj/eKrv0K4OSu/T3ARQ1qk6SxGctiPFX1OeBz3fa9wEsOcs4PgV8ZRz2aPnv37mV+fv6ANoc/aaVzNTRNhfn5eTZfsoXV6zcCDn/SZDCANTVWr9/o8CdNFNeCkKRGDGBJasQAlqRGDGBJamSkAE7yilHaJEmjG/UK+D+P2CZJGtEhh6EleTnw88DaJO8ZOvQM4Jg+C5OkaXe4ccCrgOO7804Yav8Oj6/nIEk6AocM4Kr6K+CvkvxJVd0/ppokaSaMOhPuyUkuAzYOv6eqzu6jKEmaBaMG8J8BfwhcDjzaXzmSNDtGDeB9VXVpr5VI0owZdRjanyf5l0nWJzlp/6vXyiRpyo16Bbz/SRW/NdRWwN9Z3nIkaXaMFMBVdVrfhUjSrBkpgJO87WDtVXXV8pYjSbNj1C6IFw9tPwU4B7gDMIAlPaEfP7qPhYWFA9p8VNTjRu2CeOfwfpI1wNV9FCRpenx313Y+8PUfsO6efYCPilrsSB9J9D3AfmFJh3X8uuf4qKgnMGof8J8zGPUAg0V4XgBc01dRkjQLRr0C/k9D2/uA+6tqew/1SNLMGGkiRrcozz0MVkQ7EdjbZ1GSNAtGfSLG+cDfAL8CnA/clsTlKCXpKIzaBfFvgRdX1S6AJGuBvwSu7aswSZp2o64F8aT94dv5xhLeK0k6iFGvgG9M8hng493+m4Eb+ilJkmbD4Z4J9zxgXVX9VpJ/DLyyO/Q/gY/1XZw0Cfbu3cv8/Pxj+wsLC1TVId4hDRzuCvhDwPsAqupTwKcAkvy97tg/7LE2aSLMz8+z+ZItrF6/EYAdd32RNc89k5PblqUJcLgAXldVdy1urKq7kmzspyRp8qxev/Gx2V57dt7XthhNjMPdSFtziGNPXcY6JGnmHC6Atyb5zcWNSX4DuL2fkiRpNhyuC+LdwHVJfo3HA3cOWAX8ox7rkjSFXJ7yQIcM4Kp6CPj5JL8I/FzX/D+q6pbeK5N6snjUAsx2CIyTy1MeaNT1gD8LfHYpH5zkKcDngSd333NtVb0/yWkM1hI+mcFV9Vuram+SJzNY4P3vM5jo8eaqum8p3ymNYvGohVkPgXFzecrH9Tmb7f8BZ1fVC4EzgXOTvAz4feCDVfU84FvAhd35FwLf6to/2J0n9WL/qIWTNr7gsSCWxq23AK6BR7rd47pXAWfz+BoSVwJv7LbP6/bpjp+TJH3VJ0mt9bqeQ5JjktwJ7AJuAv4P8O2q2tedsh04pds+BXgAoDu+B35yLHuSTUm2Jtm6e/fuPsuXpF71GsBV9WhVnQlsAF4C/MwyfOZlVTVXVXNr16492o+TpGbGsqJZVX2bwU28lwNrkuy/+bcB2NFt7wBOBeiOr2ZwM06SplJvAZxkbff0ZJI8Ffgl4G4GQbx/MfcLgOu77S3dPt3xW8oVTSRNsSN9KvIo1gNXJjmGQdBfU1WfTrIAXJ3kPwBfBq7ozr8C+G9JtgHfBN7SY22S1FxvAVxV88CLDtJ+L4P+4MXtP2TwyCNJmgk+1UKSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGulzKrI0FXyEkfpiAEuH4SOM1BcDWBrB/kcYScvJPmBJasQAlqRGDGBJasQAlqRGvAknLdGPH93HwsLCY/sLCwv49CwdCQNYWqLv7trOB77+A9bdsw+AHXd9kTXPPZOTG9elyWMAS0fg+HXPeWxY2p6d97UtRhPLPmBJasQAlqRGDGBJasQ+YGmRxYvvOMpBfTGApUUWL77jKAf1xQCWDmJ48R1HOagv9gFLUiNeAUtqZvGsQpitxe4NYEnNLJ5VOGuL3RvAkpoanlU4a+wDlqRGDGBJasQAlqRGDGBJasQAlqRGHAWhqTTr40s1GQxgTaVZH1+qydBbACc5FbgKWAcUcFlVfTjJScAngI3AfcD5VfWtJAE+DLwO+D7w61V1R1/1abKNsmLZLI8v1WTo8wp4H/Cvq+qOJCcAtye5Cfh14OaqujjJRcBFwHuB1wKnd6+XApd2P6Wf4Iplmga9BXBV7QR2dtvfTXI3cApwHvDq7rQrgc8xCODzgKtqcBlza5I1SdZ3nyP9BFcs06QbyyiIJBuBFwG3AeuGQvVBBl0UMAjnB4betr1rW/xZm5JsTbJ19+7d/RUtST3rPYCTHA98Enh3VX1n+Fh3tbukRw1U1WVVNVdVc2vXrl3GSiVpvHoN4CTHMQjfj1XVp7rmh5Ks746vB3Z17TuAU4fevqFrk6Sp1FsAd6MargDurqo/GDq0Bbig274AuH6o/W0ZeBmwx/5fSdOsz1EQrwDeCtyV5M6u7XeAi4FrklwI3A+c3x27gcEQtG0MhqG9vcfaJKm5PkdB/DWQJzh8zkHOL2BzX/VI0krjWhCS1IgBLEmNuBaEZsLw4jwHm7YstWAAayYML87jtGWtFHZBaGbsX5zn+Gc+q3UpEmAAS1IzBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNeJ6wJp5w4u1gwu2t7T47wLgjDPOYNWqVY0q6pcBrJk3vFg74ILtDS3+u9iz8z4u2Qxzc3ONK+uHASzx+GLtMPg/vdoZ/ruYdgawVoS9e/cyPz9/QNs0/+opgQGsFWJ+fp7Nl2xh9fqNwPT/6imBAawVZPX6jTPzq6cEDkOTpGYMYElqxACWpEbsA9ZEWDxKwskSmgYGsCbC4lESTpbQNDCANTGGR0k4WULTwACWtGJN+9oQBrCkFWva14YwgCWtaNO8NoTD0CSpkd4COMlHk+xK8pWhtpOS3JTka93PE7v2JPlIkm1J5pOc1VddkrRS9HkF/CfAuYvaLgJurqrTgZu7fYDXAqd3r03ApT3WJUkrQm8BXFWfB765qPk84Mpu+0rgjUPtV9XArcCaJOv7qk2SVoJx9wGvq6qd3faDwLpu+xTggaHztndtPyHJpiRbk2zdvXt3f5VKUs+a3YSrwTzSJc8lrarLqmququbWrl3bQ2WSNB7jDuCH9nctdD93de07gFOHztvQtUnS1Bp3AG8BLui2LwCuH2p/Wzca4mXAnqGuCkkCHp8Zt3XrVrZu3crevXtbl3RUepuIkeTjwKuBZybZDrwfuBi4JsmFwP3A+d3pNwCvA7YB3wfe3lddkibX8My4aZgV11sAV9WvPsGhcw5ybgGb+6pF0vSYpplxzoSTpEYMYElqxACWpEYMYElqxOUo1YTPeJMMYDXiM94kA1gN+Yw3zTr7gCWpEQNYkhoxgCWpEQNYkhoxgCWpEQNYkhoxgCWpEQNYkhpxIoakibT/6RjDzjjjDFatWtWooqUzgDUWrv2g5Tb8dAxgIp+QYQBrLFz7QX2Y9KdjGMAaG9d+kA7kTThJasQAlqRGDGBJasQAlqRGDGBJasRREFoWi8f5wuQNipfGzQDWslg8zncSB8Vrsk3izDgDWMtmeJyvNG6TODPOAJY0NSZtZpw34SSpEQNYkhoxgCWpEfuANRKHmUnLzwDWSBxmpkkzCcPSDGCNzGFmmiSTMCzNAJY0tVb6sDRvwklSIyvqCjjJucCHgWOAy6vq4uX+jpY3k1rfyDrU97euTerb4j7hH/3oRwAcd9xxj7WN+9/8igngJMcAlwC/BGwHvpRkS1UtHPqdS9PyZlLrG1mH+v7WtUl9W9wnvOOuL3Ls8Sey7rTHH5M1/G9+HBclKyaAgZcA26rqXoAkVwPnAcsawAez+E7pOL9nXN99uO8/XG0LCwsHPMdtz877WFg49oiPP/Lw/+XYH/6Abz7taQfdP9r3H2r/aN47afsrqZYV8Wc7/kQOZfG/+Ys//pc87aSfBuD733yQq/79v1rWi5KslEeDJ3kTcG5V/Ua3/1bgpVX1jkXnbQI2dbvPB756BF/3TODhoyi3FeseL+ser2mu++GqOndx40q6Ah5JVV0GXHY0n5Fka1VN3O/W1j1e1j1es1j3ShoFsQM4dWh/Q9cmSVNpJQXwl4DTk5yWZBXwFmBL45okqTcrpguiqvYleQfwGQbD0D5aVX/b09cdVRdGQ9Y9XtY9XjNX94q5CSdJs2YldUFI0kwxgCWpkZkK4CTnJvlqkm1JLmpdz6iSfDTJriRfaV3LqJKcmuSzSRaS/G2Sd7WuaRRJnpLkb5L8r67uf9e6pqVIckySLyf5dOtaliLJfUnuSnJnkq2t6xlVkjVJrk1yT5K7k7x8Se+flT7gbqrz/2ZoqjPwq8s91bkPSV4FPAJcVVU/17qeUSRZD6yvqjuSnADcDrxxpf/vnSTA06vqkSTHAX8NvKuqbm1c2kiSvAeYA55RVa9vXc+oktwHzFXVRE3ESHIl8IWqurwbvfW0qvr2qO+fpSvgx6Y6V9VeYP9U5xWvqj4PfLN1HUtRVTur6o5u+7vA3cApbas6vBp4pNs9rntNxFVKkg3ALwOXt65lFiRZDbwKuAKgqvYuJXxhtgL4FOCBof3tTEAgTIMkG4EXAbc1LmUk3a/xdwK7gJuqaiLqBj4E/Dbw48Z1HIkC/iLJ7d1yA5PgNGA38Mddt8/lSZ6+lA+YpQBWA0mOBz4JvLuqvtO6nlFU1aNVdSaD2ZgvSbLiu32SvB7YVVW3t67lCL2yqs4CXgts7rrdVrpjgbOAS6vqRcD3gCXdW5qlAHaq85h1faifBD5WVZ9qXc9Sdb9Ofhb4iUVUVqBXAG/o+lKvBs5O8t/bljS6qtrR/dwFXMegy3Cl2w5sH/oN6VoGgTyyWQpgpzqPUXcz6wrg7qr6g9b1jCrJ2iRruu2nMrhpe0/TokZQVe+rqg1VtZHBv+1bquqfNi5rJEme3t2opfsV/jXAih/xU1UPAg8keX7XdA5LXD53xUxF7tuYpzovqyQfB14NPDPJduD9VXVF26oO6xXAW4G7uv5UgN+pqhvalTSS9cCV3aiZJwHXVNVEDemaQOuA6wb/zeZY4E+r6sa2JY3sncDHuou6e4G3L+XNMzMMTZJWmlnqgpCkFcUAlqRGDGBJasQAlqRGDGBJasQA1kxK8qwk17auQ7PNYWiS1IhXwJp6SS5Osnlo/3eT/Jv96yt3i+/8xyRfSjKf5J937ZckeUO3fV2Sj3bb/yzJ77X4s2i6GMCaBZ8Azh/aP58DV2a7ENhTVS8GXgz8ZpLTgC8Av9Cdcwrws932LwCf77VizQQDWFOvqr4M/FTX7/tC4FscuDTpa4C3dVOmbwNOBk6nC+AkP8tgjv9D3ULzLwe+OMY/gqbUzKwFoZn3Z8CbgJ9mcEU8LMA7q+ozi9/ULcxzLoMr3pMYXD0/0i0yLx0VA1iz4hPAfwWeCfwD4MlDxz4D/Iskt1TVj5L8XWBHVX0PuBV4N3A2gyvja7uXdNTsgtBM6Fa+O4FBsO5cdPhyBl0Md3Q35v6Ixy9OvgAcW1XbgDsYXAV/YTxVa9o5DE2SGvEKWJIaMYAlqREDWJIaMYAlqREDWJIaMYAlqREDWJIa+f9dW3wSHTHsXQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.displot(np.log(df[df.exp_group == 1].view));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "156b852a-d8b6-4f50-a225-a16a081416f2",
   "metadata": {},
   "source": [
    "Распределения в в обоих группах стали заметно больше походить на нормальные. Хотя и далеки от идеального, попробуем поработать с ними."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a89fed2-3fed-4ade-aaad-76a78897e162",
   "metadata": {},
   "source": [
    "***Гипотеза:*** Выдвинем новую гипотезу, что в группе 1 в сравнении с группой 0 уменьшилось число просмотров \\\n",
    "***Нулевая гипотеза:*** число просмотров в экспериментальной группе равно или больше, нежели в контрольной группе \\\n",
    "***Альтернативная гипотеза:*** число просмотров в экспериментальной группе меньше, чем в контрольной. \\\n",
    "***Пороговое значение p-value:*** 0.05"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "a4afc2d8-5e96-4881-9b0d-a77630a3467c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MannwhitneyuResult(statistic=84514942.0, pvalue=0.48586381818421304)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Сравнение непараметричиским U-критерием Манна — Уитни\n",
    "stats.mannwhitneyu(np.log(df[df.exp_group == 0].view), np.log(df[df.exp_group == 1].view), alternative = 'greater')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "d0cc511a-ad87-4426-b09b-d7a5fe63797c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ttest_indResult(statistic=0.21340400193402262, pvalue=0.4155067663875661)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Сравнение параметричиским T-тестом\n",
    "stats.ttest_ind(np.log(df[df.exp_group == 0].view), np.log(df[df.exp_group == 1].view), equal_var = False, alternative = 'greater')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cff60f36-3667-4e49-959f-c1e802e77aff",
   "metadata": {},
   "source": [
    "Результаты сравнения количества просмотров в группах 0 и 1 критерием Манна-Уитни и t-тестом, со соответствующими значениями p-value 0.48 и 0.41, превышающими пороговое значение 0.05, не позволяют отменить нулевую гипотезу. \\\n",
    "**Вывод:** новый алгоритм не привел к снижению количества просмотров в экспериментальной группе."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f18cf743-ff0e-4fd2-9be9-080e3a9357ee",
   "metadata": {},
   "source": [
    "**Задача 3. Написать рекомендацию, будем ли мы раскатывать новый алгоритм на всех новых пользователей.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70ea626c-e433-4140-971c-83138a8d2334",
   "metadata": {},
   "source": [
    "### ВЫВОД\n",
    "По результатам A/B теста, проходившего с 2022-01-28 по 2022-02-03 включительно на группах 0 и 1, был получен статистически значимый результат: увеличение количества CTR в экспериментальной группе, при неизменившемся количестве просмотров в контрольной. Рекоменую раскатать новый алгоритм в проде на всех пользователей."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}