Unsupervised_fs_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unsupervised feature selection with global DIFFI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook are reproduced the experimental results on unsupervised feature selection with global DIFFI described in our paper [Interpretable Anomaly Detection with DIFFI: Depth-based Feature Importance for the Isolation Forest](https://arxiv.org/pdf/2007.11117.pdf).\n",
    "\n",
    "The present code represents the very first release of the DIFFI algorithm: the code in this version is not yet optimized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt \n",
    "%matplotlib inline\n",
    "from sklearn.ensemble import IsolationForest\n",
    "from sklearn.metrics import f1_score\n",
    "import interpretability_module as interp\n",
    "from utils import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loaded lympho dataset: 148 samples, 18 features.\n",
      "4.05 percent outliers.\n"
     ]
    }
   ],
   "source": [
    "dataset_id = 'lympho' # 'musk', 'satellite', 'lympho', 'letter', 'ionosphere', 'cardio'\n",
    "\n",
    "X, y, contamination = get_fs_dataset(dataset_id, seed=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the feature ranking based on the global DIFFI method\n",
    "We use $N_{fs}=5$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "max_samples (256) is greater than the total number of samples (148). max_samples will be set to n_samples for estimation.\n",
      "divide by zero encountered in true_divide\n",
      "max_samples (256) is greater than the total number of samples (148). max_samples will be set to n_samples for estimation.\n",
      "divide by zero encountered in true_divide\n",
      "max_samples (256) is greater than the total number of samples (148). max_samples will be set to n_samples for estimation.\n",
      "divide by zero encountered in true_divide\n",
      "max_samples (256) is greater than the total number of samples (148). max_samples will be set to n_samples for estimation.\n",
      "divide by zero encountered in true_divide\n",
      "max_samples (256) is greater than the total number of samples (148). max_samples will be set to n_samples for estimation.\n",
      "divide by zero encountered in true_divide\n"
     ]
    }
   ],
   "source": [
    "sorted_idx, avg_f1_ranking = diffi_ranks(X, y, n_trees=100, max_samples=256, n_iter=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the Isolation Forest\n",
    "We train the Isolation Forest by exploiting a variable number $k$ of features, ranging from $1$ to $p$ (where $p$ is the maximum number of available features, which depends on the selected dataset). For each value of $k$, we perform $n_{iter}=30$ runs of the algorithm and compute the median F1 score. For the Isolation Forest we use hyperparameters values tuned on the original dataset (just to make sure we are considering a reasonable model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_iter = 30\n",
    "\n",
    "n_feats_max = X.shape[1]\n",
    "y[y==1] = -1\n",
    "y[y==0] = 1\n",
    "hyper = fs_datasets_hyperparams(dataset_id)\n",
    "out_dict = {}\n",
    "for n_feats in range(1, n_feats_max+1):\n",
    "    X_sel_feats = X[:, sorted_idx[:n_feats]]\n",
    "    f1_all = []\n",
    "    for j in range(n_iter):\n",
    "        m = IsolationForest(**hyper, random_state=j)\n",
    "        m.fit(X_sel_feats)\n",
    "        y_pred = m.predict(X_sel_feats)\n",
    "        f1_all.append(f1_score(y, y_pred))\n",
    "    out_dict[n_feats] = np.median(f1_all)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot average F1 score as a function of the number of features exploited"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'lympho dataset')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de5yU9Xn//9ebPcDC7oLCwi6ggojs0pSK4jkVxaRqbD3m22iTtkmbmqY1x2ojtQ9jTP2aRNMcfvWb1DYmMTGx1hqjiQYT1CQ1asSgos6u4hmY5SCH3YUF9nD9/rg/g8MwuztHZmb3ej4e85h77sNnPjfMzjWfw33dMjOcc865TI0rdQWcc85VFg8czjnnsuKBwznnXFY8cDjnnMuKBw7nnHNZ8cDhnHMuKx443Kgg6TVJ7yrB+5qkoyqtbOfy4YHDuQonaU4IMtWj4X1c+fPA4ZxzLiseONyoIqlZ0i5JU5PWHSdps6QaSR+U9Kikr0jaLukVSaeE9W9K2iTpL5OO/Y6kb0r6uaRuSb+UdETK275L0kuStkm6WZLCseMk/bOk10O5t0maPEzdr5QUl7RB0l+lbDtX0mpJXaGe1yZt/lV43i6pR9LJkuZJekjSW5K2SLpd0pSk8j4jaX04pw5JZybV+SpJL4dj75R06FDvk9n/ihttPHC4UcXMOoFHgD9NWv0B4A4z6wuvTwSeBaYCPwDuAI4Hjgr7/puk+qTj3w98HpgGPA3cnvK2fxyO/4PwvmeF9R8MjzOAI4F64N/S1VvS2cAVwLuB+UDqeM1O4C+AKcC5wEclXRC2nRaep5hZvZk9Bgi4AZgJtAGHAdeG91oAXA4cb2YNob6vhTI+DlwALA3HbgNuHuZ93FhkZv7wR8U/iL743hWW3wc8GpargE7ghPD6g8BLScf9PmDAjKR1bwHHhOXvEAWdxLZ6YAA4LLw24J1J2+8ErgrLK4G/S9q2AOgDqtPU/1bgC0mvjw5lHzXE+X4V+EpYnhP2PaDcpP0vAFaH5aOATUTBqSZlvxhwZtLrlkSdM3kff4yNh7c43Gj0Y2ChpCOJfsHvMLPfJm3fmLTcC2BmqeuSWxxvJhbMrAfYSvRrPKEzaXlX0rEzgdeTtr1O9AU8I02dZya/T8pxSDpR0sOhy20H8LdELaC0JE2XdEfojuoCvp/Y38zWAp8kaoFsCvslzucI4EehG287USAZGKLObozywOFGHTPbTfTL//3AnwPfy7PIwxILoQvrUGBDBsdtIPoiTjgc6Gf/wJUQT36fsG+yHwD3ErV0JgPfJOqOgqgVkOqGsH6RmTUSdcEl9sfMfmBm7wz1M+CLYdObwDlmNiXpMcHM1g/xPm4M8sDhRqvbiLqlziP6tZ2P90h6p6RaorGOJ8zszZEOAn4IfErS3BBw/i/wX2bWn2bfO4EPSlooaSLw2ZTtDcBWM9st6QTgz5K2bQYGicZRkvfvIRrIngVcmdggaYGkZZLGA7uJWlgDYfM3gesTEwAkNUk6f5j3cWOQBw43KpnZo0Rfcr8zs9fyLO4HRF/kW4HjiFoymbiVqLXzK+BVoi/pjw1R3weIxi0eAtaG52R/B1wnqRu4hijQJI7dBVwPPBq6mE4CPgccC+wAfgrcnVTWeOALwBaibrbpwD+FbV8jatk8GN7rcaLJBEO9jxuDZOatTzc6SXoI+IGZ/WceZXwHWGdm/1ywijlX4fwKUDcqSTqe6Bf3+SPt65zLjndVuVFH0neBXwCfNLPuUtfHudHGu6qcc85lxVsczjnnsjImxjimTZtmc+bMKXU1nHOuojz11FNbzKwpdf2YCBxz5sxh1apVpa6Gc85VFEmvp1vvXVXOOeey4oHDOedcVjxwOOecy4oHDuecc1nxwOGccy4rY2JWVSnds3o9N67oYMP2XmZOqePKsxZwweJZpa6Wc87lzANHEd2zej3L715Db1+UsXr99l6W370GwIOHc65ieVdVEd24omNf0Ejo7RvgxhUdJaqRc87lzwNHEW3Y3pvVeuecqwQeOIpo5pS6rNY751wl8MBRRFeetYDa6v3/ietqqrjyrAUlqpFzzuXPA0cRXbB4Fuf9wcx9rw+dVMsNF/2+D4w75yqaB44iq60ex6TaKgA+dMocDxrOuYrn03GLLBbv4h2zJtPZtZtYZ1epq+Occ3nzFkcRDQ4aHZ3dtLU00trcQHvc72LqnKt8HjiK6I2tu9i1d4CFLY20tTTy6ls72bW3v9TVcs65vHjgKKJYPOqaaguBwww6Or3V4ZyrbB44iijW2c04wfwZ9bQ1NwLQ7oHDOVfhPHAUUSzexZFN9UyoqWL2IXXUj6/e1wpxzrlKVdTAIelsSR2S1kq6Ks32IyStlPSspEckzU7a9kVJz4XH+5LW3x7KfE7SrZJqinkO+YjFu2hriVoa48aJ1uYGDxzOuYpXtMAhqQq4GTgHWAhcKmlhym43AbeZ2SLgOuCGcOy5wLHAMcCJwJWSGsMxtwOtwO8DdcCHi3UO+eja3ce6bb20NjfsW9faEs2sMrMS1sw55/JTzBbHCcBaM3vFzPYCdwDnp+yzEFgZlh9O2r4Q+KWZ9ZvZTuAZ4GwAM7vfAuC3wGzKUGIQfGFL4751bS2NdO/pZ902T3LonKtcxQwcs4A3k16vC+uSPQNcHJYvBBokTQ3rz5E0UdI04AzgsOQDQxfVnwM/S/fmki6TtErSqs2bN+d9MtlKnlGVkFj27irnXCUrZuBQmnWpfTRXAEslrQaWAuuBfjN7ELgf+A3wQ+AxIPUCiP8H/MrMfp3uzc3sFjNbYmZLmpqa8jiN3MTiXRwysYYZjeP3rVswowEJYn4hoHOughUzcKxj/1bCbGBD8g5mtsHMLjKzxcDVYd2O8Hy9mR1jZu8mCkIvJY6T9FmgCfh0Eeufl1i8m9bmRqS34+ek8dUccehE2j31iHOughUzcDwJzJc0V1ItcAlwb/IOkqZJStRhOXBrWF8VuqyQtAhYBDwYXn8YOAu41MwGi1j/nA0kpRpJ1dbS6F1VzrmKVrTAYWb9wOXACiAG3Glmz0u6TtJ5YbfTgQ5JLwIzgOvD+hrg15JeAG4BPhDKA/hm2PcxSU9LuqZY55Cr19/aSW/fAG0tDQdsa2tp5PWtu9i5x1OPOOcqU1Gz45rZ/URjFcnrrklavgu4K81xu4lmVqUrs+wz+iauDk/X4mhtbsAs2ue4Iw452FVzzrm8+ZXjRRCLd1E1Thw1vf6AbYlg4uMczrlK5YGjCGLxLuY1TWJCTdUB22YfUkfDBE894pyrXB44iiAxoyodSbQ1N/qUXOdcxfLAUWA7evtYv7037fhGQmtLAx2d3QwOeuoR51zl8cBRYO37rhg/cEZVQltLIz2eesQ5V6E8cBRYYuxi4TAtjkRr5AUf53DOVSAPHAXW3tnNoZNqaWoYP+Q+R8+oD6lHPHA45yqPB44Ci+7B0bBfqpFUE2urmTt1kk/Jdc5VJA8cBTQwaHRs7N53m9jhRKlHfGaVc67yeOAooNfe2snuvkFahxnfSGhtbuCNrbvo3t13EGrmnHOF44GjgGIZzKhKSAyQv7jRWx3OucrigaOAYvEuqodINZKqbWZiZpUHDudcZfHAUUCxeDfzmuoZX31gqpFUMydPoNFTjzjnKpAHjgJqDzOqMiGJ1pbGfRcMOudcpfDAUSDbd+1lw47dw6YaSbWwpZF2Tz3inKswHjgKJDG1NpvA0dbSwK69A7yxdVexquWccwXngaNAEhfztWbYVQXsy6Dr4xzOuUrigaNAYvEuptXXMr1hQsbHLGhuYJwg1ukzq5xzlcMDR4HE4t1ZdVMBTKipYu60Sd7icM5VlKIGDklnS+qQtFbSVWm2HyFppaRnJT0iaXbSti9Kei483pe0fq6kJyS9JOm/JNUW8xwy0T8wSMfGblqbM++mSohSj3jgcM5VjqIFDklVwM3AOcBC4FJJC1N2uwm4zcwWAdcBN4RjzwWOBY4BTgSulJT4Of9F4CtmNh/YBvx1sc4hU6+9tZO9/YNZtzggChzrtvXS5alHnHMVopgtjhOAtWb2ipntBe4Azk/ZZyGwMiw/nLR9IfBLM+s3s53AM8DZilLOLgPuCvt9F7igiOeQkRdymFGVkLjuo8PHOZxzFaKYgWMW8GbS63VhXbJngIvD8oVAg6SpYf05kiZKmgacARwGTAW2m1n/MGUCIOkySaskrdq8eXNBTmgosXgXNVViXtPIqUZSJYKNd1c55ypFMQNHuhtSpF7pdgWwVNJqYCmwHug3sweB+4HfAD8EHgP6MywzWml2i5ktMbMlTU1NOZ5CZtrjXcxrqqe2Ovt/zubGCUyuq/HA4ZyrGMUMHOuIWgkJs4ENyTuY2QYzu8jMFgNXh3U7wvP1ZnaMmb2bKGC8BGwBpkiqHqrMUojFu4e9VexwJNHW0uD35nDOVYxiBo4ngflhFlQtcAlwb/IOkqZJStRhOXBrWF8VuqyQtAhYBDxoZkY0FvLecMxfAj8u4jmMaNvOvXR2ZZdqJFVbSyMdnd0MeOoR51wFKFrgCOMQlwMrgBhwp5k9L+k6SeeF3U4HOiS9CMwArg/ra4BfS3oBuAX4QNK4xmeAT0taSzTm8a1inUMmYjlcMZ6qrbmR3r4BXn9rZ6Gq5ZxzRVM98i65M7P7icYqktddk7R8F2/PkEreZzfRzKp0Zb5CNGOrLOSSoypV4tj2zm6OzGGA3TnnDia/cjxPsXgXTQ3jmVY/Pucy5s+op2qcfIDcOVcRPHDkKRbvyumK8WQTaqo40lOPOOcqhAeOPPQPDPLSxp6cZ1Qla21p9JlVzrmK4IEjD69s2cnegdxSjaRqa2lg/fZedvR66hHnXHnzwJGHRNdSPjOqEvYNkHt3lXOuzHngyEMs3k1t1bicUo2kavObOjnnKoQHjjzE4l0cNb2emqr8/xlnNI7nkIk1tHuyQ+dcmfPAkYdYvKsg4xuQSD3i9+ZwzpW/ES8ADDdXugT4Q2Am0As8B/wUeMDMBotawzL1Vs8eNnXv2ZcWvRDaWhq5/YnXGRg0qsaly+fonHOlN2yLQ9K3ifJH7SW6gdKlwN8BvwDOBv5X0mnFrmQ5SnQpFarFAdDa3MDuvkFe89QjzrkyNlKL48tm9lya9c8Bd4fkhYcXvlrlb9+Mqjwv/kuWfG+OQgy4O+dcMQzb4hgiaCRv32tmawtbpcrwQryL6Q3jmZpHqpFUnnrEOVcJMkpyKOlU4FrgiHCMADOzI4tXtfLWHu8uaDcVwPjqKuY1TfIryJ1zZS3T7LjfAj4FPAUMFK86laFvYJC1m3o47ejC31mwraWRJ1/dWvBynXOuUDKdjrvDzB4ws01m9lbiUdSalbGXN/eEVCOFG99IaGtpZMOO3WzftbfgZTvnXCFkGjgelnSjpJMlHZt4FLVmZay9APfgGEpisN27q5xz5SrTrqoTw/OSpHUGLCtsdSpDLN5FbfU4jpw2qeBlL9x3U6cuTp43teDlO+dcvjIKHGZ2RrErUkleiHdx9Ix6qguQaiRVU8N4pk6q9ZlVzrmylfE3n6RzJf2jpGsSjwyOOVtSh6S1kq5Ks/0ISSslPSvpkXCVemLblyQ9Lykm6euSFNZfKmlNOOZnkqZleg6FEot309pc+G4qSE494l1VzrnylFHgkPRN4H3Ax4im4v4foqm5wx1TBdwMnEN0//BLJaXeR/wm4DYzWwRcB9wQjj0FOBVYBLwDOB5YKqka+BpwRjjmWeDyTM6hUDZ372FLz56ijG8ktDY30LGxm/6BMZnNxTlX5jJtcZxiZn8BbDOzzwEnA4eNcMwJwFoze8XM9gJ3AOen7LMQWBmWH07absAEoBYYD9QAG4mCloBJoQXSCGzI8BwKor0z6kIqxoyqhLaWRvb2e+oR51x5yjRw9IbnXZJmAn3A3BGOmQW8mfR6XViX7Bng4rB8IdAgaaqZPUYUSOLhscLMYmbWB3wUWEMUMBYSXWNy0CTGHtqK1FUFb8/WesG7q5xzZSjTwPETSVOAG4HfAa8RtSCGky69q6W8voKoC2o1sBRYD/RLOgpoA2YTBZtlkk6TVEMUOBYTZep9Flie9s2lyyStkrRq8+bNGZxiZtrj3TQ3TuCQSbUFKzPVvOmTqPbUI865MpXprKrPh8X/kfQTYIKZ7RjhsHXs3501m5RuJTPbAFwEIKkeuNjMdki6DHjczHrCtgeAkwgtHzN7Oay/Ezhg0D3scwtwC8CSJUtSA1bOXoh3FbWbCqLUI0dNr/fbyDrnytJIadWXheeLEg/gXODMsDycJ4H5kuaGLLqXAPemlD9NUqIOy4lSuAO8QRgMD62MpUCMqEWyUFIi18e7w/qDYm//IC9v7inqwHiCz6xyzpWrkVocS4GHgD9Js82Au4c60Mz6JV0OrACqgFvN7HlJ1wGrzOxe4HTgBkkG/Ar4+3D4XUQXF64J7/MzM7sPQNLngF9J6gNeBz6YwXkWxMube+gbMFoPSuBo4Eer17Nt596idos551y2hg0cZvbZ8PyhXAo3s/uB+1PWXZO0fBdRkEg9bgD4yBBlfhP4Zi71yVdizGFhkbuqgH3XicTiXZxy1EG/VMU554Y0bOCQ9OnhtpvZvxa2OuUtFu9ifPU45kwtfKqRVPtu6tTZ7YHDOVdWRuqqSvy0XkB0EV5ijOJPiLqWxpRYvJujZzQUJdVIqqaG8UyrH+8zq5xzZWekrqrPAUh6EDjWzLrD62uB/y567cpMe2cXy1qnH7T3a2tp8MDhnCs7mf50PhxIvkHEXmBOwWtTxjZ172ZLz96DMqMqoa2lkZc29njqEedcWck0rfr3gN9K+hHRLKcLgduKVqsylJgaW6zkhum0tTSwd2CQV7bs5OgZxR+Qd865TGTU4jCz64EPAduA7cCHzOz/FrNi5aZ934yqg9viALy7yjlXVrIZ5Z0IdJnZ14B1kkbKVTWqxOJdzJw8gckTaw7aex45rZ6aKvmFgM65spJpWvXPAp/h7bxQNcD3i1WpchSLdx/U8Q2A2upxHDXdB8idc+Ul0xbHhcB5wE7Yl2NqzHS67+kf4OXNPbQehAv/UvnMKudcuck0cOw1MyNkt5VU/CvgysjaTT30D9pBb3FAlL59U/ce3urZc9Df2znn0sk0cNwp6d+BKZL+BvgF8B/Fq1Z5SYwxlCRwhPds7/RxDudcecg0rfpNkt4NdBFdRX6Nmf28qDUrI7F4FxNqDk6qkVSJFO6xeBeneuoR51wZyPQ6Dszs55KeSBwj6VAz21q0mpWR9s4uFsxooGpcuntTFdfU+vE0NYznBR/ncM6ViUxnVX1E0kaiO+6tAp4Kz6OemZVkRlWytpZG2n1KrnOuTGTa4rgC+D0z21LMypSjTd172LpzL63NpZtE1tbSwLdffou+gUFqDkKCReecG06m30IvA7uKWZFylZgKW8oWx8KWRvYORHcfdM65Usu0xbEc+E0Y49g3L9TMPl6UWpWRfTmqShg4km/qdDBzZTnnXDqZBo5/J7qF7BpgTKVqjcW7mDWljsl1By/VSKojmyZRWzUuGudYXLJqOOcckHng6DezYe8GOFrF4l37psSWSk3VOObPqPeZVc65spDpGMfDki6T1CLp0MRjpIMknS2pQ9JaSVel2X6EpJWSnpX0iKTZSdu+JOl5STFJX5eksL5W0i2SXpTULunijM82S7v7Bnhly86Sjm8ktDY3erJD51xZyLTF8WfheXnSOgOOHOoASVXAzcC7gXXAk5LuNbMXkna7CbjNzL4raRlwA/Dnkk4BTgUWhf3+F1gKPAJcDWwys6MljQNGDGC5uGf1eq6/P8bAoHH7468zr6meCxbPKsZbZWRgcJAtPXuYe9VPmTmljivPWlDS+jjnxq5MrxzPJYX6CcBaM3sFQNIdwPlAcuBYCHwqLD8M3JN4S2ACUAuIKBvvxrDtr4DWUK9BoOBThO9ZvZ7ld6+ht28AgK27+lh+9xqAknxZ37N6Pfev6QSif5j123tLWh/n3Ng2bOCQ9E4z+99htjcCh5vZc2k2zwLeTHq9DjgxZZ9ngIuBrxFl4G2QNNXMHpP0MBAnChz/ZmYxSVPCcZ+XdDrRNOHLzWxjSrlIugy4DODwww8f7jQPcOOKjn1BI6G3b4AbV3SU5Iv6xhUd7E25fWwp6+PGpntWr+fGFR1s2N6bV6vXyyl+OYWqy1BGGuO4WNJvJF0j6VxJJ0g6TdJfSfoe8BOgbohj0+XnsJTXVwBLJa0m6opaD/RLOgpoA2YTBaBlkk4jCnSzgUfN7FjgMaLurgPfyOwWM1tiZkuamppGOM39bdjem9X6Yiu3+rixJ9EKX7+9d79W7z2r13s5ZVZOoeoyHEXZ0ofZQToEeC/RmEML0AvEgJ+O0Bo5GbjWzM4Kr5cDmNkNQ+xfD7Sb2WxJVwITzOzzYds1wG7gRqAHaDCzQUmHAT8zs98b7hyWLFliq1ZlniHl1C88xPo0X8qzptTx6FXLMi6nUMqtPm7sGeozCGSVw21gcOjvGy+nMOUMVUYu3xeSnjKzJanrRxzjMLNtRCnUs02j/iQwP9xidj1wCW8PsicqNQ3YGsYqlgO3hk1vAH8j6QailstS4KtmZpLuA04nuq7kTPYfMymIK89asN8YB0BdTRVXnrWg0G9VkfVxY89wrduPLp2XcTn/9vBaL6fI5QxVRiF7KDLOjpstM+uXdDmwAqgCbjWz5yVdB6wys3uJAsANkgz4FfD34fC7gGVEFxwaUavivrDtM8D3JH0V2Ax8qNB1T/QFFrOPMJf6fO6+59m2q4/pDeP5p/e0+fiGO2hmTqkbstV7RRY/YH60er2XU+Ryhipj5pShRhWyV9SMeWZ2v5kdbWbzzOz6sO6aEDQws7vMbH7Y58NmtiesHzCzj5hZm5ktTL740MxeN7PTzGyRmZ1pZm8Uo+4XLJ7Fo1ct49UvnMujVy0r+Zf0BYtnccdlJwNw9bkeNNzBdeVZC6hNSbCZS6v3yrMWUFdT5eUUsZxC1WU4RWtxuMI7smkSNVUiFu/m/GNKXRs3llyweBYrYxu579k4gpxb4YVqzXs5xa/LcEYcHAeQNBH4B6Kpt38jaT6wwMx+UrCaFFG2g+Pl7Jyv/ZoZjeP5zodOKHVV3BjziTtW8+SrW/nN8jNLXRV3kAw1OJ5pV9W3ibLinhxerwP+pUB1c1loa27wmzq5kojytpU+/Y4rvUwDxzwz+xLQB2BmvaS/TsMVWWtLA51du9m2c2+pq+LGkN19A7y8uTzytrnSyzRw7JVUR7iAT9I8ku7L4Q6exP042ju91eEOnrWbehgYNFpLnCnalYdMA8dngZ8Bh0m6HVgJ/GPRauWGlPjDjXmKdXcQlcOdMF35GHFWVUhn3g5cBJxE1EX1ibF4//Fy0FQ/nqmTamnv9MDhDp5YvJsJNeOYM3VSqaviykAmV46bpHvM7DjgpwehTm4YkmhtafCuKndQtXd2sWBGQ1bpM9zolWlX1eOSji9qTVzGWpsb6ejsHjavjXOFYmY+o8rtJ9PAcQbwmKSXw9361kh6tpgVc0NrbW5gT/8gr721s9RVcWPAxq49bNvV54HD7ZPplePnFLUWLiuJP+D2eDfzmupLXBs32sXCeFprs8+ocpGMWhxm9jowBfiT8JgS1rkSOGp6PeOED5C7gyIxo6rVWxwuyChwSPoEcDswPTy+L+ljxayYG9qEmiqObKon5leQu4MgFu9m1pQ6JtfVlLoqrkxk2lX118CJZrYTQNIXie6+9/8Vq2JueK3NDTz95vZSV8ONAdHAuHdTubdlOjguIPkm3AN4ypGSamtpZN22Xrp295W6Km4U2903wCube3xg3O0n0xbHt4EnJP0ovL4A+FZxquQykRiofLGzmyVzDi1xbdxo9dLGHgbNrxh3+8t0cPxfie60txXYBnzIzL5azIq54SX+kGN+IaArIk814tLJqMUh6STgeTP7XXjdIOlEM3uiqLVzQ2qZPIHGCdW0e84qV0Sxzi7qaqo4/NCJpa6KKyOZjnF8A+hJer0zrHMlEqUeafTUI66oYvEuFjR7qhG3v4wHxy3pVoFmNkhmCRLPltQhaa2kq9JsP0LSynA1+iOSZidt+5Kk5yXFJH09JFtMPvZeSc9lWP9Rqa25gY7ObgY99YgrgijVSLd3U7kDZBo4XpH0cUk14fEJ4JXhDpBUBdxMdNX5QuBSSQtTdrsJuM3MFgHXATeEY08BTgUWAe8AjgeWJpV9Efu3gMak1pZGevb0s357b6mr4kahzq7d7Ojt86m47gCZBo6/BU4B1hPdNvZE4LIRjjkBWGtmr5jZXuAO4PyUfRYS3dsD4OGk7QZMAGqB8UANsBFAUj3wafzWtftmVvm9OVwx+MC4G0qms6o2mdklZjbdzGaY2Z+Z2aYRDpsFvJn0el1Yl+wZ4OKwfCHQIGmqmT1GFEji4bHCzGJhv88DXwZ2Dffmki6TtErSqs2bN494jpXo6BkNSH43QFccicwECzxHlUuRacqRL0lqDN1UKyVtkfSBkQ5Lsy61M/4KYKmk1URdUeuBfklHAW3AbKJgs0zSaZKOAY4ysx8xAjO7xcyWmNmSpqamkXavSJPGV3PEoRM9Z5UrihfiXcw+pI7GCZ5qxO0v066qPzKzLuCPiVoORwNXjnDMOuCwpNezgQ3JO5jZBjO7yMwWA1eHdTuIWh+Pm1mPmfUADxDdffBk4DhJrwH/Cxwt6ZEMz2FUam1upN1zVrkiaPd7cLghZBo4Ej853gP80My2ZnDMk8B8SXMl1QKXAPcm7yBpmqREHZYDt4blN4haItWSaohaIzEz+4aZzTSzOcA7gRfN7PQMz2FUam1p4NW3dtK7d2DknZ3L0O6+AV7dstMDh0sr08Bxn6R2YAmwUlITsHu4A8ysH7gcWAHEgDvN7HlJ10k6L+x2OtAh6UVgBnB9WH8X8DKwhmgc5Bkzuy/z0xo7WpsbMYOOjd7qcIXT0dkdpRrx8Q2XRkZXjpvZVSEjbpeZDUjaxYEzpNIddz9wf8q6a5KW7yIKEqnHDQAfGaHs14im6o5piamS7fEujjlsSolr40aLxLiZtzhcOpkmOcTMtiUt7yS6etyV2GGHTFD+OtUAABU3SURBVGRibZXPrHIFFYt3M6nWU4249DLtqnJlatw4saC5wa/lcAX1Qkg1Ms5Tjbg0PHCMAq3NUc6qpKwwzuXMzGiPd/mtYt2Qcg4ckloLWRGXu7aWBnb09tHZNex8BecysmHHbrp29/v4hhtSPi2OBwtWC5eX1uboD9yv53CFENsQdXsu9BxVbgjDDo5L+vpQmwCfwlMmEikhYp1dnNE6vcS1cZUuMV62oNlbHC69kWZVfQj4B2BPmm2XFr46LheT62qYNaXOWxyuINo7uzn80InUj8940qUbY0b6ZDwJPGdmv0ndIOnaotTI5aS1ucFzVrmCiMW7PJW6G9ZIYxzvBZ5Ot8HM5ha+Oi5XrS0NvLx5J3v6PfWIy92uvf28+tbOfeNmzqUzUuCoN7Nh05e78tDW0sjAoLF205i/v5XLw4sbezDzK8bd8EYKHPckFiT9T5Hr4vLgM6tcISQGxhd64HDDGClwJF82emQxK+LyM2fqRMZXj/NxDpeXWLyLSbVVzD6krtRVcWVspMBhQyy7MlNdNY6jZzR4ziqXl/Z4N60tjZ5qxA1rpMDxB5K6JHUDi8Jyl6RuSf7Ttsy0Njfsu92nc9kyM2KdPqPKjWzYwGFmVWbWaGYNZlYdlhOvvRO0zLS2NLKlZw+bu9NdduPc8NZt66XbU424DHiSw1EkcdOdDu+ucjlIDIz7VFw3Eg8co0gi9YgPkLtctHd2I0Vdns4NxwPHKDK1fjzTG8b7OIfLSSzexRGHTmSSpxpxIyhq4JB0tqQOSWslXZVm+xGSVkp6VtIjkmYnbfuSpOclxSR9XZGJkn4qqT1s+0Ix61+JWlsavcXhchKLd3k3lctI0QKHpCrgZuAcYCFwqaSFKbvdBNxmZouA64AbwrGnAKcCi4juK348sDRxjJm1AouBUyWdU6xzqERtzQ28tLGHvoHBUlfFVZCde/p5fesuHxh3GSlmi+MEYK2ZvWJme4E7gPNT9lkIrAzLDydtN2ACUAuMB2qAjWa2y8weBghl/g6YjduntaWBvQODvLrFbwnvMtexsTukGvHxDTeyYgaOWcCbSa/XhXXJngEuDssXAg2SpprZY0SBJB4eK8wslnygpCnAn/B24HG8PSPG70HuspH4vHiLw2WimIEj3aWnqVefXwEslbSaqCtqPdAv6Sigjag1MQtYJum0fQVL1cAPga+b2Stp31y6TNIqSas2b96c/9lUiHlN9VSPk19B7rISi3fRML7aU424jBQzcKwDDkt6PRvYkLyDmW0ws4vMbDFwdVi3g6j18biZ9ZhZD/AAcFLSobcAL5nZV4d6czO7xcyWmNmSpqamwpxRBaitHsdR0+tp9xaHy0KUaqQByVONuJEVM3A8CcyXNFdSLXAJcG/yDpKmSUrUYTlwa1h+g6glUi2phqg1EgvH/AswGfhkEete0aKbOnmLw2VmcNBo7+z2GVUuY0ULHGbWD1wOrCD60r/TzJ6XdJ2k88JupwMdkl4EZgDXh/V3AS8Da4jGQZ4xs/vCdN2riQbVfyfpaUkfLtY5VKrWlkbiO3azfdfeUlfFVYB123rp2eOpRlzminqlj5ndD9yfsu6apOW7iIJE6nEDwEfSrF9H+rETl6R13xXk3Zx05NQS18aVu1hnYmDcZ1S5zPiV46NQ4pejj3O4TMTiXUhvp6xxbiQeOEah6Q3jOWRijY9zuIzE4l3MmTqJibWeasRlxgPHKCSJ1uZGYh44XAbaO7u9m8plxQPHKNXa0sCLnd0MDPqNG93Qevb08/pbu2jzGVUuCx44Rqm2lkZ6+wZ4Y+uuUlfFlbGOMDDe6jOqXBY8cIxSiV+QPkDuhvNCSMHvXVUuGx44Rqn5M+oZJ3ycww2rPd5Fw4RqZk3xVCMucx44RqkJNVXMnTbJWxxuWLF4F23NjZ5qxGXFA8coFt3UyVscLr1EqhHvpnLZ8sAxirU1N/DG1l307OkvdVVcGXpz2y527R3wVCMuax44RrFE0roOb3W4NBL34PAZVS5bHjhGsdaWRM4qH+dwB3oh3s04wYIZ3lXlsuOBYxSbNaWOhvHVfjdAl1Z7vIs50yZRV1tV6qq4CuOBYxSTRGtLA+1x76pyB4p1dvkV4y4nHjhGudbmaGaVmacecW/r3t3Hm1t7fUaVy4kHjlGutaWBnj39rNvWW+qquDKSmKbtM6pcLjxwjHKJmVV+PYdLlrgw1AOHy4UHjlEucXMev4LcJXsh3k3jhGpaJk8odVVcBfLAMcrVj6/m8EMneovD7ScW76KtxVONuNwUNXBIOltSh6S1kq5Ks/0ISSslPSvpEUmzk7Z9SdLzkmKSvq7wCZd0nKQ1ocx9693QWpsb9t1X2rnBQaOjs9u7qVzOihY4JFUBNwPnAAuBSyUtTNntJuA2M1sEXAfcEI49BTgVWAS8AzgeWBqO+QZwGTA/PM4u1jmMFq0tjby2ZSe9ewdKXRVXBl7fuovevgGfUeVyVswWxwnAWjN7xcz2AncA56fssxBYGZYfTtpuwASgFhgP1AAbJbUAjWb2mEXzS28DLijiOYwKbc0NDBq8tMm7q9zbqUa8xeFyVczAMQt4M+n1urAu2TPAxWH5QqBB0lQze4wokMTDY4WZxcLx60YoEwBJl0laJWnV5s2b8z6ZSpbIReQXAjqIJkqMExztqUZcjooZONKNPaRehXYFsFTSaqKuqPVAv6SjgDZgNlFgWCbptAzLjFaa3WJmS8xsSVNTU67nMCocfuhE6mqqfJzDAdGMqrnTJjGhxlONuNwUM3CsAw5Lej0b2JC8g5ltMLOLzGwxcHVYt4Oo9fG4mfWYWQ/wAHBSKHP2cGW6A1WNE0c3e+oRF0nMqHIuV8UMHE8C8yXNlVQLXALcm7yDpGmSEnVYDtwalt8gaolUS6ohao3EzCwOdEs6Kcym+gvgx0U8h1GjrbmB9s4uTz0yxu3o7WP99l4PHC4vRQscZtYPXA6sAGLAnWb2vKTrJJ0Xdjsd6JD0IjADuD6svwt4GVhDNA7yjJndF7Z9FPhPYG3Y54FincNo0tbSyLZdfWzq3lPqqrgS6tiXasTHN1zuqotZuJndD9yfsu6apOW7iIJE6nEDwEeGKHMV0RRdl4XWcAV5LN7FjEa/Wnis8hlVrhD8yvExwnNWOYgCx5SJNTT7jweXBw8cY8TkiTXMnDzBc1aNcbHObtqaPdWIy48HjjGktaXRWxxj2MCg0dHZte+Wws7lygPHGNLa3MDaTT3s7R8sdVVcCbz21k529w36+IbLmweOMaS1pZH+QePlzT2lroorgcR1PAs9cLg8eeAYQ9qSZla5sScW76JqnDhqen2pq+IqnAeOMWTutEnUVo3zcY4xKhbv4khPNeIKwAPHGFJdNY75M+q9xTFGeaoRVygeOMaY1mafWTUW7djVx4Ydu31GlSuIol457srPwOAgm7v3MPeqnzJzSh1XnrWACxanzUw/rHtWr+fGFR1s2N5b8nLKqS7lWs6//PQFAL7161eZObkup3KcS/DAMYbcs3o996/pBKJc9Ou397L87jUAWX2R3LN6PcvvXkNvX3RHwVKWU051qYRy3tq5N6dynEumsZAtdcmSJbZq1apSV6PkTv3CQ6zf3nvA+upxYu60SRmX8+qWnfQPHvi5KUU55VSXSipn1pQ6Hr1qWcbluLFJ0lNmtiR1vbc4xpANaYIGQP+gMX9G5lM0X9qU/jqQUpRTTnWppHKG+iw4lwkPHGPIzCl1aVscs6bU8f/ef1zG5QzVcilFOeVUl0oqZ+aUuozLcC6Vz6oaQ648awF1KXP462qquPKsBRVbTjnVZTSX41wyb3GMIYnB0Hxn6pRTOeVUl9FcjnPJfHDcOedcWkMNjntXlXPOuax44HDOOZcVDxzOOeey4oHDOedcVjxwOOecy8qYmFUlaTPweo6HTwO2FKAaXk7xyymnung5Xk45lJNvGUeYWVPqyjEROPIhaVW66WheTvmVU0518XK8nHIop1B1SeVdVc4557LigcM551xWPHCM7BYvp2LKKae6eDleTjmUU6i67MfHOJxzzmXFWxzOOeey4oHDOedcVjxwDEHSrZI2SXouz3IOk/SwpJik5yV9IocyJkj6raRnQhmfy7NOVZJWS/pJHmW8JmmNpKcl5Zx6WNIUSXdJag//RifnUMaCUI/Eo0vSJ3Osz6fCv/Fzkn4oaUKO5XwilPF8NnVJ97mTdKikn0t6KTwfkmM5/yfUZ1BSRlM0hyjnxvD/9aykH0makmM5nw9lPC3pQUkzcyknadsVkkzStBzqcq2k9UmfoffkWhdJH5PUEf6tv5RLOZL+K6kur0l6OsdyjpH0eOLvVNIJI5WTETPzR5oHcBpwLPBcnuW0AMeG5QbgRWBhlmUIqA/LNcATwEl51OnTwA+An+RRxmvAtAL8O38X+HBYrgWm5FleFdBJdOFStsfOAl4F6sLrO4EP5lDOO4DngIlE97z5BTA/188d8CXgqrB8FfDFHMtpAxYAjwBL8qjPHwHVYfmLedSnMWn548A3cyknrD8MWEF0oe+wn8sh6nItcEWW/8/pyjkj/H+PD6+n53pOSdu/DFyTY30eBM4Jy+8BHsn285zu4S2OIZjZr4CtBSgnbma/C8vdQIzoCyqbMszMEjePrgmPnGY1SJoNnAv8Zy7HF5KkRqIP+7cAzGyvmW3Ps9gzgZfNLNdMAdVAnaRqoi/+DTmU0QY8bma7zKwf+CVwYSYHDvG5O58owBKeL8ilHDOLmVlHJvUYoZwHw3kBPA7MzrGcrqSXk8jgMz3M3+VXgH/Ms4ysDFHOR4EvmNmesM+mfOojScCfAj/MsRwDGsPyZHL7PB/AA8dBJGkOsJioxZDtsVWhuboJ+LmZZV1G8FWiP7DBHI9PMOBBSU9JuizHMo4ENgPfDl1n/ylpUp71uoQM/sjSMbP1wE3AG0Ac2GFmD+ZQ1HPAaZKmSppI9EvvsFzqFMwws3ioYxyYnkdZhfZXwAO5HizpeklvAu8HrsmxjPOA9Wb2TK71CC4PXWe3ZtIdOISjgT+U9ISkX0o6Ps86/SGw0cxeyvH4TwI3hn/jm4DledYH8MBx0EiqB/4H+GTKL62MmNmAmR1D9OvuBEnvyKEOfwxsMrOnsj02jVPN7FjgHODvJZ2WQxnVRE3rb5jZYmAnUVdMTiTVAucB/53j8YcQ/bqfC8wEJkn6QLblmFmMqAvn58DPgGeA/mEPqkCSriY6r9tzLcPMrjazw0IZl+dQh4nA1eQYdJJ8A5gHHEP0o+HLOZZTDRwCnARcCdwZWg25upQcfwgFHwU+Ff6NP0Vo3efLA8dBIKmGKGjcbmZ351NW6Mp5BDg7h8NPBc6T9BpwB7BM0vdzrMeG8LwJ+BGQy6DbOmBdUuvpLqJAkqtzgN+Z2cYcj38X8KqZbTazPuBu4JRcCjKzb5nZsWZ2GlH3Qa6/GAE2SmoBCM8jdn8Um6S/BP4YeL+FDvQ8/QC4OIfj5hEF+mfC53o28DtJzdkUYmYbw4+zQeA/yO3zDNFn+u7Qvfxbopb9sIP1QwndpRcB/5VjXQD+kuhzDNEPqoIMjnvgKLLwa+NbQMzM/jXHMpoSM1ck1RF9wbVnW46ZLTez2WY2h6hL5yEzy/oXtaRJkhoSy0SDpVnPPjOzTuBNSQvCqjOBF7ItJ0m+v87eAE6SNDH8v51JNCaVNUnTw/PhRH/8+dTrXqIvAMLzj/MoK2+SzgY+A5xnZrvyKGd+0svzyO0zvcbMppvZnPC5Xkc0GaUzy7q0JL28kBw+z8E9wLJQ5tFEEz5yzU77LqDdzNbleDxEYxpLw/Iy8vsB87ZCjLCPxgfRH3oc6CP6MP51juW8k2g84Fng6fB4T5ZlLAJWhzKeI4MZFhmUeTo5zqoiGpt4JjyeB67Oox7HAKvCud0DHJJjOROBt4DJef67fI7oC+w54HuE2TE5lPNroiD4DHBmPp87YCqwMvzRrwQOzbGcC8PyHmAjsCLHctYCbyZ9njOZDZWunP8J/87PAvcBs3IpJ2X7a4w8qypdXb4HrAl1uRdoyfGcaoHvh/P6HbAs13MCvgP8bZ6fnXcCT4XP4RPAcfn8fSQennLEOedcVryryjnnXFY8cDjnnMuKBw7nnHNZ8cDhnHMuKx44nHPOZcUDhxv1QsbULye9vkLStQfx/cdL+kXIUPq+lG2tYf1qSfNyKPuT4epp5w4aDxxuLNgDXDRSuu0iWgzUmNkxZpZ6FfAFwI/NbLGZvZxD2Z8kuoYlY+GKZOdy5oHDjQX9RPde/lTqBknfkfTepNc94fn0kKTuTkkvSvqCpPcrui/KmnStA0X3zbgnJMp7XNKicAX594FjQstiXtL+7yH64v+wpIfDug+E93ha0r9LqgrrvxHup7DvfiySPk6UU+vhpON7ksp/r6TvJJ3nv4b9vhiu/r9V0pOhtXN+2O/3kt7/2ZSru50DPHC4seNm4P2SJmdxzB8AnwB+H/hz4GgzO4EoJf3H0uz/OWC1mS0C/gm4zaJcXh8Gfh1aHPtaFWZ2P/BN4CtmdoakNuB9RAkkjwEGiLLGQnR1/hKiLAJLJS0ys68TpZQ4w8zOyOB8jgbeZWb/QJQY8CEzO57oHhI3hvQxfwt8Lbz/EqIrkJ3bjzdZ3ZhgZl2SbiO6YVBvhoc9aSGduaSXiW6KA1FqinRf1O8kJOozs4dCWvVsAtWZwHHAkyGhah1vJzT8U0Xp66uJbg62kCg9Rjb+28wGwvIfESW8vCK8ngAcDjwGXK3ovi13W+7pvN0o5oHDjSVfJcof9O2kdf2ElndIbFibtG1P0vJg0utB0v/tpEufnU1OHwHfNbP97pkgaS5wBXC8mW0L3U9D3dI2+f1S99mZ8l4X24E3dopJeoLoZl8rJH3YzB7K4hzcGOBdVW7MMLOtRLeD/euk1a8R/cqH6F4cNXm8xa8IXUuSTge2WHb3XlkJvDcps+6hko4guoPbTmCHpBlE6eMTuoluSZywUVKbpHEMf9fBFcDHEveKkLQ4PB8JvBK6we4l6hpzbj8eONxY82X2vz/CfxCNGfwWOJH9f5Vn61pgiaRngS/wdir0jJjZC8A/E91Z8VmiG0G1WHRnu9VEmYhvBR5NOuwW4IHE4DjRjbB+AjxElCl1KJ8nCpLPSnouvIZojOU5RXebbAVuy+Yc3Njg2XGdc85lxVsczjnnsuKBwznnXFY8cDjnnMuKBw7nnHNZ8cDhnHMuKx44nHPOZcUDh3POuaz8/whCkOuMsPDLAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.xticks(list(out_dict.keys()))\n",
    "plt.plot(list(out_dict.keys()), list(out_dict.values()), '-o')\n",
    "plt.xlabel('Num of features')\n",
    "plt.ylabel('F1 score (median)')\n",
    "plt.title('{} dataset'.format(dataset_id))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}