Predict Employee Computer Access_in_Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Science Project in Python- Given his or her job role, predict employee access needs using amazon employee database."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## When an employee at any company starts work, they first need to obtain the computer access necessary to fulfill their role. This access may allow an employee to read/manipulate resources through various applications or web portals. It is assumed that employees fulfilling the functions of a given role will access the same or similar resources. It is often the case that employees figure out the access they need as they encounter roadblocks during their daily work (e.g. not able to log into a reporting portal). A knowledgeable supervisor then takes time to manually grant the needed access in order to overcome access obstacles. As employees move throughout a company, this access discovery/recovery cycle wastes a nontrivial amount of time and money.\n",
    "\n",
    "\n",
    "## There is a considerable amount of data regarding an employee’s role within an organization and the resources to which they have access. Given the data related to current employees and their provisioned access, models can be built that automatically determine access privileges as employees enter and leave roles within a company. In this data science project, we will build an auto-access model that minimizes the human involvement required to grant or revoke employee access.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "/root/hackerday/12_employee_access_need/train.csv\n",
    "/root/hackerday/12_employee_access_need/test.csv\n",
    "/root/hackerday/12_employee_access_need/sampleSubmission.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Import the dataset\n",
    "#Perform Univariate Analysis\n",
    "#Data Transformation conversion\n",
    "#prepare model data\n",
    "#select model\n",
    "#apply model\n",
    "#evaluate results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   ACTION  RESOURCE  MGR_ID  ROLE_ROLLUP_1  ROLE_ROLLUP_2  ROLE_DEPTNAME  \\\n",
      "0       1     39353   85475         117961         118300         123472   \n",
      "1       1     17183    1540         117961         118343         123125   \n",
      "2       1     36724   14457         118219         118220         117884   \n",
      "3       1     36135    5396         117961         118343         119993   \n",
      "4       1     42680    5905         117929         117930         119569   \n",
      "\n",
      "   ROLE_TITLE  ROLE_FAMILY_DESC  ROLE_FAMILY  ROLE_CODE  \n",
      "0      117905            117906       290919     117908  \n",
      "1      118536            118536       308574     118539  \n",
      "2      117879            267952        19721     117880  \n",
      "3      118321            240983       290919     118322  \n",
      "4      119323            123932        19793     119325  \n"
     ]
    }
   ],
   "source": [
    "trdata = pd.read_csv(\"/root/hackerday/12_employee_access_need/train.csv\")\n",
    "print trdata.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "118300    4424\n",
       "118343    3945\n",
       "118327    2641\n",
       "118225    2547\n",
       "118386    1796\n",
       "Name: ROLE_ROLLUP_2, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trdata['ROLE_ROLLUP_2'].value_counts().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fc49e630090>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEbCAYAAAAh9sTfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X20HVWZ5/Hvk5uEdwmIRggwQSEiNrpsbV9mWObgC43M\ntDAzNtIzbdNgr57Rpa12awu2L7enxxa1e7R9nbW0w8LVLUq3dDCiNuHlAI74jhgIIUAbSNBceUkg\nSEjuyzN/7Gdzdirn3HvuOffce1P5fdY669ap2lW1q2rXU/vs2rfK3B0REamXBXOdARERmXkK7iIi\nNaTgLiJSQwruIiI1pOAuIlJDCu4iIjU0aXA3s1VmNmJm6yrj325md5rZ7Wb20WL8xWZ2t5ltMLMz\nBpVpERGZ3MIppl8KfBr4Uh5hZqcDrwde4O6jZvaMGH8K8EbgFGAZcK2ZrXD3iYHkXEREOpq05u7u\nNwPbKqPfAnzE3UcjzYMx/mzgcncfdfdNwD3AS2c2uyIi0o1e2txPAl5pZt8zs6aZvSTGHwNsKdJt\nIdXgRURklk3VLNNpniPc/eVm9lvAFcCzO6TVsw1EROZAL8F9C3AlgLv/0MwmzOwo4AHguCLdsTFu\nD2amgC8i0gN3t27T9tIssxp4FYCZrQAWu/tDwNeB88xssZmdQGq++UEPyxcRkT5NWnM3s8uBlcDT\nzWwz8EFgFbAqukfuBv4AwN3Xm9kVwHpgDHir65GTIiJzwmY7/qpZRkSkN9Nplumlzb1vixYtYmIi\ndX9fsGDBrA3P9vq0PfvHNmh75vfwfMlHv8Ojo6NMx5zU3NVaIyIyPWY28BuqIiIyzym4i4jUkIK7\niEgNKbiLiNSQgruISA3NSXA36/qGr4iI9EA1dxGRGlJwFxGpIQV3EZEaUnAXEakhBXcRkRpScBcR\nqSEFdxGRGlJwFxGpIQV3EZEaUnAXEakhBXcRkRqaNLib2SozG4mXYVen/ZmZTZjZkcW4i83sbjPb\nYGZnDCLDIiIytalq7pcCZ1ZHmtlxwGuB+4pxpwBvBE6JeT5nZvplICIyByYNvu5+M7CtzaT/A/x5\nZdzZwOXuPurum4B7gJfORCZFRGR6pl2zNrOzgS3u/rPKpGOALcX3LcCyPvImIiI9WjidxGZ2MPA+\nUpPMU6MnmcV7yZSIiPRnWsEdeA6wHLgtXrhxLPBjM3sZ8ABwXJH22BjX1vDwMACNRoNGozHNbIiI\n1Fuz2aTZbPY8v7lPXrk2s+XAGnc/tc20nwMvdvdH4obql0nt7MuAa4ETvbICM3OAqdYrIiItZoa7\nd/0au6m6Ql4OfBdYYWabzeyCSpKnIrS7rweuANYD3wLeWg3sIiIyO6asuc/4ClVzFxGZthmtuYuI\nyL5JwV1EpIYU3EVEakjBXUSkhhTcRURqSMFdRKSGFNxFRGpIwV1EpIYU3EVEakjBXUSkhhTcRURq\nSMFdRKSGFNxFRGpIwV1EpIYU3EVEakjBXUSkhhTcRURqSMFdRKSGFNxFRGpoqhdkrzKzETNbV4z7\nuJndaWa3mdmVZnZ4Me1iM7vbzDaY2RmDzLiIiHQ2Vc39UuDMyrhrgOe7+wuBjcDFAGZ2CvBG4JSY\n53Nmpl8GIiJzYNLg6+43A9sq49a6+0R8/T5wbAyfDVzu7qPuvgm4B3jpzGZXRES60W/N+kLgmzF8\nDLClmLYFWNbn8kVEpAc9B3cz+wtgt7t/eZJk3uvyRUSkdwt7mcnM/hA4C3h1MfoB4Lji+7Exrq3h\n4WEAGo0GjUajl2yIiNRWs9mk2Wz2PL+5T165NrPlwBp3PzW+nwn8LbDS3R8q0p0CfJnUzr4MuBY4\n0SsrMDMHmGq9IiLSYma4u3WbftKau5ldDqwEjjKzzcCHSL1jFgNrzQzgFnd/q7uvN7MrgPXAGPDW\namAXEZHZMWXNfcZXqJq7iMi0Tbfmrn7oIiI1pOAuIlJDCu4iIjWk4C4iUkMK7iIiNaTgLiJSQwru\nIiI1pOAuIlJDCu4iIjWk4C4iUkMK7iIiNaTgLiJSQwruIiI1pOAuIlJDCu4iIjWk4C4iUkMK7iIi\nNaTgLiJSQwruIiI1NGlwN7NVZjZiZuuKcUea2Voz22hm15jZkmLaxWZ2t5ltMLMzBplxERHpbKqa\n+6XAmZVxFwFr3X0FcF18x8xOAd4InBLzfM7M9MtARGQOTBp83f1mYFtl9OuBy2L4MuCcGD4buNzd\nR919E3AP8NKZy6qIiHSrl5r1UncfieERYGkMHwNsKdJtAZb1kTcREenRwn5mdnc3M58sSacJw8PD\nADQaDRqNRj/ZEBGpnWazSbPZ7Hl+c58sNoOZLQfWuPup8X0D0HD3rWZ2NHCDu59sZhcBuPslke7b\nwIfc/fuV5Xmk6znTIiL7GzPD3a3b9L00y3wdOD+GzwdWF+PPM7PFZnYCcBLwgx6WLyIifZq0WcbM\nLgdWAkeZ2Wbgg8AlwBVm9mZgE3AugLuvN7MrgPXAGPBWV/VcRGROTNksM+MrVLOMiMi0zUazjIiI\nzHMK7iIiNaTgLiJSQwruIiI1pOAuIlJDCu4iIjWk4C4iUkMK7iIiNaTgLiJSQwruIiI1pOAuIlJD\nCu4iIjWk4C4iUkMK7iIiNaTgLiJSQwruIiI1pOAuIlJDCu4iIjWk4C4iUkM9B3cze5eZ3W5m68zs\ny2Z2gJkdaWZrzWyjmV1jZktmMrMiItKdnoK7mS0D3g682N1PBYaA84CLgLXuvgK4Lr6LiMgs66dZ\nZiFwsJktBA4GfgG8Hrgspl8GnNNf9kREpBc9BXd3fwD4W+B+UlDf7u5rgaXuPhLJRoClM5JLERGZ\nloW9zGRmR5Bq6cuBR4F/MrPfL9O4u5uZd1rG8PAwAI1Gg0aj0Us2RERqq9ls0mw2e57f3DvG384z\nmf0u8Nvu/kfx/U3Ay4FXAae7+1YzOxq4wd1PrszrAL2sV0Rkf2VmuLt1m77XNvf7gJeb2UFmZsBr\ngPXAGuD8SHM+sLrH5YuISB96qrkDmNkw8EZgDPgJ8EfAYcAVwPHAJuBcd99emU81dxGRaZpuzb3n\n4N4rBXcRkembrWYZERGZxxTcRURqSMFdRKSGFNxFRGpIwV1EpIYU3EVEakjBXUSkhhTcRURqSMFd\nRKSGFNxFRGpIwV1EpIYU3EVEakjBXUSkhhTcRURqSMFdRKSGFNxFRGpIwV1EpIYU3EVEakjBXUSk\nhnoO7ma2xMz+2czuNLP1ZvYyMzvSzNaa2UYzu8bMlsxkZkVEpDv91Nz/Dvimuz8PeAGwAbgIWOvu\nK4Dr4ruIiMwyc/fpz2R2OHCruz+7Mn4DsNLdR8zsWUDT3U+upHGAXtYrIrK/MjPc3bpN32vN/QTg\nQTO71Mx+YmZfMLNDgKXuPhJpRoClk2VUREQGY2Ef8/0m8DZ3/6GZfZJKE4y7e66ldzI8PEyj0aDR\naPSYDRGRemo2mzSbzZ7n77VZ5lnALe5+Qnw/DbgYeDZwurtvNbOjgRs6NcuAmmZERLo1K80y7r4V\n2GxmK2LUa4A7gDXA+THufGB1L8sXEZH+9FRzBzCzFwJfBBYD9wIXAEPAFcDxwCbgXHffXplPNXcR\nkWmabs295+DeKwV3EZHpm63eMiIiMo8puIuI1JCCu4hIDSm4i4jU0JwG9+Hh4blcvYhIbc1pbxlQ\njxkRkW6ot4yIiCi4i4jUkYK7iEgNKbiLiNSQgruISA3NeXDXSztERGbenAd3ERGZeQruIiI1pOAu\nIlJD8yK4q91dRGRmzYvgLiIiM0vBXUSkhuZNcD/wwAPnOgsiIrXRV3A3syEzu9XM1sT3I81srZlt\nNLNrzGxJt8vatWtXP1kREZFCvzX3dwDrgfzc3ouAte6+ArguvndNN1ZFRGZGz8HdzI4FzgK+COSo\n/Hrgshi+DDinr9yJiEhP+qm5fwJ4DzBRjFvq7iMxPAIs7WP5IiLSo4W9zGRm/wn4lbvfamaNdmnc\n3atvXRIRke40m02azWbP8/f0mj0z+2vgTcAYcCDwNOBK4LeAhrtvNbOjgRvc/eTKvJOuUK/dExHZ\n26y8Zs/d3+fux7n7CcB5wPXu/ibg68D5kex8YHUvyxcRkf7MVD/3XN2+BHitmW0EXhXfp0U9ZkRE\n+tdTs0xfK+yiHX5oaIixsbHZyI6IyD5hVpplBm18fHyusyAisk+bl8FdRET6M2+Du9reRUR6N2+D\nu4iI9E7BXUSkhuZ1cFfTjIhIb+Z1cAdYsGDeZ1FEZN6Z95FTjyMQEZm+eR/cQc0zIiLTtU8EdxER\nmZ59JribGUuWdP3WPhGR/dq8fLbMZNQGLyL7o1o8W2Yyan8XEZnaPhfcIQX4RqMx19kQEZm39rlm\nmZKaaERkf1H7ZpmSmmhERNrbp4M7pAC/cGFP7/kWEamtfT64Q3q5h5mpJi8iEmoR3EVEZE89BXcz\nO87MbjCzO8zsdjP7kxh/pJmtNbONZnaNmc36fx3pn51ERHrsLWNmzwKe5e4/NbNDgR8D5wAXAA+5\n+8fM7L3AEe5+UWXeWe3ioh41IlIHs9Jbxt23uvtPY/hx4E5gGfB64LJIdhkp4IuIyCzru83dzJYD\nLwK+Dyx195GYNAIs7Xf5/co3WtWjRkT2J31FvGiS+RrwDnffUfZWcXef7SaYyeQeNQBDQ0OMjY3N\ncY5ERDprNps0m82e5+/5P1TNbBHwDeBb7v7JGLcBaLj7VjM7GrjB3U+uzDdvAj7AypUr+9qBIiKz\nYbpt7r3eUDVSm/rD7v6uYvzHYtxHzewiYMlc31Dthm66ish8N1vB/TTgJuBnQF7AxcAPgCuA44FN\nwLnuvr0y77yNpAryIjJfzUpw78d8Du5VCvYiMl/sVw8OExGR9tQ/cBLtnlWj2ryI7AtUc5+m3G8+\nf4aHh+c6SyIie1Gb+4Cohi8iM0lt7iIiojb3Qen0bHnV6EVkNii4zzIFfRGZDWqWERGpIQX3eaLa\nCyd/9OIREemFesvUgJp0ROpPvWX2Q51q/dVPo9GY66yKyCxRzX0/o1q+yL5JNXeZVLe1fNX0RfZt\nCu7S0Y033jiti4EexSAyf6hZRmaUmn1EBmO6zTL6JyaZUZ3+SatbujiIzAwFd5lX+r04tKMLhuyP\nFNyl9gZxwZiKLigy12b8hqqZnWlmG8zsbjN770wvX0REpjajN1TNbAi4C3gN8ADwQ+D33P3OIo2q\nNCKzbGhoCIAFCxYwMTHR9XAv88y34fmSj36HR0dH5+4F2Wb2CuBD7n5mfL8IwN0vKdIouIuI9GAu\n/4lpGbC5+L4lxomIyCya6eCuWrmIyDww08H9AeC44vtxpNq7iIjMopluc19IuqH6auAXwA+o3FAV\nEZHBm9F+7u4+ZmZvA/4VGAL+XoFdRGT2zfqzZUREZPD0VEgRkRoa6OMHzGwNMAGcRWqm6ff/wL3P\nZXQzf/4p0026TmlmOp+d8tRpPRPF9KEp1jMBPBnfD2yTvt06cn52AmPA04r1VisM1XEe8+Sy92vg\n4Bi2mF5Nn9c/EcNT7duJYjinfZK0feW4CWAUOKDN/BPsud/Lc2W0GN4Z+R9i720t8zkG7I513Q0s\njmUui3kMGAceAw6LaXlfLZpkW6vy/vLKuE4VuV2xjoMr+R2P+RZWxu0GHgaOiWVOxPiF7F1m8/Fa\nQPsynPfxUPwdjemLKumrZXA05qmWk5yuuv27SfvbirTjtMqhkY7j4kh7UDFvNf9eWU7++zhwaGUb\n8363yPNYrHdRpH8Y2Aq8JNY5WbkeBT7p7n8+SZo9DKxZxszOILW9i4jIzPi1ux86dbLBBvcNwHMH\nsnARkf1Ut/+lOsg298maA0REZIAGGdxXDXDZIiIyiYEFd3f/CPCFQS1fRGQ/9Gi3CQfaz93MXgD8\nDukRwCcBR5HuFOe72eXd5gW0LjZlb498t3qCdCc7fy/vlo8Xf4eKNPnu/o6Y/7CYfkCkyb0iiPS7\nY3q+gz5KqzdEtWdRvtuet6e8Q26kXgh5Ww4olr+QvS+qeV/kHh0LivHlNuXllcNlr4TcFPZkDC+O\n72VvgZx+IvKf91PuvVFOz70ghmJf7Cb1btkFLCH1Dmi3LWXeRmObymNe7eVQrm+o2N6y50c+TqOR\n123A02Nc7mlitHrflL0myl5DZa8KijRj7FmmcrqJNvNS7Lv7gWfE+g8rlvk4rV4gYzEt97rI21ku\nu+wJNFZMy9u9C3gEOJpWmct5KnuH5HL9EHB4sS/L/bGN9B/ky2JfLSId59yD48DIw9NJZTf3nBmq\nLOeXkaYsW3n9Wd7mfK5Ue6o8Riqvj5N64Dwc6zycVvnN8zjpfMz5JbZjEbC0kj73XKr2bHk09uUz\naJWbIVoxJPfEWUQr5uRpC2N4NKYfVGxftXdO3t4tpGO2OJb7JLAduC/y8TzgmZU8lrEpn583Ap9x\n96vo0iBvqN4KvJD+uz+KiEhyu7uf2k3CQba5L0GBXURkJv1GtwkHGdy3D3DZIiIyCT1+QESkhgYZ\n3O8a4LJFRGQSgwzutwB/Q3rt3lh8dlXSVO8yl/JzPya745t7Y7QbXy5/vDK+U/p+VJ9HMlPr8Mrf\ncrjsLdPNuqr7pdP+7XbcVHIPh251Op5PFOOry2u3X6bSab+1W8ZEh2nVslsOj9PeZOW93FfV45uf\nSZKn5Z5FY23WX03fLg8TRZpcBmbqmFfXM5V2290pXZnGSb1SZirv7ZbT7vsYe+tmXb2eP6O09uO5\nwIu6nXmQvWUejUwdjm6siki9VR9u1m+6dn4FPNfdu7qfOcia+7+hHjMisn/oNs71Ew+fSfr/ha7o\nhqqIyL6j62d2KbiLiNTQIIP7eQNctoiITGLQz5Z5kr3fciMiItP3M+AIdz++m8SDbpZRYBeR+WRw\ntdneTCc/p7J3d/KOBl1zn287UkRknzYf3sQkIiJzRMFdRKSGZiu4q3lGRGQWDTq4v5rWG3y6ec6E\niIh01vUN1YEGd3e/ntbrwn4NrGbPB0ONkF47dROth0blhwFB54cP5deQPVmkyfOPkl7Z9TiTP9zp\nico4r3zablKnbZ0iTb8PYOo1L5Mtc7oP3Jpq3/Sbp+mabBuqD6LqJV8zfRz7MdV+l8GZT/v9EeA/\nd5t4oL1l4KkeM/ldgCIi0rsJd+/qEQQDDbhmtm421iMisp/oOpYOOug+c+okIiIy0wYd3K/uML7T\niwRERGQGDPqG6oXA14D/AjwH2Ab8AfA5YCPpZsWdtHrTrAVuI918HSfdMC3f4DQWaR+KafcX0xx4\nlHQjdRvwdmBnpAf4OemGxGNFuvWkG7A7aP+Glac2hdZN2/INMLtIN2Z30bpg5fz8upK3PN94TGv3\nlpf8hp3qzcHRyHO78fmtOtlYJe1EkebvSPstz78r0v5b5C3fqH4ihqd6O031hnd5s3yy3lFTTWv3\n1qEJ0rHL8z4Reb45tunRYhm5PD3O3m/HKt9EtLsYl/fnOLCVdIyqb9cqe32VbzGCvfdTPo47i+VQ\nGYbWPhtnz7dNlct/tBiXl7GhmC8fs4lIW769J789yIEHY/jJYjk537mDAcV81eGJyvdcnqFVdnK5\nandTu9NN6nLftJve7s1U1fxV8z+T8rZ10ukcGZ8kTfWNY+U2jhXjcqyZAD4A/HUX+QVm4YbqHisz\n+yvgY+6+I75/F7gE+B3SG0ZeGeMvJAWi3wU+7e4nmdlVwFXuvirSfBy4htQTZxXwDmAdsBz4v8Cb\nY7lXA4e4+2oze527f8vM3gH8mbsfb2afJ70K8DeB5wGHAAtJO/Qg4HpgJXAw6cE9m0kXlTcDi0kH\nYgT4OulCdSzwP4CT3f3xyOeTwJHAl4C3xd8TgE+SLjqfAv4UOBH4ZWzT0thtj5IukN8ADgMawEnA\n80kn61GkJ3AOAycDB5IuYk+P7cg3s3NPonHgHuCIyNOi2NaHSCflu4DtwKHA/wJ+I+YZJxW0I2Ke\nfIItiP00WixrM+nCeRCwLPKRp38POA54ecxrxbQsB8AdwCuBT8f2HhCfRcC/AH/s7o+Y2TlxfD9L\nago8Ffh3tC66C2N78r9tGyno3h/HbS3wPHf/VFEefuru38wZMrObSWVgJalsfsndbzSzG4CXAP8b\nuBL4LHB0HJcjgTXAHwMXARfGuOybMc8zY1tHSefD/cA/uPvfmNnbSeXsJaTXVh4f++R1pLI+Htvz\nGVK5/0vgjFh/voB9D7iUVE4bpLL0VdJFfRmtl+ocHuuqypWsQyvjJ0jl9YFYxyGkitU20vmRt/WQ\nWP4C0nE7BnhFsQxiWvn6w3xxHIo85Yvbl4H7SMfv5bGvT41xB0ReT4xl3kUqa4ew53PQR2N9Q+zd\n2SO/KakcP1r8zcs8gs7PVs8Vpf9OOp+2xXwvI51bK2I9Jxfb+QTpfL4z9s9JwDuBT8R2bXP3ZR3W\n1567z4sPcGHl+6p24zvMu6oY/hNSgVtNOuDn5GUAt5KC4PeATcBHSMH7A6TumO9vlyfSSbkqPh3z\nA/xjHMSf5HUX07bHen9crPdfIv37q9sa67yw0zJj2hbSxWE1qUA9RiqAG0mF8x9ju64E7qvk9e/b\n5P8q4N7I4zWk4JiDeH5n573s+etpdUz7NPA/Y1/eB1xQXU8cm7tinodinptj2n0x/6mR71/GuCdI\nNem833bE8E3AldMoIzcV6/9JrP9TUR7ujfW0LQ+V+bbHcM7PLtIvhB3AjaST8wPAd4Br4zhdUDmu\nB8d2DpNenfZjUiXk+ph/JykgfKRY3j2kk/+pcl0s8774u5pUxh6J478r/j4S+Vsdx/IuUvm/OtJs\nLvJ/fbl/gZti2bfEPsu/aB4nlbfcjfnRKA83kspk+YslvwP3sVjvlbGsd8axfoI9z9dbYtyTkb8H\ngS8Cv4j1PxR5zscg76ObgPfTOm/yMvLFIq+//BWet+UJWpUYj3Fjse+2x7J/kct0fK6iFUuujnxs\nonUe/rKyPI/9fBPpAuuxr3aRysE34vvumP9G0i+0z8Qxu3JaMXWug3pRQDdXvo+1G99h3rFi+HZg\nSwwvB35EuupBCu63k2pyW2JHHx7TDgLWtcsTe77ku2N+4oAcGunzut8Z0yZivQfn9Ua6LXm95bJj\n2uZOy4zvO4FDI30uqNti+esj7XuisOzutM8q+d8ceRyPgrWe9FLeCdLJNRaFdgJ4oFj37lj35hje\n3OHY5Pz+ItJ5fC/nGY39Mkbrfx/eE9t1W4w/KG9Tl2XE8/ojj3n9v4zvP+1UHirz3VXs1x1xDMZI\ntdRxWmXvoLxN1eNa2R9jsb/z+4bztu+O5efl3QHcUSnXuWztLvbbQloB7XDgtBgeL8rhj+JYTUT+\nFxb5P7zcv8Xx8SgLuUw6KaDuiOF1pBr8OK3mzjvj+w7SxSzv73K5o8Vw3q4ceHeSmnN30/r14DHP\nOlrnUrnP11XyuB44hT2bc/K54rFfH6iMc9KF3Iv5DgJ25jJNq2zmWDIRf734jJICd87/r2J/XEXr\nf3RuL7Yx5/dWWi8wz9u2hco5PNVntptl1lVGPT9Pir9lZoy04c9198XxbPjS4kraXZXxd5B+2kD6\nWfMw6ecZpFrQc0k7Mv+0OonuHlGcC8RJbaYdEPlYHGkWkH565Z93eZrR+vnXjTwf05in3fLL9efv\npQNo/fR8Pq026btJzTPTNZ1tnK5dpPyWZaRsF+32ZcW5zOWyWf7UPpFUVk5iz+PW7fKr65ruPE8C\nQ0X5ry6jLK95f+wqxlePd55/FzP/OO5yvwzyuLdbX909df67e/f3SWe5dj5CqgUuj88EqQ16O6km\n8YbiMx5p8k+hcVLb8mnxeQS4oEj7IHAW6Yr7cMybx42Q2q0nSFfFk0k1twVFnh6mdfXeCVxG+nn1\ncdJPzvHi86Ji2acVH4/tGSm2MbdlO+lxDI+Qfja+gdbP1nHg3TH8buAtkZ8JUrD9w1hmvqm4g9aV\n/sORPgeq7cC3imV/lNZPw7zf8vacVvnkn5IjwA9JteTbSPc+nNaN4PJzL62fnY8W2/pgrCun+zyt\nm3mXFfkra1Q7Yn+XTUFOKhvbSbW6vL/OjH2wnFYZcVLb9hsi/Y7I12di2j2xvx6O/I2Qauz5ZmRZ\nHl5AunAsJzUT5OaNO2i1KTeL7XuMVGN8ONb9gcjfg7TKRS7nOW+5PD0W494Sx+ChmPeEYtt2k9rI\ny3L3CKn5Jx9bJ/0H445Y3oOxvo20fhHkcui0ao8OXBF/17SZ9hfFccg3/LZFvn8W0zaSaqZjpF98\ned5fRbpcE3+wOK7lMS6PU/n5BK0OEOV/ouc85HPiLbTK1ERlGT+urLOXT/4lsLPN+J2TTMvnzXnF\n+L8s8rilOP5l/sdJ59aDcWx/xTRr7guZXVeTfpbfCmBm98QGXgkc7+7/nBOa2RZ332RmN8aoe4Gn\nu/tXYvpq4B53v9nMtgDXkQr12cCnYt41Me564L8B/wH4kbs/aWY3uvuEmV1N+sn9bVKht1jXF4HF\n7v4eMzsW+PeRj3xjaQ2ww92/U+T5ftIJfr27b4pxXyfdHF3q7tdFvu+OfP+c9NPybuD7pPa278e0\nVwIvJV2Evg38NvBi0om/jVTz+hzwXXffGjduHyedZGfFtq4iBZt/BRa5+3di/UuBF5d5j7xeSevi\ndnXkB9KJvoXU9n0MqW003+T6U+AG0gl9d6Q/KvK5k1So8zK/RrrR+0VSgT2QdEPsB6Ta8W2xv1cA\nryIFtNyb537STUEntaf+CLi2UkYeA57p7peY2Vm0bkp/lXST+73A/wOeXWz2BbHcz1bKw53A78Xy\nzyUF8m2R79zb63TSTdGVcfz+gXTR2UmqRFwb+f+PFOU8tmUp6WLyoljG8e7+eTNrkH7qH+TuPy+2\nLbfJv4god3EsLwU+GN9/Hvv91aT2/K2xnn8Cfh94LfC2KIc7aAXkV8Z895HK2umkC9oLgcPc/cPR\nGeJrMe0+0kVwZ6z/GuC/xjoXkIL0F0hl+yvA04BzSWX5zcDFpJu+o6QmkWNIF5evxvjnxDa/y9NN\n7mfFus6K8bfE38Wki/BI7Lt3k873Z5B+gS0hlb0fksrkEaTz1yJv+cb6RHzP+28FKdieSCqD62Jf\nPT/y+zraaF9sAAAAYElEQVRagfqu2M8nxPwrY95DSDHiFcBj7v4VM/swrYrTGaTz8uOxn4+PZWyM\n+S4m3X94Xxzbm0lNQ12b1WYZERGZHXosgIhIDSm4i4jUkIK7iEgNKbiLiNSQgruISA39fyaXa5SD\nJtykAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc49e680fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "trdata['MGR_ID'].value_counts().plot(kind='bar')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from numpy import array\n",
    "from sklearn import metrics, linear_model\n",
    "from sklearn.cross_validation import KFold\n",
    "from scipy import sparse\n",
    "from itertools import combinations\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import sys\n",
    "import random\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def dict_encode(encoding, value):\n",
    "    if not value in encoding:\n",
    "        encoding[value] = {'code': len(encoding)+1, 'count': 0}\n",
    "    enc = encoding[value]\n",
    "    enc['count'] += 1\n",
    "    encoding[value] = enc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def dict_decode(encoding, value, min_occurs):\n",
    "    enc = encoding[value]\n",
    "    if enc['count'] < min_occurs:\n",
    "        return -1\n",
    "    else:\n",
    "        return enc['code']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def group_data(data, degree, min_occurs):\n",
    "    \"\"\" \n",
    "    numpy.array -> numpy.array\n",
    "    \n",
    "    Groups all columns of data into all combinations of degree\n",
    "    \"\"\"\n",
    "    m, n = data.shape\n",
    "    encoding = dict()\n",
    "    for indexes in combinations(range(n), degree):\n",
    "        for v in data[:, indexes]:\n",
    "            dict_encode(encoding, tuple(v))\n",
    "    new_data = []\n",
    "    for indexes in combinations(range(n), degree):\n",
    "        new_data.append([dict_decode(encoding, tuple(v), min_occurs) for v in data[:, indexes]])\n",
    "    return array(new_data).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def one_hot_encoder(data, keymap=None):\n",
    "    \"\"\"\n",
    "     one_hot_encoder takes data matrix with categorical columns and\n",
    "     converts it to a sparse binary matrix.\n",
    "     \n",
    "     Returns sparse binary matrix and keymap mapping categories to indicies.\n",
    "     If a keymap is supplied on input it will be used instead of creating one\n",
    "     and any categories appearing in the data that are not in the keymap are\n",
    "     ignored\n",
    "     \"\"\"\n",
    "    if keymap is None:\n",
    "        keymap = []\n",
    "        for col in data.T:\n",
    "            uniques = set(list(col))\n",
    "            keymap.append(dict((key, i) for i, key in enumerate(uniques)))\n",
    "    total_pts = data.shape[0]\n",
    "    outdat = []\n",
    "    for i, col in enumerate(data.T):\n",
    "        km = keymap[i]\n",
    "        num_labels = len(km)\n",
    "        spmat = sparse.lil_matrix((total_pts, num_labels))\n",
    "        for j, val in enumerate(col):\n",
    "            if val in km:\n",
    "                spmat[j, km[val]] = 1\n",
    "        outdat.append(spmat)\n",
    "    outdat = sparse.hstack(outdat).tocsr()\n",
    "    return outdat, keymap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_test_submission(filename, prediction):\n",
    "    content = ['id,ACTION']\n",
    "    for i, p in enumerate(prediction):\n",
    "        content.append('%i,%f' % (i + 1, p))\n",
    "    f = open(filename, 'w')\n",
    "    f.write('\\n'.join(content))\n",
    "    f.close()\n",
    "    print('Saved')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def cv_loop(X, y, model, N, seed):\n",
    "    mean_auc = 0.\n",
    "    k_fold = KFold(len(y), N, indices=True, shuffle=True,\n",
    "                   random_state=seed)\n",
    "    for train_ix, test_ix in k_fold:\n",
    "        model.fit(X[train_ix], y[train_ix])\n",
    "        preds = model.predict_proba(X[test_ix])[:, 1]\n",
    "        auc = metrics.auc_score(y[test_ix], preds)\n",
    "        #print(\"AUC (fold %d/%d): %f\" % (i + 1, N, auc))\n",
    "        mean_auc += auc\n",
    "    return mean_auc / N"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'submit': 'lr/data.lr.11.1.test.pred.csv', 'train': '-f', 'good_features': None, 'test': '/home/ubuntu/.local/share/jupyter/runtime/kernel-404c5bd7-dfd1-418d-a845-1a214ec43e49.json', 'seed': 123, 'min_occurs': 3}\n",
      "Reading train dataset...\n",
      "Reading test dataset...\n",
      "Transforming data (32769 instances)..."
     ]
    }
   ],
   "source": [
    "def main(train, test, submit, seed, min_occurs, good_features):\n",
    "\n",
    "    start_time = time.time()\n",
    "    print(\"Reading train dataset...\")\n",
    "    train_data = pd.read_csv(\"/root/hackerday/12_employee_access_need/train.csv\")\n",
    "    #print(train_data.head())\n",
    "\n",
    "    print(\"Reading test dataset...\")\n",
    "    test_data = pd.read_csv(\"/root/hackerday/12_employee_access_need/test.csv\")\n",
    "    #print(test_data.head())\n",
    "\n",
    "    all_data = np.vstack((train_data.ix[:, 1:], test_data.ix[:, 1:]))\n",
    "\n",
    "    num_train = np.shape(train_data)[0]\n",
    "    # Transform data\n",
    "    print(\"Transforming data (%i instances)...\" % num_train)\n",
    "    d_2 = group_data(all_data, degree=2, min_occurs=min_occurs)\n",
    "    d_3 = group_data(all_data, degree=3, min_occurs=min_occurs)\n",
    "    d_4 = group_data(all_data, degree=4, min_occurs=min_occurs)\n",
    "    d_5 = group_data(all_data, degree=5, min_occurs=min_occurs)\n",
    "    d_6 = group_data(all_data, degree=6, min_occurs=min_occurs)\n",
    "    d_7 = group_data(all_data, degree=7, min_occurs=min_occurs)\n",
    "\n",
    "    y = array(train_data.ACTION)\n",
    "    #X_train_all = np.hstack((all_data[:num_train], d_2[:num_train], d_3[:num_train],\n",
    "    #                         d_4[:num_train], d_7[:num_train]))\n",
    "    #X_test_all = np.hstack((all_data[num_train:], d_2[num_train:], d_3[num_train:],\n",
    "    #                        d_4[num_train:], d_7[num_train:]))\n",
    "    X_train_all = np.hstack((all_data[:num_train], d_2[:num_train], d_3[:num_train], d_4[:num_train],\n",
    "                             d_5[:num_train], d_6[:num_train], d_7[:num_train]))\n",
    "    X_test_all = np.hstack((all_data[num_train:], d_2[num_train:], d_3[num_train:], d_4[num_train:],\n",
    "                            d_5[num_train:], d_6[num_train:], d_7[num_train:]))\n",
    "    num_features = X_train_all.shape[1]\n",
    "    print(\"Total number of categorical features %i\" % num_features)\n",
    "\n",
    "    rnd = random.Random()\n",
    "    rnd.seed(seed*num_features)\n",
    "\n",
    "    model = linear_model.LogisticRegression()\n",
    "    model.C = 0.5 + rnd.random()*3.5\n",
    "    print(\"Logistic C parameter: %f\" % model.C)\n",
    "\n",
    "    # Xts holds one hot encodings for each individual feature in memory\n",
    "    # speeding up feature selection \n",
    "    Xts = [one_hot_encoder(X_train_all[:, [i]])[0] for i in range(num_features)]\n",
    "\n",
    "    print(\"Performing aproximate greedy feature selection...\")\n",
    "    N = 10\n",
    "    if good_features is None:\n",
    "        score_hist = []\n",
    "        good_features = set([])\n",
    "\n",
    "        # Feature selection loop\n",
    "        f_remain = range(len(Xts))\n",
    "        cur_best_score = -1\n",
    "        cur_best_score_thres = 1.0\n",
    "        while len(score_hist) < 2 or score_hist[-1][0] > score_hist[-2][0]:\n",
    "            scores = []\n",
    "            f_shuff = list(f_remain)\n",
    "            rnd.shuffle(f_shuff)\n",
    "            n_thres = 0.3679*len(f_remain)\n",
    "            i = 0\n",
    "            iter_best_score = -1\n",
    "            for f in f_shuff:\n",
    "\n",
    "                i += 1\n",
    "                feats = list(good_features) + [f]\n",
    "                Xt = sparse.hstack([Xts[j] for j in feats]).tocsr()\n",
    "                score = cv_loop(Xt, y, model, N, seed)\n",
    "                if score < (cur_best_score*cur_best_score_thres):\n",
    "                    f_remain.remove(f)\n",
    "                    print(\"Discarded: %i (AUC = %f) \" % (f, score))\n",
    "                else:\n",
    "                    scores.append((score, f))\n",
    "                    if score > iter_best_score:\n",
    "                        iter_best_score = score\n",
    "                        if i > n_thres and iter_best_score > cur_best_score:\n",
    "                            print(\"Early stop on iter %i\" % i)\n",
    "                            break\n",
    "            if len(scores) > 0:\n",
    "                best_score = sorted(scores)[-1]\n",
    "                f_remain.remove(best_score[1])\n",
    "                if best_score[0] > cur_best_score:\n",
    "                    good_features.add(best_score[1])\n",
    "                    score_hist.append(best_score)\n",
    "                    cur_best_score = best_score[0]\n",
    "                print(\"Current features: %s (AUC = %f, remain = %i) \" %\n",
    "                      (list(good_features), best_score[0], len(f_remain)))\n",
    "            else:\n",
    "                break\n",
    "\n",
    "    good_features = sorted(list(good_features))\n",
    "    print(\"Selected features %s\" % good_features)\n",
    "\n",
    "    print(\"Performing hyperparameter selection...\")\n",
    "    # Hyperparameter selection loop\n",
    "    Xt = sparse.hstack([Xts[j] for j in good_features]).tocsr()\n",
    "    score_hist = []\n",
    "    score = cv_loop(Xt, y, model, N, seed)\n",
    "    score_hist.append((score, model.C))\n",
    "    \n",
    "    Cvals = np.logspace(-3, 4, 20, base=2)\n",
    "    for C in Cvals:\n",
    "        model.C = C\n",
    "        score = cv_loop(Xt, y, model, N, seed)\n",
    "        score_hist.append((score, C))\n",
    "        print(\"C: %f Mean AUC: %f\" % (C, score))\n",
    "    model.C = sorted(score_hist)[-1][1]\n",
    "    score = sorted(score_hist)[-1][0]\n",
    "    print(\"Best (C, AUC): (%f, %f)\" % (model.C, score))\n",
    "\n",
    "    print(\"Performing One Hot Encoding on entire dataset...\")\n",
    "    Xt = np.vstack((X_train_all[:, good_features], X_test_all[:, good_features]))\n",
    "    Xt, keymap = one_hot_encoder(Xt)\n",
    "    X_train = Xt[:num_train]\n",
    "    X_test = Xt[num_train:]\n",
    "\n",
    "    print(\"Training full model...\")\n",
    "    model.fit(X_train, y)\n",
    "\n",
    "    print(\"Making prediction and saving results...\")\n",
    "    preds = model.predict_proba(X_test)[:, 1]\n",
    "    create_test_submission(submit, preds)\n",
    "    print(\"Total time %f minutes\" % ((time.time() - start_time)/60.0))\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "\n",
    "    name_def = '11.1'\n",
    "    args = {'train': 'lr/data.lr.' + name_def + '.tr.csv',\n",
    "            'test': 'lr/data.lr.' + name_def + '.test.csv',\n",
    "            'submit': 'lr/data.lr.' + name_def + '.test.pred.csv',\n",
    "            'seed': 123,\n",
    "            'min_occurs': 3,\n",
    "            'good_features': []}\n",
    "    if len(sys.argv) >= 2:\n",
    "        args['train'] = sys.argv[1]\n",
    "\n",
    "    if len(sys.argv) >= 3:\n",
    "        args['test'] = sys.argv[2]\n",
    "\n",
    "    if len(sys.argv) >= 4:\n",
    "        args['submit'] = sys.argv[3]\n",
    "\n",
    "    if len(sys.argv) >= 5:\n",
    "        args['seed'] = int(sys.argv[4])\n",
    "\n",
    "    if len(sys.argv) >= 6:\n",
    "        args['min_occurs'] = int(sys.argv[5])\n",
    "\n",
    "    if len(sys.argv) >= 7:\n",
    "        args['good_features'] = [int(val) for val in sys.argv[6].split(',')]\n",
    "\n",
    "    if len(args['good_features']) == 0:\n",
    "        args['good_features'] = None\n",
    "\n",
    "    print(args)\n",
    "    main(**args)\n",
    "                            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:Anaconda3]",
   "language": "python",
   "name": "conda-env-Anaconda3-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}