Exploratory Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Issues with the Aftershocks study  \n",
    "\n",
    "This notebook shows three items:\n",
    "1. Replicating the results in the paper\n",
    "2. Replicating the results in the paper, but showing the results on both test and train.  Puzzingly, the scores for the test set are higher than the train set.\n",
    "3. Replicating similar results using only 1500 rows of data with 2 epochs (The original paper used 4.7 million rows of data).\n",
    "4. One source of potential leakage in how test/train is constructed\n",
    "\n",
    "Everything is in this notebook, except you must download the data and place it in an *adjoining* folder named Data (e.g., the line in python in TrainModel.py is:   `trainFile = '../Data/Training.h5`) the files are available [here](https://drive.google.com/drive/folders/1lAHfdjFd-Uv3wJcA0Tk2ViDIZeFt0mCA). The only files needed are the training and testing files. The authors have already split the data into training and a test file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Replicating the reults"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "WARNING:tensorflow:From /Users/rajiv.shah/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1349: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "keep_dims is deprecated, use keepdims instead\n",
      "Epoch 1/5\n",
      "2019-04-30 12:51:43.051357: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n",
      "Epoch 00001: val_loss improved from inf to 0.60753, saving model to ../Data/TheBestWeights.h5\n",
      " - 2s - loss: 0.6931 - binary_accuracy: 0.5973 - val_loss: 0.6075 - val_binary_accuracy: 0.7145\n",
      "Epoch 2/5\n",
      "Epoch 00002: val_loss did not improve\n",
      " - 0s - loss: 0.6426 - binary_accuracy: 0.6657 - val_loss: 0.6109 - val_binary_accuracy: 0.7189\n",
      "Epoch 3/5\n",
      "Epoch 00003: val_loss improved from 0.60753 to 0.60744, saving model to ../Data/TheBestWeights.h5\n",
      " - 1s - loss: 0.6273 - binary_accuracy: 0.6823 - val_loss: 0.6074 - val_binary_accuracy: 0.7206\n",
      "Epoch 4/5\n",
      "Epoch 00004: val_loss improved from 0.60744 to 0.60259, saving model to ../Data/TheBestWeights.h5\n",
      " - 1s - loss: 0.6205 - binary_accuracy: 0.6902 - val_loss: 0.6026 - val_binary_accuracy: 0.7218\n",
      "Epoch 5/5\n",
      "Epoch 00005: val_loss improved from 0.60259 to 0.60203, saving model to ../Data/TheBestWeights.h5\n",
      " - 1s - loss: 0.6134 - binary_accuracy: 0.6971 - val_loss: 0.6020 - val_binary_accuracy: 0.7224\n",
      "Figure(640x480)\n",
      "Using TensorFlow backend.\n",
      "WARNING:tensorflow:From /Users/rajiv.shah/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1349: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "keep_dims is deprecated, use keepdims instead\n",
      "2019-04-30 12:51:50.854208: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n",
      "merged AUC on testing data set: 0.8456169271512985\n"
     ]
    }
   ],
   "source": [
    "!python TrainModel.py\n",
    "!python EvalModel.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This should end after 5 Epochs with and AUC of 0.849607780685832 on the testing dataset.  This score is based on validating the external testing dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Test and Train Scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "WARNING:tensorflow:From /Users/rajiv.shah/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1349: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "keep_dims is deprecated, use keepdims instead\n",
      "2019-04-14 17:02:44.269431: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n",
      "merged AUC on testing data set: 0.849607780685832\n",
      "length of testing dataset:  1378120\n",
      "merged AUC on training data set: 0.7594855621142795\n",
      "length of training dataset:  4743090\n"
     ]
    }
   ],
   "source": [
    "##I modified the Eval File to add the training results, now lets look at that again\n",
    "!python EvalModel2.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Hmm, the training dataset has a lower score than the test set.***  This should set off some alarm bells and is worthy of further investgation and explanation.  Typically, most machine learning algorithms will do better on the training set and poorer when generalizing to a unseen test set.  The fact that this generalized with improved performance suggests some sort of target leakage where the model knows something about the data in the test set.  Again, for experienced data scientists, this result is a concern.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Training with 1500 Rows of data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the TrainModel.py - I changed the batch size to 300 and limited it to 5 steps - Thus 1500 rows over 2 epoaches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training file:,  ../Data/Training.h5\n",
      "WARNING:tensorflow:From /Users/rajiv.shah/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1349: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "keep_dims is deprecated, use keepdims instead\n",
      "Epoch 1/2\n",
      "Epoch 00001: val_loss improved from inf to 0.68046, saving model to ../Data/TheBestWeights.h5\n",
      " - 1s - loss: 0.7118 - binary_accuracy: 0.5427 - val_loss: 0.6805 - val_binary_accuracy: 0.6784\n",
      "Epoch 2/2\n",
      "Epoch 00002: val_loss improved from 0.68046 to 0.66744, saving model to ../Data/TheBestWeights.h5\n",
      " - 0s - loss: 0.7161 - binary_accuracy: 0.5740 - val_loss: 0.6674 - val_binary_accuracy: 0.7195\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEWCAYAAACKSkfIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XuYFOWZ9/HvzXAYEOTsqgwKEiIM\ngooTTF4gHDyhRlGDLKeoJIbEN8YYl6ysGpcQyRLDKkGJkY1oVGDWV6JiIpCDKBgVGHyRwyBCEHUA\ndRwBRXB14N4/qmamZ5hDD3RPT3f9PtfV13RXPV39VDf8qvq5u6rM3RERkWhokuoOiIhIw1Hoi4hE\niEJfRCRCFPoiIhGi0BcRiRCFvohIhCj0pV7MLMvM9pvZKYlsm0pm9iUzS/hvl83sfDPbEfN4i5kN\njqftUbzW78zstqN9fi3LvcvMHkn0ciV1mqa6A5JcZrY/5mEr4H+AQ+Hj77n7/Posz90PAa0T3TYK\n3P30RCzHzK4HJrj70JhlX5+IZUvmU+hnOHcvD91wT/J6d/9rTe3NrKm7lzZE30Sk4Wl4J+LCr+//\nbWYLzewTYIKZfc3MXjWzvWa228xmm1mzsH1TM3Mz6xY+fjycv8TMPjGzV8yse33bhvMvNrM3zWyf\nmd1nZn83s+tq6Hc8ffyemW0zsz1mNjvmuVlmdq+ZlZjZdmBELe/P7WaWX2XaHDO7J7x/vZltDtfn\nH+FeeE3LKjKzoeH9Vmb2WNi3TcA5VdreYWbbw+VuMrPLw+l9gfuBweHQ2Ycx7+3UmOd/P1z3EjN7\n2sxOiue9qYuZXRn2Z6+ZPW9mp8fMu83MdpnZx2b2Rsy6ftXMXgunv29mv4r39SQJ3F23iNyAHcD5\nVabdBXwOXEawE9AS+ApwLsE3wdOAN4Ebw/ZNAQe6hY8fBz4E8oBmwH8Djx9F2xOAT4CR4bxbgC+A\n62pYl3j6+AzQFugGfFS27sCNwCYgB+gIrAj+K1T7OqcB+4HjYpb9AZAXPr4sbGPAcOAg0C+cdz6w\nI2ZZRcDQ8P5M4AWgPXAqUFil7WjgpPAzGRf24Z/CedcDL1Tp5+PA1PD+hWEfzwKygd8Az8fz3lSz\n/ncBj4T3e4f9GB5+RrcBW8L7fYC3gRPDtt2B08L7a4Cx4f02wLmp/r8Q5Zv29AXgJXd/1t0Pu/tB\nd1/j7qvcvdTdtwNzgSG1PP9Jdy9w9y+A+QRhU9+23wDWufsz4bx7CTYQ1Yqzj//h7vvcfQdBwJa9\n1mjgXncvcvcSYEYtr7Md2EiwMQK4ANjj7gXh/GfdfbsHngf+BlRbrK1iNHCXu+9x97cJ9t5jX/cJ\nd98dfiYLCDbYeXEsF2A88Dt3X+funwFTgCFmlhPTpqb3pjZjgMXu/nz4Gc0g2HCcC5QSbGD6hEOE\nb4XvHQQb755m1tHdP3H3VXGuhySBQl8A3o19YGa9zOxPZvaemX0MTAM61fL892LuH6D24m1NbU+O\n7Ye7O8GecbXi7GNcr0Wwh1qbBcDY8P648HFZP75hZqvM7CMz20uwl13be1XmpNr6YGbXmdnr4TDK\nXqBXnMuFYP3Kl+fuHwN7gC4xberzmdW03MMEn1EXd98C/AvB5/BBOFx4Yth0IpALbDGz1WZ2SZzr\nIUmg0BcIvu7HepBg7/ZL7n48cCfB8EUy7SYYbgHAzIzKIVXVsfRxN9A15nFdPyl9AjjfzLoQ7PEv\nCPvYEngS+A+CoZd2wJ/j7Md7NfXBzE4DHgBuADqGy30jZrl1/bx0F8GQUdny2hAMI+2Mo1/1WW4T\ngs9sJ4C7P+7uAwmGdrII3hfcfYu7jyEYwvtPYJGZZR9jX+QoKfSlOm2AfcCnZtYb+F4DvOYfgf5m\ndpmZNQV+BHROUh+fAG42sy5m1hG4tbbG7v4e8BLwCLDF3beGs1oAzYFi4JCZfQM4rx59uM3M2llw\nHMONMfNaEwR7McH277sEe/pl3gdyygrX1VgIfMfM+plZC4LwXenuNX5zqkefLzezoeFr/4SgDrPK\nzHqb2bDw9Q6Gt8MEK/AtM+sUfjPYF67b4WPsixwlhb5U51+Aawn+Qz9IUHBNKnd/H/hn4B6gBOgB\n/H+C4woS3ccHCMbeNxAUGZ+M4zkLCAqz5UM77r4X+DHwFEExdBTBxise/07wjWMHsAR4NGa564H7\ngNVhm9OB2HHwvwBbgffNLHaYpuz5SwmGWZ4Kn38KwTj/MXH3TQTv+QMEG6QRwOXh+H4L4G6COsx7\nBN8sbg+fegmw2YJfh80E/tndPz/W/sjRsWDoVKRxMbMsguGEUe6+MtX9EckU2tOXRsPMRoTDHS2A\nnxL86mN1irslklEU+tKYDAK2EwwdXARc6e41De+IyFHQ8I6ISIRoT19EJEIa3QnXOnXq5N26dUt1\nN0RE0sratWs/dPfafuYMNMLQ79atGwUFBanuhohIWjGzuo4sBzS8IyISKQp9EZEIUeiLiESIQl9E\nJEIU+iIiEaLQFxFJsfnzoVs3aNIk+Dt/fvJeq9H9ZFNEJErmz4dJk+DAgeDx228HjwHGH/O5UY+k\nPX0RkRS6/faKwC9z4EAwPRkU+iIiKfTOO/WbfqwU+iIiKXRKDRfrrGn6sVLoi4ik0PTp0KpV5Wmt\nWgXTk0GhLyKSQuPHw9y5cOqpYBb8nTs3OUVc0K93RERSbvz45IV8VdrTFxGJEIW+iEiEKPRFRCJE\noS8iEiEKfRGRCFHoi4hEiEJfRCRCFPoiIhGi0BcRiRCFvohIhGRM6DfklWdERNJVRpx7p6GvPCMi\nkq4yYk+/oa88IyKSrjIi9Bv6yjMiIukqI0K/oa88IyKSrjIi9Bv6yjMiIukqI0K/oa88IyKSruIK\nfTMbYWZbzGybmU2pZv69ZrYuvL1pZntj5i01s71m9sdEdryq8eNhxw44fDj4q8AXETlSnT/ZNLMs\nYA5wAVAErDGzxe5eWNbG3X8c0/6HwNkxi/gV0Ar4XqI6LSIiRyeePf0BwDZ33+7unwP5wMha2o8F\nFpY9cPe/AZ8cUy9FRCQh4gn9LsC7MY+LwmlHMLNTge7A8/XphJlNMrMCMysoLi6uz1NFRKQeEl3I\nHQM86e6H6vMkd5/r7nnunte5c+cEd0lERMrEE/o7ga4xj3PCadUZQ8zQjoiINC7xhP4aoKeZdTez\n5gTBvrhqIzPrBbQHXklsF0VEJFHqDH13LwVuBJYBm4En3H2TmU0zs8tjmo4B8t3dY59vZiuB/wec\nZ2ZFZnZR4rovIiL1YVUyOuXy8vK8oKAg1d0QEUkrZrbW3fPqapcRR+SKiEh8FPoiIhGi0BcRiRCF\nvohIhCj0RUQiRKEvIhIhCn0RkQhR6IuIRIhCX0QkQhT6IiIRotAXEYkQhb6ISIQo9EVEIkShLyIS\nIQp9EZEIUeiLiESIQl9EJEIU+iIiEaLQFxGJEIW+iEiEKPRFRCJEoS8iEiEKfRGRCFHoi4hEiEJf\nRCRCFPoiIhGi0BcRiRCFvohIhCj0RUQiRKEvIhIhCn0RkQiJK/TNbISZbTGzbWY2pZr595rZuvD2\nppntjZl3rZltDW/XJrLzIiJSP03ramBmWcAc4AKgCFhjZovdvbCsjbv/OKb9D4Gzw/sdgH8H8gAH\n1obP3ZPQtRARkbjEs6c/ANjm7tvd/XMgHxhZS/uxwMLw/kXAX9z9ozDo/wKMOJYOi4jI0Ysn9LsA\n78Y8LgqnHcHMTgW6A8/X97kiIpJ8iS7kjgGedPdD9XmSmU0yswIzKyguLk5wl0REpEw8ob8T6Brz\nOCecVp0xVAztxP1cd5/r7nnunte5c+c4uiQiIkcjntBfA/Q0s+5m1pwg2BdXbWRmvYD2wCsxk5cB\nF5pZezNrD1wYThMRkRSo89c77l5qZjcShHUWMM/dN5nZNKDA3cs2AGOAfHf3mOd+ZGY/J9hwAExz\n948SuwoiIhIvi8noRiEvL88LCgpS3Q0RkbRiZmvdPa+udjoiV0QkQhT6IiIRotAXEYkQhb6ISIQo\n9EVEIkShLyISIQp9EZEIUeiLiESIQl9EJEIU+iIiEaLQFxGJEIW+iEiEKPRFRCJEoS8iEiEKfRGR\nCFHoi4hEiEJfRCRCFPoiIhGi0BcRiRCFvohIhCj0RUQiRKEvIhIhCn0RkQhR6IuIRIhCX0QkQhT6\nIiIRotAXEYkQhb6ISIQo9EVEIkShLyISIQp9EZEIUeiLiERIXKFvZiPMbIuZbTOzKTW0GW1mhWa2\nycwWxEz/pZltDG//nKiOi4hI/TWtq4GZZQFzgAuAImCNmS1298KYNj2BfwMGuvseMzshnH4p0B84\nC2gBvGBmS9z948SvioiI1CWePf0BwDZ33+7unwP5wMgqbb4LzHH3PQDu/kE4PRdY4e6l7v4psB4Y\nkZiui4hIfcUT+l2Ad2MeF4XTYn0Z+LKZ/d3MXjWzsmB/HRhhZq3MrBMwDOha9QXMbJKZFZhZQXFx\ncf3XQkRE4lLn8E49ltMTGArkACvMrK+7/9nMvgK8DBQDrwCHqj7Z3ecCcwHy8vI8QX0SEZEq4tnT\n30nlvfOccFqsImCxu3/h7m8BbxJsBHD36e5+lrtfAFg4T0REUiCe0F8D9DSz7mbWHBgDLK7S5mmC\nvXzCYZwvA9vNLMvMOobT+wH9gD8nqO8iIlJPdQ7vuHupmd0ILAOygHnuvsnMpgEF7r44nHehmRUS\nDN/8xN1LzCwbWGlmAB8DE9y9NFkrIyIitTP3xjWEnpeX5wUFBanuhohIWjGzte6eV1c7HZErIhIh\nCn0RkQhR6IuIRIhCX0QkQhT6IiIRotAXEYkQhb6ISIQo9EVEIkShLyISIQp9EZEIUeiLiESIQl9E\nJEIU+iIiEaLQFxGJEIW+iEiEKPRFRCJEoS8iEiEKfRGRCFHoi4hEiEJfRCRCFPoiIhGi0BcRiRCF\nvohIhCj0RUQiRKEvIhIhCn0RkQhR6IuIRIhCX0QkQhT6IiIRotAXEYkQhb6ISIQo9EVEIiSu0Dez\nEWa2xcy2mdmUGtqMNrNCM9tkZgtipt8dTttsZrPNzBLV+Vj/8z/wla/AtdfCzJmwdCns3AnuyXg1\nEZH01LSuBmaWBcwBLgCKgDVmttjdC2Pa9AT+DRjo7nvM7IRw+v8BBgL9wqYvAUOAFxK5EgB79kCH\nDvDXv8Kjj1ZMb98ezjgD+vat/Lddu0T3QESk8asz9IEBwDZ33w5gZvnASKAwps13gTnuvgfA3T8I\npzuQDTQHDGgGvJ+Yrld24omwbFlwv6QENm4Mbhs2BH8ffxw+/riifU7OkRuD3r0hOzsZvRMRaRzi\nCf0uwLsxj4uAc6u0+TKAmf0dyAKmuvtSd3/FzJYDuwlC/35331z1BcxsEjAJ4JRTTqn3SlTVsSMM\nGRLcyrhDUVHFRqDs7/PPw+efB22aNIGePY/cGPToAVlZx9wtEZGUiyf0411OT2AokAOsMLO+QCeg\ndzgN4C9mNtjdV8Y+2d3nAnMB8vLykjIKbwZduwa3Sy6pmF5aCtu2Vd4YrF8Pf/hDRT0gOxtyc4/c\nGJx8crBcEZF0EU/o7wS6xjzOCafFKgJWufsXwFtm9iYVG4FX3X0/gJktAb4GrKSRaNoUevUKbldf\nXTH9wAHYvLnyxqCmekHsxuCMM4LpIiKNUTyhvwboaWbdCcJ+DDCuSpungbHAw2bWiWC4ZztwGvBd\nM/sPguGdIcCsBPU9qVq1gnPOCW6xSkpg06bKG4MFC2Dfvoo2XboEG4HYbwW9ekHLlg27DiIiVdUZ\n+u5eamY3AssIxuvnufsmM5sGFLj74nDehWZWCBwCfuLuJWb2JDAc2EBQ1F3q7s8ma2UaQseO8PWv\nB7cyNdULli8PfkoKqheISONg3sh+yJ6Xl+cFBQWp7kZCVFcv2LgxmKZ6gYgkkpmtdfe8Otsp9Bte\ndfWCjRth166KNqoXiEh9xBv6ifr1jtRDIuoFsRuD3r1VLxCR+Cj0G5Ha6gWx3wg2bDiyXvClLx25\nMfjSl1QvEJHKFPqNXOzxBRdfXDG9rF5QthGo6fiC3r2P3Bh06aJ6gUhUaUw/w8RTL2jX7sgNgeoF\nIulNY/oRpXqBiNRGoR8RqheICCj0Iy3eesHGjaoXiGQKjelL3MrqBVW/GVStF1R3/QLVC0SSS2P6\nknA11Qs++ujIDUF19YLqrl+geoFIw1LoyzHr0CH+esELLxxZL6i6MVC9QCR5FPqSFPWpF2zYAE89\nVVEvaNEiOB+R6gUiiacxfWkUVC8QOTYa05e0Ule9IHZjoHqByNFT6EujVlO9YOfOitNP1Kde0KNH\ncLU0kajSP39JO2aQkxPcjrZeUHVjoHqBRIXG9CXj1adeUHVjoHqBpIuMuojKF198QVFREZ999lmK\neiX1kZ2dTU5ODs2aNUt1V2pVXb1gw4bK9YKTT668EVC9QBqrjCrkFhUV0aZNG7p164bpO3ij5u6U\nlJRQVFRE9+7dU92dWqleIFGUFv9EP/vsMwV+mjAzOnbsSHFxcaq7clRUL5BMlxahDyjw00gmflZN\nm0KvXsFt1KiK6QcPQmFh5Y3B3/4Gjz1W0Ub1AmlM0ib0RRqjli3jP75g4UL47W8r2lStF5xxRvBN\nQfUCSaaMDP358+H22+Gdd+CUU2D6dBg//uiXV1JSwnnnnQfAe++9R1ZWFp07dwZg9erVNG/evM5l\nTJw4kSlTpnD66afX2GbOnDm0a9eO8cfS2dCgQYO4//77Oeuss455WVJ/ddULYjcG999fuV7Qo0f1\n1y9QvUASIeP+Gc2fD5MmBT/TA3j77eAxHH3wd+zYkXXr1gEwdepUWrduzeTJkyu1cXfcnSZNmlS7\njIcffrjO1/nBD35wdB2UtFBbveAf/6i8MVC9QJKl+oRKY7ffXhH4ZQ4cCKYn2rZt28jNzWX8+PH0\n6dOH3bt3M2nSJPLy8ujTpw/Tpk0rbzto0CDWrVtHaWkp7dq1Y8qUKZx55pl87Wtf44MPPgDgjjvu\nYNasWeXtp0yZwoABAzj99NN5+eWXAfj000/55je/SW5uLqNGjSIvL698g1STxx9/nL59+3LGGWdw\n2223AVBaWsq3vvWt8umzZ88G4N577yU3N5d+/foxYcKEhL9ncqSmTeH004NawdSpsGgRvPkmfPop\nFBTAI4/AjTfCCScE9YJ//Ve45JLgZHYdOsDgwXDDDfCb38CKFcHQkkhNMm5P/5136jf9WL3xxhs8\n+uij5OUFP4+dMWMGHTp0oLS0lGHDhjFq1Chyc3MrPWffvn0MGTKEGTNmcMsttzBv3jymTJlyxLLd\nndWrV7N48WKmTZvG0qVLue+++zjxxBNZtGgRr7/+Ov3796+1f0VFRdxxxx0UFBTQtm1bzj//fP74\nxz/SuXNnPvzwQzZs2ADA3r17Abj77rt5++23ad68efk0SQ3VCyQZMi70TzklGNKpbnoy9OjRozzw\nARYuXMhDDz1EaWkpu3btorCw8IjQb9myJReH3+/POeccVq5cWe2yr7rqqvI2O3bsAOCll17i1ltv\nBeDMM8+kT58+tfZv1apVDB8+nE6dOgEwbtw4VqxYwa233sqWLVu46aabuPTSS7nwwgsB6NOnDxMm\nTGDkyJFcccUV9Xw3pCGoXiDHIuM+6unTK4/pQ3AGx+nTk/N6xx13XPn9rVu38utf/5rVq1fTrl07\nJkyYUO1RxLGF36ysLEpLS6tddosWLepsc7Q6duzI+vXrWbJkCXPmzGHRokXMnTuXZcuW8eKLL7J4\n8WJ+8YtfsH79erJ0RZNGrz71go0b4emn4fDhoE2LFtVf7zgnR/WCTJRxoV9WrE3kr3fi9fHHH9Om\nTRuOP/54du/ezbJlyxgxYkRCX2PgwIE88cQTDB48mA0bNlBYWFhr+3PPPZfJkydTUlJC27Ztyc/P\nZ/LkyRQXF5Odnc3VV19Nz549uf766zl06BBFRUUMHz6cQYMG0bVrVw4cOECbNm0Sug7ScMrqBWU1\ngzIHDwbnI4rdGDz/fOXjC9q2rf76BR06NPx6SOJkXOhDEPANEfJV9e/fn9zcXHr16sWpp57KwIED\nE/4aP/zhD7nmmmvIzc0tv7Vt27bG9jk5Ofz85z9n6NChuDuXXXYZl156Ka+99hrf+c53cHfMjF/+\n8peUlpYybtw4PvnkEw4fPszkyZMV+BmqZUvo3z+4xaqpXlD1fERVNwaqF6SPtDjh2ubNm+ndu3eK\netS4lJaWUlpaSnZ2Nlu3buXCCy9k69atNG1kg7L6zDJHTfWCwsKKeoFZUBtQvSB1EnrCNTMbAfwa\nyAJ+5+4zqmkzGpgKOPC6u48zs2HAvTHNegFj3P3peF5XjrR//37OO+88SktLcXcefPDBRhf4kllU\nL8gsde7pm1kW8CZwAVAErAHGunthTJuewBPAcHffY2YnuPsHVZbTAdgG5Lh7lV/SV9CefmbQZxZd\n1dULNm4Mvi2UUb0g8RK5pz8A2Obu28MF5wMjgdgK4neBOe6+B6Bq4IdGAUtqC3wRSX+11Qs2baq8\nMYinXtC7d/ALPEmMeEK/C/BuzOMi4Nwqbb4MYGZ/JxgCmuruS6u0GQPcc5T9FJE0V3b08ODBFdPK\n6gWxp5+oenxBWb2g6sZA9YKjk6i3rCnQExgK5AArzKyvu+8FMLOTgL7AsuqebGaTgEkApyTrKCoR\naXRi6wWxv26uqV7wzDNH1guqbgxUL6hdPKG/E+ga8zgnnBarCFjl7l8Ab5nZmwQbgTXh/NHAU+H8\nI7j7XGAuBGP68XdfRDJRfY4vWL4cHn+8ok1ZvSD2EpeqF1SI54Rra4CeZtbdzJoTDNMsrtLmaYK9\nfMysE8Fwz/aY+WOBhcfc2xQZNmwYy5ZV/pIya9Ysbrjhhlqf17p1awB27drFqNh/uTGGDh1KXReC\nnzVrFgdiDjG+5JJLEnJenKlTpzJz5sxjXo5IQymrF1x7LfzqV7B0KRQVQUlJcLK5OXNg3LhgTz8/\nPzhR3ZAh0LFjcEbSiy6CyZODk9itXXvkyRmjoM49fXcvNbMbCYZmsoB57r7JzKYBBe6+OJx3oZkV\nAoeAn7h7CYCZdSP4pvBiclYh+caOHUt+fj4XXXRR+bT8/HzuvvvuuJ5/8skn8+STTx7168+aNYsJ\nEybQKqxmPffcc0e9LJFMFE+9oOxv1OsFca2Wuz8HPFdl2p0x9x24JbxVfe4OgmJwQtx8M9RxJuF6\nO+ssCM9oXK1Ro0Zxxx138Pnnn9O8eXN27NjBrl27GDx4MPv372fkyJHs2bOHL774grvuuouRI0dW\nev6OHTv4xje+wcaNGzl48CATJ07k9ddfp1evXhw8eLC83Q033MCaNWs4ePAgo0aN4mc/+xmzZ89m\n165dDBs2jE6dOrF8+XK6detGQUEBnTp14p577mHevHkAXH/99dx8883s2LGDiy++mEGDBvHyyy/T\npUsXnnnmGVrWcsjkunXr+P73v8+BAwfo0aMH8+bNo3379syePZvf/va3NG3alNzcXPLz83nxxRf5\n0Y9+BASXRlyxYoWO3JVGp656QdWNQVTqBRm6LUusDh06MGDAAJYsWcLIkSPJz89n9OjRmBnZ2dk8\n9dRTHH/88Xz44Yd89atf5fLLL6/xOrEPPPAArVq1YvPmzaxfv77SqZGnT59Ohw4dOHToEOeddx7r\n16/npptu4p577mH58uXlZ8oss3btWh5++GFWrVqFu3PuuecyZMgQ2rdvz9atW1m4cCH/9V//xejR\no1m0aFGt58e/5ppruO+++xgyZAh33nknP/vZz5g1axYzZszgrbfeokWLFuVDSjNnzmTOnDkMHDiQ\n/fv3k52dnYB3WaRhxNYLvvnNiulHUy8o+5tO9YK0C/3a9siTqWyIpyz0H3roISA45/1tt93GihUr\naNKkCTt37uT999/nxBNPrHY5K1as4KabbgKgX79+9OvXr3zeE088wdy5cyktLWX37t0UFhZWml/V\nSy+9xJVXXll+ps+rrrqKlStXcvnll9O9e/fySyXGnpq5Ovv27WPv3r0MGTIEgGuvvZarr766vI/j\nx4/niiuuKD/V8sCBA7nlllsYP348V111FTk5OfG8hSKNWn2OL8jPhwcfrGhz0knVX7+gMR5fkHah\nnyojR47kxz/+Ma+99hoHDhzgnPDKFvPnz6e4uJi1a9fSrFkzunXrVu3plOvy1ltvMXPmTNasWUP7\n9u257rrrjmo5ZcpOywzBqZljh5Hq409/+hMrVqzg2WefZfr06WzYsIEpU6Zw6aWX8txzzzFw4ECW\nLVtGr169jrqvIo1ZfeoFc+ZUrhdUd/2Cnj1TWy9Q6MepdevWDBs2jG9/+9uMHTu2fPq+ffs44YQT\naNasGcuXL+ft6q7gEuPrX/86CxYsYPjw4WzcuJH169cDwWmZjzvuONq2bcv777/PkiVLGDp0KABt\n2rThk08+OWJ4Z/DgwVx33XVMmTIFd+epp57isdhz48apbdu2tG/fnpUrVzJ48GAee+wxhgwZwuHD\nh3n33XcZNmwYgwYNIj8/n/3791NSUkLfvn3p27cva9as4Y033lDoS6QcS72gefOK8xGlol6g0K+H\nsWPHcuWVV5Kfn18+bfz48Vx22WX07duXvLy8OsPvhhtuYOLEifTu3ZvevXuXf2M488wzOfvss+nV\nqxddu3atdFrmSZMmMWLECE4++WSWL19ePr1///5cd911DBgwAAgKuWeffXatQzk1+f3vf19eyD3t\ntNN4+OGHOXToEBMmTGDfvn24OzfddBPt2rXjpz/9KcuXL6dJkyb06dOn/CpgIlFXV70gdmNQXb1g\nxIhg6CiZdGplSQp9ZiJ127On8vUL2rWDX/zi6JaV0FMri4hI4rVvf2S9INniOSJXREQyRNqEfmMb\nhpKa6bMSabzSIvSzs7MpKSlRmKQBd6ekpEQHbIk0Umkxpp+Tk0NRURHFxcWp7orEITs7WwdsiTRS\naRH6zZo1o3v37qnuhohI2kuL4R0REUkMhb6ISIQo9EVEIqTRHZFrZsVA7SewqV0n4MMEdSddRG2d\no7a+oHWOimNZ51PdvXNdjRpd6B8rMyuI51DkTBK1dY7a+oLWOSoaYp01vCMiEiEKfRGRCMnE0J+b\n6g6kQNTWOWrrC1rnqEj6OmfNZ8oRAAAEgklEQVTcmL6IiNQsE/f0RUSkBgp9EZEIScvQN7N5ZvaB\nmW2sYb6Z2Wwz22Zm682sf3Xt0kkc6zw+XNcNZvaymZ3Z0H1MtLrWOabdV8ys1MxGNVTfkiGe9TWz\noWa2zsw2mdmLDdm/ZIjj33VbM3vWzF4P13liQ/cx0cysq5ktN7PCcJ1+VE2bpGVYWoY+8Agwopb5\nFwM9w9sk4IEG6FOyPULt6/wWMMTd+wI/JzOKYI9Q+zpjZlnAL4E/N0SHkuwRallfM2sH/Aa43N37\nAFc3UL+S6RFq/4x/ABS6+5nAUOA/zax5A/QrmUqBf3H3XOCrwA/MLLdKm6RlWFqGvruvAD6qpclI\n4FEPvAq0M7OTGqZ3yVHXOrv7y+6+J3z4KpD25zaO43MG+CGwCPgg+T1KrjjWdxzwB3d/J2wfhXV2\noI2ZGdA6bFvaEH1LFnff7e6vhfc/ATYDXao0S1qGpWXox6EL8G7M4yKOfFMz2XeAJanuRLKZWRfg\nSjLjm1w8vgy0N7MXzGytmV2T6g41gPuB3sAuYAPwI3c/nNouJY6ZdQPOBlZVmZW0DEuL8+lL/Mxs\nGEHoD0p1XxrALOBWdz8c7AhmvKbAOcB5QEvgFTN71d3fTG23kuoiYB0wHOgB/MXMVrr7x6nt1rEz\ns9YE31Jvbsj1ydTQ3wl0jXmcE07LaGbWD/gdcLG7l6S6Pw0gD8gPA78TcImZlbr706ntVtIUASXu\n/inwqZmtAM4EMjn0JwIzPDigaJuZvQX0AlantlvHxsyaEQT+fHf/QzVNkpZhmTq8sxi4JqyAfxXY\n5+67U92pZDKzU4A/AN/K8D2/cu7e3d27uXs34Eng/2Zw4AM8Awwys6Zm1go4l2A8OJO9Q/DNBjP7\nJ+B0YHtKe3SMwvrEQ8Bmd7+nhmZJy7C03NM3s4UElfxOZlYE/DvQDMDdfws8B1wCbAMOEOwtpLU4\n1vlOoCPwm3DPtzTdz1AYxzpnlLrW1903m9lSYD1wGPidu9f6c9bGLo7P+OfAI2a2ATCC4bx0P93y\nQOBbwAYzWxdOuw04BZKfYToNg4hIhGTq8I6IiFRDoS8iEiEKfRGRCFHoi4hEiEJfRCRCFPoigJnt\nT3UfRBqCQl9EJEIU+iI1MLNuZvZ8eD7zv4VHPWNmV5vZxvAc7yvCaX3MbHV4rvv1ZtYztb0XqZ4O\nzhIhGN5x99ZVpj0LPOnuvzezbxOcx/6K8OjQEe6+08zaufteM7sPeNXd54fne89y94MpWBWRWmlP\nX6RmXwMWhPcfo+LMpX8nODXAd4GscNorwG1mditwqgJfGiuFvkg9ufv3gTsIzoK41sw6uvsC4HLg\nIPCcmQ1PZR9FaqLQF6nZy8CY8P54YCWAmfVw91XufidQDHQ1s9OA7e4+m+BsmP1S0WGRumhMXwQw\ns8MEV2cqcw/B+c4fJjhXfzEw0d3fMbM/EFy71IC/ATcDtxKcOfEL4D1gnLvXdalHkQan0BcRiRAN\n74iIRIhCX0QkQhT6IiIRotAXEYkQhb6ISIQo9EVEIkShLyISIf8LgvQoJXBCM2wAAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x12a6522b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "WARNING:tensorflow:From /Users/rajiv.shah/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1349: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "keep_dims is deprecated, use keepdims instead\n",
      "2019-04-30 13:06:02.101678: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n",
      "merged AUC on testing data set: 0.8370979493580865\n",
      "length of testing dataset:  1378120\n",
      "merged AUC on training data set: 0.7642133320322944\n",
      "length of training dataset:  4743090\n"
     ]
    }
   ],
   "source": [
    "import copy\n",
    "import gc\n",
    "import numpy as np\n",
    "import pdb as check\n",
    "import random\n",
    "import modelfunctions\n",
    "from keras.callbacks import ModelCheckpoint\n",
    "import matplotlib.pyplot as plt\n",
    "np.random.seed(42)\n",
    "\n",
    "\n",
    "#set name of weights file and training/testing files\n",
    "weightFile = '../Data/TheBestWeights.h5'\n",
    "trainFile = '../Data/Training.h5' \n",
    "testFile = '../Data/Testing.h5'\n",
    "print (\"training file:, \", trainFile)\n",
    "\n",
    "#names of features\n",
    "field_names_in = ['stresses_full_xx', 'stresses_full_yy', 'stresses_full_xy', 'stresses_full_xz','stresses_full_yz','stresses_full_zz']\n",
    "\n",
    "#name of label\n",
    "field_names_out = 'aftershocksyn'\n",
    "\n",
    "#load training data set\n",
    "IN_Train, OUT_Train = modelfunctions.LoadInputs(trainFile, field_names_in, field_names_out)\n",
    "\n",
    "#pull out some validation data with positive grid cells upsampled so that validation data has equal numbers of positive and negative samples\n",
    "posidx = np.where(OUT_Train==1)\n",
    "numpos = np.size(posidx)\n",
    "negidx = np.where(OUT_Train==0)\n",
    "numneg = np.size(negidx)\n",
    "\n",
    "#divide data into positive and negative samples\n",
    "POSdata = np.column_stack((IN_Train[posidx,:][0], OUT_Train[posidx].T))\n",
    "NEGdata = np.column_stack((IN_Train[negidx,:][0], OUT_Train[negidx].T))\n",
    "\n",
    "np.random.seed(42) #shuffle order of samples (same way every time to ensure validation data does not change)\n",
    "np.random.shuffle(POSdata)\n",
    "np.random.shuffle(NEGdata)\n",
    "np.random.seed() #reseed\n",
    "\n",
    "cutoff = int(round(numpos/10)) #validation data consists of a random 10% of positive samples, and same number of randomly selected negative samples\n",
    "\n",
    "#positive validation samples\n",
    "val_in1 = copy.copy(POSdata[:cutoff,:len(field_names_in)*2])\n",
    "val_out1 = copy.copy(POSdata[:cutoff,len(field_names_in)*2])\n",
    "\n",
    "#negative validation samples\n",
    "val_in2 = copy.copy(NEGdata[:cutoff,:len(field_names_in)*2])\n",
    "val_out2 = copy.copy(NEGdata[:cutoff,len(field_names_in)*2])\n",
    "\n",
    "#merge to obtain entire validation data set\n",
    "val_in = np.row_stack((val_in1, val_in2))\n",
    "val_out = np.append(val_out1, val_out2)\n",
    "\n",
    "#the remaining data set is the training data set\n",
    "POSdataFinal = copy.copy(POSdata[cutoff:,:])\n",
    "NEGdataFinal = copy.copy(NEGdata[cutoff:,:])\n",
    "\n",
    "shapepos = np.shape(POSdataFinal)\n",
    "\n",
    "#set hyperparameters\n",
    "batch_size = 300\n",
    "#steps_per_epoch = int(round((shapepos[0])/batch_size)) #total number of steps (batches of samples) before declaring one epoch finished and starting the next epoch, equal to the number of samples in dataset divided by the batch size\n",
    "steps_per_epoch = 5\n",
    "epoch_num = 2\n",
    "posstart = 0\n",
    "negstart = 0\n",
    "\n",
    "#train\n",
    "model = modelfunctions.create_model()\n",
    "checkpointer = ModelCheckpoint(filepath=weightFile, monitor = 'val_loss', verbose=2, save_best_only = True)\n",
    "history = model.fit_generator(modelfunctions.generate_data(POSdataFinal, NEGdataFinal, batch_size, posstart, negstart), steps_per_epoch, validation_data = (val_in, val_out),  callbacks = [checkpointer], verbose=2, epochs=epoch_num)\n",
    "\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = list(range(1,len(loss)+1))\n",
    "plt.plot(epochs, loss, 'bo', label = 'Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label = 'Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.xlabel('Loss')\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "!python EvalModel2.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***We get the almost the same performance after passing 1500 rows of data in 2 epochs -- something is amiss.***  Using such a small amount of data to get good performance is rare.  Typically the more data, the better the performance.  Using a little data suggests either a leakage issue or that it is faily easy for the model to identify the signal in the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Leakage across similar aftershocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv(\"names_train_ids.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The name is the original filename and original is the authors placement of the data in either test or train.  As you can see, some aftershocks (e.g., 1985NAHANN01HART_grid.csv)  with the same name are split between test and train.  This could be a source of the potential leakage.  The easiest way is to use group partitioning, build the models, and compare the performance.  In this case, after using group partitioning my performance drops to an AUC of 0.77.  I am happy to share how I partitioned the dataset (I actually used 10 different randomly selected trials using group partitioning).  You can use the ID field for group partitioning.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Original</th>\n",
       "      <th>Change</th>\n",
       "      <th>ID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1968HYUGAx01YAGI_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1968TOKACH01NAGA_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1969GIFUxK01TAKE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1971SANFER01HEAT_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1974IZUxHA01TAKE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1974PERUCE01HART_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1978MIYAGI01YAMA_grid.csv</td>\n",
       "      <td>TEST</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1978TABASI01HART_grid.csv</td>\n",
       "      <td>TEST</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1979COYOTE01LIUx_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1979IMPERI01ARCH_grid.csv</td>\n",
       "      <td>TEST</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1979IMPERI01HART_grid.csv</td>\n",
       "      <td>TEST</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1979IMPERI01OLSO_grid.csv</td>\n",
       "      <td>TEST</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1979PETATL01MEND_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1980IZUxHA01TAKE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1981PLAYAA01MEND_grid.csv</td>\n",
       "      <td>TEST</td>\n",
       "      <td>NaN</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1982NEWBRU01HART_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1983BORAHP01MEND_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>1983JAPANE01FUKU_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1984MORGAN01HART_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1984NAGANO01TAKE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>1985CENTRA01MEND_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>1985MICHOA01MEND_grid.csv</td>\n",
       "      <td>TEST</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>1985NAHANN01HART_grid.csv</td>\n",
       "      <td>TEST</td>\n",
       "      <td>issue</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>1985NAHANN02HART_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>1985ZIHUAT01MEND_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>1986NORTHP01HART_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>1986NORTHP01MEND_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>1987ELMORE01LARS_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>1987SUPERS01LARS_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>1987SUPERS01WALD_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169</th>\n",
       "      <td>2010HAITIx02HAYE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>170</th>\n",
       "      <td>2010MAULEC01HAYE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>171</th>\n",
       "      <td>2010NORTHE01HAYE_grid.csv</td>\n",
       "      <td>TEST</td>\n",
       "      <td>issue</td>\n",
       "      <td>172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172</th>\n",
       "      <td>2010NORTHE02HAYE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>173</th>\n",
       "      <td>2010VANUAT01HAYE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>174</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174</th>\n",
       "      <td>2011KERMAD01HAYE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>175</th>\n",
       "      <td>2011KERMAD02HAYE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>176</th>\n",
       "      <td>2011OFFSHO01HAYE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>177</th>\n",
       "      <td>2011TOHOKU01FUJI_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>2011TOHOKU01SATA_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>179</th>\n",
       "      <td>2011TOHOKU01YUEx_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>180</th>\n",
       "      <td>2011TOHOKU02FUJI_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>181</th>\n",
       "      <td>2011TOHOKU03SATA_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>182</th>\n",
       "      <td>2011VANTUR01HAYE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183</th>\n",
       "      <td>2011VANTUR01SHAO_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>184</th>\n",
       "      <td>2011VANUAT01HAYE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>185</th>\n",
       "      <td>2012BRAWLE01WEIx_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>186</th>\n",
       "      <td>2012BRAWLE02WEIx_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>187</th>\n",
       "      <td>2012COSTAR01HAYE_grid.csv</td>\n",
       "      <td>TEST</td>\n",
       "      <td>NaN</td>\n",
       "      <td>188</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>188</th>\n",
       "      <td>2012COSTAR01YUEx_grid.csv</td>\n",
       "      <td>TEST</td>\n",
       "      <td>NaN</td>\n",
       "      <td>188</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>189</th>\n",
       "      <td>2012EASTOF01HAYE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>190</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>190</th>\n",
       "      <td>2012MASSET01LAYx_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>191</th>\n",
       "      <td>2012MASSET01SHAO_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>192</th>\n",
       "      <td>2012MASSET01WEIx_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>193</th>\n",
       "      <td>2012OAXACA01HAYE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>194</th>\n",
       "      <td>2012OAXACA01WEIx_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>2012SUMATR01HAYE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>2012SUMATR01SHAO_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>2012SUMATR02HAYE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>2012SUMATR03HAYE_grid.csv</td>\n",
       "      <td>TRAIN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>196</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>199 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                          Name Original Change   ID\n",
       "0    1968HYUGAx01YAGI_grid.csv    TRAIN    NaN    1\n",
       "1    1968TOKACH01NAGA_grid.csv    TRAIN    NaN    2\n",
       "2    1969GIFUxK01TAKE_grid.csv    TRAIN    NaN    3\n",
       "3    1971SANFER01HEAT_grid.csv    TRAIN    NaN    4\n",
       "4    1974IZUxHA01TAKE_grid.csv    TRAIN    NaN    5\n",
       "5    1974PERUCE01HART_grid.csv    TRAIN    NaN    6\n",
       "6    1978MIYAGI01YAMA_grid.csv     TEST    NaN    7\n",
       "7    1978TABASI01HART_grid.csv     TEST    NaN    8\n",
       "8    1979COYOTE01LIUx_grid.csv    TRAIN    NaN    9\n",
       "9    1979IMPERI01ARCH_grid.csv     TEST    NaN   10\n",
       "10   1979IMPERI01HART_grid.csv     TEST    NaN   10\n",
       "11   1979IMPERI01OLSO_grid.csv     TEST    NaN   10\n",
       "12   1979PETATL01MEND_grid.csv    TRAIN    NaN   13\n",
       "13   1980IZUxHA01TAKE_grid.csv    TRAIN    NaN   14\n",
       "14   1981PLAYAA01MEND_grid.csv     TEST    NaN   15\n",
       "15   1982NEWBRU01HART_grid.csv    TRAIN    NaN   16\n",
       "16   1983BORAHP01MEND_grid.csv    TRAIN    NaN   17\n",
       "17   1983JAPANE01FUKU_grid.csv    TRAIN    NaN   18\n",
       "18   1984MORGAN01HART_grid.csv    TRAIN    NaN   19\n",
       "19   1984NAGANO01TAKE_grid.csv    TRAIN    NaN   20\n",
       "20   1985CENTRA01MEND_grid.csv    TRAIN    NaN   21\n",
       "21   1985MICHOA01MEND_grid.csv     TEST    NaN   22\n",
       "22   1985NAHANN01HART_grid.csv     TEST  issue   23\n",
       "23   1985NAHANN02HART_grid.csv    TRAIN    NaN   23\n",
       "24   1985ZIHUAT01MEND_grid.csv    TRAIN    NaN   25\n",
       "25   1986NORTHP01HART_grid.csv    TRAIN    NaN   26\n",
       "26   1986NORTHP01MEND_grid.csv    TRAIN    NaN   26\n",
       "27   1987ELMORE01LARS_grid.csv    TRAIN    NaN   28\n",
       "28   1987SUPERS01LARS_grid.csv    TRAIN    NaN   29\n",
       "29   1987SUPERS01WALD_grid.csv    TRAIN    NaN   29\n",
       "..                         ...      ...    ...  ...\n",
       "169  2010HAITIx02HAYE_grid.csv    TRAIN    NaN  167\n",
       "170  2010MAULEC01HAYE_grid.csv    TRAIN    NaN  171\n",
       "171  2010NORTHE01HAYE_grid.csv     TEST  issue  172\n",
       "172  2010NORTHE02HAYE_grid.csv    TRAIN    NaN  172\n",
       "173  2010VANUAT01HAYE_grid.csv    TRAIN    NaN  174\n",
       "174  2011KERMAD01HAYE_grid.csv    TRAIN    NaN  175\n",
       "175  2011KERMAD02HAYE_grid.csv    TRAIN    NaN  175\n",
       "176  2011OFFSHO01HAYE_grid.csv    TRAIN    NaN  177\n",
       "177  2011TOHOKU01FUJI_grid.csv    TRAIN    NaN  178\n",
       "178  2011TOHOKU01SATA_grid.csv    TRAIN    NaN  178\n",
       "179  2011TOHOKU01YUEx_grid.csv    TRAIN    NaN  178\n",
       "180  2011TOHOKU02FUJI_grid.csv    TRAIN    NaN  178\n",
       "181  2011TOHOKU03SATA_grid.csv    TRAIN    NaN  178\n",
       "182  2011VANTUR01HAYE_grid.csv    TRAIN    NaN  183\n",
       "183  2011VANTUR01SHAO_grid.csv    TRAIN    NaN  183\n",
       "184  2011VANUAT01HAYE_grid.csv    TRAIN    NaN  183\n",
       "185  2012BRAWLE01WEIx_grid.csv    TRAIN    NaN  186\n",
       "186  2012BRAWLE02WEIx_grid.csv    TRAIN    NaN  186\n",
       "187  2012COSTAR01HAYE_grid.csv     TEST    NaN  188\n",
       "188  2012COSTAR01YUEx_grid.csv     TEST    NaN  188\n",
       "189  2012EASTOF01HAYE_grid.csv    TRAIN    NaN  190\n",
       "190  2012MASSET01LAYx_grid.csv    TRAIN    NaN  191\n",
       "191  2012MASSET01SHAO_grid.csv    TRAIN    NaN  191\n",
       "192  2012MASSET01WEIx_grid.csv    TRAIN    NaN  191\n",
       "193  2012OAXACA01HAYE_grid.csv    TRAIN    NaN  194\n",
       "194  2012OAXACA01WEIx_grid.csv    TRAIN    NaN  194\n",
       "195  2012SUMATR01HAYE_grid.csv    TRAIN    NaN  196\n",
       "196  2012SUMATR01SHAO_grid.csv    TRAIN    NaN  196\n",
       "197  2012SUMATR02HAYE_grid.csv    TRAIN    NaN  196\n",
       "198  2012SUMATR03HAYE_grid.csv    TRAIN    NaN  196\n",
       "\n",
       "[199 rows x 4 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}