4-machine-learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 機械学習をやってみる\n",
    "\n",
    "## この章で扱うもの\n",
    "\n",
    "- 前章で作った各種データを再利用\n",
    "- 機械学習を行ってみる\n",
    "- 学習結果の評価を行う\n",
    "- 機械学習の改善方法\n",
    "\n",
    "\n",
    "## この章で取り扱う手順\n",
    "\n",
    "- 学習データとテストデータの分割\n",
    "- sklearnを用いて学習\n",
    "- 評価\n",
    "- 交差検証\n",
    "- グリットサーチ\n",
    "- 各種モデルを試す\n",
    "- 評価結果の再確認\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_pickle(\"df.db\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 学習データとテストデータの分割\n",
    "\n",
    "http://qiita.com/terapyon/items/8f8d3518ee8eeb4f96b2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>報告数</th>\n",
       "      <th>流行</th>\n",
       "      <th>増加</th>\n",
       "      <th>平均気温(℃)</th>\n",
       "      <th>最高気温(℃)</th>\n",
       "      <th>最低気温(℃)</th>\n",
       "      <th>平均湿度(％)</th>\n",
       "      <th>最小相対湿度(％)</th>\n",
       "      <th>平均現地気圧(hPa)</th>\n",
       "      <th>降水量の合計(mm)</th>\n",
       "      <th>日照時間(時間)</th>\n",
       "      <th>平均風速(m/s)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2014-01-01</th>\n",
       "      <td>0.178571</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9.8</td>\n",
       "      <td>13.7</td>\n",
       "      <td>3.9</td>\n",
       "      <td>54.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>1005.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.2</td>\n",
       "      <td>5.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-02</th>\n",
       "      <td>0.178571</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>12.9</td>\n",
       "      <td>4.4</td>\n",
       "      <td>41.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>1011.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.1</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-03</th>\n",
       "      <td>0.178571</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.9</td>\n",
       "      <td>9.9</td>\n",
       "      <td>2.7</td>\n",
       "      <td>43.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>1014.9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.1</td>\n",
       "      <td>1.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-04</th>\n",
       "      <td>0.178571</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6.7</td>\n",
       "      <td>11.5</td>\n",
       "      <td>2.1</td>\n",
       "      <td>47.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>1009.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.9</td>\n",
       "      <td>2.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-05</th>\n",
       "      <td>0.178571</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4.4</td>\n",
       "      <td>6.9</td>\n",
       "      <td>2.3</td>\n",
       "      <td>40.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>1016.6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 報告数  流行  増加  平均気温(℃)  最高気温(℃)  最低気温(℃)  平均湿度(％)  最小相対湿度(％)  \\\n",
       "2014-01-01  0.178571   0   0      9.8     13.7      3.9     54.0       37.0   \n",
       "2014-01-02  0.178571   0   0      8.0     12.9      4.4     41.0       26.0   \n",
       "2014-01-03  0.178571   0   0      5.9      9.9      2.7     43.0       32.0   \n",
       "2014-01-04  0.178571   0   0      6.7     11.5      2.1     47.0       29.0   \n",
       "2014-01-05  0.178571   0   0      4.4      6.9      2.3     40.0       28.0   \n",
       "\n",
       "            平均現地気圧(hPa)  降水量の合計(mm)  日照時間(時間)  平均風速(m/s)  \n",
       "2014-01-01       1005.3         0.0       9.2        5.3  \n",
       "2014-01-02       1011.3         0.0       9.1        3.0  \n",
       "2014-01-03       1014.9         0.0       4.1        1.6  \n",
       "2014-01-04       1009.5         0.0       5.9        2.4  \n",
       "2014-01-05       1016.6         0.0       1.1        2.5  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = df.iloc[:, 3:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>平均気温(℃)</th>\n",
       "      <th>最高気温(℃)</th>\n",
       "      <th>最低気温(℃)</th>\n",
       "      <th>平均湿度(％)</th>\n",
       "      <th>最小相対湿度(％)</th>\n",
       "      <th>平均現地気圧(hPa)</th>\n",
       "      <th>降水量の合計(mm)</th>\n",
       "      <th>日照時間(時間)</th>\n",
       "      <th>平均風速(m/s)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2014-01-01</th>\n",
       "      <td>9.8</td>\n",
       "      <td>13.7</td>\n",
       "      <td>3.9</td>\n",
       "      <td>54.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>1005.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.2</td>\n",
       "      <td>5.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-02</th>\n",
       "      <td>8.0</td>\n",
       "      <td>12.9</td>\n",
       "      <td>4.4</td>\n",
       "      <td>41.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>1011.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.1</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-03</th>\n",
       "      <td>5.9</td>\n",
       "      <td>9.9</td>\n",
       "      <td>2.7</td>\n",
       "      <td>43.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>1014.9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.1</td>\n",
       "      <td>1.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-04</th>\n",
       "      <td>6.7</td>\n",
       "      <td>11.5</td>\n",
       "      <td>2.1</td>\n",
       "      <td>47.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>1009.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.9</td>\n",
       "      <td>2.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-05</th>\n",
       "      <td>4.4</td>\n",
       "      <td>6.9</td>\n",
       "      <td>2.3</td>\n",
       "      <td>40.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>1016.6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            平均気温(℃)  最高気温(℃)  最低気温(℃)  平均湿度(％)  最小相対湿度(％)  平均現地気圧(hPa)  \\\n",
       "2014-01-01      9.8     13.7      3.9     54.0       37.0       1005.3   \n",
       "2014-01-02      8.0     12.9      4.4     41.0       26.0       1011.3   \n",
       "2014-01-03      5.9      9.9      2.7     43.0       32.0       1014.9   \n",
       "2014-01-04      6.7     11.5      2.1     47.0       29.0       1009.5   \n",
       "2014-01-05      4.4      6.9      2.3     40.0       28.0       1016.6   \n",
       "\n",
       "            降水量の合計(mm)  日照時間(時間)  平均風速(m/s)  \n",
       "2014-01-01         0.0       9.2        5.3  \n",
       "2014-01-02         0.0       9.1        3.0  \n",
       "2014-01-03         0.0       4.1        1.6  \n",
       "2014-01-04         0.0       5.9        2.4  \n",
       "2014-01-05         0.0       1.1        2.5  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y = df['流行']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2014-01-01    0\n",
       "2014-01-02    0\n",
       "2014-01-03    0\n",
       "2014-01-04    0\n",
       "2014-01-05    0\n",
       "Name: 流行, dtype: int32"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_val, y_train, y_val = train_test_split(X, y, train_size=0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ロジスティック回帰"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clf = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train_pred = clf.predict(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 正答率を確認"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.86986301369863017"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_train, y_train_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_val_pred = clf.predict(X_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8954545454545455"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_val, y_val_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 混同行列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cm = confusion_matrix(y_val, y_val_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[184,   9],\n",
       "       [ 14,  13]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[664,  50],\n",
       "       [ 64,  98]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm_t = confusion_matrix(y_train, y_train_pred)\n",
    "cm_t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 混同行列の評価\n",
    "\n",
    "```\n",
    "[[TN, FP],\n",
    " [FN, TP]]\n",
    " ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 適合率(precision)・再現率(recall)・F値(f1-score)\n",
    "\n",
    "## 適合率\n",
    " \n",
    "- P(今回の場合は、流行していない) に判定された率 (178 / (178+14) = 0.93)\n",
    "- N(今回の場合は、流行している) に判定された率 (17 / (17+11) = 0.61)\n",
    " \n",
    "## 再現率\n",
    " \n",
    "- Tと正しく予測できた割合 (178 / (178+11) = 0.94)\n",
    "- Fと正しく予測できた割合 (17 / (17+14) = 0.55)\n",
    " \n",
    " \n",
    "## F値\n",
    " \n",
    "`2 / (1/適合率+1/再現率) = 2 * 適合率 * 再現率 / (適合率+再現率）`\n",
    " \n",
    "- 0のF値 `(2 * 0.93 * 0.94 / (0.93 + 0.94) = 0.93`\n",
    "- 1のF値 `(2 * 0.61 * 0.55 / (0.61 + 0.55) = 0.58`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.93      0.95      0.94       193\n",
      "          1       0.59      0.48      0.53        27\n",
      "\n",
      "avg / total       0.89      0.90      0.89       220\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_val, y_val_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## レポート関係を関数化し再利用可能にする"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def report(y, pred):\n",
    "    print(accuracy_score(y, pred))\n",
    "    cm = confusion_matrix(y, pred)\n",
    "    print(cm)\n",
    "    cr = classification_report(y, pred)\n",
    "    print(cr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.869863013699\n",
      "[[664  50]\n",
      " [ 64  98]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.91      0.93      0.92       714\n",
      "          1       0.66      0.60      0.63       162\n",
      "\n",
      "avg / total       0.87      0.87      0.87       876\n",
      "\n"
     ]
    }
   ],
   "source": [
    "report(y_train, y_train_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 学習から評価までを関数化\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def fit_to_pred(clf, X_train, X_val, y_train, y_val):\n",
    "    # 学習\n",
    "    clf.fit(X_train, y_train)\n",
    "    \n",
    "    # 学習データで評価\n",
    "    y_train_pred = clf.predict(X_train)\n",
    "    print(\"y_train_pred: \")\n",
    "    report(y_train, y_train_pred)\n",
    "    \n",
    "    # テストデータで評価\n",
    "    y_val_pred = clf.predict(X_val)\n",
    "    print(\"y_val_pred: \")\n",
    "    report(y_val, y_val_pred)\n",
    "    \n",
    "    # 学習済みデータを返す\n",
    "    return clf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train_pred: \n",
      "0.869863013699\n",
      "[[664  50]\n",
      " [ 64  98]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.91      0.93      0.92       714\n",
      "          1       0.66      0.60      0.63       162\n",
      "\n",
      "avg / total       0.87      0.87      0.87       876\n",
      "\n",
      "y_val_pred: \n",
      "0.895454545455\n",
      "[[184   9]\n",
      " [ 14  13]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.93      0.95      0.94       193\n",
      "          1       0.59      0.48      0.53        27\n",
      "\n",
      "avg / total       0.89      0.90      0.89       220\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = LogisticRegression()\n",
    "fit_to_pred(clf, X_train, X_val, y_train, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# その他の機械学習アルゴリズム\n",
    "\n",
    "- サポートベクターマシン SVC\n",
    "- カーネルSVM\n",
    "- 決定木 DecisionTreeClassifier\n",
    "- ランダムフォレスト RandomForestClassifier\n",
    "- k近傍"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train_pred: \n",
      "0.876712328767\n",
      "[[658  56]\n",
      " [ 52 110]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.93      0.92      0.92       714\n",
      "          1       0.66      0.68      0.67       162\n",
      "\n",
      "avg / total       0.88      0.88      0.88       876\n",
      "\n",
      "y_val_pred: \n",
      "0.890909090909\n",
      "[[182  11]\n",
      " [ 13  14]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.93      0.94      0.94       193\n",
      "          1       0.56      0.52      0.54        27\n",
      "\n",
      "avg / total       0.89      0.89      0.89       220\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape=None, degree=3, gamma='auto', kernel='linear',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svc = SVC(kernel=\"linear\")\n",
    "fit_to_pred(svc, X_train, X_val, y_train, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train_pred: \n",
      "0.996575342466\n",
      "[[714   0]\n",
      " [  3 159]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       1.00      1.00      1.00       714\n",
      "          1       1.00      0.98      0.99       162\n",
      "\n",
      "avg / total       1.00      1.00      1.00       876\n",
      "\n",
      "y_val_pred: \n",
      "0.877272727273\n",
      "[[193   0]\n",
      " [ 27   0]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.88      1.00      0.93       193\n",
      "          1       0.00      0.00      0.00        27\n",
      "\n",
      "avg / total       0.77      0.88      0.82       220\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/terapyon/dev/ipython/ossaj/env/lib/python3.6/site-packages/sklearn/metrics/classification.py:1113: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape=None, degree=3, gamma='auto', kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "k_svc = SVC(kernel=\"rbf\")\n",
    "fit_to_pred(k_svc, X_train, X_val, y_train, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train_pred: \n",
      "0.875570776256\n",
      "[[653  61]\n",
      " [ 48 114]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.93      0.91      0.92       714\n",
      "          1       0.65      0.70      0.68       162\n",
      "\n",
      "avg / total       0.88      0.88      0.88       876\n",
      "\n",
      "y_val_pred: \n",
      "0.877272727273\n",
      "[[181  12]\n",
      " [ 15  12]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.92      0.94      0.93       193\n",
      "          1       0.50      0.44      0.47        27\n",
      "\n",
      "avg / total       0.87      0.88      0.87       220\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=2,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            presort=False, random_state=None, splitter='best')"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree = DecisionTreeClassifier(max_depth=2)\n",
    "fit_to_pred(tree, X_train, X_val, y_train, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train_pred: \n",
      "0.988584474886\n",
      "[[712   2]\n",
      " [  8 154]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.99      1.00      0.99       714\n",
      "          1       0.99      0.95      0.97       162\n",
      "\n",
      "avg / total       0.99      0.99      0.99       876\n",
      "\n",
      "y_val_pred: \n",
      "0.877272727273\n",
      "[[182  11]\n",
      " [ 16  11]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.92      0.94      0.93       193\n",
      "          1       0.50      0.41      0.45        27\n",
      "\n",
      "avg / total       0.87      0.88      0.87       220\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            n_estimators=10, n_jobs=1, oob_score=False, random_state=None,\n",
       "            verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf = RandomForestClassifier()\n",
    "fit_to_pred(rf, X_train, X_val, y_train, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train_pred: \n",
      "0.891552511416\n",
      "[[673  41]\n",
      " [ 54 108]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.93      0.94      0.93       714\n",
      "          1       0.72      0.67      0.69       162\n",
      "\n",
      "avg / total       0.89      0.89      0.89       876\n",
      "\n",
      "y_val_pred: \n",
      "0.863636363636\n",
      "[[182  11]\n",
      " [ 19   8]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.91      0.94      0.92       193\n",
      "          1       0.42      0.30      0.35        27\n",
      "\n",
      "avg / total       0.85      0.86      0.85       220\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "           metric_params=None, n_jobs=1, n_neighbors=5, p=2,\n",
       "           weights='uniform')"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn = KNeighborsClassifier()\n",
    "fit_to_pred(knn, X_train, X_val, y_train, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 交差検証(クロスバリデーション)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cv = KFold(5, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.80454545,  0.89954338,  0.88584475,  0.86757991,  0.89041096])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = LogisticRegression()\n",
    "cross_val_score(clf, X, y, cv=cv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.85909091,  0.80821918,  0.82191781,  0.83105023,  0.84018265])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "k_svc = SVC(kernel=\"rbf\")\n",
    "cross_val_score(k_svc, X, y, cv=cv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.83181818,  0.85388128,  0.84018265,  0.87214612,  0.8173516 ])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf = RandomForestClassifier()\n",
    "cross_val_score(rf, X, y, cv=cv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## F1-score で評価"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.59459459,  0.66666667,  0.65060241,  0.55882353,  0.58333333])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = LogisticRegression()\n",
    "cross_val_score(clf, X, y, cv=cv, scoring=\"f1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/terapyon/dev/ipython/ossaj/env/lib/python3.6/site-packages/sklearn/metrics/classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "/Users/terapyon/dev/ipython/ossaj/env/lib/python3.6/site-packages/sklearn/metrics/classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([ 0.        ,  0.08695652,  0.0625    ,  0.        ,  0.06060606])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "k_svc = SVC(kernel=\"rbf\")\n",
    "cross_val_score(k_svc, X, y, cv=cv, scoring=\"f1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.47457627,  0.49122807,  0.54320988,  0.52173913,  0.47887324])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf = RandomForestClassifier()\n",
    "cross_val_score(rf, X, y, cv=cv, scoring=\"f1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# グリッドサーチ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "param_grid = {'max_depth': [2, 3, 4, 5, 10, 15, 20, 30], 'n_estimators': [2, 3, 4, 5, 10, 20, 30, 40]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rf = RandomForestClassifier(max_depth=2, n_estimators=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grid_search = GridSearchCV(rf, param_grid, cv=cv, n_jobs=-1, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 64 candidates, totalling 320 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done 320 out of 320 | elapsed:    4.7s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=KFold(n_splits=5, random_state=None, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=2, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            n_estimators=2, n_jobs=1, oob_score=False, random_state=None,\n",
       "            verbose=0, warm_start=False),\n",
       "       fit_params={}, iid=True, n_jobs=-1,\n",
       "       param_grid={'max_depth': [2, 3, 4, 5, 10, 15, 20, 30], 'n_estimators': [2, 3, 4, 5, 10, 20, 30, 40]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "       scoring=None, verbose=1)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.87226277372262773"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': 2, 'n_estimators': 40}"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 64 candidates, totalling 320 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done 320 out of 320 | elapsed:    5.0s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=KFold(n_splits=5, random_state=None, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=2, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            n_estimators=2, n_jobs=1, oob_score=False, random_state=None,\n",
       "            verbose=0, warm_start=False),\n",
       "       fit_params={}, iid=True, n_jobs=-1,\n",
       "       param_grid={'max_depth': [2, 3, 4, 5, 10, 15, 20, 30], 'n_estimators': [2, 3, 4, 5, 10, 20, 30, 40]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "       scoring='f1', verbose=1)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf = RandomForestClassifier(max_depth=2, n_estimators=2)\n",
    "grid_search = GridSearchCV(rf, param_grid, cv=cv, n_jobs=-1, verbose=1, scoring='f1')\n",
    "grid_search.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.633797541829\n",
      "{'max_depth': 3, 'n_estimators': 10}\n"
     ]
    }
   ],
   "source": [
    "print(grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.65789474,  0.54054054,  0.69230769,  0.61290323,  0.64516129])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf = RandomForestClassifier(max_depth=3, n_estimators=40)\n",
    "cross_val_score(rf, X, y, cv=cv, scoring=\"f1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 最終確認"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train_pred: \n",
      "0.883561643836\n",
      "[[657  57]\n",
      " [ 45 117]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.94      0.92      0.93       714\n",
      "          1       0.67      0.72      0.70       162\n",
      "\n",
      "avg / total       0.89      0.88      0.89       876\n",
      "\n",
      "y_val_pred: \n",
      "0.881818181818\n",
      "[[182  11]\n",
      " [ 15  12]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.92      0.94      0.93       193\n",
      "          1       0.52      0.44      0.48        27\n",
      "\n",
      "avg / total       0.87      0.88      0.88       220\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=3, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            n_estimators=40, n_jobs=1, oob_score=False, random_state=None,\n",
       "            verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf = RandomForestClassifier(max_depth=3, n_estimators=40)\n",
    "fit_to_pred(rf, X_train, X_val, y_train, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1,\n",
       "       0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int32)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf.predict(X_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.externals import joblib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['clf_rf.db']"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(rf, \"clf_rf.db\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}