[E-02]ClassifierMetrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0c336236",
   "metadata": {},
   "source": [
    "# 손글씨를 분류해 봅시다"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee75e1fc",
   "metadata": {},
   "source": [
    "### 1. 필요한 모듈 import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "id": "a9b034fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c727ac0b",
   "metadata": {},
   "source": [
    "### 2. 손글씨 데이터셋 준비"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "f76de5f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "digits = load_digits() "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b88966c1",
   "metadata": {},
   "source": [
    "### 3. 데이터 이해하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "id": "482e7e45",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Target Names : [0 1 2 3 4 5 6 7 8 9]\n"
     ]
    }
   ],
   "source": [
    "digits_feature = digits.feature_names # Feature Data \n",
    "# print(f'features 갯수 : {len(digits_feature)}')\n",
    "# print(f'feature 데이터 : {digits_feature}') # \n",
    "\n",
    "digits_data = digits.data # data\n",
    "# print(digits_data)\n",
    "\n",
    "digits_label = digits.target # Label Data\n",
    "\n",
    "digits_target = digits.target_names # Target Names\n",
    "\n",
    "print(f'Target Names : {digits_target}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "f50099df",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".. _digits_dataset:\n",
      "\n",
      "Optical recognition of handwritten digits dataset\n",
      "--------------------------------------------------\n",
      "\n",
      "**Data Set Characteristics:**\n",
      "\n",
      "    :Number of Instances: 5620\n",
      "    :Number of Attributes: 64\n",
      "    :Attribute Information: 8x8 image of integer pixels in the range 0..16.\n",
      "    :Missing Attribute Values: None\n",
      "    :Creator: E. Alpaydin (alpaydin '@' boun.edu.tr)\n",
      "    :Date: July; 1998\n",
      "\n",
      "This is a copy of the test set of the UCI ML hand-written digits datasets\n",
      "https://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits\n",
      "\n",
      "The data set contains images of hand-written digits: 10 classes where\n",
      "each class refers to a digit.\n",
      "\n",
      "Preprocessing programs made available by NIST were used to extract\n",
      "normalized bitmaps of handwritten digits from a preprinted form. From a\n",
      "total of 43 people, 30 contributed to the training set and different 13\n",
      "to the test set. 32x32 bitmaps are divided into nonoverlapping blocks of\n",
      "4x4 and the number of on pixels are counted in each block. This generates\n",
      "an input matrix of 8x8 where each element is an integer in the range\n",
      "0..16. This reduces dimensionality and gives invariance to small\n",
      "distortions.\n",
      "\n",
      "For info on NIST preprocessing routines, see M. D. Garris, J. L. Blue, G.\n",
      "T. Candela, D. L. Dimmick, J. Geist, P. J. Grother, S. A. Janet, and C.\n",
      "L. Wilson, NIST Form-Based Handprint Recognition System, NISTIR 5469,\n",
      "1994.\n",
      "\n",
      ".. topic:: References\n",
      "\n",
      "  - C. Kaynak (1995) Methods of Combining Multiple Classifiers and Their\n",
      "    Applications to Handwritten Digit Recognition, MSc Thesis, Institute of\n",
      "    Graduate Studies in Science and Engineering, Bogazici University.\n",
      "  - E. Alpaydin, C. Kaynak (1998) Cascading Classifiers, Kybernetika.\n",
      "  - Ken Tang and Ponnuthurai N. Suganthan and Xi Yao and A. Kai Qin.\n",
      "    Linear dimensionalityreduction using relevance weighted LDA. School of\n",
      "    Electrical and Electronic Engineering Nanyang Technological University.\n",
      "    2005.\n",
      "  - Claudio Gentile. A New Approximate Maximal Margin Classification\n",
      "    Algorithm. NIPS. 2000.\n"
     ]
    }
   ],
   "source": [
    "print(digits.DESCR) # 데이터 Describe\n",
    "\n",
    "# 데이터셋은 손글씨 이미지를 포함하고 있음 \n",
    "# 이미지는 8x8 이미지 픽셀로 이뤄져 있음\n",
    "# 10개 분류의 숫자를 의미함"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28ea70d4",
   "metadata": {},
   "source": [
    "### 4. train, test 데이터 분리"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "id": "507b8eb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "digits_X_train, digits_X_test, digits_y_train, digits_y_test = train_test_split(digits.data,\n",
    "                                                                                digits_label,\n",
    "                                                                                test_size=0.2,\n",
    "                                                                                random_state=7)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e42586be",
   "metadata": {},
   "source": [
    "#### 모델 함수 준비"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdec63a9",
   "metadata": {},
   "source": [
    "###### Decision Tree "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "id": "60bfcad5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "    \n",
    "def runDecisionTree(X_train, y_train, X_test):\n",
    "    decision_tree = DecisionTreeClassifier(random_state=32)\n",
    "    decision_tree.fit(X_train, y_train)\n",
    "    y_pred = decision_tree.predict(X_test)\n",
    "    return y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37765cc1",
   "metadata": {},
   "source": [
    "###### Random Forest 사용해 보기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "id": "ac530aeb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "def runRandomForest(X_train, y_train, X_test):\n",
    "    random_forest = RandomForestClassifier(random_state=32)\n",
    "    random_forest.fit(X_train, y_train)\n",
    "    y_pred = random_forest.predict(X_test)\n",
    "    return y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14426eb1",
   "metadata": {},
   "source": [
    "###### SVM 사용해 보기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "id": "fa593080",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import svm\n",
    "\n",
    "def runSVM(X_train, y_train, X_test):\n",
    "    svm_model = svm.SVC()\n",
    "    svm_model.fit(X_train, y_train)\n",
    "    y_pred = svm_model.predict(X_test)\n",
    "    return y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee6d2292",
   "metadata": {},
   "source": [
    "###### SGD Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "id": "0e2acc7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import SGDClassifier\n",
    "\n",
    "def runSGDClassifier(X_train, y_train, X_test):\n",
    "    sgd_model = SGDClassifier()\n",
    "    sgd_model.fit(X_train, y_train)\n",
    "    y_pred = sgd_model.predict(X_test)\n",
    "    return y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9936c9e1",
   "metadata": {},
   "source": [
    "###### Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "id": "7c661b56",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "def runLogisticRegression(X_train, y_train, X_test, max_iter=6000):\n",
    "    logistic_model = LogisticRegression(solver='lbfgs', max_iter=max_iter)\n",
    "    logistic_model.fit(X_train, y_train)\n",
    "    y_pred = logistic_model.predict(X_test)\n",
    "    return y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9541656c",
   "metadata": {},
   "source": [
    "### 5. 다양한 모델로 학습시켜보기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "id": "0cef4e60",
   "metadata": {},
   "outputs": [],
   "source": [
    "digits_y_pred_decision_tree = runDecisionTree(digits_X_train, digits_y_train, digits_X_test) # Decision Tree\n",
    "digits_y_pred_random_forest = runRandomForest(digits_X_train, digits_y_train, digits_X_test) # Random Forest\n",
    "digits_y_pred_svm = runSVM(digits_X_train, digits_y_train, digits_X_test) # SVM\n",
    "digits_y_pred_sgd = runSGDClassifier(digits_X_train, digits_y_train, digits_X_test) # SGD Classifier\n",
    "digits_y_pred_logistic_regression = runLogisticRegression(digits_X_train, digits_y_train, digits_X_test) # Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bf20037",
   "metadata": {},
   "source": [
    "### 6. 모델을 평가해보기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "id": "dc346c03",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision Tree\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      0.98      0.99        43\n",
      "           1       0.81      0.81      0.81        42\n",
      "           2       0.79      0.82      0.80        40\n",
      "           3       0.79      0.91      0.85        34\n",
      "           4       0.83      0.95      0.89        37\n",
      "           5       0.90      0.96      0.93        28\n",
      "           6       0.84      0.93      0.88        28\n",
      "           7       0.96      0.82      0.89        33\n",
      "           8       0.88      0.65      0.75        43\n",
      "           9       0.78      0.78      0.78        32\n",
      "\n",
      "    accuracy                           0.86       360\n",
      "   macro avg       0.86      0.86      0.86       360\n",
      "weighted avg       0.86      0.86      0.85       360\n",
      "\n",
      "-----------------------------------------------------------\n",
      "Random Forest\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      0.98      0.99        43\n",
      "           1       0.93      1.00      0.97        42\n",
      "           2       1.00      1.00      1.00        40\n",
      "           3       1.00      1.00      1.00        34\n",
      "           4       0.93      1.00      0.96        37\n",
      "           5       0.90      0.96      0.93        28\n",
      "           6       1.00      0.96      0.98        28\n",
      "           7       0.94      0.97      0.96        33\n",
      "           8       1.00      0.84      0.91        43\n",
      "           9       0.94      0.94      0.94        32\n",
      "\n",
      "    accuracy                           0.96       360\n",
      "   macro avg       0.96      0.96      0.96       360\n",
      "weighted avg       0.97      0.96      0.96       360\n",
      "\n",
      "-----------------------------------------------------------\n",
      "SVM\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00        43\n",
      "           1       0.95      1.00      0.98        42\n",
      "           2       1.00      1.00      1.00        40\n",
      "           3       1.00      1.00      1.00        34\n",
      "           4       1.00      1.00      1.00        37\n",
      "           5       0.93      1.00      0.97        28\n",
      "           6       1.00      1.00      1.00        28\n",
      "           7       1.00      1.00      1.00        33\n",
      "           8       1.00      0.93      0.96        43\n",
      "           9       1.00      0.97      0.98        32\n",
      "\n",
      "    accuracy                           0.99       360\n",
      "   macro avg       0.99      0.99      0.99       360\n",
      "weighted avg       0.99      0.99      0.99       360\n",
      "\n",
      "-----------------------------------------------------------\n",
      "SDG Classifier\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00        43\n",
      "           1       0.94      0.76      0.84        42\n",
      "           2       0.95      1.00      0.98        40\n",
      "           3       0.91      0.94      0.93        34\n",
      "           4       1.00      0.95      0.97        37\n",
      "           5       0.90      0.96      0.93        28\n",
      "           6       0.96      0.93      0.95        28\n",
      "           7       0.97      0.94      0.95        33\n",
      "           8       0.91      0.91      0.91        43\n",
      "           9       0.79      0.97      0.87        32\n",
      "\n",
      "    accuracy                           0.93       360\n",
      "   macro avg       0.93      0.94      0.93       360\n",
      "weighted avg       0.94      0.93      0.93       360\n",
      "\n",
      "-----------------------------------------------------------\n",
      "Logistic Regression\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00        43\n",
      "           1       0.95      0.95      0.95        42\n",
      "           2       0.98      1.00      0.99        40\n",
      "           3       0.94      0.97      0.96        34\n",
      "           4       1.00      1.00      1.00        37\n",
      "           5       0.79      0.96      0.87        28\n",
      "           6       1.00      0.96      0.98        28\n",
      "           7       0.97      0.97      0.97        33\n",
      "           8       0.92      0.84      0.88        43\n",
      "           9       0.97      0.88      0.92        32\n",
      "\n",
      "    accuracy                           0.95       360\n",
      "   macro avg       0.95      0.95      0.95       360\n",
      "weighted avg       0.96      0.95      0.95       360\n",
      "\n",
      "-----------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print('Decision Tree')\n",
    "print(classification_report(digits_y_test, digits_y_pred_decision_tree))\n",
    "print('-----------------------------------------------------------')\n",
    "\n",
    "print('Random Forest')\n",
    "print(classification_report(digits_y_test, digits_y_pred_random_forest))\n",
    "print('-----------------------------------------------------------')\n",
    "\n",
    "print('SVM')\n",
    "print(classification_report(digits_y_test, digits_y_pred_svm))\n",
    "print('-----------------------------------------------------------')\n",
    "\n",
    "print('SDG Classifier')\n",
    "print(classification_report(digits_y_test, digits_y_pred_sgd))\n",
    "print('-----------------------------------------------------------')\n",
    "\n",
    "print('Logistic Regression')\n",
    "print(classification_report(digits_y_test, digits_y_pred_logistic_regression))\n",
    "print('-----------------------------------------------------------')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "id": "78441970",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[43,  0,  0,  0,  0,  0,  0,  0,  0,  0],\n",
       "       [ 0, 42,  0,  0,  0,  0,  0,  0,  0,  0],\n",
       "       [ 0,  0, 40,  0,  0,  0,  0,  0,  0,  0],\n",
       "       [ 0,  0,  0, 34,  0,  0,  0,  0,  0,  0],\n",
       "       [ 0,  0,  0,  0, 37,  0,  0,  0,  0,  0],\n",
       "       [ 0,  0,  0,  0,  0, 28,  0,  0,  0,  0],\n",
       "       [ 0,  0,  0,  0,  0,  0, 28,  0,  0,  0],\n",
       "       [ 0,  0,  0,  0,  0,  0,  0, 33,  0,  0],\n",
       "       [ 0,  2,  0,  0,  0,  1,  0,  0, 40,  0],\n",
       "       [ 0,  0,  0,  0,  0,  1,  0,  0,  0, 31]])"
      ]
     },
     "execution_count": 208,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 가장 평가 수치가 높은 svm 기준 confusion 행렬 확인\n",
    "\n",
    "confusion_matrix(digits_y_test, digits_y_pred_svm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a79db97a",
   "metadata": {},
   "source": [
    "### 평가지표\n",
    "    \n",
    "손글씨 문제에서는 ***Accuracy*** 평가지표가 중요합니다.\n",
    "\n",
    "물론 데이터가 불균형한 경우에는 Accuracy에 허점이 있기 때문에\n",
    "\n",
    "학습 데이터 분류가 어떻게 분포되어 있는지 확인하는게 좋습니다.\n",
    "\n",
    "그래서 학습 데이터(=y_train)가 균형적으로 고루 분포되어 있는지 확인해보면, "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "id": "cf4dfca5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "0    135\n",
      "1    140\n",
      "2    137\n",
      "3    149\n",
      "4    144\n",
      "5    154\n",
      "6    153\n",
      "7    146\n",
      "8    131\n",
      "9    148\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "digits_y_train_df = pd.DataFrame(data=digits_y_train)\n",
    "train_grouped = digits_y_train_df.groupby(0)\n",
    "print(train_grouped.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2e85293",
   "metadata": {},
   "source": [
    "가장 적은 수는 8 (131개), 가장 많은 수는 5 (154개)로서, 크게 치중되는 것 없이 고루 분포되어 있습니다. \n",
    "\n",
    "또한 손글씨 데이터에서는 더 중요한 것을 따로 고르기보단, 각 분류가 올바르게 맞았는지를 보기 위함이 중요하다 생각합니다. \n",
    "\n",
    "1. 데이터가 클래스 별 고루 분포되어 있고\n",
    "2. 올바르게 잘 판단했는지 (TP, TN가 높은) \n",
    "\n",
    "따라서, 위의 두 기준으로 ***Accuracy*** 를 선택했습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "795f3598",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6c680a0",
   "metadata": {},
   "source": [
    "# 와인을 분류해 봅시다"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28f5e8c6",
   "metadata": {},
   "source": [
    "### 1. 필요한 모듈 import하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "db941373",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_wine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "256f7cd6",
   "metadata": {},
   "source": [
    "### 2. 필요한 데이터셋 준비"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "60f70a64",
   "metadata": {},
   "outputs": [],
   "source": [
    "wine = load_wine()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa1f2ba3",
   "metadata": {},
   "source": [
    "### 3. 데이터 이해하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "id": "18846726",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Target Names : [0 1 2 3 4 5 6 7 8 9]\n"
     ]
    }
   ],
   "source": [
    "wine_feature = wine.feature_names # Feature Data\n",
    "\n",
    "wine_label = wine.target # Label Data\n",
    "\n",
    "wine_target = wine.target_names # Target Names\n",
    "print(f'Target Names : {target_names}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "id": "4d9403ad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alcohol</th>\n",
       "      <th>malic_acid</th>\n",
       "      <th>ash</th>\n",
       "      <th>alcalinity_of_ash</th>\n",
       "      <th>magnesium</th>\n",
       "      <th>total_phenols</th>\n",
       "      <th>flavanoids</th>\n",
       "      <th>nonflavanoid_phenols</th>\n",
       "      <th>proanthocyanins</th>\n",
       "      <th>color_intensity</th>\n",
       "      <th>hue</th>\n",
       "      <th>od280/od315_of_diluted_wines</th>\n",
       "      <th>proline</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>14.23</td>\n",
       "      <td>1.71</td>\n",
       "      <td>2.43</td>\n",
       "      <td>15.6</td>\n",
       "      <td>127.0</td>\n",
       "      <td>2.80</td>\n",
       "      <td>3.06</td>\n",
       "      <td>0.28</td>\n",
       "      <td>2.29</td>\n",
       "      <td>5.64</td>\n",
       "      <td>1.04</td>\n",
       "      <td>3.92</td>\n",
       "      <td>1065.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>13.20</td>\n",
       "      <td>1.78</td>\n",
       "      <td>2.14</td>\n",
       "      <td>11.2</td>\n",
       "      <td>100.0</td>\n",
       "      <td>2.65</td>\n",
       "      <td>2.76</td>\n",
       "      <td>0.26</td>\n",
       "      <td>1.28</td>\n",
       "      <td>4.38</td>\n",
       "      <td>1.05</td>\n",
       "      <td>3.40</td>\n",
       "      <td>1050.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.16</td>\n",
       "      <td>2.36</td>\n",
       "      <td>2.67</td>\n",
       "      <td>18.6</td>\n",
       "      <td>101.0</td>\n",
       "      <td>2.80</td>\n",
       "      <td>3.24</td>\n",
       "      <td>0.30</td>\n",
       "      <td>2.81</td>\n",
       "      <td>5.68</td>\n",
       "      <td>1.03</td>\n",
       "      <td>3.17</td>\n",
       "      <td>1185.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>14.37</td>\n",
       "      <td>1.95</td>\n",
       "      <td>2.50</td>\n",
       "      <td>16.8</td>\n",
       "      <td>113.0</td>\n",
       "      <td>3.85</td>\n",
       "      <td>3.49</td>\n",
       "      <td>0.24</td>\n",
       "      <td>2.18</td>\n",
       "      <td>7.80</td>\n",
       "      <td>0.86</td>\n",
       "      <td>3.45</td>\n",
       "      <td>1480.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>13.24</td>\n",
       "      <td>2.59</td>\n",
       "      <td>2.87</td>\n",
       "      <td>21.0</td>\n",
       "      <td>118.0</td>\n",
       "      <td>2.80</td>\n",
       "      <td>2.69</td>\n",
       "      <td>0.39</td>\n",
       "      <td>1.82</td>\n",
       "      <td>4.32</td>\n",
       "      <td>1.04</td>\n",
       "      <td>2.93</td>\n",
       "      <td>735.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>173</th>\n",
       "      <td>13.71</td>\n",
       "      <td>5.65</td>\n",
       "      <td>2.45</td>\n",
       "      <td>20.5</td>\n",
       "      <td>95.0</td>\n",
       "      <td>1.68</td>\n",
       "      <td>0.61</td>\n",
       "      <td>0.52</td>\n",
       "      <td>1.06</td>\n",
       "      <td>7.70</td>\n",
       "      <td>0.64</td>\n",
       "      <td>1.74</td>\n",
       "      <td>740.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174</th>\n",
       "      <td>13.40</td>\n",
       "      <td>3.91</td>\n",
       "      <td>2.48</td>\n",
       "      <td>23.0</td>\n",
       "      <td>102.0</td>\n",
       "      <td>1.80</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.43</td>\n",
       "      <td>1.41</td>\n",
       "      <td>7.30</td>\n",
       "      <td>0.70</td>\n",
       "      <td>1.56</td>\n",
       "      <td>750.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>175</th>\n",
       "      <td>13.27</td>\n",
       "      <td>4.28</td>\n",
       "      <td>2.26</td>\n",
       "      <td>20.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>1.59</td>\n",
       "      <td>0.69</td>\n",
       "      <td>0.43</td>\n",
       "      <td>1.35</td>\n",
       "      <td>10.20</td>\n",
       "      <td>0.59</td>\n",
       "      <td>1.56</td>\n",
       "      <td>835.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>176</th>\n",
       "      <td>13.17</td>\n",
       "      <td>2.59</td>\n",
       "      <td>2.37</td>\n",
       "      <td>20.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>1.65</td>\n",
       "      <td>0.68</td>\n",
       "      <td>0.53</td>\n",
       "      <td>1.46</td>\n",
       "      <td>9.30</td>\n",
       "      <td>0.60</td>\n",
       "      <td>1.62</td>\n",
       "      <td>840.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>177</th>\n",
       "      <td>14.13</td>\n",
       "      <td>4.10</td>\n",
       "      <td>2.74</td>\n",
       "      <td>24.5</td>\n",
       "      <td>96.0</td>\n",
       "      <td>2.05</td>\n",
       "      <td>0.76</td>\n",
       "      <td>0.56</td>\n",
       "      <td>1.35</td>\n",
       "      <td>9.20</td>\n",
       "      <td>0.61</td>\n",
       "      <td>1.60</td>\n",
       "      <td>560.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>178 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     alcohol  malic_acid   ash  alcalinity_of_ash  magnesium  total_phenols  \\\n",
       "0      14.23        1.71  2.43               15.6      127.0           2.80   \n",
       "1      13.20        1.78  2.14               11.2      100.0           2.65   \n",
       "2      13.16        2.36  2.67               18.6      101.0           2.80   \n",
       "3      14.37        1.95  2.50               16.8      113.0           3.85   \n",
       "4      13.24        2.59  2.87               21.0      118.0           2.80   \n",
       "..       ...         ...   ...                ...        ...            ...   \n",
       "173    13.71        5.65  2.45               20.5       95.0           1.68   \n",
       "174    13.40        3.91  2.48               23.0      102.0           1.80   \n",
       "175    13.27        4.28  2.26               20.0      120.0           1.59   \n",
       "176    13.17        2.59  2.37               20.0      120.0           1.65   \n",
       "177    14.13        4.10  2.74               24.5       96.0           2.05   \n",
       "\n",
       "     flavanoids  nonflavanoid_phenols  proanthocyanins  color_intensity   hue  \\\n",
       "0          3.06                  0.28             2.29             5.64  1.04   \n",
       "1          2.76                  0.26             1.28             4.38  1.05   \n",
       "2          3.24                  0.30             2.81             5.68  1.03   \n",
       "3          3.49                  0.24             2.18             7.80  0.86   \n",
       "4          2.69                  0.39             1.82             4.32  1.04   \n",
       "..          ...                   ...              ...              ...   ...   \n",
       "173        0.61                  0.52             1.06             7.70  0.64   \n",
       "174        0.75                  0.43             1.41             7.30  0.70   \n",
       "175        0.69                  0.43             1.35            10.20  0.59   \n",
       "176        0.68                  0.53             1.46             9.30  0.60   \n",
       "177        0.76                  0.56             1.35             9.20  0.61   \n",
       "\n",
       "     od280/od315_of_diluted_wines  proline  label  \n",
       "0                            3.92   1065.0      0  \n",
       "1                            3.40   1050.0      0  \n",
       "2                            3.17   1185.0      0  \n",
       "3                            3.45   1480.0      0  \n",
       "4                            2.93    735.0      0  \n",
       "..                            ...      ...    ...  \n",
       "173                          1.74    740.0      2  \n",
       "174                          1.56    750.0      2  \n",
       "175                          1.56    835.0      2  \n",
       "176                          1.62    840.0      2  \n",
       "177                          1.60    560.0      2  \n",
       "\n",
       "[178 rows x 14 columns]"
      ]
     },
     "execution_count": 257,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 데이터 & feature & label 이해\n",
    "\n",
    "wine_data_df = pd.DataFrame(data=wine.data, columns=wine_feature)\n",
    "wine_data_df['label'] = wine_label\n",
    "wine_data_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "id": "0e6dc18b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    71\n",
       "0    59\n",
       "2    48\n",
       "dtype: int64"
      ]
     },
     "execution_count": 256,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Label 데이터 확인\n",
    "# :Class Distribution: class_0 (59), class_1 (71), class_2 (48)\n",
    "\n",
    "wine_label_df = pd.Series(wine_label)\n",
    "wine_label_df.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "52421502",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".. _wine_dataset:\n",
      "\n",
      "Wine recognition dataset\n",
      "------------------------\n",
      "\n",
      "**Data Set Characteristics:**\n",
      "\n",
      "    :Number of Instances: 178 (50 in each of three classes)\n",
      "    :Number of Attributes: 13 numeric, predictive attributes and the class\n",
      "    :Attribute Information:\n",
      " \t\t- Alcohol\n",
      " \t\t- Malic acid\n",
      " \t\t- Ash\n",
      "\t\t- Alcalinity of ash  \n",
      " \t\t- Magnesium\n",
      "\t\t- Total phenols\n",
      " \t\t- Flavanoids\n",
      " \t\t- Nonflavanoid phenols\n",
      " \t\t- Proanthocyanins\n",
      "\t\t- Color intensity\n",
      " \t\t- Hue\n",
      " \t\t- OD280/OD315 of diluted wines\n",
      " \t\t- Proline\n",
      "\n",
      "    - class:\n",
      "            - class_0\n",
      "            - class_1\n",
      "            - class_2\n",
      "\t\t\n",
      "    :Summary Statistics:\n",
      "    \n",
      "    ============================= ==== ===== ======= =====\n",
      "                                   Min   Max   Mean     SD\n",
      "    ============================= ==== ===== ======= =====\n",
      "    Alcohol:                      11.0  14.8    13.0   0.8\n",
      "    Malic Acid:                   0.74  5.80    2.34  1.12\n",
      "    Ash:                          1.36  3.23    2.36  0.27\n",
      "    Alcalinity of Ash:            10.6  30.0    19.5   3.3\n",
      "    Magnesium:                    70.0 162.0    99.7  14.3\n",
      "    Total Phenols:                0.98  3.88    2.29  0.63\n",
      "    Flavanoids:                   0.34  5.08    2.03  1.00\n",
      "    Nonflavanoid Phenols:         0.13  0.66    0.36  0.12\n",
      "    Proanthocyanins:              0.41  3.58    1.59  0.57\n",
      "    Colour Intensity:              1.3  13.0     5.1   2.3\n",
      "    Hue:                          0.48  1.71    0.96  0.23\n",
      "    OD280/OD315 of diluted wines: 1.27  4.00    2.61  0.71\n",
      "    Proline:                       278  1680     746   315\n",
      "    ============================= ==== ===== ======= =====\n",
      "\n",
      "    :Missing Attribute Values: None\n",
      "    :Class Distribution: class_0 (59), class_1 (71), class_2 (48)\n",
      "    :Creator: R.A. Fisher\n",
      "    :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)\n",
      "    :Date: July, 1988\n",
      "\n",
      "This is a copy of UCI ML Wine recognition datasets.\n",
      "https://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data\n",
      "\n",
      "The data is the results of a chemical analysis of wines grown in the same\n",
      "region in Italy by three different cultivators. There are thirteen different\n",
      "measurements taken for different constituents found in the three types of\n",
      "wine.\n",
      "\n",
      "Original Owners: \n",
      "\n",
      "Forina, M. et al, PARVUS - \n",
      "An Extendible Package for Data Exploration, Classification and Correlation. \n",
      "Institute of Pharmaceutical and Food Analysis and Technologies,\n",
      "Via Brigata Salerno, 16147 Genoa, Italy.\n",
      "\n",
      "Citation:\n",
      "\n",
      "Lichman, M. (2013). UCI Machine Learning Repository\n",
      "[https://archive.ics.uci.edu/ml]. Irvine, CA: University of California,\n",
      "School of Information and Computer Science. \n",
      "\n",
      ".. topic:: References\n",
      "\n",
      "  (1) S. Aeberhard, D. Coomans and O. de Vel, \n",
      "  Comparison of Classifiers in High Dimensional Settings, \n",
      "  Tech. Rep. no. 92-02, (1992), Dept. of Computer Science and Dept. of  \n",
      "  Mathematics and Statistics, James Cook University of North Queensland. \n",
      "  (Also submitted to Technometrics). \n",
      "\n",
      "  The data was used with many others for comparing various \n",
      "  classifiers. The classes are separable, though only RDA \n",
      "  has achieved 100% correct classification. \n",
      "  (RDA : 100%, QDA 99.4%, LDA 98.9%, 1NN 96.1% (z-transformed data)) \n",
      "  (All results using the leave-one-out technique) \n",
      "\n",
      "  (2) S. Aeberhard, D. Coomans and O. de Vel, \n",
      "  \"THE CLASSIFICATION PERFORMANCE OF RDA\" \n",
      "  Tech. Rep. no. 92-01, (1992), Dept. of Computer Science and Dept. of \n",
      "  Mathematics and Statistics, James Cook University of North Queensland. \n",
      "  (Also submitted to Journal of Chemometrics).\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(wine.DESCR) # 데이터 Describe"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20133b2b",
   "metadata": {},
   "source": [
    "### 4. train, test 데이터 분리"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "155b3b6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_X_train, wine_X_test, wine_y_train, wine_y_test = train_test_split(wine.data,\n",
    "                                                                        wine_label,\n",
    "                                                                        test_size=0.2,\n",
    "                                                                        random_state=7)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b57ba31",
   "metadata": {},
   "source": [
    "### 5. 다양한 모델로 학습시켜보기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "id": "1efd9e7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_y_pred_decision_tree = runDecisionTree(wine_X_train, wine_y_train, wine_X_test) # Decision Tree\n",
    "wine_y_pred_random_forest = runRandomForest(wine_X_train, wine_y_train, wine_X_test) # Random Forest\n",
    "wine_y_pred_svm = runSVM(wine_X_train, wine_y_train, wine_X_test) # SVM\n",
    "wine_y_pred_sgd = runSGDClassifier(wine_X_train, wine_y_train, wine_X_test) # SGD Classifier\n",
    "wine_y_pred_logistic_regression = runLogisticRegression(wine_X_train, wine_y_train, wine_X_test, 3000) # Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a7fa70c",
   "metadata": {},
   "source": [
    "### 6. 모델을 평가해보기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "id": "5faa50bc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision Tree\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00         7\n",
      "           1       0.89      1.00      0.94        17\n",
      "           2       1.00      0.83      0.91        12\n",
      "\n",
      "    accuracy                           0.94        36\n",
      "   macro avg       0.96      0.94      0.95        36\n",
      "weighted avg       0.95      0.94      0.94        36\n",
      "\n",
      "-----------------------------------------------------------\n",
      "Random Forest\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00         7\n",
      "           1       1.00      1.00      1.00        17\n",
      "           2       1.00      1.00      1.00        12\n",
      "\n",
      "    accuracy                           1.00        36\n",
      "   macro avg       1.00      1.00      1.00        36\n",
      "weighted avg       1.00      1.00      1.00        36\n",
      "\n",
      "-----------------------------------------------------------\n",
      "SVM\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.86      0.86      0.86         7\n",
      "           1       0.58      0.88      0.70        17\n",
      "           2       0.33      0.08      0.13        12\n",
      "\n",
      "    accuracy                           0.61        36\n",
      "   macro avg       0.59      0.61      0.56        36\n",
      "weighted avg       0.55      0.61      0.54        36\n",
      "\n",
      "-----------------------------------------------------------\n",
      "SDG Classifier\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      1.00      0.93         7\n",
      "           1       0.71      0.88      0.79        17\n",
      "           2       0.86      0.50      0.63        12\n",
      "\n",
      "    accuracy                           0.78        36\n",
      "   macro avg       0.82      0.79      0.78        36\n",
      "weighted avg       0.79      0.78      0.76        36\n",
      "\n",
      "-----------------------------------------------------------\n",
      "Logistic Regression\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00         7\n",
      "           1       0.94      1.00      0.97        17\n",
      "           2       1.00      0.92      0.96        12\n",
      "\n",
      "    accuracy                           0.97        36\n",
      "   macro avg       0.98      0.97      0.98        36\n",
      "weighted avg       0.97      0.97      0.97        36\n",
      "\n",
      "-----------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print('Decision Tree')\n",
    "print(classification_report(wine_y_test, wine_y_pred_decision_tree))\n",
    "print('-----------------------------------------------------------')\n",
    "\n",
    "print('Random Forest')\n",
    "print(classification_report(wine_y_test, wine_y_pred_random_forest))\n",
    "print('-----------------------------------------------------------')\n",
    "\n",
    "print('SVM')\n",
    "print(classification_report(wine_y_test, wine_y_pred_svm))\n",
    "print('-----------------------------------------------------------')\n",
    "\n",
    "print('SDG Classifier')\n",
    "print(classification_report(wine_y_test, wine_y_pred_sgd))\n",
    "print('-----------------------------------------------------------')\n",
    "\n",
    "print('Logistic Regression')\n",
    "print(classification_report(wine_y_test, wine_y_pred_logistic_regression))\n",
    "print('-----------------------------------------------------------')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d196c607",
   "metadata": {},
   "source": [
    "##### Confusion Matrix 확인"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "id": "8ba6e46e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 7,  0,  0],\n",
       "       [ 0, 17,  0],\n",
       "       [ 0,  0, 12]])"
      ]
     },
     "execution_count": 249,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(wine_y_test, wine_y_pred_random_forest)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd694e8a",
   "metadata": {},
   "source": [
    "```\n",
    "         C0  C1  C2\n",
    "    C0 [[ 7,  0,  0],\n",
    "    C1  [ 0, 17,  0],\n",
    "    C2  [ 0,  0, 12]]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 258,
   "id": "c1fec5bb",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "0    52\n",
      "1    54\n",
      "2    36\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "wine_y_train_df = pd.DataFrame(data=wine_y_train)\n",
    "wine_train_grouped = wine_y_train_df.groupby(0)\n",
    "print(wine_train_grouped.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a10226b5",
   "metadata": {},
   "source": [
    "### 평가지표 선택\n",
    "    \n",
    "와인 문제에서 평가지표는 정확도만 보고 판단할 수 없습니다.\n",
    "\n",
    "와인 문제는 Class0, Class1, Class2 각 와인을 분류하는 문제이고,\n",
    "\n",
    "특성(feature)이 다를 뿐이지 와인 분류마다 중요도 차이가 있지 않습니다.\n",
    "\n",
    ">The data is the results of a chemical analysis of wines grown in the same region in Italy by three different cultivators. There are thirteen different measurements taken for different constituents found in the three types of wine. (DESCR에서 확인한 내용)\n",
    "\n",
    "이러한 와인문제는 잘 분류했는지 못했는지 파악하해야 하며, 이 경우 정확도가 중요합니다.\n",
    "\n",
    "그러나, \n",
    "\n",
    "class_2 데이터가 다른 class_1, class_2에 비해 \n",
    "\n",
    "50~67% 정도 데이터 갯수가 적어 클래스 간 데이터 불균형이 있는 상태입니다.\n",
    "\n",
    "따라서, 정확도에 허점이 발생할 수 있으므로 **Precision, Recall, F1-score 등 다양한 평가지표를 살펴보는 것이 중요합니다.**\n",
    "\n",
    "\n",
    "\n",
    "번외) 굳이 1개 평가지표를 선택한다면 \n",
    "\n",
    "이처럼 클래스가 불균형한 경우 *F1-score* 를 보는 것이 중요하다 생각합니다.\n",
    "\n",
    "F1-score은 조화평균으로 지표를 평가하며 잘못 분류된 사례를 더 잘 측정하기 때문입니다.\n",
    "\n",
    "(관련 링크 참고 : https://medium.com/analytics-vidhya/accuracy-vs-f1-score-6258237beca2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b005f9e",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "335a8814",
   "metadata": {},
   "source": [
    "# 유방암 여부를 진단해 봅시다"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b69bdd2d",
   "metadata": {},
   "source": [
    "### 1. 필요한 모듈 import하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "3dc1fe11",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "226169b0",
   "metadata": {},
   "source": [
    "### 2. 유방암 데이터셋 준비"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "cce1a299",
   "metadata": {},
   "outputs": [],
   "source": [
    "cancer = load_breast_cancer()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "120e44a6",
   "metadata": {},
   "source": [
    "### 3. 데이터 이해하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "id": "70368a7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "cancer_feature = cancer.feature_names # Feature Data\n",
    "\n",
    "cancer_label = cancer.target # Label Data\n",
    "\n",
    "cancer_target = cancer.target_names # Target Names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "id": "6ad9e8a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "cancer_df = pd.DataFrame(data=cancer.data, columns=cancer_feature)\n",
    "cancer_df['label'] = cancer_label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "id": "08c04f2f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean radius</th>\n",
       "      <th>mean texture</th>\n",
       "      <th>mean perimeter</th>\n",
       "      <th>mean area</th>\n",
       "      <th>mean smoothness</th>\n",
       "      <th>mean compactness</th>\n",
       "      <th>mean concavity</th>\n",
       "      <th>mean concave points</th>\n",
       "      <th>mean symmetry</th>\n",
       "      <th>mean fractal dimension</th>\n",
       "      <th>...</th>\n",
       "      <th>worst texture</th>\n",
       "      <th>worst perimeter</th>\n",
       "      <th>worst area</th>\n",
       "      <th>worst smoothness</th>\n",
       "      <th>worst compactness</th>\n",
       "      <th>worst concavity</th>\n",
       "      <th>worst concave points</th>\n",
       "      <th>worst symmetry</th>\n",
       "      <th>worst fractal dimension</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>17.990</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.30010</td>\n",
       "      <td>0.147100</td>\n",
       "      <td>0.2419</td>\n",
       "      <td>0.07871</td>\n",
       "      <td>...</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.16220</td>\n",
       "      <td>0.66560</td>\n",
       "      <td>0.71190</td>\n",
       "      <td>0.26540</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20.570</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.08690</td>\n",
       "      <td>0.070170</td>\n",
       "      <td>0.1812</td>\n",
       "      <td>0.05667</td>\n",
       "      <td>...</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.12380</td>\n",
       "      <td>0.18660</td>\n",
       "      <td>0.24160</td>\n",
       "      <td>0.18600</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>19.690</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.19740</td>\n",
       "      <td>0.127900</td>\n",
       "      <td>0.2069</td>\n",
       "      <td>0.05999</td>\n",
       "      <td>...</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.14440</td>\n",
       "      <td>0.42450</td>\n",
       "      <td>0.45040</td>\n",
       "      <td>0.24300</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11.420</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.24140</td>\n",
       "      <td>0.105200</td>\n",
       "      <td>0.2597</td>\n",
       "      <td>0.09744</td>\n",
       "      <td>...</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.20980</td>\n",
       "      <td>0.86630</td>\n",
       "      <td>0.68690</td>\n",
       "      <td>0.25750</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20.290</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.19800</td>\n",
       "      <td>0.104300</td>\n",
       "      <td>0.1809</td>\n",
       "      <td>0.05883</td>\n",
       "      <td>...</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.13740</td>\n",
       "      <td>0.20500</td>\n",
       "      <td>0.40000</td>\n",
       "      <td>0.16250</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>13.540</td>\n",
       "      <td>14.36</td>\n",
       "      <td>87.46</td>\n",
       "      <td>566.3</td>\n",
       "      <td>0.09779</td>\n",
       "      <td>0.08129</td>\n",
       "      <td>0.06664</td>\n",
       "      <td>0.047810</td>\n",
       "      <td>0.1885</td>\n",
       "      <td>0.05766</td>\n",
       "      <td>...</td>\n",
       "      <td>19.26</td>\n",
       "      <td>99.70</td>\n",
       "      <td>711.2</td>\n",
       "      <td>0.14400</td>\n",
       "      <td>0.17730</td>\n",
       "      <td>0.23900</td>\n",
       "      <td>0.12880</td>\n",
       "      <td>0.2977</td>\n",
       "      <td>0.07259</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>13.080</td>\n",
       "      <td>15.71</td>\n",
       "      <td>85.63</td>\n",
       "      <td>520.0</td>\n",
       "      <td>0.10750</td>\n",
       "      <td>0.12700</td>\n",
       "      <td>0.04568</td>\n",
       "      <td>0.031100</td>\n",
       "      <td>0.1967</td>\n",
       "      <td>0.06811</td>\n",
       "      <td>...</td>\n",
       "      <td>20.49</td>\n",
       "      <td>96.09</td>\n",
       "      <td>630.5</td>\n",
       "      <td>0.13120</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.18900</td>\n",
       "      <td>0.07283</td>\n",
       "      <td>0.3184</td>\n",
       "      <td>0.08183</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>9.504</td>\n",
       "      <td>12.44</td>\n",
       "      <td>60.34</td>\n",
       "      <td>273.9</td>\n",
       "      <td>0.10240</td>\n",
       "      <td>0.06492</td>\n",
       "      <td>0.02956</td>\n",
       "      <td>0.020760</td>\n",
       "      <td>0.1815</td>\n",
       "      <td>0.06905</td>\n",
       "      <td>...</td>\n",
       "      <td>15.66</td>\n",
       "      <td>65.13</td>\n",
       "      <td>314.9</td>\n",
       "      <td>0.13240</td>\n",
       "      <td>0.11480</td>\n",
       "      <td>0.08867</td>\n",
       "      <td>0.06227</td>\n",
       "      <td>0.2450</td>\n",
       "      <td>0.07773</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>13.030</td>\n",
       "      <td>18.42</td>\n",
       "      <td>82.61</td>\n",
       "      <td>523.8</td>\n",
       "      <td>0.08983</td>\n",
       "      <td>0.03766</td>\n",
       "      <td>0.02562</td>\n",
       "      <td>0.029230</td>\n",
       "      <td>0.1467</td>\n",
       "      <td>0.05863</td>\n",
       "      <td>...</td>\n",
       "      <td>22.81</td>\n",
       "      <td>84.46</td>\n",
       "      <td>545.9</td>\n",
       "      <td>0.09701</td>\n",
       "      <td>0.04619</td>\n",
       "      <td>0.04833</td>\n",
       "      <td>0.05013</td>\n",
       "      <td>0.1987</td>\n",
       "      <td>0.06169</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>8.196</td>\n",
       "      <td>16.84</td>\n",
       "      <td>51.71</td>\n",
       "      <td>201.9</td>\n",
       "      <td>0.08600</td>\n",
       "      <td>0.05943</td>\n",
       "      <td>0.01588</td>\n",
       "      <td>0.005917</td>\n",
       "      <td>0.1769</td>\n",
       "      <td>0.06503</td>\n",
       "      <td>...</td>\n",
       "      <td>21.96</td>\n",
       "      <td>57.26</td>\n",
       "      <td>242.2</td>\n",
       "      <td>0.12970</td>\n",
       "      <td>0.13570</td>\n",
       "      <td>0.06880</td>\n",
       "      <td>0.02564</td>\n",
       "      <td>0.3105</td>\n",
       "      <td>0.07409</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    mean radius  mean texture  mean perimeter  mean area  mean smoothness  \\\n",
       "0        17.990         10.38          122.80     1001.0          0.11840   \n",
       "1        20.570         17.77          132.90     1326.0          0.08474   \n",
       "2        19.690         21.25          130.00     1203.0          0.10960   \n",
       "3        11.420         20.38           77.58      386.1          0.14250   \n",
       "4        20.290         14.34          135.10     1297.0          0.10030   \n",
       "19       13.540         14.36           87.46      566.3          0.09779   \n",
       "20       13.080         15.71           85.63      520.0          0.10750   \n",
       "21        9.504         12.44           60.34      273.9          0.10240   \n",
       "37       13.030         18.42           82.61      523.8          0.08983   \n",
       "46        8.196         16.84           51.71      201.9          0.08600   \n",
       "\n",
       "    mean compactness  mean concavity  mean concave points  mean symmetry  \\\n",
       "0            0.27760         0.30010             0.147100         0.2419   \n",
       "1            0.07864         0.08690             0.070170         0.1812   \n",
       "2            0.15990         0.19740             0.127900         0.2069   \n",
       "3            0.28390         0.24140             0.105200         0.2597   \n",
       "4            0.13280         0.19800             0.104300         0.1809   \n",
       "19           0.08129         0.06664             0.047810         0.1885   \n",
       "20           0.12700         0.04568             0.031100         0.1967   \n",
       "21           0.06492         0.02956             0.020760         0.1815   \n",
       "37           0.03766         0.02562             0.029230         0.1467   \n",
       "46           0.05943         0.01588             0.005917         0.1769   \n",
       "\n",
       "    mean fractal dimension  ...  worst texture  worst perimeter  worst area  \\\n",
       "0                  0.07871  ...          17.33           184.60      2019.0   \n",
       "1                  0.05667  ...          23.41           158.80      1956.0   \n",
       "2                  0.05999  ...          25.53           152.50      1709.0   \n",
       "3                  0.09744  ...          26.50            98.87       567.7   \n",
       "4                  0.05883  ...          16.67           152.20      1575.0   \n",
       "19                 0.05766  ...          19.26            99.70       711.2   \n",
       "20                 0.06811  ...          20.49            96.09       630.5   \n",
       "21                 0.06905  ...          15.66            65.13       314.9   \n",
       "37                 0.05863  ...          22.81            84.46       545.9   \n",
       "46                 0.06503  ...          21.96            57.26       242.2   \n",
       "\n",
       "    worst smoothness  worst compactness  worst concavity  \\\n",
       "0            0.16220            0.66560          0.71190   \n",
       "1            0.12380            0.18660          0.24160   \n",
       "2            0.14440            0.42450          0.45040   \n",
       "3            0.20980            0.86630          0.68690   \n",
       "4            0.13740            0.20500          0.40000   \n",
       "19           0.14400            0.17730          0.23900   \n",
       "20           0.13120            0.27760          0.18900   \n",
       "21           0.13240            0.11480          0.08867   \n",
       "37           0.09701            0.04619          0.04833   \n",
       "46           0.12970            0.13570          0.06880   \n",
       "\n",
       "    worst concave points  worst symmetry  worst fractal dimension  label  \n",
       "0                0.26540          0.4601                  0.11890      0  \n",
       "1                0.18600          0.2750                  0.08902      0  \n",
       "2                0.24300          0.3613                  0.08758      0  \n",
       "3                0.25750          0.6638                  0.17300      0  \n",
       "4                0.16250          0.2364                  0.07678      0  \n",
       "19               0.12880          0.2977                  0.07259      1  \n",
       "20               0.07283          0.3184                  0.08183      1  \n",
       "21               0.06227          0.2450                  0.07773      1  \n",
       "37               0.05013          0.1987                  0.06169      1  \n",
       "46               0.02564          0.3105                  0.07409      1  \n",
       "\n",
       "[10 rows x 31 columns]"
      ]
     },
     "execution_count": 219,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cancer_df.groupby('label').head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "id": "9d1387be",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cancer label names : ['malignant' 'benign']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1    357\n",
       "0    212\n",
       "dtype: int64"
      ]
     },
     "execution_count": 226,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(f'cancer label names : {cancer_target}')\n",
    "\n",
    "cancer_label_df = pd.Series(cancer_label)\n",
    "cancer_label_df.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "id": "f1f022b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".. _breast_cancer_dataset:\n",
      "\n",
      "Breast cancer wisconsin (diagnostic) dataset\n",
      "--------------------------------------------\n",
      "\n",
      "**Data Set Characteristics:**\n",
      "\n",
      "    :Number of Instances: 569\n",
      "\n",
      "    :Number of Attributes: 30 numeric, predictive attributes and the class\n",
      "\n",
      "    :Attribute Information:\n",
      "        - radius (mean of distances from center to points on the perimeter)\n",
      "        - texture (standard deviation of gray-scale values)\n",
      "        - perimeter\n",
      "        - area\n",
      "        - smoothness (local variation in radius lengths)\n",
      "        - compactness (perimeter^2 / area - 1.0)\n",
      "        - concavity (severity of concave portions of the contour)\n",
      "        - concave points (number of concave portions of the contour)\n",
      "        - symmetry\n",
      "        - fractal dimension (\"coastline approximation\" - 1)\n",
      "\n",
      "        The mean, standard error, and \"worst\" or largest (mean of the three\n",
      "        worst/largest values) of these features were computed for each image,\n",
      "        resulting in 30 features.  For instance, field 0 is Mean Radius, field\n",
      "        10 is Radius SE, field 20 is Worst Radius.\n",
      "\n",
      "        - class:\n",
      "                - WDBC-Malignant\n",
      "                - WDBC-Benign\n",
      "\n",
      "    :Summary Statistics:\n",
      "\n",
      "    ===================================== ====== ======\n",
      "                                           Min    Max\n",
      "    ===================================== ====== ======\n",
      "    radius (mean):                        6.981  28.11\n",
      "    texture (mean):                       9.71   39.28\n",
      "    perimeter (mean):                     43.79  188.5\n",
      "    area (mean):                          143.5  2501.0\n",
      "    smoothness (mean):                    0.053  0.163\n",
      "    compactness (mean):                   0.019  0.345\n",
      "    concavity (mean):                     0.0    0.427\n",
      "    concave points (mean):                0.0    0.201\n",
      "    symmetry (mean):                      0.106  0.304\n",
      "    fractal dimension (mean):             0.05   0.097\n",
      "    radius (standard error):              0.112  2.873\n",
      "    texture (standard error):             0.36   4.885\n",
      "    perimeter (standard error):           0.757  21.98\n",
      "    area (standard error):                6.802  542.2\n",
      "    smoothness (standard error):          0.002  0.031\n",
      "    compactness (standard error):         0.002  0.135\n",
      "    concavity (standard error):           0.0    0.396\n",
      "    concave points (standard error):      0.0    0.053\n",
      "    symmetry (standard error):            0.008  0.079\n",
      "    fractal dimension (standard error):   0.001  0.03\n",
      "    radius (worst):                       7.93   36.04\n",
      "    texture (worst):                      12.02  49.54\n",
      "    perimeter (worst):                    50.41  251.2\n",
      "    area (worst):                         185.2  4254.0\n",
      "    smoothness (worst):                   0.071  0.223\n",
      "    compactness (worst):                  0.027  1.058\n",
      "    concavity (worst):                    0.0    1.252\n",
      "    concave points (worst):               0.0    0.291\n",
      "    symmetry (worst):                     0.156  0.664\n",
      "    fractal dimension (worst):            0.055  0.208\n",
      "    ===================================== ====== ======\n",
      "\n",
      "    :Missing Attribute Values: None\n",
      "\n",
      "    :Class Distribution: 212 - Malignant, 357 - Benign\n",
      "\n",
      "    :Creator:  Dr. William H. Wolberg, W. Nick Street, Olvi L. Mangasarian\n",
      "\n",
      "    :Donor: Nick Street\n",
      "\n",
      "    :Date: November, 1995\n",
      "\n",
      "This is a copy of UCI ML Breast Cancer Wisconsin (Diagnostic) datasets.\n",
      "https://goo.gl/U2Uwz2\n",
      "\n",
      "Features are computed from a digitized image of a fine needle\n",
      "aspirate (FNA) of a breast mass.  They describe\n",
      "characteristics of the cell nuclei present in the image.\n",
      "\n",
      "Separating plane described above was obtained using\n",
      "Multisurface Method-Tree (MSM-T) [K. P. Bennett, \"Decision Tree\n",
      "Construction Via Linear Programming.\" Proceedings of the 4th\n",
      "Midwest Artificial Intelligence and Cognitive Science Society,\n",
      "pp. 97-101, 1992], a classification method which uses linear\n",
      "programming to construct a decision tree.  Relevant features\n",
      "were selected using an exhaustive search in the space of 1-4\n",
      "features and 1-3 separating planes.\n",
      "\n",
      "The actual linear program used to obtain the separating plane\n",
      "in the 3-dimensional space is that described in:\n",
      "[K. P. Bennett and O. L. Mangasarian: \"Robust Linear\n",
      "Programming Discrimination of Two Linearly Inseparable Sets\",\n",
      "Optimization Methods and Software 1, 1992, 23-34].\n",
      "\n",
      "This database is also available through the UW CS ftp server:\n",
      "\n",
      "ftp ftp.cs.wisc.edu\n",
      "cd math-prog/cpo-dataset/machine-learn/WDBC/\n",
      "\n",
      ".. topic:: References\n",
      "\n",
      "   - W.N. Street, W.H. Wolberg and O.L. Mangasarian. Nuclear feature extraction \n",
      "     for breast tumor diagnosis. IS&T/SPIE 1993 International Symposium on \n",
      "     Electronic Imaging: Science and Technology, volume 1905, pages 861-870,\n",
      "     San Jose, CA, 1993.\n",
      "   - O.L. Mangasarian, W.N. Street and W.H. Wolberg. Breast cancer diagnosis and \n",
      "     prognosis via linear programming. Operations Research, 43(4), pages 570-577, \n",
      "     July-August 1995.\n",
      "   - W.H. Wolberg, W.N. Street, and O.L. Mangasarian. Machine learning techniques\n",
      "     to diagnose breast cancer from fine-needle aspirates. Cancer Letters 77 (1994) \n",
      "     163-171.\n"
     ]
    }
   ],
   "source": [
    "print(cancer.DESCR) # 데이터 Describe"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ebbd15f",
   "metadata": {},
   "source": [
    "### 4. train, test 데이터 분리"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "097877cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "cancer_X_train, cancer_X_test, cancer_y_train, cancer_y_test = train_test_split(cancer.data,\n",
    "                                                                                cancer_label,\n",
    "                                                                                test_size=0.2,\n",
    "                                                                                random_state=7)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e61e7d5",
   "metadata": {},
   "source": [
    "### 5. 다양한 모델로 학습시켜보기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "id": "4d4c0f3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "cancer_y_pred_decision_tree = runDecisionTree(cancer_X_train, cancer_y_train, cancer_X_test) # Decision Tree\n",
    "cancer_y_pred_random_forest = runRandomForest(cancer_X_train, cancer_y_train, cancer_X_test) # Random Forest\n",
    "cancer_y_pred_svm = runSVM(cancer_X_train, cancer_y_train, cancer_X_test) # SVM\n",
    "cancer_y_pred_sgd = runSGDClassifier(cancer_X_train, cancer_y_train, cancer_X_test) # SGD Classifier\n",
    "cancer_y_pred_logistic_regression = runLogisticRegression(cancer_X_train, cancer_y_train, cancer_X_test, 2000) # Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a488ea9e",
   "metadata": {},
   "source": [
    "### 6. 모델을 평가해보기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "id": "35c4246d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision Tree\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "   malignant       0.92      0.82      0.87        40\n",
      "      benign       0.91      0.96      0.93        74\n",
      "\n",
      "    accuracy                           0.91       114\n",
      "   macro avg       0.91      0.89      0.90       114\n",
      "weighted avg       0.91      0.91      0.91       114\n",
      "\n",
      "-----------------------------------------------------------\n",
      "Random Forest\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "   malignant       1.00      1.00      1.00        40\n",
      "      benign       1.00      1.00      1.00        74\n",
      "\n",
      "    accuracy                           1.00       114\n",
      "   macro avg       1.00      1.00      1.00       114\n",
      "weighted avg       1.00      1.00      1.00       114\n",
      "\n",
      "-----------------------------------------------------------\n",
      "SVM\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "   malignant       1.00      0.72      0.84        40\n",
      "      benign       0.87      1.00      0.93        74\n",
      "\n",
      "    accuracy                           0.90       114\n",
      "   macro avg       0.94      0.86      0.89       114\n",
      "weighted avg       0.92      0.90      0.90       114\n",
      "\n",
      "-----------------------------------------------------------\n",
      "SDG Classifier\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "   malignant       0.64      0.95      0.77        40\n",
      "      benign       0.96      0.72      0.82        74\n",
      "\n",
      "    accuracy                           0.80       114\n",
      "   macro avg       0.80      0.83      0.79       114\n",
      "weighted avg       0.85      0.80      0.80       114\n",
      "\n",
      "-----------------------------------------------------------\n",
      "Logistic Regression\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "   malignant       1.00      0.85      0.92        40\n",
      "      benign       0.93      1.00      0.96        74\n",
      "\n",
      "    accuracy                           0.95       114\n",
      "   macro avg       0.96      0.93      0.94       114\n",
      "weighted avg       0.95      0.95      0.95       114\n",
      "\n",
      "-----------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print('Decision Tree')\n",
    "print(classification_report(cancer_y_test, cancer_y_pred_decision_tree, target_names=cancer_target))\n",
    "print('-----------------------------------------------------------')\n",
    "\n",
    "print('Random Forest')\n",
    "print(classification_report(cancer_y_test, cancer_y_pred_random_forest, target_names=cancer_target))\n",
    "print('-----------------------------------------------------------')\n",
    "\n",
    "print('SVM')\n",
    "print(classification_report(cancer_y_test, cancer_y_pred_svm, target_names=cancer_target))\n",
    "print('-----------------------------------------------------------')\n",
    "\n",
    "print('SDG Classifier')\n",
    "print(classification_report(cancer_y_test, cancer_y_pred_sgd, target_names=cancer_target))\n",
    "print('-----------------------------------------------------------')\n",
    "\n",
    "print('Logistic Regression')\n",
    "print(classification_report(cancer_y_test, cancer_y_pred_logistic_regression, target_names=cancer_target))\n",
    "print('-----------------------------------------------------------')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "id": "ec97438c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "confusion_matrix : \n",
      "[[40  0]\n",
      " [ 0 74]]\n",
      "\n",
      "y_test 각 label 별 개수 : \n",
      "1    74\n",
      "0    40\n",
      "dtype: int64\n",
      "\n",
      "y_pred 각 label 별 개수 : \n",
      "1    74\n",
      "0    40\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# 가장 평가 수치가 높은 Random Forest 기준 confusion 행렬 확인 (+ y 데이터 톺파보기)\n",
    "\n",
    "confusion = confusion_matrix(cancer_y_test, cancer_y_pred_random_forest)\n",
    "print(f'confusion_matrix : \\n{confusion}\\n')\n",
    "\n",
    "cancer_y_test_df = pd.Series(cancer_y_test)\n",
    "print(f'y_test 각 label 별 개수 : \\n{cancer_y_test_df.value_counts()}\\n')\n",
    "\n",
    "cancer_y_pred_random_forest_df = pd.Series(cancer_y_pred_random_forest)\n",
    "print(f'y_pred 각 label 별 개수 : \\n{cancer_y_pred_random_forest_df.value_counts()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34d479eb",
   "metadata": {},
   "source": [
    "##### Label 데이터 - 0, 1 확인\n",
    "1. cancer.DESC에서는 다음과 같이 작성되어 있었고,\n",
    "    ```\n",
    "        :Class Distribution: 212 - Malignant, 357 - Benign\n",
    "    ```\n",
    "2. 실제 데이터셋의 라벨 개수는 다음과 같습니다.\n",
    "    ```\n",
    "        cancer_label_df = pd.Series(cancer_label)\n",
    "        cancer_label_df.value_counts()\n",
    "        # 1    357   -> Benign\n",
    "        # 0    212   -> Malignant\n",
    "    ```\n",
    "3. 따라서, **Malignant[0], Benign[1]**, 배열 순으로 class가 적용됨을 확인함\n",
    "\n",
    "##### Confusion Matrix 확인\n",
    "   - Malignant[0] : Positive\n",
    "   - Benign[1] : Negative\n",
    "   ```\n",
    "            0   1\n",
    "        0 [[40  0]\n",
    "        1 [ 0 74]]\n",
    "    ```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "524a47c7",
   "metadata": {},
   "source": [
    "### 평가지표 선택\n",
    "    \n",
    "유방암 분류에서는 ***Recall*** 평가지표가 중요합니다. \n",
    "\n",
    "유방암 분류 데이터를 살펴보면,\n",
    "\n",
    "- Malignant : 암 (Positive)\n",
    "- Benign : 암이 아닌 종양 (Negative)\n",
    "\n",
    "분류로 나뉘는데요.\n",
    "\n",
    "암 환자에게 암이라고 정확하게 맞추는 것도 중요하지만, \n",
    "\n",
    "오진을 하더라도 Malignant을 Benign(**Negative**)이라 잘못 오진(**False**)을 하면\n",
    "\n",
    "이는 가장 큰 문제를 발생시키기 때문에\n",
    "\n",
    "유방암 문제에서는 ***FN***가 가장 낮은 지표를 찾아야 합니다.\n",
    "\n",
    "따라서, FP가 가장 낮을 때 점수가 가장 높은 Recall을 선택했습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b89bd51",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e484d2d",
   "metadata": {},
   "source": [
    "# 회고\n",
    "\n",
    "### 1. 어려웠던 점\n",
    "- 이미지를 이해할 때, 데이터 포인트 하나하나가 내가 보던 그 이미지로 연결이 잘 안되어 이해가 바로 안됐던 것 같습니다.\n",
    "- 데이터마다 적절한 평가지표를 선정해야하는데, 의사결정 기준을 도출하는 것부터 지표 선정까지 전반적으로 어려웠던 것 같습니다.\n",
    "\n",
    "### 2. 알아낸 점 혹은 아직 모호한 점\n",
    "\n",
    "##### 알아낸 점\n",
    "- 라벨 배열에 입력된 순(인덱스 순서)대로 평가지표에 순서대로 나타난다는 것을 알았습니다.\n",
    "- 데이터, attributes, classes 등 데이터에 관한 모든 내용들이 평가지표에 영향을 미칠 수 있다는 점을 알았습니다.\n",
    "\n",
    "##### 모호한 점\n",
    "- 이미지 픽셀 8x8을 flatten해서 64로 만들고, 이게 attributes라는 점이 잘 와닿지 않았습니다.\n",
    "\n",
    "### 3. 시도한 것들\n",
    "- LogisticRegression(solver='lbfgs', max_iter=max_iter)할 때, max_iter관련 오류가 발생했었습니다. 검색한 결과, iter를 다 돌아도 처리가 완료되지 않아 발생하는 문제였고, 조금씩 숫자를 올려보면서 적정 iter 값을 찾아 문제를 해결했습니다.\n",
    "- 수업 때 배운 train data의 균형/불균형성을 확인하고 싶었습니다. 추가적으로 데이터를 살펴보고자 pandas DataFrame으로 train data의 각 class별 갯수를 groupby하여 데이터 균형/불균형성을 확인해보았습니다. \n",
    "- 평가지표를 선택할 때 0과 1이 어떤걸 지칭하는지 헷갈려서 데이터 갯수를 맞춰봤습니다. DESC에 작성된 라벨 갯수와 pandas.Series(cancer_label)의 value_counts()로 각 라벨의 갯수를 비교하고, Confusion Matrix 갯수도 맞춰보면서 Malignant -> 0, Benign -> 1 을 눈으로 직접 확인했습니다.\n",
    "- 데이터가 불균형한 경우에는 어떤 평가지표를 선택하는 것이 좋은지 리서치했습니다.\n",
    "\n",
    "### 4. 루브릭 지표를 달성하지 못한 부분과 이유\n",
    "- 루브릭 지표 모두 달성했습니다.\n",
    "\n",
    "### 5. 자기 다짐\n",
    "- 모델을 적용하고 평가지표을 도출하는 과정을 빠르게 해보긴 했지만, 사실 각 모델에 대해 명확히 이해하지 못한건 사실입니다.\n",
    "- 용어도 다 낯설고 온갖 다 처음 배우는 거라 그런걸테니.. 앞으로 하나씩 배우고 쌓아가면서 여러 모델들의 이해도를 높여야겠다 생각했습니다."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python ds38",
   "language": "python",
   "name": "ds38"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}