5 - 다중 감성 분석.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ben Trevett 의 [Multi-class Sentiment Analysis](https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/5%20-%20Multi-class%20Sentiment%20Analysis.ipynb) 튜토리얼을 연습해본다.\n",
    "\n",
    "이 튜토리얼에서는 Convolutional Neural Networks for Sentence Classification 논문에서 제시된 CNN 기반 모델을 이용해서 다중 감정 분석을 실시한다. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 전처리\n",
    "\n",
    "레이블의 갯수가 두개 이상인 데이터의 분류를 해보자. 그런데 한글 데이터셋을 못구해서... 일단 영어 데이터로 진행하겠다. 이 경우는 6개의 클래스가 있는 `TREC` 데이터를 분석할 것이며, 따라서 `LABEL` 필드에서 `dtype`을 선언할 필요가 없다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torchtext import data\n",
    "from torchtext import datasets\n",
    "import random\n",
    "\n",
    "SEED = 1234\n",
    "\n",
    "torch.manual_seed(SEED)\n",
    "torch.backends.cudnn.determinisitic = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEXT = data.Field(tokenize='spacy')\n",
    "LABEL = data.LabelField()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data, test_data = datasets.TREC.splits(TEXT, LABEL, fine_grained=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data, valid_data = train_data.split(random_state=random.seed(SEED))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3816, 1636, 500)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_data), len(valid_data), len(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "데이터를 살펴보자."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': ['Who',\n",
       "  'is',\n",
       "  'section',\n",
       "  'manager',\n",
       "  'for',\n",
       "  'guidance',\n",
       "  'and',\n",
       "  'control',\n",
       "  'systems',\n",
       "  'at',\n",
       "  'JPL',\n",
       "  '?'],\n",
       " 'label': 'HUM'}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vars(train_data[-3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': ['What',\n",
       "  'year',\n",
       "  'did',\n",
       "  'Jack',\n",
       "  'Nicklaus',\n",
       "  'join',\n",
       "  'the',\n",
       "  'Professional',\n",
       "  'Golfers',\n",
       "  'Association',\n",
       "  'tour',\n",
       "  '?'],\n",
       " 'label': 'NUM'}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vars(train_data[-2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "단어장을 만들자. 훈련 데이터 갯수가 많지 않아서 최대 갯수 설정은 할 필요가 없긴 하나 그냥 하겠다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      ".vector_cache/glove.6B.zip: 862MB [17:28, 822kB/s]                                \n",
      "100%|█████████▉| 399754/400000 [00:30<00:00, 39243.72it/s]"
     ]
    }
   ],
   "source": [
    "MAX_VOCAB_SIZE = 25000\n",
    "\n",
    "TEXT.build_vocab(train_data, \n",
    "                max_size = MAX_VOCAB_SIZE,\n",
    "                vectors = 'glove.6B.100d',\n",
    "                unk_init = torch.Tensor.normal_\n",
    "                )\n",
    "LABEL.build_vocab(train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "데이터의 라벨은 다음과 같은 6개이다.\n",
    "\n",
    "* `HUM` for questions about humans\n",
    "* `ENTY` for questions about entities\n",
    "* `DESC` for questions asking you for a description\n",
    "* `NUM` for questions where the answer is numerical\n",
    "* `LOC` for questions where the answer is a location\n",
    "* `ABBR` for questions asking about abbreviations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "defaultdict(None, {'HUM': 0, 'ENTY': 1, 'DESC': 2, 'NUM': 3, 'LOC': 4, 'ABBR': 5})\n"
     ]
    }
   ],
   "source": [
    "print(LABEL.vocab.stoi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이터레이터 설정하자"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 64\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
    "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(\n",
    "    (train_data, valid_data, test_data),\n",
    "    batch_size = BATCH_SIZE,\n",
    "    device = device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([19, 64])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(iter(train_iterator)).text.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CNN 모델을 사용해보자."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class CNN(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim, dropout, pad_idx):\n",
    "        super().__init__()\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
    "        self.convs = nn.ModuleList([nn.Conv2d(in_channels=1,\n",
    "                                             out_channels=n_filters,\n",
    "                                             kernel_size = (fs, embedding_dim))\n",
    "                                    for fs in filter_sizes])\n",
    "        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self, text):\n",
    "        #text = [sent_len, batch_size]\n",
    "        text = text.permute(1,0)\n",
    "        #text = [batch_size, sent_len]\n",
    "        \n",
    "        embedded = self.embedding(text)\n",
    "        #embedded = [batch_size, sent_len, emb_dim]\n",
    "        \n",
    "        embedded = embedded.unsqueeze(1)\n",
    "        #embedded = [batch_size, 1, sent_len, emb_dim]\n",
    "        \n",
    "        conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs]\n",
    "        #conv_n = [batch_size, n_filters, sent_len - fs[n]]\n",
    "        \n",
    "        pooled = [F.max_pool1d(conv, conv.shape[2]).squeeze(2) for conv in conved]\n",
    "        #pooled_n = [batch_size, n_filters]\n",
    "        \n",
    "        cat = self.dropout(torch.cat(pooled, dim=1))\n",
    "        #cat = [batch_size, n_filters * len(filter_size)]\n",
    "        \n",
    "        return self.fc(cat)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "하이퍼파라미터 설정하자. 데이터셋 사이즈가 작으므로 필터 사이즈를 작게 하자."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "INPUT_DIM = len(TEXT.vocab)\n",
    "EMBEDDING_DIM = 100\n",
    "N_FILTERS = 100\n",
    "FILTER_SIZES = [2,3,4]\n",
    "OUTPUT_DIM = len(LABEL.vocab)\n",
    "DROPOUT = 0.5\n",
    "PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = CNN(INPUT_DIM, EMBEDDING_DIM, N_FILTERS, FILTER_SIZES, OUTPUT_DIM, DROPOUT, PAD_IDX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "파라미터 갯수를 세보자."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "이 모델은 842,406개의 훈련 가능한 파라미터가 있습니다.\n"
     ]
    }
   ],
   "source": [
    "def count_parameters(model):\n",
    "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
    "\n",
    "print(f'이 모델은 {count_parameters(model):,}개의 훈련 가능한 파라미터가 있습니다.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "사전훈련된 단어 벡터를 적용하자."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.1117, -0.4966,  0.1631,  ...,  1.2647, -0.2753, -0.1325],\n",
       "        [-0.8555, -0.7208,  1.3755,  ...,  0.0825, -1.1314,  0.3997],\n",
       "        [ 0.1638,  0.6046,  1.0789,  ..., -0.3140,  0.1844,  0.3624],\n",
       "        ...,\n",
       "        [-0.3110, -0.3398,  1.0308,  ...,  0.5317,  0.2836, -0.0640],\n",
       "        [ 0.0091,  0.2810,  0.7356,  ..., -0.7508,  0.8967, -0.7631],\n",
       "        [ 0.4306,  1.2011,  0.0873,  ...,  0.8817,  0.3722,  0.3458]])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pretrained_embeddings = TEXT.vocab.vectors\n",
    "model.embedding.weight.data.copy_(pretrained_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "unk_token과 pad_token의 벡터를 0으로 만들자."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]\n",
    "model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)\n",
    "model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이 모델은 기존처럼 하나의 확률값을 출력으로 하는 것이 아니라 각 클래스별 확률을 출력하므로, 손실 함수를 기존 `BCEWithLogitsLoss`에서 `CrossEntropyLoss`로 수정해야한다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "model = model.to(device)\n",
    "criterion = criterion.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "각 클래스별 스코어를 출력으로 받으면 가장 높은 점수를 가지는 클래스를 반환하도록 하는 정확도 함수를 만들자. 다만 배치로 작업하고 있으므로, 배치의 평균으로 구하자."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def categorical_accuracy(preds, y):\n",
    "    \"\"\"배치별 정확도를 출력하는 함수\"\"\"\n",
    "    max_preds = preds.argmax(dim=1, keepdim=True)\n",
    "    correct = max_preds.squeeze(1).eq(y)\n",
    "    return correct.sum() / torch.FloatTensor([y.shape[0]])    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[5.0000, 0.1000, 0.3000],\n",
       "        [2.3000, 3.1000, 0.7000]])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred = torch.tensor([[5.0,0.1,0.3],[2.3,3.1,0.7]])\n",
    "pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([2]), torch.Size([2, 1]))"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred.argmax(dim=1).shape, pred.argmax(dim=1, keepdim=True).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([False,  True])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred.argmax(dim=1, keepdim=True).squeeze(1).eq(torch.tensor([1,1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "훈련 함수를 만들자."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, iterator, optimizer, criterion):\n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "    model.train()\n",
    "    \n",
    "    for batch in iterator:\n",
    "        optimizer.zero_grad()\n",
    "        predictions = model(batch.text)\n",
    "        loss = criterion(predictions, batch.label)\n",
    "        acc = categorical_accuracy(predictions, batch.label)\n",
    "        \n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "        epoch_loss += loss.item()\n",
    "        epoch_acc += acc.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "eval함수도 비슷"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def eval(model, iterator, criterion):\n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "    model.eval()\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        for batch in iterator:\n",
    "            predictions = model(batch.text)\n",
    "            loss = criterion(predictions, batch.label)\n",
    "            acc = categorical_accuracy(predictions, batch.label)\n",
    "\n",
    "            epoch_loss += loss.item()\n",
    "            epoch_acc += acc.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "훈련시키자!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Epoch Time: 0m 1s\n",
      "\tTrain Loss: 0.292 | Train Acc: 91.53%\n",
      "\tValid Loss: 0.163 | Valid Acc: 96.73%\n",
      "Epoch: 02 | Epoch Time: 0m 1s\n",
      "\tTrain Loss: 0.219 | Train Acc: 93.82%\n",
      "\tValid Loss: 0.117 | Valid Acc: 98.16%\n",
      "Epoch: 03 | Epoch Time: 0m 1s\n",
      "\tTrain Loss: 0.167 | Train Acc: 95.46%\n",
      "\tValid Loss: 0.084 | Valid Acc: 98.96%\n",
      "Epoch: 04 | Epoch Time: 0m 1s\n",
      "\tTrain Loss: 0.134 | Train Acc: 96.47%\n",
      "\tValid Loss: 0.065 | Valid Acc: 98.85%\n",
      "Epoch: 05 | Epoch Time: 0m 1s\n",
      "\tTrain Loss: 0.104 | Train Acc: 97.44%\n",
      "\tValid Loss: 0.046 | Valid Acc: 99.48%\n"
     ]
    }
   ],
   "source": [
    "N_EPOCHS = 5\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(N_EPOCHS):\n",
    "    start_time = time.time()\n",
    "    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)\n",
    "    valid_loss, valid_acc = eval(model, train_iterator, criterion)\n",
    "    \n",
    "    end_time = time.time()\n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "    \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'tut5-model.pt')\n",
    "        \n",
    "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
    "    print(f'\\tValid Loss: {valid_loss:.3f} | Valid Acc: {valid_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "테스트 셋에 검증해보자."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Loss: 0.311 | Test Acc: 88.81%\n"
     ]
    }
   ],
   "source": [
    "model.load_state_dict(torch.load('tut5-model.pt'))\n",
    "\n",
    "test_loss, test_acc = eval(model, test_iterator, criterion)\n",
    "\n",
    "print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "임의의 문장에 대해 판정해보자."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en')\n",
    "\n",
    "def predict_class(model, sentence, min_len=4):\n",
    "    model.eval()\n",
    "    tokenized = [tok.text for tok in nlp.tokenizer(sentence)]\n",
    "    if len(tokenized) < min_len:\n",
    "        tokenized += ['<pad>'] * (min_len - len(tokenized))\n",
    "    indexed = [TEXT.vocab.stoi[t] for t in tokenized]\n",
    "    tensor = torch.LongTensor(indexed).to(device)\n",
    "    tensor = tensor.unsqueeze(1)\n",
    "    preds = model(tensor)\n",
    "    max_preds = preds.argmax(dim=1)\n",
    "    return max_preds.item()    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "실제로 확인해보자."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted class is: 0 = HUM\n"
     ]
    }
   ],
   "source": [
    "q = 'The one who told me the truth is Inhwan Lee.'\n",
    "pred_class = predict_class(model, q)\n",
    "print(f'Predicted class is: {pred_class} = {LABEL.vocab.itos[pred_class]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted class is: 4 = LOC\n"
     ]
    }
   ],
   "source": [
    "q = 'The central part of Korea is called Gyung-gi do.'\n",
    "pred_class = predict_class(model, q)\n",
    "print(f'Predicted class is: {pred_class} = {LABEL.vocab.itos[pred_class]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "굿굿!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}