pruning-bert.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `nn.utils.prune` 모듈로 BERT 파라미터 Pruning 해보기\n",
    "\n",
    "최근 **자연어 처리**를 비롯한 많은 분야에서 태스크 성능을 높이기 위해 모델의 크기를 늘리는 연구를 하고 있습니다. 그리고 이를 통해 비약적인 성능의 개선을 이끌어 내긴 했지만, 지나치게 커진 모델 크기로 인해 **실제 어플리케이션**에 배포를 하기 어려워졌다는 **사이드 이펙트**가 생겨났습니다.\n",
    "\n",
    "그러나 이전부터 모델의 크기 문제를 타파하기 위한 해결책으로 다양한 모델 압축 기법들이 연구되어 왔고, 모델이 예측 값을 내는데 큰 영향을 미치지 않는 파라미터들을 가지치기 하는 **Pruning**도 이러한 모델 압축 기법 중 하나입니다.\n",
    "\n",
    "본 튜토리얼에서는 [**Pruning Tutorial**](https://pytorch.org/tutorials/intermediate/pruning_tutorial.html#pruning-a-module)을 참조해 `torch.nn.utils.prune` 모듈을 활용한 **BERT** 파라미터 **Pruning** 사용 예에 대해 살펴보도록 합니다. \n",
    "\n",
    "**BERT**의 경우 **Transformer** 모델을 기반으로 하지만, 최근 많은 연구들에서 **Multi-Head Attention** 혹은 더 나아가 무의미한 **Attention map**의 효능에 대해 의문을 품고 있습니다. 그리고 이에 따라 헤드의 갯수를 줄인다거나, 어텐션 매트릭스를 **Pruning** 한다거나 등의 다양한 시도가 행해지고 있습니다. \n",
    "\n",
    "우리는 이 중, 어텐션 매트릭스를 **Pruning** 해보는 시간을 본 튜토리얼을 통해 가져볼 것입니다.\n",
    "\n",
    "_p.s. Model Compression 기법들은 BERT와 같은 큰 모델의 등장 이전에도 꾸준히 연구되어 오던 분야입니다._\n",
    "\n",
    "_p.s. 본 튜토리얼을 주제로 한 [**포스트**](https://huffon.github.io/2020/03/15/torch-pruning/)도 읽어보심을 추천드립니다._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "먼저 튜토리얼에 사용될 라이브러리들을 임포트합니다. 본 튜토리얼을 수행하기 위해서는 **1.4.0** 버전 이상의 `torch`가 필요합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.4.0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn.utils.prune as prune\n",
    "\n",
    "torch.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 모델 로드\n",
    "\n",
    "본 튜토리얼에서는 **SKT Brain**이 훈련시키고 [@monologg](https://github.com/monologg)님이 Hugging face 사의 [**transformer**](https://github.com/huggingface/transformers) 라이브러리에 등록한 **KoBERT** 모델을 활용합니다. \n",
    "\n",
    "해당 모델에 대해 더 자세한 설명을 원하시는 분은 [**본 저장소**](https://github.com/SKTBrain/KoBERT)를 참조하시면 좋을 것 같습니다.\n",
    "\n",
    "이제 모델을 불러오도록 합니다. 사전에 모델이 설치되어 있지 않았다면 아래 명령어를 통해 모델이 다운로드 및 로드가 됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from transformers import BertModel\n",
    "model = BertModel.from_pretrained('monologg/kobert')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 모델 조사\n",
    "\n",
    "**Pruning**이 아직 적용되지 않은 **KoBERT** 모델을 먼저 살펴보도록 합니다.\n",
    "\n",
    "해당 모델은 아래에서 살펴볼 수 있듯 `embeddings`, `pooler`를 비롯한 12개의 `BertEncoder` 모듈들로 구성되어 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BertModel(\n",
       "  (embeddings): BertEmbeddings(\n",
       "    (word_embeddings): Embedding(8002, 768, padding_idx=0)\n",
       "    (position_embeddings): Embedding(512, 768)\n",
       "    (token_type_embeddings): Embedding(2, 768)\n",
       "    (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "    (dropout): Dropout(p=0.1, inplace=False)\n",
       "  )\n",
       "  (encoder): BertEncoder(\n",
       "    (layer): ModuleList(\n",
       "      (0): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (1): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (2): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (3): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (4): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (5): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (6): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (7): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (8): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (9): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (10): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (11): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (pooler): BertPooler(\n",
       "    (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "    (activation): Tanh()\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "우리는 위 모듈들 중**인코더 레이어**들에만 관심이 있으므로 해당 모듈만 분석해봅시다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ModuleList(\n",
       "  (0): BertLayer(\n",
       "    (attention): BertAttention(\n",
       "      (self): BertSelfAttention(\n",
       "        (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "      (output): BertSelfOutput(\n",
       "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "    )\n",
       "    (intermediate): BertIntermediate(\n",
       "      (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "    )\n",
       "    (output): BertOutput(\n",
       "      (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "  )\n",
       "  (1): BertLayer(\n",
       "    (attention): BertAttention(\n",
       "      (self): BertSelfAttention(\n",
       "        (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "      (output): BertSelfOutput(\n",
       "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "    )\n",
       "    (intermediate): BertIntermediate(\n",
       "      (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "    )\n",
       "    (output): BertOutput(\n",
       "      (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "  )\n",
       "  (2): BertLayer(\n",
       "    (attention): BertAttention(\n",
       "      (self): BertSelfAttention(\n",
       "        (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "      (output): BertSelfOutput(\n",
       "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "    )\n",
       "    (intermediate): BertIntermediate(\n",
       "      (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "    )\n",
       "    (output): BertOutput(\n",
       "      (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "  )\n",
       "  (3): BertLayer(\n",
       "    (attention): BertAttention(\n",
       "      (self): BertSelfAttention(\n",
       "        (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "      (output): BertSelfOutput(\n",
       "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "    )\n",
       "    (intermediate): BertIntermediate(\n",
       "      (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "    )\n",
       "    (output): BertOutput(\n",
       "      (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "  )\n",
       "  (4): BertLayer(\n",
       "    (attention): BertAttention(\n",
       "      (self): BertSelfAttention(\n",
       "        (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "      (output): BertSelfOutput(\n",
       "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "    )\n",
       "    (intermediate): BertIntermediate(\n",
       "      (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "    )\n",
       "    (output): BertOutput(\n",
       "      (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "  )\n",
       "  (5): BertLayer(\n",
       "    (attention): BertAttention(\n",
       "      (self): BertSelfAttention(\n",
       "        (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "      (output): BertSelfOutput(\n",
       "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "    )\n",
       "    (intermediate): BertIntermediate(\n",
       "      (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "    )\n",
       "    (output): BertOutput(\n",
       "      (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "  )\n",
       "  (6): BertLayer(\n",
       "    (attention): BertAttention(\n",
       "      (self): BertSelfAttention(\n",
       "        (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "      (output): BertSelfOutput(\n",
       "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "    )\n",
       "    (intermediate): BertIntermediate(\n",
       "      (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "    )\n",
       "    (output): BertOutput(\n",
       "      (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "  )\n",
       "  (7): BertLayer(\n",
       "    (attention): BertAttention(\n",
       "      (self): BertSelfAttention(\n",
       "        (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "      (output): BertSelfOutput(\n",
       "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "    )\n",
       "    (intermediate): BertIntermediate(\n",
       "      (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "    )\n",
       "    (output): BertOutput(\n",
       "      (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "  )\n",
       "  (8): BertLayer(\n",
       "    (attention): BertAttention(\n",
       "      (self): BertSelfAttention(\n",
       "        (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "      (output): BertSelfOutput(\n",
       "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "    )\n",
       "    (intermediate): BertIntermediate(\n",
       "      (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "    )\n",
       "    (output): BertOutput(\n",
       "      (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "  )\n",
       "  (9): BertLayer(\n",
       "    (attention): BertAttention(\n",
       "      (self): BertSelfAttention(\n",
       "        (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "      (output): BertSelfOutput(\n",
       "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "    )\n",
       "    (intermediate): BertIntermediate(\n",
       "      (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "    )\n",
       "    (output): BertOutput(\n",
       "      (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "  )\n",
       "  (10): BertLayer(\n",
       "    (attention): BertAttention(\n",
       "      (self): BertSelfAttention(\n",
       "        (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "      (output): BertSelfOutput(\n",
       "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "    )\n",
       "    (intermediate): BertIntermediate(\n",
       "      (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "    )\n",
       "    (output): BertOutput(\n",
       "      (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "  )\n",
       "  (11): BertLayer(\n",
       "    (attention): BertAttention(\n",
       "      (self): BertSelfAttention(\n",
       "        (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "      (output): BertSelfOutput(\n",
       "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "    )\n",
       "    (intermediate): BertIntermediate(\n",
       "      (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "    )\n",
       "    (output): BertOutput(\n",
       "      (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.encoder.layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "앞으로 나올 예제 진행을 위해 모든 레이어를 사용할 필요는 없습니다.\n",
    "\n",
    "먼저, 12개의 인코더 레이어 중 **최하단 레이어**만 살펴보도록 합시다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BertLayer(\n",
       "  (attention): BertAttention(\n",
       "    (self): BertSelfAttention(\n",
       "      (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "      (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "      (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "    (output): BertSelfOutput(\n",
       "      (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "  )\n",
       "  (intermediate): BertIntermediate(\n",
       "    (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "  )\n",
       "  (output): BertOutput(\n",
       "    (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "    (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "    (dropout): Dropout(p=0.1, inplace=False)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.encoder.layer[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "마지막으로 우리는 위 구성 중 `attention` 모듈에만 관심이 있으므로 `attention` 모듈을 살펴봅시다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BertSelfAttention(\n",
       "  (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "  (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "  (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "  (dropout): Dropout(p=0.1, inplace=False)\n",
       ")"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.encoder.layer[0].attention.self"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`BertSelfAttention`은 `query`, `key`, `value` 그리고 `dropout`으로 구성되어 있습니다. \n",
    "\n",
    "이제 q, k, v에 **Pruning**을 진행하기 앞서 `key`에 **Pruning**을 적용하는 연습을 해보기 위해 실험 모듈을 `key`로 지정합니다.\n",
    "\n",
    "현재 `key`의 `named_parameters()` 내에는 `weight`와 `bias`가 존재하며, `named_buffers()`에는 아무것도 존재하지 않습니다.\n",
    "\n",
    "<br/>\n",
    "\n",
    "_cf. What is the difference between register_parameter and register_buffer in PyTorch?_\n",
    "\n",
    "> This is typically used to register a buffer that should not to be considered a model parameter. <br/> For example, BatchNorm’s running_mean is not a parameter, but is part of the persistent state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "module = model.encoder.layer[0].attention.self.key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('weight', Parameter containing:\n",
       "  tensor([[ 0.0267,  0.0427, -0.0713,  ..., -0.0344,  0.0055,  0.0277],\n",
       "          [ 0.0228,  0.0752, -0.0429,  ...,  0.0860,  0.1789,  0.0289],\n",
       "          [-0.0079, -0.0309, -0.0990,  ..., -0.0316,  0.0335, -0.0635],\n",
       "          ...,\n",
       "          [ 0.0040, -0.0065, -0.0924,  ..., -0.0338, -0.0250, -0.1278],\n",
       "          [ 0.0134,  0.0512,  0.0694,  ..., -0.0096, -0.0297,  0.0294],\n",
       "          [-0.0243, -0.0592, -0.0535,  ..., -0.0318, -0.0714,  0.0376]],\n",
       "         requires_grad=True)), ('bias', Parameter containing:\n",
       "  tensor([ 5.4592e-08,  2.9247e-08, -2.0385e-08, -2.8358e-07, -9.8012e-08,\n",
       "          -5.8178e-08, -1.3005e-07, -5.6738e-08, -2.4501e-08, -4.8126e-08,\n",
       "           8.5868e-08, -6.5552e-08, -1.5294e-07, -6.0779e-08, -1.0340e-08,\n",
       "          -1.0260e-07, -1.4049e-07, -5.1941e-08, -1.7012e-07, -6.5405e-08,\n",
       "          -1.5491e-07, -1.1515e-07, -2.5359e-08,  4.9711e-08,  1.3979e-07,\n",
       "          -8.6575e-08, -8.9431e-08,  1.3025e-07, -2.3191e-07, -6.7193e-08,\n",
       "           1.1002e-07, -6.4796e-08, -1.4740e-07, -1.9011e-08,  2.2245e-07,\n",
       "          -1.4329e-07, -2.8459e-08,  7.1192e-08, -5.4687e-08, -1.0020e-07,\n",
       "          -1.4137e-07, -4.4915e-08, -2.6025e-08, -2.9728e-07, -8.7707e-08,\n",
       "          -3.9435e-08, -5.6708e-08, -4.0600e-08,  1.3973e-07, -2.7347e-07,\n",
       "          -1.3187e-07,  1.3556e-08, -1.4675e-08,  4.6401e-08, -5.2453e-08,\n",
       "          -6.1351e-08, -8.3668e-08,  1.2962e-07,  2.1639e-07,  7.9439e-08,\n",
       "          -4.9919e-08, -6.5474e-08,  1.9281e-08, -4.2483e-08,  1.6391e-07,\n",
       "          -3.8939e-07, -1.1450e-07, -3.8202e-07, -2.1552e-07,  3.9563e-07,\n",
       "           3.0277e-07,  1.6662e-07,  1.6972e-07, -1.3551e-07, -1.0182e-07,\n",
       "           2.8733e-07, -7.0817e-08, -3.8927e-07,  3.3062e-07,  1.4522e-07,\n",
       "          -1.2677e-07,  2.0442e-08, -1.0237e-08, -3.1870e-07,  4.2408e-08,\n",
       "           2.6782e-07,  1.2641e-08, -5.0443e-08,  1.5858e-07, -3.8922e-07,\n",
       "          -1.9792e-08, -3.1154e-08,  1.8157e-07,  6.6326e-09,  9.1542e-08,\n",
       "           4.1303e-09, -3.1614e-07, -1.3147e-07, -2.2811e-07,  1.1269e-07,\n",
       "          -2.5746e-07,  1.4798e-07,  4.5211e-08, -2.6614e-08,  3.9736e-07,\n",
       "          -2.6781e-07, -2.2600e-07,  3.2725e-07, -2.3893e-07,  2.4561e-07,\n",
       "          -2.0945e-07, -4.9683e-07, -6.8960e-08, -1.0985e-07, -1.9155e-08,\n",
       "           2.7619e-07, -7.8056e-08,  6.6799e-07,  3.3654e-07,  7.9260e-08,\n",
       "           1.8013e-07,  3.6409e-07,  5.8692e-08,  2.9595e-07,  1.3185e-07,\n",
       "          -1.5979e-07, -3.2274e-07,  1.1820e-07, -6.7861e-08,  6.2519e-08,\n",
       "           1.7694e-08,  1.9007e-07, -1.2253e-08,  9.4781e-08,  1.1964e-07,\n",
       "          -1.5201e-07, -2.0908e-07,  2.1751e-07,  1.3110e-07, -7.6005e-08,\n",
       "          -7.2351e-08, -1.1120e-07,  1.1059e-08, -1.1045e-08,  8.7099e-08,\n",
       "           1.6686e-07, -1.0117e-07, -2.8408e-07, -5.9637e-08,  8.9130e-09,\n",
       "          -2.2259e-08, -2.8968e-08,  7.7852e-08,  9.6652e-08, -9.8998e-08,\n",
       "          -1.9577e-07, -7.1940e-08,  1.5101e-08,  7.1112e-08,  3.1372e-08,\n",
       "           1.1861e-07,  2.4077e-08,  1.6269e-07,  3.9347e-09, -3.9538e-09,\n",
       "          -7.8518e-08,  1.1114e-07,  1.1625e-07,  1.1282e-08,  5.3560e-08,\n",
       "          -7.6504e-08, -6.0267e-08, -1.1799e-08,  8.6065e-08,  3.1305e-08,\n",
       "          -4.4863e-08, -2.1979e-07,  2.2306e-08, -5.4133e-09, -2.5605e-07,\n",
       "           6.2860e-08, -7.5285e-08, -1.1065e-07, -1.5208e-07, -3.0479e-08,\n",
       "           4.9316e-08,  6.7766e-08,  1.0247e-07, -9.5905e-08, -8.5965e-08,\n",
       "           2.1083e-08,  2.2197e-07,  5.0754e-08, -2.0659e-07, -1.0336e-07,\n",
       "          -9.2374e-08, -5.1872e-08, -4.9566e-07,  2.9554e-08,  2.0362e-07,\n",
       "          -2.6180e-09,  3.6847e-08, -1.4453e-07, -2.3543e-07,  1.4833e-07,\n",
       "           9.3070e-08,  7.9025e-08,  1.1281e-08,  1.0711e-07,  9.2672e-08,\n",
       "           4.4602e-08,  6.2573e-08,  1.8780e-08,  3.1025e-08, -4.8195e-08,\n",
       "          -1.5173e-08,  1.1228e-07, -1.7452e-07, -2.5543e-08, -1.2804e-07,\n",
       "          -3.5630e-07,  1.3448e-08, -2.9886e-08,  1.5605e-08, -9.5864e-08,\n",
       "           4.6114e-08,  1.2234e-07, -2.8767e-07,  1.9457e-08,  9.4918e-08,\n",
       "           1.6320e-07, -5.9371e-08, -1.8035e-07,  5.4930e-08, -7.1661e-08,\n",
       "           6.3309e-08,  2.8829e-08,  7.0693e-08,  4.5149e-08,  5.5471e-08,\n",
       "          -3.9404e-08,  7.4631e-08, -7.3951e-08,  4.0582e-08,  3.2118e-07,\n",
       "          -2.2044e-07, -7.3377e-09, -6.1323e-08,  1.4230e-07,  6.0617e-09,\n",
       "           7.6287e-08,  2.1655e-07,  8.8323e-09,  1.6086e-08, -4.0542e-08,\n",
       "          -1.1028e-07,  7.0081e-08,  1.3175e-07,  1.4077e-07, -1.0854e-07,\n",
       "           5.4646e-08,  2.7159e-08,  3.7363e-08,  5.6481e-09, -1.8462e-07,\n",
       "           5.1507e-08, -2.4222e-07, -1.7078e-07,  2.6885e-08, -3.7777e-08,\n",
       "          -5.4775e-08, -3.3475e-08,  1.8043e-08, -2.6002e-08,  1.0623e-07,\n",
       "          -2.9295e-08,  1.1175e-08,  3.4062e-07, -9.4083e-08, -1.3122e-08,\n",
       "           1.2139e-08,  4.2698e-08,  5.5309e-08, -3.2358e-08, -1.2346e-07,\n",
       "           2.6396e-09,  1.1075e-07, -4.0735e-08, -1.7003e-07,  4.3231e-08,\n",
       "          -1.5420e-07, -1.5166e-08,  2.9520e-08, -3.1258e-10,  1.0987e-07,\n",
       "           9.1626e-08, -7.3850e-08, -4.4047e-08,  1.2439e-07,  1.1803e-07,\n",
       "           1.7810e-07, -3.2616e-09, -7.2168e-08, -9.8500e-08,  1.8917e-07,\n",
       "           6.1860e-09, -4.9969e-09, -8.3460e-10, -1.4427e-08, -1.4171e-07,\n",
       "          -1.8042e-07,  5.7994e-08,  8.9171e-08, -5.3812e-08, -1.1186e-08,\n",
       "           1.2069e-07, -4.1306e-08,  3.8549e-08, -2.6021e-08,  5.5914e-08,\n",
       "           2.0648e-07,  2.2912e-08, -7.7152e-08,  5.2057e-08, -2.8716e-07,\n",
       "          -9.0171e-08, -2.3865e-08,  2.3779e-09, -1.6981e-07, -7.5877e-08,\n",
       "           2.6154e-07, -6.5519e-08, -1.1477e-07, -9.7396e-08, -2.5694e-08,\n",
       "          -1.0117e-07, -7.5133e-09, -1.1856e-07,  2.6017e-07, -5.2667e-08,\n",
       "           1.0610e-08,  1.9343e-08, -3.6094e-08,  1.1599e-07, -5.0230e-08,\n",
       "          -3.3382e-07,  2.1465e-07, -2.1061e-07, -3.4796e-09,  1.8067e-08,\n",
       "          -3.9974e-08,  5.6306e-08,  9.8680e-08,  1.4153e-07, -8.4846e-08,\n",
       "           2.4724e-07, -4.6981e-08,  2.4887e-07, -2.6745e-07,  9.5903e-08,\n",
       "          -2.8251e-07, -2.5444e-07, -8.0403e-08, -3.3439e-08,  1.4418e-07,\n",
       "           1.8163e-08, -1.1089e-07, -1.7096e-07,  4.3642e-09,  1.1574e-07,\n",
       "           6.3896e-08, -2.0590e-07,  2.8891e-08,  9.9786e-08,  1.1999e-07,\n",
       "           2.5415e-07, -7.2567e-08, -6.5372e-09,  1.6629e-07, -1.2370e-07,\n",
       "          -2.9782e-08,  1.8479e-07, -3.4119e-07, -4.5975e-08, -2.2810e-07,\n",
       "           3.2521e-07,  3.5040e-07, -2.5999e-07,  5.7259e-08, -9.0412e-08,\n",
       "          -1.6055e-07, -2.1732e-08,  1.2432e-07, -1.1365e-07,  4.9454e-08,\n",
       "           1.7514e-07,  8.0093e-08, -2.0677e-07,  1.1858e-07, -2.5169e-07,\n",
       "          -1.3862e-07, -3.5102e-08,  4.9936e-07,  1.2548e-07,  2.8573e-07,\n",
       "          -2.6150e-07,  2.4200e-07,  1.4818e-07,  8.9595e-08, -1.0800e-08,\n",
       "           1.1552e-08,  3.3775e-08,  4.6794e-07, -1.4711e-07, -9.5149e-08,\n",
       "          -6.9137e-08,  1.7641e-07,  1.9165e-07,  1.4456e-07, -6.6487e-08,\n",
       "          -1.7922e-07,  7.5856e-10,  1.7300e-07,  2.0469e-08, -4.2650e-07,\n",
       "          -3.5908e-08, -6.5810e-08, -3.7828e-08,  6.3509e-08,  3.0708e-07,\n",
       "           1.0413e-07, -5.5479e-08, -1.8942e-07,  9.0802e-08, -9.6585e-09,\n",
       "          -2.8615e-08,  4.9223e-08,  5.2463e-08, -7.0254e-08,  2.7888e-07,\n",
       "           4.1418e-08, -3.4391e-08,  5.9028e-09,  1.3962e-07,  2.4130e-07,\n",
       "          -3.6720e-07, -4.4258e-08,  1.4455e-07, -1.6950e-09, -8.9174e-08,\n",
       "          -4.3999e-08, -1.0885e-07,  2.7086e-07, -2.1972e-07,  1.0028e-07,\n",
       "          -2.0821e-07, -9.5543e-08, -5.7404e-08,  1.2352e-07, -6.8864e-08,\n",
       "           1.2697e-07,  1.2360e-07,  2.7013e-07, -8.9481e-09, -1.3829e-07,\n",
       "           8.3031e-08, -2.8425e-07, -2.9014e-07, -1.2196e-07, -1.0869e-07,\n",
       "          -2.1149e-07,  3.4489e-08, -1.4093e-07, -7.7525e-08,  2.1968e-07,\n",
       "          -5.0631e-08, -1.9246e-07,  1.3715e-07,  1.3899e-07,  1.1157e-07,\n",
       "          -3.3254e-08,  2.9268e-07,  1.3354e-07, -2.5957e-07, -6.3511e-09,\n",
       "           2.7985e-07,  2.5703e-07,  1.1644e-07,  3.3019e-07,  4.4714e-07,\n",
       "           2.6136e-07,  3.5940e-07, -1.8085e-07,  3.3423e-08, -3.8664e-07,\n",
       "          -1.5279e-07, -1.2050e-07,  4.2058e-07,  3.2963e-07,  9.5716e-08,\n",
       "           3.4245e-07,  1.7248e-07, -3.2550e-07, -1.5293e-07, -5.5632e-07,\n",
       "          -3.7536e-07,  9.1865e-08,  4.3624e-07, -3.6344e-07, -5.6842e-08,\n",
       "           3.4795e-07, -6.1592e-08, -3.4271e-08,  6.1030e-08, -1.0758e-07,\n",
       "          -1.8395e-07, -9.9803e-08, -5.4088e-09, -5.4233e-08, -9.2269e-09,\n",
       "           1.2381e-07, -2.5438e-07,  9.8197e-08, -9.5377e-08, -1.8550e-07,\n",
       "           1.3236e-08, -8.2342e-09, -9.9295e-08, -1.6971e-07, -1.0325e-07,\n",
       "          -9.7483e-08,  1.2700e-08, -1.7809e-08, -1.3853e-07,  3.0244e-07,\n",
       "          -7.3157e-09, -1.8094e-07,  1.0635e-07, -1.7925e-07, -1.3679e-07,\n",
       "           1.0447e-07, -6.1985e-08, -5.2934e-08,  2.6372e-08,  8.0918e-08,\n",
       "          -2.1866e-07,  1.0449e-07, -5.6597e-08,  3.6664e-08,  6.6950e-08,\n",
       "          -2.4457e-07, -8.0017e-08, -6.0411e-08, -8.5286e-08,  1.9282e-07,\n",
       "          -4.1565e-08, -1.9870e-07,  9.5402e-08,  2.8738e-08,  7.7193e-08,\n",
       "           9.0123e-08,  1.4046e-07, -1.0890e-07, -2.4012e-07, -1.7943e-07,\n",
       "          -3.9861e-08,  1.0663e-07, -2.3715e-07, -2.1413e-08,  7.6687e-08,\n",
       "           1.6217e-07, -4.2227e-07, -1.6835e-07, -1.8471e-07,  3.1524e-08,\n",
       "           2.0459e-07, -1.3775e-07, -1.6749e-08, -2.6716e-07,  2.5601e-08,\n",
       "          -1.4088e-07,  2.6604e-07, -1.6206e-07,  1.4477e-07, -1.3148e-07,\n",
       "           1.6999e-07, -1.3299e-07,  1.3983e-07,  4.5710e-08,  9.2621e-08,\n",
       "           3.8669e-08, -5.3204e-08, -4.5011e-09,  6.9478e-08, -7.7081e-08,\n",
       "          -1.6432e-07,  1.0455e-07, -1.9308e-07, -9.5926e-08,  4.8385e-08,\n",
       "           1.2267e-07, -9.1898e-08,  5.9563e-08, -1.3781e-07,  8.6945e-09,\n",
       "          -1.3107e-07, -7.0969e-08,  1.0931e-07, -1.1680e-07, -4.9075e-08,\n",
       "           3.6988e-08,  5.1331e-08,  2.0270e-08,  1.5435e-08,  6.8575e-09,\n",
       "          -1.4435e-07,  2.8441e-07, -1.9271e-07,  1.2306e-07, -1.3534e-07,\n",
       "          -9.7325e-08,  1.7742e-07,  2.1128e-08,  1.6122e-07,  7.7396e-08,\n",
       "           3.6802e-08,  2.2261e-08,  9.2789e-08,  6.7343e-09,  1.1408e-07,\n",
       "          -6.3088e-08,  1.0185e-08,  1.1353e-07,  1.5471e-07, -1.0849e-07,\n",
       "           6.8370e-08,  1.1572e-07,  1.1224e-07, -1.8016e-07,  3.3457e-07,\n",
       "           7.3493e-08, -2.0950e-07,  1.0402e-07,  4.4845e-08,  1.3230e-07,\n",
       "           3.0573e-09, -8.2084e-08,  8.3373e-08, -5.0208e-08, -1.0386e-07,\n",
       "          -2.6320e-07, -7.6536e-08, -3.7370e-07, -9.8152e-08,  9.9287e-08,\n",
       "          -2.1932e-07, -3.4836e-09,  1.0537e-07, -8.9699e-08,  2.3610e-07,\n",
       "          -3.4294e-08,  6.3627e-08, -2.9412e-07, -2.5985e-07,  7.2934e-08,\n",
       "           5.5913e-08,  2.4384e-08,  1.7635e-09,  6.7586e-08, -2.1892e-07,\n",
       "           3.4711e-08,  3.8239e-08, -2.2601e-07, -4.3452e-08,  3.3282e-07,\n",
       "           6.9132e-08, -1.2726e-07,  6.9563e-08, -6.7884e-09, -3.0566e-07,\n",
       "          -1.1005e-07,  3.8111e-08, -4.2025e-08, -3.9029e-08, -7.7983e-08,\n",
       "           2.1044e-07,  1.9684e-07,  1.3060e-07, -3.5063e-08, -6.6682e-08,\n",
       "          -1.0049e-07,  1.8091e-07,  2.7957e-07, -1.2904e-08, -6.2500e-09,\n",
       "           3.6601e-07,  6.7805e-09,  1.0812e-08, -1.8425e-07, -3.1444e-07,\n",
       "           1.7924e-07, -2.8672e-08,  5.2108e-08,  2.2712e-07,  1.3329e-07,\n",
       "           4.3251e-08,  6.7528e-08, -2.7907e-08, -1.0495e-08,  8.0822e-08,\n",
       "           1.2289e-08,  4.0666e-08,  6.9507e-08,  9.8026e-08, -2.8796e-08,\n",
       "           8.2916e-08, -4.6638e-09, -4.2722e-08,  4.5378e-08, -1.3197e-07,\n",
       "           1.7895e-08,  1.2429e-07, -1.0082e-07,  1.1847e-08, -3.8076e-08,\n",
       "          -9.2748e-08, -2.2646e-07,  3.6654e-08, -3.7660e-08, -3.4484e-09,\n",
       "          -1.8338e-07,  6.5280e-08, -6.8788e-08, -1.0414e-07, -1.5916e-07,\n",
       "           1.2688e-07,  2.1599e-09, -2.3045e-07, -5.5640e-08,  7.3935e-08,\n",
       "           4.5443e-08,  1.2464e-07, -1.1818e-07,  1.2796e-07, -3.8161e-08,\n",
       "          -1.8330e-07, -1.7057e-07,  2.0435e-08, -6.2736e-08,  3.6503e-08,\n",
       "           7.8009e-08,  1.1035e-07,  1.0935e-07,  4.8004e-08,  1.6678e-08,\n",
       "          -3.6434e-08, -1.6552e-07, -6.1854e-08, -8.3859e-08,  1.6588e-08,\n",
       "           2.4265e-07,  1.1019e-07,  5.2357e-08, -5.1577e-08, -1.1036e-07,\n",
       "          -1.4878e-07, -2.7722e-08, -8.8569e-08], requires_grad=True))]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(module.named_parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(module.named_buffers())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 모델 Pruning\n",
    "\n",
    "모듈을 Pruning 하기 위해서는 **자신이 직접 구현한 기법** 혹은 `torch.nn.utils.prune` 내에 존재하는 Pruning 기법 중 하나를 활용해야 합니다. \n",
    "\n",
    "그리고 해당 기법을 활용해 Pruning을 적용할 **모듈명**과 **파라미터명**을 상세해줍니다.\n",
    "\n",
    "아래 예제에서는 `key` 내 `weight` 중 임의로 **30%**의 파라미터에 Pruning을 적용합니다. \n",
    "\n",
    "이때 첫 번째 인자로는 **모듈**이, 두 번째인  `name`의 인자로는 Pruning을 적용할 **모듈 내 파라미터명**이, 그리고 마지막 `amount`의 인자로는 **0 에서 1 사이의 소수** (Pruning이 적용될 퍼센티지) 혹은 **양의 정수**(Pruning을 적용할 파라미터 개수)가 사용됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear(in_features=768, out_features=768, bias=True)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prune.random_unstructured(module, name=\"weight\", amount=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`utils.prune` 모듈의 **Pruning**은 `weight`를 모듈 내 파라미터에서 제거한 후, `weight_orig`로 대체함으로써 적용됩니다. \n",
    "\n",
    "이때, `weight_orig`는 Pruning이 적용되지 않은 텐서입니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('bias', Parameter containing:\n",
       "  tensor([ 5.4592e-08,  2.9247e-08, -2.0385e-08, -2.8358e-07, -9.8012e-08,\n",
       "          -5.8178e-08, -1.3005e-07, -5.6738e-08, -2.4501e-08, -4.8126e-08,\n",
       "           8.5868e-08, -6.5552e-08, -1.5294e-07, -6.0779e-08, -1.0340e-08,\n",
       "          -1.0260e-07, -1.4049e-07, -5.1941e-08, -1.7012e-07, -6.5405e-08,\n",
       "          -1.5491e-07, -1.1515e-07, -2.5359e-08,  4.9711e-08,  1.3979e-07,\n",
       "          -8.6575e-08, -8.9431e-08,  1.3025e-07, -2.3191e-07, -6.7193e-08,\n",
       "           1.1002e-07, -6.4796e-08, -1.4740e-07, -1.9011e-08,  2.2245e-07,\n",
       "          -1.4329e-07, -2.8459e-08,  7.1192e-08, -5.4687e-08, -1.0020e-07,\n",
       "          -1.4137e-07, -4.4915e-08, -2.6025e-08, -2.9728e-07, -8.7707e-08,\n",
       "          -3.9435e-08, -5.6708e-08, -4.0600e-08,  1.3973e-07, -2.7347e-07,\n",
       "          -1.3187e-07,  1.3556e-08, -1.4675e-08,  4.6401e-08, -5.2453e-08,\n",
       "          -6.1351e-08, -8.3668e-08,  1.2962e-07,  2.1639e-07,  7.9439e-08,\n",
       "          -4.9919e-08, -6.5474e-08,  1.9281e-08, -4.2483e-08,  1.6391e-07,\n",
       "          -3.8939e-07, -1.1450e-07, -3.8202e-07, -2.1552e-07,  3.9563e-07,\n",
       "           3.0277e-07,  1.6662e-07,  1.6972e-07, -1.3551e-07, -1.0182e-07,\n",
       "           2.8733e-07, -7.0817e-08, -3.8927e-07,  3.3062e-07,  1.4522e-07,\n",
       "          -1.2677e-07,  2.0442e-08, -1.0237e-08, -3.1870e-07,  4.2408e-08,\n",
       "           2.6782e-07,  1.2641e-08, -5.0443e-08,  1.5858e-07, -3.8922e-07,\n",
       "          -1.9792e-08, -3.1154e-08,  1.8157e-07,  6.6326e-09,  9.1542e-08,\n",
       "           4.1303e-09, -3.1614e-07, -1.3147e-07, -2.2811e-07,  1.1269e-07,\n",
       "          -2.5746e-07,  1.4798e-07,  4.5211e-08, -2.6614e-08,  3.9736e-07,\n",
       "          -2.6781e-07, -2.2600e-07,  3.2725e-07, -2.3893e-07,  2.4561e-07,\n",
       "          -2.0945e-07, -4.9683e-07, -6.8960e-08, -1.0985e-07, -1.9155e-08,\n",
       "           2.7619e-07, -7.8056e-08,  6.6799e-07,  3.3654e-07,  7.9260e-08,\n",
       "           1.8013e-07,  3.6409e-07,  5.8692e-08,  2.9595e-07,  1.3185e-07,\n",
       "          -1.5979e-07, -3.2274e-07,  1.1820e-07, -6.7861e-08,  6.2519e-08,\n",
       "           1.7694e-08,  1.9007e-07, -1.2253e-08,  9.4781e-08,  1.1964e-07,\n",
       "          -1.5201e-07, -2.0908e-07,  2.1751e-07,  1.3110e-07, -7.6005e-08,\n",
       "          -7.2351e-08, -1.1120e-07,  1.1059e-08, -1.1045e-08,  8.7099e-08,\n",
       "           1.6686e-07, -1.0117e-07, -2.8408e-07, -5.9637e-08,  8.9130e-09,\n",
       "          -2.2259e-08, -2.8968e-08,  7.7852e-08,  9.6652e-08, -9.8998e-08,\n",
       "          -1.9577e-07, -7.1940e-08,  1.5101e-08,  7.1112e-08,  3.1372e-08,\n",
       "           1.1861e-07,  2.4077e-08,  1.6269e-07,  3.9347e-09, -3.9538e-09,\n",
       "          -7.8518e-08,  1.1114e-07,  1.1625e-07,  1.1282e-08,  5.3560e-08,\n",
       "          -7.6504e-08, -6.0267e-08, -1.1799e-08,  8.6065e-08,  3.1305e-08,\n",
       "          -4.4863e-08, -2.1979e-07,  2.2306e-08, -5.4133e-09, -2.5605e-07,\n",
       "           6.2860e-08, -7.5285e-08, -1.1065e-07, -1.5208e-07, -3.0479e-08,\n",
       "           4.9316e-08,  6.7766e-08,  1.0247e-07, -9.5905e-08, -8.5965e-08,\n",
       "           2.1083e-08,  2.2197e-07,  5.0754e-08, -2.0659e-07, -1.0336e-07,\n",
       "          -9.2374e-08, -5.1872e-08, -4.9566e-07,  2.9554e-08,  2.0362e-07,\n",
       "          -2.6180e-09,  3.6847e-08, -1.4453e-07, -2.3543e-07,  1.4833e-07,\n",
       "           9.3070e-08,  7.9025e-08,  1.1281e-08,  1.0711e-07,  9.2672e-08,\n",
       "           4.4602e-08,  6.2573e-08,  1.8780e-08,  3.1025e-08, -4.8195e-08,\n",
       "          -1.5173e-08,  1.1228e-07, -1.7452e-07, -2.5543e-08, -1.2804e-07,\n",
       "          -3.5630e-07,  1.3448e-08, -2.9886e-08,  1.5605e-08, -9.5864e-08,\n",
       "           4.6114e-08,  1.2234e-07, -2.8767e-07,  1.9457e-08,  9.4918e-08,\n",
       "           1.6320e-07, -5.9371e-08, -1.8035e-07,  5.4930e-08, -7.1661e-08,\n",
       "           6.3309e-08,  2.8829e-08,  7.0693e-08,  4.5149e-08,  5.5471e-08,\n",
       "          -3.9404e-08,  7.4631e-08, -7.3951e-08,  4.0582e-08,  3.2118e-07,\n",
       "          -2.2044e-07, -7.3377e-09, -6.1323e-08,  1.4230e-07,  6.0617e-09,\n",
       "           7.6287e-08,  2.1655e-07,  8.8323e-09,  1.6086e-08, -4.0542e-08,\n",
       "          -1.1028e-07,  7.0081e-08,  1.3175e-07,  1.4077e-07, -1.0854e-07,\n",
       "           5.4646e-08,  2.7159e-08,  3.7363e-08,  5.6481e-09, -1.8462e-07,\n",
       "           5.1507e-08, -2.4222e-07, -1.7078e-07,  2.6885e-08, -3.7777e-08,\n",
       "          -5.4775e-08, -3.3475e-08,  1.8043e-08, -2.6002e-08,  1.0623e-07,\n",
       "          -2.9295e-08,  1.1175e-08,  3.4062e-07, -9.4083e-08, -1.3122e-08,\n",
       "           1.2139e-08,  4.2698e-08,  5.5309e-08, -3.2358e-08, -1.2346e-07,\n",
       "           2.6396e-09,  1.1075e-07, -4.0735e-08, -1.7003e-07,  4.3231e-08,\n",
       "          -1.5420e-07, -1.5166e-08,  2.9520e-08, -3.1258e-10,  1.0987e-07,\n",
       "           9.1626e-08, -7.3850e-08, -4.4047e-08,  1.2439e-07,  1.1803e-07,\n",
       "           1.7810e-07, -3.2616e-09, -7.2168e-08, -9.8500e-08,  1.8917e-07,\n",
       "           6.1860e-09, -4.9969e-09, -8.3460e-10, -1.4427e-08, -1.4171e-07,\n",
       "          -1.8042e-07,  5.7994e-08,  8.9171e-08, -5.3812e-08, -1.1186e-08,\n",
       "           1.2069e-07, -4.1306e-08,  3.8549e-08, -2.6021e-08,  5.5914e-08,\n",
       "           2.0648e-07,  2.2912e-08, -7.7152e-08,  5.2057e-08, -2.8716e-07,\n",
       "          -9.0171e-08, -2.3865e-08,  2.3779e-09, -1.6981e-07, -7.5877e-08,\n",
       "           2.6154e-07, -6.5519e-08, -1.1477e-07, -9.7396e-08, -2.5694e-08,\n",
       "          -1.0117e-07, -7.5133e-09, -1.1856e-07,  2.6017e-07, -5.2667e-08,\n",
       "           1.0610e-08,  1.9343e-08, -3.6094e-08,  1.1599e-07, -5.0230e-08,\n",
       "          -3.3382e-07,  2.1465e-07, -2.1061e-07, -3.4796e-09,  1.8067e-08,\n",
       "          -3.9974e-08,  5.6306e-08,  9.8680e-08,  1.4153e-07, -8.4846e-08,\n",
       "           2.4724e-07, -4.6981e-08,  2.4887e-07, -2.6745e-07,  9.5903e-08,\n",
       "          -2.8251e-07, -2.5444e-07, -8.0403e-08, -3.3439e-08,  1.4418e-07,\n",
       "           1.8163e-08, -1.1089e-07, -1.7096e-07,  4.3642e-09,  1.1574e-07,\n",
       "           6.3896e-08, -2.0590e-07,  2.8891e-08,  9.9786e-08,  1.1999e-07,\n",
       "           2.5415e-07, -7.2567e-08, -6.5372e-09,  1.6629e-07, -1.2370e-07,\n",
       "          -2.9782e-08,  1.8479e-07, -3.4119e-07, -4.5975e-08, -2.2810e-07,\n",
       "           3.2521e-07,  3.5040e-07, -2.5999e-07,  5.7259e-08, -9.0412e-08,\n",
       "          -1.6055e-07, -2.1732e-08,  1.2432e-07, -1.1365e-07,  4.9454e-08,\n",
       "           1.7514e-07,  8.0093e-08, -2.0677e-07,  1.1858e-07, -2.5169e-07,\n",
       "          -1.3862e-07, -3.5102e-08,  4.9936e-07,  1.2548e-07,  2.8573e-07,\n",
       "          -2.6150e-07,  2.4200e-07,  1.4818e-07,  8.9595e-08, -1.0800e-08,\n",
       "           1.1552e-08,  3.3775e-08,  4.6794e-07, -1.4711e-07, -9.5149e-08,\n",
       "          -6.9137e-08,  1.7641e-07,  1.9165e-07,  1.4456e-07, -6.6487e-08,\n",
       "          -1.7922e-07,  7.5856e-10,  1.7300e-07,  2.0469e-08, -4.2650e-07,\n",
       "          -3.5908e-08, -6.5810e-08, -3.7828e-08,  6.3509e-08,  3.0708e-07,\n",
       "           1.0413e-07, -5.5479e-08, -1.8942e-07,  9.0802e-08, -9.6585e-09,\n",
       "          -2.8615e-08,  4.9223e-08,  5.2463e-08, -7.0254e-08,  2.7888e-07,\n",
       "           4.1418e-08, -3.4391e-08,  5.9028e-09,  1.3962e-07,  2.4130e-07,\n",
       "          -3.6720e-07, -4.4258e-08,  1.4455e-07, -1.6950e-09, -8.9174e-08,\n",
       "          -4.3999e-08, -1.0885e-07,  2.7086e-07, -2.1972e-07,  1.0028e-07,\n",
       "          -2.0821e-07, -9.5543e-08, -5.7404e-08,  1.2352e-07, -6.8864e-08,\n",
       "           1.2697e-07,  1.2360e-07,  2.7013e-07, -8.9481e-09, -1.3829e-07,\n",
       "           8.3031e-08, -2.8425e-07, -2.9014e-07, -1.2196e-07, -1.0869e-07,\n",
       "          -2.1149e-07,  3.4489e-08, -1.4093e-07, -7.7525e-08,  2.1968e-07,\n",
       "          -5.0631e-08, -1.9246e-07,  1.3715e-07,  1.3899e-07,  1.1157e-07,\n",
       "          -3.3254e-08,  2.9268e-07,  1.3354e-07, -2.5957e-07, -6.3511e-09,\n",
       "           2.7985e-07,  2.5703e-07,  1.1644e-07,  3.3019e-07,  4.4714e-07,\n",
       "           2.6136e-07,  3.5940e-07, -1.8085e-07,  3.3423e-08, -3.8664e-07,\n",
       "          -1.5279e-07, -1.2050e-07,  4.2058e-07,  3.2963e-07,  9.5716e-08,\n",
       "           3.4245e-07,  1.7248e-07, -3.2550e-07, -1.5293e-07, -5.5632e-07,\n",
       "          -3.7536e-07,  9.1865e-08,  4.3624e-07, -3.6344e-07, -5.6842e-08,\n",
       "           3.4795e-07, -6.1592e-08, -3.4271e-08,  6.1030e-08, -1.0758e-07,\n",
       "          -1.8395e-07, -9.9803e-08, -5.4088e-09, -5.4233e-08, -9.2269e-09,\n",
       "           1.2381e-07, -2.5438e-07,  9.8197e-08, -9.5377e-08, -1.8550e-07,\n",
       "           1.3236e-08, -8.2342e-09, -9.9295e-08, -1.6971e-07, -1.0325e-07,\n",
       "          -9.7483e-08,  1.2700e-08, -1.7809e-08, -1.3853e-07,  3.0244e-07,\n",
       "          -7.3157e-09, -1.8094e-07,  1.0635e-07, -1.7925e-07, -1.3679e-07,\n",
       "           1.0447e-07, -6.1985e-08, -5.2934e-08,  2.6372e-08,  8.0918e-08,\n",
       "          -2.1866e-07,  1.0449e-07, -5.6597e-08,  3.6664e-08,  6.6950e-08,\n",
       "          -2.4457e-07, -8.0017e-08, -6.0411e-08, -8.5286e-08,  1.9282e-07,\n",
       "          -4.1565e-08, -1.9870e-07,  9.5402e-08,  2.8738e-08,  7.7193e-08,\n",
       "           9.0123e-08,  1.4046e-07, -1.0890e-07, -2.4012e-07, -1.7943e-07,\n",
       "          -3.9861e-08,  1.0663e-07, -2.3715e-07, -2.1413e-08,  7.6687e-08,\n",
       "           1.6217e-07, -4.2227e-07, -1.6835e-07, -1.8471e-07,  3.1524e-08,\n",
       "           2.0459e-07, -1.3775e-07, -1.6749e-08, -2.6716e-07,  2.5601e-08,\n",
       "          -1.4088e-07,  2.6604e-07, -1.6206e-07,  1.4477e-07, -1.3148e-07,\n",
       "           1.6999e-07, -1.3299e-07,  1.3983e-07,  4.5710e-08,  9.2621e-08,\n",
       "           3.8669e-08, -5.3204e-08, -4.5011e-09,  6.9478e-08, -7.7081e-08,\n",
       "          -1.6432e-07,  1.0455e-07, -1.9308e-07, -9.5926e-08,  4.8385e-08,\n",
       "           1.2267e-07, -9.1898e-08,  5.9563e-08, -1.3781e-07,  8.6945e-09,\n",
       "          -1.3107e-07, -7.0969e-08,  1.0931e-07, -1.1680e-07, -4.9075e-08,\n",
       "           3.6988e-08,  5.1331e-08,  2.0270e-08,  1.5435e-08,  6.8575e-09,\n",
       "          -1.4435e-07,  2.8441e-07, -1.9271e-07,  1.2306e-07, -1.3534e-07,\n",
       "          -9.7325e-08,  1.7742e-07,  2.1128e-08,  1.6122e-07,  7.7396e-08,\n",
       "           3.6802e-08,  2.2261e-08,  9.2789e-08,  6.7343e-09,  1.1408e-07,\n",
       "          -6.3088e-08,  1.0185e-08,  1.1353e-07,  1.5471e-07, -1.0849e-07,\n",
       "           6.8370e-08,  1.1572e-07,  1.1224e-07, -1.8016e-07,  3.3457e-07,\n",
       "           7.3493e-08, -2.0950e-07,  1.0402e-07,  4.4845e-08,  1.3230e-07,\n",
       "           3.0573e-09, -8.2084e-08,  8.3373e-08, -5.0208e-08, -1.0386e-07,\n",
       "          -2.6320e-07, -7.6536e-08, -3.7370e-07, -9.8152e-08,  9.9287e-08,\n",
       "          -2.1932e-07, -3.4836e-09,  1.0537e-07, -8.9699e-08,  2.3610e-07,\n",
       "          -3.4294e-08,  6.3627e-08, -2.9412e-07, -2.5985e-07,  7.2934e-08,\n",
       "           5.5913e-08,  2.4384e-08,  1.7635e-09,  6.7586e-08, -2.1892e-07,\n",
       "           3.4711e-08,  3.8239e-08, -2.2601e-07, -4.3452e-08,  3.3282e-07,\n",
       "           6.9132e-08, -1.2726e-07,  6.9563e-08, -6.7884e-09, -3.0566e-07,\n",
       "          -1.1005e-07,  3.8111e-08, -4.2025e-08, -3.9029e-08, -7.7983e-08,\n",
       "           2.1044e-07,  1.9684e-07,  1.3060e-07, -3.5063e-08, -6.6682e-08,\n",
       "          -1.0049e-07,  1.8091e-07,  2.7957e-07, -1.2904e-08, -6.2500e-09,\n",
       "           3.6601e-07,  6.7805e-09,  1.0812e-08, -1.8425e-07, -3.1444e-07,\n",
       "           1.7924e-07, -2.8672e-08,  5.2108e-08,  2.2712e-07,  1.3329e-07,\n",
       "           4.3251e-08,  6.7528e-08, -2.7907e-08, -1.0495e-08,  8.0822e-08,\n",
       "           1.2289e-08,  4.0666e-08,  6.9507e-08,  9.8026e-08, -2.8796e-08,\n",
       "           8.2916e-08, -4.6638e-09, -4.2722e-08,  4.5378e-08, -1.3197e-07,\n",
       "           1.7895e-08,  1.2429e-07, -1.0082e-07,  1.1847e-08, -3.8076e-08,\n",
       "          -9.2748e-08, -2.2646e-07,  3.6654e-08, -3.7660e-08, -3.4484e-09,\n",
       "          -1.8338e-07,  6.5280e-08, -6.8788e-08, -1.0414e-07, -1.5916e-07,\n",
       "           1.2688e-07,  2.1599e-09, -2.3045e-07, -5.5640e-08,  7.3935e-08,\n",
       "           4.5443e-08,  1.2464e-07, -1.1818e-07,  1.2796e-07, -3.8161e-08,\n",
       "          -1.8330e-07, -1.7057e-07,  2.0435e-08, -6.2736e-08,  3.6503e-08,\n",
       "           7.8009e-08,  1.1035e-07,  1.0935e-07,  4.8004e-08,  1.6678e-08,\n",
       "          -3.6434e-08, -1.6552e-07, -6.1854e-08, -8.3859e-08,  1.6588e-08,\n",
       "           2.4265e-07,  1.1019e-07,  5.2357e-08, -5.1577e-08, -1.1036e-07,\n",
       "          -1.4878e-07, -2.7722e-08, -8.8569e-08], requires_grad=True)),\n",
       " ('weight_orig', Parameter containing:\n",
       "  tensor([[ 0.0267,  0.0427, -0.0713,  ..., -0.0344,  0.0055,  0.0277],\n",
       "          [ 0.0228,  0.0752, -0.0429,  ...,  0.0860,  0.1789,  0.0289],\n",
       "          [-0.0079, -0.0309, -0.0990,  ..., -0.0316,  0.0335, -0.0635],\n",
       "          ...,\n",
       "          [ 0.0040, -0.0065, -0.0924,  ..., -0.0338, -0.0250, -0.1278],\n",
       "          [ 0.0134,  0.0512,  0.0694,  ..., -0.0096, -0.0297,  0.0294],\n",
       "          [-0.0243, -0.0592, -0.0535,  ..., -0.0318, -0.0714,  0.0376]],\n",
       "         requires_grad=True))]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(module.named_parameters())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 모듈의 `named_buffers()`를 출력하면 Pruning 기법에 의해 생성된 **Pruning mask**가 `weight_mask`라는 이름으로 생성된 것을 확인할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('weight_mask', tensor([[1., 1., 1.,  ..., 1., 1., 1.],\n",
       "          [1., 0., 0.,  ..., 0., 1., 1.],\n",
       "          [0., 0., 1.,  ..., 1., 1., 1.],\n",
       "          ...,\n",
       "          [1., 1., 1.,  ..., 1., 1., 1.],\n",
       "          [1., 1., 1.,  ..., 1., 1., 1.],\n",
       "          [1., 1., 0.,  ..., 1., 0., 1.]]))]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(module.named_buffers())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch에서 모델에 순전파를 적용하기 위해서는 `weight` 속성이 존재해야 합니다.\n",
    "\n",
    "`weight`는 원래의 `weight_orig` 값에 앞서 생성한 **Pruning mask**를 적용해 계산합니다.\n",
    "\n",
    "그리고 해당 계산 결과가 `weight`라는 이름의 **속성**으로 모듈에 저장되게 됩니다.\n",
    "\n",
    "이제 `weight`는 더 이상 모듈의 **파라미터**로 관리되는 것이 아니라 모듈의 **속성(attribute)** 값으로 관리되게 되는 것입니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0267,  0.0427, -0.0713,  ..., -0.0344,  0.0055,  0.0277],\n",
       "        [ 0.0228,  0.0000, -0.0000,  ...,  0.0000,  0.1789,  0.0289],\n",
       "        [-0.0000, -0.0000, -0.0990,  ..., -0.0316,  0.0335, -0.0635],\n",
       "        ...,\n",
       "        [ 0.0040, -0.0065, -0.0924,  ..., -0.0338, -0.0250, -0.1278],\n",
       "        [ 0.0134,  0.0512,  0.0694,  ..., -0.0096, -0.0297,  0.0294],\n",
       "        [-0.0243, -0.0592, -0.0000,  ..., -0.0318, -0.0000,  0.0376]],\n",
       "       grad_fn=<MulBackward0>)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "module.weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([768, 768]), tensor(176947))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "module.weight.size(), (module.weight == 0).sum()  # 30% 파라미터가 Pruned !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "마지막으로 `nn.utils.prune`은 앞서 적용한 **Pruning**을 순전파 이전에 적용하기 위해 `forward_pre_hooks`라는 속성을 사용합니다. \n",
    "\n",
    "우리는 앞서 `weight` 파라미터에만 **Pruning**을 적용했기 때문에 아래와 같이 하나의 **pre_hook**이 생성된 것을 확인할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([(0, <torch.nn.utils.prune.RandomUnstructured at 0x26e66135a58>)])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "module._forward_pre_hooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "모듈의 파라미터, 버퍼, 훅 그리고 속성이 어떻게 변하는지 다시 한 번 확인하기 위해, `key`의 `bias`에도 **Pruning**을 적용해보도록 합시다. \n",
    "\n",
    "이번에는 `bias`의 50개의 파라미터에 **Pruning**을 적용해보도록 합니다. \n",
    "\n",
    "참고로 `l1_unstructured`는 인자로 받은 파라미터에서 **L1 노름** 기준으로 가장 영향력이 작은 `amount`개의 파라미터를 **Pruning**하도록 구현되어 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear(in_features=768, out_features=768, bias=True)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prune.l1_unstructured(module, name='bias', amount=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 우리는 모듈의 `named_parameters()`에 `weight_orig` 뿐만 아니라 `bias_orig`가 함께 포함되어 있음을 예상할 수 있습니다. \n",
    "\n",
    "그리고 `named_buffers()`에는 `weight_mask`와 더불어 `bias_mask`가 포함되어 있겠죠. \n",
    "\n",
    "**Pruning**이 적용된 두 텐서는 앞서 언급한 것과 마찬가지로 이제 모듈의 **속성**으로서 관리되게 됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('weight_orig', Parameter containing:\n",
       "  tensor([[ 0.0267,  0.0427, -0.0713,  ..., -0.0344,  0.0055,  0.0277],\n",
       "          [ 0.0228,  0.0752, -0.0429,  ...,  0.0860,  0.1789,  0.0289],\n",
       "          [-0.0079, -0.0309, -0.0990,  ..., -0.0316,  0.0335, -0.0635],\n",
       "          ...,\n",
       "          [ 0.0040, -0.0065, -0.0924,  ..., -0.0338, -0.0250, -0.1278],\n",
       "          [ 0.0134,  0.0512,  0.0694,  ..., -0.0096, -0.0297,  0.0294],\n",
       "          [-0.0243, -0.0592, -0.0535,  ..., -0.0318, -0.0714,  0.0376]],\n",
       "         requires_grad=True)), ('bias_orig', Parameter containing:\n",
       "  tensor([ 5.4592e-08,  2.9247e-08, -2.0385e-08, -2.8358e-07, -9.8012e-08,\n",
       "          -5.8178e-08, -1.3005e-07, -5.6738e-08, -2.4501e-08, -4.8126e-08,\n",
       "           8.5868e-08, -6.5552e-08, -1.5294e-07, -6.0779e-08, -1.0340e-08,\n",
       "          -1.0260e-07, -1.4049e-07, -5.1941e-08, -1.7012e-07, -6.5405e-08,\n",
       "          -1.5491e-07, -1.1515e-07, -2.5359e-08,  4.9711e-08,  1.3979e-07,\n",
       "          -8.6575e-08, -8.9431e-08,  1.3025e-07, -2.3191e-07, -6.7193e-08,\n",
       "           1.1002e-07, -6.4796e-08, -1.4740e-07, -1.9011e-08,  2.2245e-07,\n",
       "          -1.4329e-07, -2.8459e-08,  7.1192e-08, -5.4687e-08, -1.0020e-07,\n",
       "          -1.4137e-07, -4.4915e-08, -2.6025e-08, -2.9728e-07, -8.7707e-08,\n",
       "          -3.9435e-08, -5.6708e-08, -4.0600e-08,  1.3973e-07, -2.7347e-07,\n",
       "          -1.3187e-07,  1.3556e-08, -1.4675e-08,  4.6401e-08, -5.2453e-08,\n",
       "          -6.1351e-08, -8.3668e-08,  1.2962e-07,  2.1639e-07,  7.9439e-08,\n",
       "          -4.9919e-08, -6.5474e-08,  1.9281e-08, -4.2483e-08,  1.6391e-07,\n",
       "          -3.8939e-07, -1.1450e-07, -3.8202e-07, -2.1552e-07,  3.9563e-07,\n",
       "           3.0277e-07,  1.6662e-07,  1.6972e-07, -1.3551e-07, -1.0182e-07,\n",
       "           2.8733e-07, -7.0817e-08, -3.8927e-07,  3.3062e-07,  1.4522e-07,\n",
       "          -1.2677e-07,  2.0442e-08, -1.0237e-08, -3.1870e-07,  4.2408e-08,\n",
       "           2.6782e-07,  1.2641e-08, -5.0443e-08,  1.5858e-07, -3.8922e-07,\n",
       "          -1.9792e-08, -3.1154e-08,  1.8157e-07,  6.6326e-09,  9.1542e-08,\n",
       "           4.1303e-09, -3.1614e-07, -1.3147e-07, -2.2811e-07,  1.1269e-07,\n",
       "          -2.5746e-07,  1.4798e-07,  4.5211e-08, -2.6614e-08,  3.9736e-07,\n",
       "          -2.6781e-07, -2.2600e-07,  3.2725e-07, -2.3893e-07,  2.4561e-07,\n",
       "          -2.0945e-07, -4.9683e-07, -6.8960e-08, -1.0985e-07, -1.9155e-08,\n",
       "           2.7619e-07, -7.8056e-08,  6.6799e-07,  3.3654e-07,  7.9260e-08,\n",
       "           1.8013e-07,  3.6409e-07,  5.8692e-08,  2.9595e-07,  1.3185e-07,\n",
       "          -1.5979e-07, -3.2274e-07,  1.1820e-07, -6.7861e-08,  6.2519e-08,\n",
       "           1.7694e-08,  1.9007e-07, -1.2253e-08,  9.4781e-08,  1.1964e-07,\n",
       "          -1.5201e-07, -2.0908e-07,  2.1751e-07,  1.3110e-07, -7.6005e-08,\n",
       "          -7.2351e-08, -1.1120e-07,  1.1059e-08, -1.1045e-08,  8.7099e-08,\n",
       "           1.6686e-07, -1.0117e-07, -2.8408e-07, -5.9637e-08,  8.9130e-09,\n",
       "          -2.2259e-08, -2.8968e-08,  7.7852e-08,  9.6652e-08, -9.8998e-08,\n",
       "          -1.9577e-07, -7.1940e-08,  1.5101e-08,  7.1112e-08,  3.1372e-08,\n",
       "           1.1861e-07,  2.4077e-08,  1.6269e-07,  3.9347e-09, -3.9538e-09,\n",
       "          -7.8518e-08,  1.1114e-07,  1.1625e-07,  1.1282e-08,  5.3560e-08,\n",
       "          -7.6504e-08, -6.0267e-08, -1.1799e-08,  8.6065e-08,  3.1305e-08,\n",
       "          -4.4863e-08, -2.1979e-07,  2.2306e-08, -5.4133e-09, -2.5605e-07,\n",
       "           6.2860e-08, -7.5285e-08, -1.1065e-07, -1.5208e-07, -3.0479e-08,\n",
       "           4.9316e-08,  6.7766e-08,  1.0247e-07, -9.5905e-08, -8.5965e-08,\n",
       "           2.1083e-08,  2.2197e-07,  5.0754e-08, -2.0659e-07, -1.0336e-07,\n",
       "          -9.2374e-08, -5.1872e-08, -4.9566e-07,  2.9554e-08,  2.0362e-07,\n",
       "          -2.6180e-09,  3.6847e-08, -1.4453e-07, -2.3543e-07,  1.4833e-07,\n",
       "           9.3070e-08,  7.9025e-08,  1.1281e-08,  1.0711e-07,  9.2672e-08,\n",
       "           4.4602e-08,  6.2573e-08,  1.8780e-08,  3.1025e-08, -4.8195e-08,\n",
       "          -1.5173e-08,  1.1228e-07, -1.7452e-07, -2.5543e-08, -1.2804e-07,\n",
       "          -3.5630e-07,  1.3448e-08, -2.9886e-08,  1.5605e-08, -9.5864e-08,\n",
       "           4.6114e-08,  1.2234e-07, -2.8767e-07,  1.9457e-08,  9.4918e-08,\n",
       "           1.6320e-07, -5.9371e-08, -1.8035e-07,  5.4930e-08, -7.1661e-08,\n",
       "           6.3309e-08,  2.8829e-08,  7.0693e-08,  4.5149e-08,  5.5471e-08,\n",
       "          -3.9404e-08,  7.4631e-08, -7.3951e-08,  4.0582e-08,  3.2118e-07,\n",
       "          -2.2044e-07, -7.3377e-09, -6.1323e-08,  1.4230e-07,  6.0617e-09,\n",
       "           7.6287e-08,  2.1655e-07,  8.8323e-09,  1.6086e-08, -4.0542e-08,\n",
       "          -1.1028e-07,  7.0081e-08,  1.3175e-07,  1.4077e-07, -1.0854e-07,\n",
       "           5.4646e-08,  2.7159e-08,  3.7363e-08,  5.6481e-09, -1.8462e-07,\n",
       "           5.1507e-08, -2.4222e-07, -1.7078e-07,  2.6885e-08, -3.7777e-08,\n",
       "          -5.4775e-08, -3.3475e-08,  1.8043e-08, -2.6002e-08,  1.0623e-07,\n",
       "          -2.9295e-08,  1.1175e-08,  3.4062e-07, -9.4083e-08, -1.3122e-08,\n",
       "           1.2139e-08,  4.2698e-08,  5.5309e-08, -3.2358e-08, -1.2346e-07,\n",
       "           2.6396e-09,  1.1075e-07, -4.0735e-08, -1.7003e-07,  4.3231e-08,\n",
       "          -1.5420e-07, -1.5166e-08,  2.9520e-08, -3.1258e-10,  1.0987e-07,\n",
       "           9.1626e-08, -7.3850e-08, -4.4047e-08,  1.2439e-07,  1.1803e-07,\n",
       "           1.7810e-07, -3.2616e-09, -7.2168e-08, -9.8500e-08,  1.8917e-07,\n",
       "           6.1860e-09, -4.9969e-09, -8.3460e-10, -1.4427e-08, -1.4171e-07,\n",
       "          -1.8042e-07,  5.7994e-08,  8.9171e-08, -5.3812e-08, -1.1186e-08,\n",
       "           1.2069e-07, -4.1306e-08,  3.8549e-08, -2.6021e-08,  5.5914e-08,\n",
       "           2.0648e-07,  2.2912e-08, -7.7152e-08,  5.2057e-08, -2.8716e-07,\n",
       "          -9.0171e-08, -2.3865e-08,  2.3779e-09, -1.6981e-07, -7.5877e-08,\n",
       "           2.6154e-07, -6.5519e-08, -1.1477e-07, -9.7396e-08, -2.5694e-08,\n",
       "          -1.0117e-07, -7.5133e-09, -1.1856e-07,  2.6017e-07, -5.2667e-08,\n",
       "           1.0610e-08,  1.9343e-08, -3.6094e-08,  1.1599e-07, -5.0230e-08,\n",
       "          -3.3382e-07,  2.1465e-07, -2.1061e-07, -3.4796e-09,  1.8067e-08,\n",
       "          -3.9974e-08,  5.6306e-08,  9.8680e-08,  1.4153e-07, -8.4846e-08,\n",
       "           2.4724e-07, -4.6981e-08,  2.4887e-07, -2.6745e-07,  9.5903e-08,\n",
       "          -2.8251e-07, -2.5444e-07, -8.0403e-08, -3.3439e-08,  1.4418e-07,\n",
       "           1.8163e-08, -1.1089e-07, -1.7096e-07,  4.3642e-09,  1.1574e-07,\n",
       "           6.3896e-08, -2.0590e-07,  2.8891e-08,  9.9786e-08,  1.1999e-07,\n",
       "           2.5415e-07, -7.2567e-08, -6.5372e-09,  1.6629e-07, -1.2370e-07,\n",
       "          -2.9782e-08,  1.8479e-07, -3.4119e-07, -4.5975e-08, -2.2810e-07,\n",
       "           3.2521e-07,  3.5040e-07, -2.5999e-07,  5.7259e-08, -9.0412e-08,\n",
       "          -1.6055e-07, -2.1732e-08,  1.2432e-07, -1.1365e-07,  4.9454e-08,\n",
       "           1.7514e-07,  8.0093e-08, -2.0677e-07,  1.1858e-07, -2.5169e-07,\n",
       "          -1.3862e-07, -3.5102e-08,  4.9936e-07,  1.2548e-07,  2.8573e-07,\n",
       "          -2.6150e-07,  2.4200e-07,  1.4818e-07,  8.9595e-08, -1.0800e-08,\n",
       "           1.1552e-08,  3.3775e-08,  4.6794e-07, -1.4711e-07, -9.5149e-08,\n",
       "          -6.9137e-08,  1.7641e-07,  1.9165e-07,  1.4456e-07, -6.6487e-08,\n",
       "          -1.7922e-07,  7.5856e-10,  1.7300e-07,  2.0469e-08, -4.2650e-07,\n",
       "          -3.5908e-08, -6.5810e-08, -3.7828e-08,  6.3509e-08,  3.0708e-07,\n",
       "           1.0413e-07, -5.5479e-08, -1.8942e-07,  9.0802e-08, -9.6585e-09,\n",
       "          -2.8615e-08,  4.9223e-08,  5.2463e-08, -7.0254e-08,  2.7888e-07,\n",
       "           4.1418e-08, -3.4391e-08,  5.9028e-09,  1.3962e-07,  2.4130e-07,\n",
       "          -3.6720e-07, -4.4258e-08,  1.4455e-07, -1.6950e-09, -8.9174e-08,\n",
       "          -4.3999e-08, -1.0885e-07,  2.7086e-07, -2.1972e-07,  1.0028e-07,\n",
       "          -2.0821e-07, -9.5543e-08, -5.7404e-08,  1.2352e-07, -6.8864e-08,\n",
       "           1.2697e-07,  1.2360e-07,  2.7013e-07, -8.9481e-09, -1.3829e-07,\n",
       "           8.3031e-08, -2.8425e-07, -2.9014e-07, -1.2196e-07, -1.0869e-07,\n",
       "          -2.1149e-07,  3.4489e-08, -1.4093e-07, -7.7525e-08,  2.1968e-07,\n",
       "          -5.0631e-08, -1.9246e-07,  1.3715e-07,  1.3899e-07,  1.1157e-07,\n",
       "          -3.3254e-08,  2.9268e-07,  1.3354e-07, -2.5957e-07, -6.3511e-09,\n",
       "           2.7985e-07,  2.5703e-07,  1.1644e-07,  3.3019e-07,  4.4714e-07,\n",
       "           2.6136e-07,  3.5940e-07, -1.8085e-07,  3.3423e-08, -3.8664e-07,\n",
       "          -1.5279e-07, -1.2050e-07,  4.2058e-07,  3.2963e-07,  9.5716e-08,\n",
       "           3.4245e-07,  1.7248e-07, -3.2550e-07, -1.5293e-07, -5.5632e-07,\n",
       "          -3.7536e-07,  9.1865e-08,  4.3624e-07, -3.6344e-07, -5.6842e-08,\n",
       "           3.4795e-07, -6.1592e-08, -3.4271e-08,  6.1030e-08, -1.0758e-07,\n",
       "          -1.8395e-07, -9.9803e-08, -5.4088e-09, -5.4233e-08, -9.2269e-09,\n",
       "           1.2381e-07, -2.5438e-07,  9.8197e-08, -9.5377e-08, -1.8550e-07,\n",
       "           1.3236e-08, -8.2342e-09, -9.9295e-08, -1.6971e-07, -1.0325e-07,\n",
       "          -9.7483e-08,  1.2700e-08, -1.7809e-08, -1.3853e-07,  3.0244e-07,\n",
       "          -7.3157e-09, -1.8094e-07,  1.0635e-07, -1.7925e-07, -1.3679e-07,\n",
       "           1.0447e-07, -6.1985e-08, -5.2934e-08,  2.6372e-08,  8.0918e-08,\n",
       "          -2.1866e-07,  1.0449e-07, -5.6597e-08,  3.6664e-08,  6.6950e-08,\n",
       "          -2.4457e-07, -8.0017e-08, -6.0411e-08, -8.5286e-08,  1.9282e-07,\n",
       "          -4.1565e-08, -1.9870e-07,  9.5402e-08,  2.8738e-08,  7.7193e-08,\n",
       "           9.0123e-08,  1.4046e-07, -1.0890e-07, -2.4012e-07, -1.7943e-07,\n",
       "          -3.9861e-08,  1.0663e-07, -2.3715e-07, -2.1413e-08,  7.6687e-08,\n",
       "           1.6217e-07, -4.2227e-07, -1.6835e-07, -1.8471e-07,  3.1524e-08,\n",
       "           2.0459e-07, -1.3775e-07, -1.6749e-08, -2.6716e-07,  2.5601e-08,\n",
       "          -1.4088e-07,  2.6604e-07, -1.6206e-07,  1.4477e-07, -1.3148e-07,\n",
       "           1.6999e-07, -1.3299e-07,  1.3983e-07,  4.5710e-08,  9.2621e-08,\n",
       "           3.8669e-08, -5.3204e-08, -4.5011e-09,  6.9478e-08, -7.7081e-08,\n",
       "          -1.6432e-07,  1.0455e-07, -1.9308e-07, -9.5926e-08,  4.8385e-08,\n",
       "           1.2267e-07, -9.1898e-08,  5.9563e-08, -1.3781e-07,  8.6945e-09,\n",
       "          -1.3107e-07, -7.0969e-08,  1.0931e-07, -1.1680e-07, -4.9075e-08,\n",
       "           3.6988e-08,  5.1331e-08,  2.0270e-08,  1.5435e-08,  6.8575e-09,\n",
       "          -1.4435e-07,  2.8441e-07, -1.9271e-07,  1.2306e-07, -1.3534e-07,\n",
       "          -9.7325e-08,  1.7742e-07,  2.1128e-08,  1.6122e-07,  7.7396e-08,\n",
       "           3.6802e-08,  2.2261e-08,  9.2789e-08,  6.7343e-09,  1.1408e-07,\n",
       "          -6.3088e-08,  1.0185e-08,  1.1353e-07,  1.5471e-07, -1.0849e-07,\n",
       "           6.8370e-08,  1.1572e-07,  1.1224e-07, -1.8016e-07,  3.3457e-07,\n",
       "           7.3493e-08, -2.0950e-07,  1.0402e-07,  4.4845e-08,  1.3230e-07,\n",
       "           3.0573e-09, -8.2084e-08,  8.3373e-08, -5.0208e-08, -1.0386e-07,\n",
       "          -2.6320e-07, -7.6536e-08, -3.7370e-07, -9.8152e-08,  9.9287e-08,\n",
       "          -2.1932e-07, -3.4836e-09,  1.0537e-07, -8.9699e-08,  2.3610e-07,\n",
       "          -3.4294e-08,  6.3627e-08, -2.9412e-07, -2.5985e-07,  7.2934e-08,\n",
       "           5.5913e-08,  2.4384e-08,  1.7635e-09,  6.7586e-08, -2.1892e-07,\n",
       "           3.4711e-08,  3.8239e-08, -2.2601e-07, -4.3452e-08,  3.3282e-07,\n",
       "           6.9132e-08, -1.2726e-07,  6.9563e-08, -6.7884e-09, -3.0566e-07,\n",
       "          -1.1005e-07,  3.8111e-08, -4.2025e-08, -3.9029e-08, -7.7983e-08,\n",
       "           2.1044e-07,  1.9684e-07,  1.3060e-07, -3.5063e-08, -6.6682e-08,\n",
       "          -1.0049e-07,  1.8091e-07,  2.7957e-07, -1.2904e-08, -6.2500e-09,\n",
       "           3.6601e-07,  6.7805e-09,  1.0812e-08, -1.8425e-07, -3.1444e-07,\n",
       "           1.7924e-07, -2.8672e-08,  5.2108e-08,  2.2712e-07,  1.3329e-07,\n",
       "           4.3251e-08,  6.7528e-08, -2.7907e-08, -1.0495e-08,  8.0822e-08,\n",
       "           1.2289e-08,  4.0666e-08,  6.9507e-08,  9.8026e-08, -2.8796e-08,\n",
       "           8.2916e-08, -4.6638e-09, -4.2722e-08,  4.5378e-08, -1.3197e-07,\n",
       "           1.7895e-08,  1.2429e-07, -1.0082e-07,  1.1847e-08, -3.8076e-08,\n",
       "          -9.2748e-08, -2.2646e-07,  3.6654e-08, -3.7660e-08, -3.4484e-09,\n",
       "          -1.8338e-07,  6.5280e-08, -6.8788e-08, -1.0414e-07, -1.5916e-07,\n",
       "           1.2688e-07,  2.1599e-09, -2.3045e-07, -5.5640e-08,  7.3935e-08,\n",
       "           4.5443e-08,  1.2464e-07, -1.1818e-07,  1.2796e-07, -3.8161e-08,\n",
       "          -1.8330e-07, -1.7057e-07,  2.0435e-08, -6.2736e-08,  3.6503e-08,\n",
       "           7.8009e-08,  1.1035e-07,  1.0935e-07,  4.8004e-08,  1.6678e-08,\n",
       "          -3.6434e-08, -1.6552e-07, -6.1854e-08, -8.3859e-08,  1.6588e-08,\n",
       "           2.4265e-07,  1.1019e-07,  5.2357e-08, -5.1577e-08, -1.1036e-07,\n",
       "          -1.4878e-07, -2.7722e-08, -8.8569e-08], requires_grad=True))]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(module.named_parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('weight_mask', tensor([[1., 1., 1.,  ..., 1., 1., 1.],\n",
       "          [1., 0., 0.,  ..., 0., 1., 1.],\n",
       "          [0., 0., 1.,  ..., 1., 1., 1.],\n",
       "          ...,\n",
       "          [1., 1., 1.,  ..., 1., 1., 1.],\n",
       "          [1., 1., 1.,  ..., 1., 1., 1.],\n",
       "          [1., 1., 0.,  ..., 1., 0., 1.]])),\n",
       " ('bias_mask',\n",
       "  tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 0., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 0., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 0., 1., 1.,\n",
       "          0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 0.,\n",
       "          0., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 0.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 0., 1., 1.,\n",
       "          1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 0., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]))]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(module.named_buffers())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 5.4592e-08,  2.9247e-08, -2.0385e-08, -2.8358e-07, -9.8012e-08,\n",
       "        -5.8178e-08, -1.3005e-07, -5.6738e-08, -2.4501e-08, -4.8126e-08,\n",
       "         8.5868e-08, -6.5552e-08, -1.5294e-07, -6.0779e-08, -0.0000e+00,\n",
       "        -1.0260e-07, -1.4049e-07, -5.1941e-08, -1.7012e-07, -6.5405e-08,\n",
       "        -1.5491e-07, -1.1515e-07, -2.5359e-08,  4.9711e-08,  1.3979e-07,\n",
       "        -8.6575e-08, -8.9431e-08,  1.3025e-07, -2.3191e-07, -6.7193e-08,\n",
       "         1.1002e-07, -6.4796e-08, -1.4740e-07, -1.9011e-08,  2.2245e-07,\n",
       "        -1.4329e-07, -2.8459e-08,  7.1192e-08, -5.4687e-08, -1.0020e-07,\n",
       "        -1.4137e-07, -4.4915e-08, -2.6025e-08, -2.9728e-07, -8.7707e-08,\n",
       "        -3.9435e-08, -5.6708e-08, -4.0600e-08,  1.3973e-07, -2.7347e-07,\n",
       "        -1.3187e-07,  1.3556e-08, -1.4675e-08,  4.6401e-08, -5.2453e-08,\n",
       "        -6.1351e-08, -8.3668e-08,  1.2962e-07,  2.1639e-07,  7.9439e-08,\n",
       "        -4.9919e-08, -6.5474e-08,  1.9281e-08, -4.2483e-08,  1.6391e-07,\n",
       "        -3.8939e-07, -1.1450e-07, -3.8202e-07, -2.1552e-07,  3.9563e-07,\n",
       "         3.0277e-07,  1.6662e-07,  1.6972e-07, -1.3551e-07, -1.0182e-07,\n",
       "         2.8733e-07, -7.0817e-08, -3.8927e-07,  3.3062e-07,  1.4522e-07,\n",
       "        -1.2677e-07,  2.0442e-08, -0.0000e+00, -3.1870e-07,  4.2408e-08,\n",
       "         2.6782e-07,  1.2641e-08, -5.0443e-08,  1.5858e-07, -3.8922e-07,\n",
       "        -1.9792e-08, -3.1154e-08,  1.8157e-07,  0.0000e+00,  9.1542e-08,\n",
       "         0.0000e+00, -3.1614e-07, -1.3147e-07, -2.2811e-07,  1.1269e-07,\n",
       "        -2.5746e-07,  1.4798e-07,  4.5211e-08, -2.6614e-08,  3.9736e-07,\n",
       "        -2.6781e-07, -2.2600e-07,  3.2725e-07, -2.3893e-07,  2.4561e-07,\n",
       "        -2.0945e-07, -4.9683e-07, -6.8960e-08, -1.0985e-07, -1.9155e-08,\n",
       "         2.7619e-07, -7.8056e-08,  6.6799e-07,  3.3654e-07,  7.9260e-08,\n",
       "         1.8013e-07,  3.6409e-07,  5.8692e-08,  2.9595e-07,  1.3185e-07,\n",
       "        -1.5979e-07, -3.2274e-07,  1.1820e-07, -6.7861e-08,  6.2519e-08,\n",
       "         1.7694e-08,  1.9007e-07, -1.2253e-08,  9.4781e-08,  1.1964e-07,\n",
       "        -1.5201e-07, -2.0908e-07,  2.1751e-07,  1.3110e-07, -7.6005e-08,\n",
       "        -7.2351e-08, -1.1120e-07,  1.1059e-08, -1.1045e-08,  8.7099e-08,\n",
       "         1.6686e-07, -1.0117e-07, -2.8408e-07, -5.9637e-08,  0.0000e+00,\n",
       "        -2.2259e-08, -2.8968e-08,  7.7852e-08,  9.6652e-08, -9.8998e-08,\n",
       "        -1.9577e-07, -7.1940e-08,  1.5101e-08,  7.1112e-08,  3.1372e-08,\n",
       "         1.1861e-07,  2.4077e-08,  1.6269e-07,  0.0000e+00, -0.0000e+00,\n",
       "        -7.8518e-08,  1.1114e-07,  1.1625e-07,  1.1282e-08,  5.3560e-08,\n",
       "        -7.6504e-08, -6.0267e-08, -1.1799e-08,  8.6065e-08,  3.1305e-08,\n",
       "        -4.4863e-08, -2.1979e-07,  2.2306e-08, -0.0000e+00, -2.5605e-07,\n",
       "         6.2860e-08, -7.5285e-08, -1.1065e-07, -1.5208e-07, -3.0479e-08,\n",
       "         4.9316e-08,  6.7766e-08,  1.0247e-07, -9.5905e-08, -8.5965e-08,\n",
       "         2.1083e-08,  2.2197e-07,  5.0754e-08, -2.0659e-07, -1.0336e-07,\n",
       "        -9.2374e-08, -5.1872e-08, -4.9566e-07,  2.9554e-08,  2.0362e-07,\n",
       "        -0.0000e+00,  3.6847e-08, -1.4453e-07, -2.3543e-07,  1.4833e-07,\n",
       "         9.3070e-08,  7.9025e-08,  1.1281e-08,  1.0711e-07,  9.2672e-08,\n",
       "         4.4602e-08,  6.2573e-08,  1.8780e-08,  3.1025e-08, -4.8195e-08,\n",
       "        -1.5173e-08,  1.1228e-07, -1.7452e-07, -2.5543e-08, -1.2804e-07,\n",
       "        -3.5630e-07,  1.3448e-08, -2.9886e-08,  1.5605e-08, -9.5864e-08,\n",
       "         4.6114e-08,  1.2234e-07, -2.8767e-07,  1.9457e-08,  9.4918e-08,\n",
       "         1.6320e-07, -5.9371e-08, -1.8035e-07,  5.4930e-08, -7.1661e-08,\n",
       "         6.3309e-08,  2.8829e-08,  7.0693e-08,  4.5149e-08,  5.5471e-08,\n",
       "        -3.9404e-08,  7.4631e-08, -7.3951e-08,  4.0582e-08,  3.2118e-07,\n",
       "        -2.2044e-07, -0.0000e+00, -6.1323e-08,  1.4230e-07,  0.0000e+00,\n",
       "         7.6287e-08,  2.1655e-07,  0.0000e+00,  1.6086e-08, -4.0542e-08,\n",
       "        -1.1028e-07,  7.0081e-08,  1.3175e-07,  1.4077e-07, -1.0854e-07,\n",
       "         5.4646e-08,  2.7159e-08,  3.7363e-08,  0.0000e+00, -1.8462e-07,\n",
       "         5.1507e-08, -2.4222e-07, -1.7078e-07,  2.6885e-08, -3.7777e-08,\n",
       "        -5.4775e-08, -3.3475e-08,  1.8043e-08, -2.6002e-08,  1.0623e-07,\n",
       "        -2.9295e-08,  1.1175e-08,  3.4062e-07, -9.4083e-08, -1.3122e-08,\n",
       "         1.2139e-08,  4.2698e-08,  5.5309e-08, -3.2358e-08, -1.2346e-07,\n",
       "         0.0000e+00,  1.1075e-07, -4.0735e-08, -1.7003e-07,  4.3231e-08,\n",
       "        -1.5420e-07, -1.5166e-08,  2.9520e-08, -0.0000e+00,  1.0987e-07,\n",
       "         9.1626e-08, -7.3850e-08, -4.4047e-08,  1.2439e-07,  1.1803e-07,\n",
       "         1.7810e-07, -0.0000e+00, -7.2168e-08, -9.8500e-08,  1.8917e-07,\n",
       "         0.0000e+00, -0.0000e+00, -0.0000e+00, -1.4427e-08, -1.4171e-07,\n",
       "        -1.8042e-07,  5.7994e-08,  8.9171e-08, -5.3812e-08, -1.1186e-08,\n",
       "         1.2069e-07, -4.1306e-08,  3.8549e-08, -2.6021e-08,  5.5914e-08,\n",
       "         2.0648e-07,  2.2912e-08, -7.7152e-08,  5.2057e-08, -2.8716e-07,\n",
       "        -9.0171e-08, -2.3865e-08,  0.0000e+00, -1.6981e-07, -7.5877e-08,\n",
       "         2.6154e-07, -6.5519e-08, -1.1477e-07, -9.7396e-08, -2.5694e-08,\n",
       "        -1.0117e-07, -0.0000e+00, -1.1856e-07,  2.6017e-07, -5.2667e-08,\n",
       "         0.0000e+00,  1.9343e-08, -3.6094e-08,  1.1599e-07, -5.0230e-08,\n",
       "        -3.3382e-07,  2.1465e-07, -2.1061e-07, -0.0000e+00,  1.8067e-08,\n",
       "        -3.9974e-08,  5.6306e-08,  9.8680e-08,  1.4153e-07, -8.4846e-08,\n",
       "         2.4724e-07, -4.6981e-08,  2.4887e-07, -2.6745e-07,  9.5903e-08,\n",
       "        -2.8251e-07, -2.5444e-07, -8.0403e-08, -3.3439e-08,  1.4418e-07,\n",
       "         1.8163e-08, -1.1089e-07, -1.7096e-07,  0.0000e+00,  1.1574e-07,\n",
       "         6.3896e-08, -2.0590e-07,  2.8891e-08,  9.9786e-08,  1.1999e-07,\n",
       "         2.5415e-07, -7.2567e-08, -0.0000e+00,  1.6629e-07, -1.2370e-07,\n",
       "        -2.9782e-08,  1.8479e-07, -3.4119e-07, -4.5975e-08, -2.2810e-07,\n",
       "         3.2521e-07,  3.5040e-07, -2.5999e-07,  5.7259e-08, -9.0412e-08,\n",
       "        -1.6055e-07, -2.1732e-08,  1.2432e-07, -1.1365e-07,  4.9454e-08,\n",
       "         1.7514e-07,  8.0093e-08, -2.0677e-07,  1.1858e-07, -2.5169e-07,\n",
       "        -1.3862e-07, -3.5102e-08,  4.9936e-07,  1.2548e-07,  2.8573e-07,\n",
       "        -2.6150e-07,  2.4200e-07,  1.4818e-07,  8.9595e-08, -1.0800e-08,\n",
       "         1.1552e-08,  3.3775e-08,  4.6794e-07, -1.4711e-07, -9.5149e-08,\n",
       "        -6.9137e-08,  1.7641e-07,  1.9165e-07,  1.4456e-07, -6.6487e-08,\n",
       "        -1.7922e-07,  0.0000e+00,  1.7300e-07,  2.0469e-08, -4.2650e-07,\n",
       "        -3.5908e-08, -6.5810e-08, -3.7828e-08,  6.3509e-08,  3.0708e-07,\n",
       "         1.0413e-07, -5.5479e-08, -1.8942e-07,  9.0802e-08, -0.0000e+00,\n",
       "        -2.8615e-08,  4.9223e-08,  5.2463e-08, -7.0254e-08,  2.7888e-07,\n",
       "         4.1418e-08, -3.4391e-08,  0.0000e+00,  1.3962e-07,  2.4130e-07,\n",
       "        -3.6720e-07, -4.4258e-08,  1.4455e-07, -0.0000e+00, -8.9174e-08,\n",
       "        -4.3999e-08, -1.0885e-07,  2.7086e-07, -2.1972e-07,  1.0028e-07,\n",
       "        -2.0821e-07, -9.5543e-08, -5.7404e-08,  1.2352e-07, -6.8864e-08,\n",
       "         1.2697e-07,  1.2360e-07,  2.7013e-07, -0.0000e+00, -1.3829e-07,\n",
       "         8.3031e-08, -2.8425e-07, -2.9014e-07, -1.2196e-07, -1.0869e-07,\n",
       "        -2.1149e-07,  3.4489e-08, -1.4093e-07, -7.7525e-08,  2.1968e-07,\n",
       "        -5.0631e-08, -1.9246e-07,  1.3715e-07,  1.3899e-07,  1.1157e-07,\n",
       "        -3.3254e-08,  2.9268e-07,  1.3354e-07, -2.5957e-07, -0.0000e+00,\n",
       "         2.7985e-07,  2.5703e-07,  1.1644e-07,  3.3019e-07,  4.4714e-07,\n",
       "         2.6136e-07,  3.5940e-07, -1.8085e-07,  3.3423e-08, -3.8664e-07,\n",
       "        -1.5279e-07, -1.2050e-07,  4.2058e-07,  3.2963e-07,  9.5716e-08,\n",
       "         3.4245e-07,  1.7248e-07, -3.2550e-07, -1.5293e-07, -5.5632e-07,\n",
       "        -3.7536e-07,  9.1865e-08,  4.3624e-07, -3.6344e-07, -5.6842e-08,\n",
       "         3.4795e-07, -6.1592e-08, -3.4271e-08,  6.1030e-08, -1.0758e-07,\n",
       "        -1.8395e-07, -9.9803e-08, -0.0000e+00, -5.4233e-08, -0.0000e+00,\n",
       "         1.2381e-07, -2.5438e-07,  9.8197e-08, -9.5377e-08, -1.8550e-07,\n",
       "         1.3236e-08, -0.0000e+00, -9.9295e-08, -1.6971e-07, -1.0325e-07,\n",
       "        -9.7483e-08,  1.2700e-08, -1.7809e-08, -1.3853e-07,  3.0244e-07,\n",
       "        -0.0000e+00, -1.8094e-07,  1.0635e-07, -1.7925e-07, -1.3679e-07,\n",
       "         1.0447e-07, -6.1985e-08, -5.2934e-08,  2.6372e-08,  8.0918e-08,\n",
       "        -2.1866e-07,  1.0449e-07, -5.6597e-08,  3.6664e-08,  6.6950e-08,\n",
       "        -2.4457e-07, -8.0017e-08, -6.0411e-08, -8.5286e-08,  1.9282e-07,\n",
       "        -4.1565e-08, -1.9870e-07,  9.5402e-08,  2.8738e-08,  7.7193e-08,\n",
       "         9.0123e-08,  1.4046e-07, -1.0890e-07, -2.4012e-07, -1.7943e-07,\n",
       "        -3.9861e-08,  1.0663e-07, -2.3715e-07, -2.1413e-08,  7.6687e-08,\n",
       "         1.6217e-07, -4.2227e-07, -1.6835e-07, -1.8471e-07,  3.1524e-08,\n",
       "         2.0459e-07, -1.3775e-07, -1.6749e-08, -2.6716e-07,  2.5601e-08,\n",
       "        -1.4088e-07,  2.6604e-07, -1.6206e-07,  1.4477e-07, -1.3148e-07,\n",
       "         1.6999e-07, -1.3299e-07,  1.3983e-07,  4.5710e-08,  9.2621e-08,\n",
       "         3.8669e-08, -5.3204e-08, -0.0000e+00,  6.9478e-08, -7.7081e-08,\n",
       "        -1.6432e-07,  1.0455e-07, -1.9308e-07, -9.5926e-08,  4.8385e-08,\n",
       "         1.2267e-07, -9.1898e-08,  5.9563e-08, -1.3781e-07,  0.0000e+00,\n",
       "        -1.3107e-07, -7.0969e-08,  1.0931e-07, -1.1680e-07, -4.9075e-08,\n",
       "         3.6988e-08,  5.1331e-08,  2.0270e-08,  1.5435e-08,  0.0000e+00,\n",
       "        -1.4435e-07,  2.8441e-07, -1.9271e-07,  1.2306e-07, -1.3534e-07,\n",
       "        -9.7325e-08,  1.7742e-07,  2.1128e-08,  1.6122e-07,  7.7396e-08,\n",
       "         3.6802e-08,  2.2261e-08,  9.2789e-08,  0.0000e+00,  1.1408e-07,\n",
       "        -6.3088e-08,  0.0000e+00,  1.1353e-07,  1.5471e-07, -1.0849e-07,\n",
       "         6.8370e-08,  1.1572e-07,  1.1224e-07, -1.8016e-07,  3.3457e-07,\n",
       "         7.3493e-08, -2.0950e-07,  1.0402e-07,  4.4845e-08,  1.3230e-07,\n",
       "         0.0000e+00, -8.2084e-08,  8.3373e-08, -5.0208e-08, -1.0386e-07,\n",
       "        -2.6320e-07, -7.6536e-08, -3.7370e-07, -9.8152e-08,  9.9287e-08,\n",
       "        -2.1932e-07, -0.0000e+00,  1.0537e-07, -8.9699e-08,  2.3610e-07,\n",
       "        -3.4294e-08,  6.3627e-08, -2.9412e-07, -2.5985e-07,  7.2934e-08,\n",
       "         5.5913e-08,  2.4384e-08,  0.0000e+00,  6.7586e-08, -2.1892e-07,\n",
       "         3.4711e-08,  3.8239e-08, -2.2601e-07, -4.3452e-08,  3.3282e-07,\n",
       "         6.9132e-08, -1.2726e-07,  6.9563e-08, -0.0000e+00, -3.0566e-07,\n",
       "        -1.1005e-07,  3.8111e-08, -4.2025e-08, -3.9029e-08, -7.7983e-08,\n",
       "         2.1044e-07,  1.9684e-07,  1.3060e-07, -3.5063e-08, -6.6682e-08,\n",
       "        -1.0049e-07,  1.8091e-07,  2.7957e-07, -1.2904e-08, -0.0000e+00,\n",
       "         3.6601e-07,  0.0000e+00,  1.0812e-08, -1.8425e-07, -3.1444e-07,\n",
       "         1.7924e-07, -2.8672e-08,  5.2108e-08,  2.2712e-07,  1.3329e-07,\n",
       "         4.3251e-08,  6.7528e-08, -2.7907e-08, -0.0000e+00,  8.0822e-08,\n",
       "         1.2289e-08,  4.0666e-08,  6.9507e-08,  9.8026e-08, -2.8796e-08,\n",
       "         8.2916e-08, -0.0000e+00, -4.2722e-08,  4.5378e-08, -1.3197e-07,\n",
       "         1.7895e-08,  1.2429e-07, -1.0082e-07,  1.1847e-08, -3.8076e-08,\n",
       "        -9.2748e-08, -2.2646e-07,  3.6654e-08, -3.7660e-08, -0.0000e+00,\n",
       "        -1.8338e-07,  6.5280e-08, -6.8788e-08, -1.0414e-07, -1.5916e-07,\n",
       "         1.2688e-07,  0.0000e+00, -2.3045e-07, -5.5640e-08,  7.3935e-08,\n",
       "         4.5443e-08,  1.2464e-07, -1.1818e-07,  1.2796e-07, -3.8161e-08,\n",
       "        -1.8330e-07, -1.7057e-07,  2.0435e-08, -6.2736e-08,  3.6503e-08,\n",
       "         7.8009e-08,  1.1035e-07,  1.0935e-07,  4.8004e-08,  1.6678e-08,\n",
       "        -3.6434e-08, -1.6552e-07, -6.1854e-08, -8.3859e-08,  1.6588e-08,\n",
       "         2.4265e-07,  1.1019e-07,  5.2357e-08, -5.1577e-08, -1.1036e-07,\n",
       "        -1.4878e-07, -2.7722e-08, -8.8569e-08], grad_fn=<MulBackward0>)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "module.bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([768]), tensor(50))"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "module.bias.size(), (module.bias == 0).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "모듈의 `_forward_pre_hooks`에도 이제 2개의 **pre_hook**이 존재하는 것을 확인할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([(0, <torch.nn.utils.prune.RandomUnstructured at 0x26e66135a58>),\n",
       "             (1, <torch.nn.utils.prune.L1Unstructured at 0x26e661049b0>)])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "module._forward_pre_hooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pruning 중첩\n",
    "\n",
    "모듈 내 같은 파라미터에 Pruning을 여러 번 적용할 수도 있습니다. Pruning을 여러 번 적용한다는 것은 다양한 **Pruning mask**를 중첩해서 사용하겠다는 의미입니다. 그리고 중첩된 마스크 결과 값은 `PruningContainer` 객체의 `compute_mask` 메서드에 의해 관리됩니다.\n",
    "\n",
    "아래 예제는 이전에 Pruning을 적용한 `key`의 `weight`에 또 다른 Pruning을 적용하는 예를 보여줍니다. `ln_structured`는 텐서의 `dim`**-th axis**에서 L`n` 노름을 기준으로 영향력이 작은 `amount`만큼의 파라미터를 **Pruning** 하는 기법입니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear(in_features=768, out_features=768, bias=True)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prune.ln_structured(module, name='weight', amount=0.3, n=2, dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 모듈의 `weight`는 앞서 적용한 `random_unstructured`와 `ln_structured`가 함께 적용된 `weight_mask`를 통해 중첩 **Pruning**이 적용되었습니다. \n",
    "\n",
    "앞서 출력한 `weight_mask` 보다 **Pruning**이 적용될 0의 갯수가 많아진 것을 확인할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 0., 1.,  ..., 1., 1., 1.],\n",
       "        [1., 0., 0.,  ..., 0., 1., 1.],\n",
       "        [0., 0., 1.,  ..., 1., 1., 1.],\n",
       "        ...,\n",
       "        [1., 0., 1.,  ..., 1., 1., 1.],\n",
       "        [1., 0., 1.,  ..., 1., 1., 1.],\n",
       "        [1., 0., 0.,  ..., 1., 0., 1.]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(module.named_buffers())[0][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0267,  0.0000, -0.0713,  ..., -0.0344,  0.0055,  0.0277],\n",
       "        [ 0.0228,  0.0000, -0.0000,  ...,  0.0000,  0.1789,  0.0289],\n",
       "        [-0.0000, -0.0000, -0.0990,  ..., -0.0316,  0.0335, -0.0635],\n",
       "        ...,\n",
       "        [ 0.0040, -0.0000, -0.0924,  ..., -0.0338, -0.0250, -0.1278],\n",
       "        [ 0.0134,  0.0000,  0.0694,  ..., -0.0096, -0.0297,  0.0294],\n",
       "        [-0.0243, -0.0000, -0.0000,  ..., -0.0318, -0.0000,  0.0376]],\n",
       "       grad_fn=<MulBackward0>)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "module.weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 `weight`에 중첩 **Pruning**을 적용하는 객체는 `torch.nn.utils.prune.PruningContainer`가 됩니다. \n",
    "\n",
    "해당 컨테이너는 `weight` 파라미터에 적용된 **Pruning** 기법들의 내역을 저장합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch.nn.utils.prune.PruningContainer at 0x26e66104438>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for hook in module._forward_pre_hooks.values():\n",
    "    if hook._tensor_name == 'weight':\n",
    "        break\n",
    "\n",
    "hook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<torch.nn.utils.prune.RandomUnstructured at 0x26e66135a58>,\n",
       " <torch.nn.utils.prune.LnStructured at 0x26e661044e0>]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(hook)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pruning 모델 시리얼라이즈\n",
    "\n",
    "마스크 버퍼(`_mask`)와 파라미터의 원래 값(`_orig`) 등 **Pruning**에 사용되는 모든 관련된 텐서들은 각 모듈의 `state_dict`에 저장되기 때문에 쉽게 저장 및 시리얼라이즈 될 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "odict_keys(['weight_orig', 'bias_orig', 'weight_mask', 'bias_mask'])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "module.state_dict().keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pruning 영구 적용\n",
    "\n",
    "우리는 `weight_orig`와 `weight_mask`, 그리고 **pre_hook** 등 **Pruning**에 사용된 모듈들을 제거해 Pruning을 영구적으로 적용할 수 있습니다. \n",
    "\n",
    "그리고 이를 위해서는 `remove` 메서드를 사용해야 합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('weight_orig', Parameter containing:\n",
       "  tensor([[ 0.0267,  0.0427, -0.0713,  ..., -0.0344,  0.0055,  0.0277],\n",
       "          [ 0.0228,  0.0752, -0.0429,  ...,  0.0860,  0.1789,  0.0289],\n",
       "          [-0.0079, -0.0309, -0.0990,  ..., -0.0316,  0.0335, -0.0635],\n",
       "          ...,\n",
       "          [ 0.0040, -0.0065, -0.0924,  ..., -0.0338, -0.0250, -0.1278],\n",
       "          [ 0.0134,  0.0512,  0.0694,  ..., -0.0096, -0.0297,  0.0294],\n",
       "          [-0.0243, -0.0592, -0.0535,  ..., -0.0318, -0.0714,  0.0376]],\n",
       "         requires_grad=True)), ('bias_orig', Parameter containing:\n",
       "  tensor([ 5.4592e-08,  2.9247e-08, -2.0385e-08, -2.8358e-07, -9.8012e-08,\n",
       "          -5.8178e-08, -1.3005e-07, -5.6738e-08, -2.4501e-08, -4.8126e-08,\n",
       "           8.5868e-08, -6.5552e-08, -1.5294e-07, -6.0779e-08, -1.0340e-08,\n",
       "          -1.0260e-07, -1.4049e-07, -5.1941e-08, -1.7012e-07, -6.5405e-08,\n",
       "          -1.5491e-07, -1.1515e-07, -2.5359e-08,  4.9711e-08,  1.3979e-07,\n",
       "          -8.6575e-08, -8.9431e-08,  1.3025e-07, -2.3191e-07, -6.7193e-08,\n",
       "           1.1002e-07, -6.4796e-08, -1.4740e-07, -1.9011e-08,  2.2245e-07,\n",
       "          -1.4329e-07, -2.8459e-08,  7.1192e-08, -5.4687e-08, -1.0020e-07,\n",
       "          -1.4137e-07, -4.4915e-08, -2.6025e-08, -2.9728e-07, -8.7707e-08,\n",
       "          -3.9435e-08, -5.6708e-08, -4.0600e-08,  1.3973e-07, -2.7347e-07,\n",
       "          -1.3187e-07,  1.3556e-08, -1.4675e-08,  4.6401e-08, -5.2453e-08,\n",
       "          -6.1351e-08, -8.3668e-08,  1.2962e-07,  2.1639e-07,  7.9439e-08,\n",
       "          -4.9919e-08, -6.5474e-08,  1.9281e-08, -4.2483e-08,  1.6391e-07,\n",
       "          -3.8939e-07, -1.1450e-07, -3.8202e-07, -2.1552e-07,  3.9563e-07,\n",
       "           3.0277e-07,  1.6662e-07,  1.6972e-07, -1.3551e-07, -1.0182e-07,\n",
       "           2.8733e-07, -7.0817e-08, -3.8927e-07,  3.3062e-07,  1.4522e-07,\n",
       "          -1.2677e-07,  2.0442e-08, -1.0237e-08, -3.1870e-07,  4.2408e-08,\n",
       "           2.6782e-07,  1.2641e-08, -5.0443e-08,  1.5858e-07, -3.8922e-07,\n",
       "          -1.9792e-08, -3.1154e-08,  1.8157e-07,  6.6326e-09,  9.1542e-08,\n",
       "           4.1303e-09, -3.1614e-07, -1.3147e-07, -2.2811e-07,  1.1269e-07,\n",
       "          -2.5746e-07,  1.4798e-07,  4.5211e-08, -2.6614e-08,  3.9736e-07,\n",
       "          -2.6781e-07, -2.2600e-07,  3.2725e-07, -2.3893e-07,  2.4561e-07,\n",
       "          -2.0945e-07, -4.9683e-07, -6.8960e-08, -1.0985e-07, -1.9155e-08,\n",
       "           2.7619e-07, -7.8056e-08,  6.6799e-07,  3.3654e-07,  7.9260e-08,\n",
       "           1.8013e-07,  3.6409e-07,  5.8692e-08,  2.9595e-07,  1.3185e-07,\n",
       "          -1.5979e-07, -3.2274e-07,  1.1820e-07, -6.7861e-08,  6.2519e-08,\n",
       "           1.7694e-08,  1.9007e-07, -1.2253e-08,  9.4781e-08,  1.1964e-07,\n",
       "          -1.5201e-07, -2.0908e-07,  2.1751e-07,  1.3110e-07, -7.6005e-08,\n",
       "          -7.2351e-08, -1.1120e-07,  1.1059e-08, -1.1045e-08,  8.7099e-08,\n",
       "           1.6686e-07, -1.0117e-07, -2.8408e-07, -5.9637e-08,  8.9130e-09,\n",
       "          -2.2259e-08, -2.8968e-08,  7.7852e-08,  9.6652e-08, -9.8998e-08,\n",
       "          -1.9577e-07, -7.1940e-08,  1.5101e-08,  7.1112e-08,  3.1372e-08,\n",
       "           1.1861e-07,  2.4077e-08,  1.6269e-07,  3.9347e-09, -3.9538e-09,\n",
       "          -7.8518e-08,  1.1114e-07,  1.1625e-07,  1.1282e-08,  5.3560e-08,\n",
       "          -7.6504e-08, -6.0267e-08, -1.1799e-08,  8.6065e-08,  3.1305e-08,\n",
       "          -4.4863e-08, -2.1979e-07,  2.2306e-08, -5.4133e-09, -2.5605e-07,\n",
       "           6.2860e-08, -7.5285e-08, -1.1065e-07, -1.5208e-07, -3.0479e-08,\n",
       "           4.9316e-08,  6.7766e-08,  1.0247e-07, -9.5905e-08, -8.5965e-08,\n",
       "           2.1083e-08,  2.2197e-07,  5.0754e-08, -2.0659e-07, -1.0336e-07,\n",
       "          -9.2374e-08, -5.1872e-08, -4.9566e-07,  2.9554e-08,  2.0362e-07,\n",
       "          -2.6180e-09,  3.6847e-08, -1.4453e-07, -2.3543e-07,  1.4833e-07,\n",
       "           9.3070e-08,  7.9025e-08,  1.1281e-08,  1.0711e-07,  9.2672e-08,\n",
       "           4.4602e-08,  6.2573e-08,  1.8780e-08,  3.1025e-08, -4.8195e-08,\n",
       "          -1.5173e-08,  1.1228e-07, -1.7452e-07, -2.5543e-08, -1.2804e-07,\n",
       "          -3.5630e-07,  1.3448e-08, -2.9886e-08,  1.5605e-08, -9.5864e-08,\n",
       "           4.6114e-08,  1.2234e-07, -2.8767e-07,  1.9457e-08,  9.4918e-08,\n",
       "           1.6320e-07, -5.9371e-08, -1.8035e-07,  5.4930e-08, -7.1661e-08,\n",
       "           6.3309e-08,  2.8829e-08,  7.0693e-08,  4.5149e-08,  5.5471e-08,\n",
       "          -3.9404e-08,  7.4631e-08, -7.3951e-08,  4.0582e-08,  3.2118e-07,\n",
       "          -2.2044e-07, -7.3377e-09, -6.1323e-08,  1.4230e-07,  6.0617e-09,\n",
       "           7.6287e-08,  2.1655e-07,  8.8323e-09,  1.6086e-08, -4.0542e-08,\n",
       "          -1.1028e-07,  7.0081e-08,  1.3175e-07,  1.4077e-07, -1.0854e-07,\n",
       "           5.4646e-08,  2.7159e-08,  3.7363e-08,  5.6481e-09, -1.8462e-07,\n",
       "           5.1507e-08, -2.4222e-07, -1.7078e-07,  2.6885e-08, -3.7777e-08,\n",
       "          -5.4775e-08, -3.3475e-08,  1.8043e-08, -2.6002e-08,  1.0623e-07,\n",
       "          -2.9295e-08,  1.1175e-08,  3.4062e-07, -9.4083e-08, -1.3122e-08,\n",
       "           1.2139e-08,  4.2698e-08,  5.5309e-08, -3.2358e-08, -1.2346e-07,\n",
       "           2.6396e-09,  1.1075e-07, -4.0735e-08, -1.7003e-07,  4.3231e-08,\n",
       "          -1.5420e-07, -1.5166e-08,  2.9520e-08, -3.1258e-10,  1.0987e-07,\n",
       "           9.1626e-08, -7.3850e-08, -4.4047e-08,  1.2439e-07,  1.1803e-07,\n",
       "           1.7810e-07, -3.2616e-09, -7.2168e-08, -9.8500e-08,  1.8917e-07,\n",
       "           6.1860e-09, -4.9969e-09, -8.3460e-10, -1.4427e-08, -1.4171e-07,\n",
       "          -1.8042e-07,  5.7994e-08,  8.9171e-08, -5.3812e-08, -1.1186e-08,\n",
       "           1.2069e-07, -4.1306e-08,  3.8549e-08, -2.6021e-08,  5.5914e-08,\n",
       "           2.0648e-07,  2.2912e-08, -7.7152e-08,  5.2057e-08, -2.8716e-07,\n",
       "          -9.0171e-08, -2.3865e-08,  2.3779e-09, -1.6981e-07, -7.5877e-08,\n",
       "           2.6154e-07, -6.5519e-08, -1.1477e-07, -9.7396e-08, -2.5694e-08,\n",
       "          -1.0117e-07, -7.5133e-09, -1.1856e-07,  2.6017e-07, -5.2667e-08,\n",
       "           1.0610e-08,  1.9343e-08, -3.6094e-08,  1.1599e-07, -5.0230e-08,\n",
       "          -3.3382e-07,  2.1465e-07, -2.1061e-07, -3.4796e-09,  1.8067e-08,\n",
       "          -3.9974e-08,  5.6306e-08,  9.8680e-08,  1.4153e-07, -8.4846e-08,\n",
       "           2.4724e-07, -4.6981e-08,  2.4887e-07, -2.6745e-07,  9.5903e-08,\n",
       "          -2.8251e-07, -2.5444e-07, -8.0403e-08, -3.3439e-08,  1.4418e-07,\n",
       "           1.8163e-08, -1.1089e-07, -1.7096e-07,  4.3642e-09,  1.1574e-07,\n",
       "           6.3896e-08, -2.0590e-07,  2.8891e-08,  9.9786e-08,  1.1999e-07,\n",
       "           2.5415e-07, -7.2567e-08, -6.5372e-09,  1.6629e-07, -1.2370e-07,\n",
       "          -2.9782e-08,  1.8479e-07, -3.4119e-07, -4.5975e-08, -2.2810e-07,\n",
       "           3.2521e-07,  3.5040e-07, -2.5999e-07,  5.7259e-08, -9.0412e-08,\n",
       "          -1.6055e-07, -2.1732e-08,  1.2432e-07, -1.1365e-07,  4.9454e-08,\n",
       "           1.7514e-07,  8.0093e-08, -2.0677e-07,  1.1858e-07, -2.5169e-07,\n",
       "          -1.3862e-07, -3.5102e-08,  4.9936e-07,  1.2548e-07,  2.8573e-07,\n",
       "          -2.6150e-07,  2.4200e-07,  1.4818e-07,  8.9595e-08, -1.0800e-08,\n",
       "           1.1552e-08,  3.3775e-08,  4.6794e-07, -1.4711e-07, -9.5149e-08,\n",
       "          -6.9137e-08,  1.7641e-07,  1.9165e-07,  1.4456e-07, -6.6487e-08,\n",
       "          -1.7922e-07,  7.5856e-10,  1.7300e-07,  2.0469e-08, -4.2650e-07,\n",
       "          -3.5908e-08, -6.5810e-08, -3.7828e-08,  6.3509e-08,  3.0708e-07,\n",
       "           1.0413e-07, -5.5479e-08, -1.8942e-07,  9.0802e-08, -9.6585e-09,\n",
       "          -2.8615e-08,  4.9223e-08,  5.2463e-08, -7.0254e-08,  2.7888e-07,\n",
       "           4.1418e-08, -3.4391e-08,  5.9028e-09,  1.3962e-07,  2.4130e-07,\n",
       "          -3.6720e-07, -4.4258e-08,  1.4455e-07, -1.6950e-09, -8.9174e-08,\n",
       "          -4.3999e-08, -1.0885e-07,  2.7086e-07, -2.1972e-07,  1.0028e-07,\n",
       "          -2.0821e-07, -9.5543e-08, -5.7404e-08,  1.2352e-07, -6.8864e-08,\n",
       "           1.2697e-07,  1.2360e-07,  2.7013e-07, -8.9481e-09, -1.3829e-07,\n",
       "           8.3031e-08, -2.8425e-07, -2.9014e-07, -1.2196e-07, -1.0869e-07,\n",
       "          -2.1149e-07,  3.4489e-08, -1.4093e-07, -7.7525e-08,  2.1968e-07,\n",
       "          -5.0631e-08, -1.9246e-07,  1.3715e-07,  1.3899e-07,  1.1157e-07,\n",
       "          -3.3254e-08,  2.9268e-07,  1.3354e-07, -2.5957e-07, -6.3511e-09,\n",
       "           2.7985e-07,  2.5703e-07,  1.1644e-07,  3.3019e-07,  4.4714e-07,\n",
       "           2.6136e-07,  3.5940e-07, -1.8085e-07,  3.3423e-08, -3.8664e-07,\n",
       "          -1.5279e-07, -1.2050e-07,  4.2058e-07,  3.2963e-07,  9.5716e-08,\n",
       "           3.4245e-07,  1.7248e-07, -3.2550e-07, -1.5293e-07, -5.5632e-07,\n",
       "          -3.7536e-07,  9.1865e-08,  4.3624e-07, -3.6344e-07, -5.6842e-08,\n",
       "           3.4795e-07, -6.1592e-08, -3.4271e-08,  6.1030e-08, -1.0758e-07,\n",
       "          -1.8395e-07, -9.9803e-08, -5.4088e-09, -5.4233e-08, -9.2269e-09,\n",
       "           1.2381e-07, -2.5438e-07,  9.8197e-08, -9.5377e-08, -1.8550e-07,\n",
       "           1.3236e-08, -8.2342e-09, -9.9295e-08, -1.6971e-07, -1.0325e-07,\n",
       "          -9.7483e-08,  1.2700e-08, -1.7809e-08, -1.3853e-07,  3.0244e-07,\n",
       "          -7.3157e-09, -1.8094e-07,  1.0635e-07, -1.7925e-07, -1.3679e-07,\n",
       "           1.0447e-07, -6.1985e-08, -5.2934e-08,  2.6372e-08,  8.0918e-08,\n",
       "          -2.1866e-07,  1.0449e-07, -5.6597e-08,  3.6664e-08,  6.6950e-08,\n",
       "          -2.4457e-07, -8.0017e-08, -6.0411e-08, -8.5286e-08,  1.9282e-07,\n",
       "          -4.1565e-08, -1.9870e-07,  9.5402e-08,  2.8738e-08,  7.7193e-08,\n",
       "           9.0123e-08,  1.4046e-07, -1.0890e-07, -2.4012e-07, -1.7943e-07,\n",
       "          -3.9861e-08,  1.0663e-07, -2.3715e-07, -2.1413e-08,  7.6687e-08,\n",
       "           1.6217e-07, -4.2227e-07, -1.6835e-07, -1.8471e-07,  3.1524e-08,\n",
       "           2.0459e-07, -1.3775e-07, -1.6749e-08, -2.6716e-07,  2.5601e-08,\n",
       "          -1.4088e-07,  2.6604e-07, -1.6206e-07,  1.4477e-07, -1.3148e-07,\n",
       "           1.6999e-07, -1.3299e-07,  1.3983e-07,  4.5710e-08,  9.2621e-08,\n",
       "           3.8669e-08, -5.3204e-08, -4.5011e-09,  6.9478e-08, -7.7081e-08,\n",
       "          -1.6432e-07,  1.0455e-07, -1.9308e-07, -9.5926e-08,  4.8385e-08,\n",
       "           1.2267e-07, -9.1898e-08,  5.9563e-08, -1.3781e-07,  8.6945e-09,\n",
       "          -1.3107e-07, -7.0969e-08,  1.0931e-07, -1.1680e-07, -4.9075e-08,\n",
       "           3.6988e-08,  5.1331e-08,  2.0270e-08,  1.5435e-08,  6.8575e-09,\n",
       "          -1.4435e-07,  2.8441e-07, -1.9271e-07,  1.2306e-07, -1.3534e-07,\n",
       "          -9.7325e-08,  1.7742e-07,  2.1128e-08,  1.6122e-07,  7.7396e-08,\n",
       "           3.6802e-08,  2.2261e-08,  9.2789e-08,  6.7343e-09,  1.1408e-07,\n",
       "          -6.3088e-08,  1.0185e-08,  1.1353e-07,  1.5471e-07, -1.0849e-07,\n",
       "           6.8370e-08,  1.1572e-07,  1.1224e-07, -1.8016e-07,  3.3457e-07,\n",
       "           7.3493e-08, -2.0950e-07,  1.0402e-07,  4.4845e-08,  1.3230e-07,\n",
       "           3.0573e-09, -8.2084e-08,  8.3373e-08, -5.0208e-08, -1.0386e-07,\n",
       "          -2.6320e-07, -7.6536e-08, -3.7370e-07, -9.8152e-08,  9.9287e-08,\n",
       "          -2.1932e-07, -3.4836e-09,  1.0537e-07, -8.9699e-08,  2.3610e-07,\n",
       "          -3.4294e-08,  6.3627e-08, -2.9412e-07, -2.5985e-07,  7.2934e-08,\n",
       "           5.5913e-08,  2.4384e-08,  1.7635e-09,  6.7586e-08, -2.1892e-07,\n",
       "           3.4711e-08,  3.8239e-08, -2.2601e-07, -4.3452e-08,  3.3282e-07,\n",
       "           6.9132e-08, -1.2726e-07,  6.9563e-08, -6.7884e-09, -3.0566e-07,\n",
       "          -1.1005e-07,  3.8111e-08, -4.2025e-08, -3.9029e-08, -7.7983e-08,\n",
       "           2.1044e-07,  1.9684e-07,  1.3060e-07, -3.5063e-08, -6.6682e-08,\n",
       "          -1.0049e-07,  1.8091e-07,  2.7957e-07, -1.2904e-08, -6.2500e-09,\n",
       "           3.6601e-07,  6.7805e-09,  1.0812e-08, -1.8425e-07, -3.1444e-07,\n",
       "           1.7924e-07, -2.8672e-08,  5.2108e-08,  2.2712e-07,  1.3329e-07,\n",
       "           4.3251e-08,  6.7528e-08, -2.7907e-08, -1.0495e-08,  8.0822e-08,\n",
       "           1.2289e-08,  4.0666e-08,  6.9507e-08,  9.8026e-08, -2.8796e-08,\n",
       "           8.2916e-08, -4.6638e-09, -4.2722e-08,  4.5378e-08, -1.3197e-07,\n",
       "           1.7895e-08,  1.2429e-07, -1.0082e-07,  1.1847e-08, -3.8076e-08,\n",
       "          -9.2748e-08, -2.2646e-07,  3.6654e-08, -3.7660e-08, -3.4484e-09,\n",
       "          -1.8338e-07,  6.5280e-08, -6.8788e-08, -1.0414e-07, -1.5916e-07,\n",
       "           1.2688e-07,  2.1599e-09, -2.3045e-07, -5.5640e-08,  7.3935e-08,\n",
       "           4.5443e-08,  1.2464e-07, -1.1818e-07,  1.2796e-07, -3.8161e-08,\n",
       "          -1.8330e-07, -1.7057e-07,  2.0435e-08, -6.2736e-08,  3.6503e-08,\n",
       "           7.8009e-08,  1.1035e-07,  1.0935e-07,  4.8004e-08,  1.6678e-08,\n",
       "          -3.6434e-08, -1.6552e-07, -6.1854e-08, -8.3859e-08,  1.6588e-08,\n",
       "           2.4265e-07,  1.1019e-07,  5.2357e-08, -5.1577e-08, -1.1036e-07,\n",
       "          -1.4878e-07, -2.7722e-08, -8.8569e-08], requires_grad=True))]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(module.named_parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('weight_mask', tensor([[1., 0., 1.,  ..., 1., 1., 1.],\n",
       "          [1., 0., 0.,  ..., 0., 1., 1.],\n",
       "          [0., 0., 1.,  ..., 1., 1., 1.],\n",
       "          ...,\n",
       "          [1., 0., 1.,  ..., 1., 1., 1.],\n",
       "          [1., 0., 1.,  ..., 1., 1., 1.],\n",
       "          [1., 0., 0.,  ..., 1., 0., 1.]])),\n",
       " ('bias_mask',\n",
       "  tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 0., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 0., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 0., 1., 1.,\n",
       "          0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 0.,\n",
       "          0., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 0.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 0., 1., 1.,\n",
       "          1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 0., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]))]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(module.named_buffers())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0267,  0.0000, -0.0713,  ..., -0.0344,  0.0055,  0.0277],\n",
       "        [ 0.0228,  0.0000, -0.0000,  ...,  0.0000,  0.1789,  0.0289],\n",
       "        [-0.0000, -0.0000, -0.0990,  ..., -0.0316,  0.0335, -0.0635],\n",
       "        ...,\n",
       "        [ 0.0040, -0.0000, -0.0924,  ..., -0.0338, -0.0250, -0.1278],\n",
       "        [ 0.0134,  0.0000,  0.0694,  ..., -0.0096, -0.0297,  0.0294],\n",
       "        [-0.0243, -0.0000, -0.0000,  ..., -0.0318, -0.0000,  0.0376]],\n",
       "       grad_fn=<MulBackward0>)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "module.weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "위 코드들에서는 여전히 `weight_orig`와 `weight_mask`가 존재합니다. 이제 이들을 제거하고 **Pruning**을 영구 적용해보도록 합시다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('bias_orig', Parameter containing:\n",
       "  tensor([ 5.4592e-08,  2.9247e-08, -2.0385e-08, -2.8358e-07, -9.8012e-08,\n",
       "          -5.8178e-08, -1.3005e-07, -5.6738e-08, -2.4501e-08, -4.8126e-08,\n",
       "           8.5868e-08, -6.5552e-08, -1.5294e-07, -6.0779e-08, -1.0340e-08,\n",
       "          -1.0260e-07, -1.4049e-07, -5.1941e-08, -1.7012e-07, -6.5405e-08,\n",
       "          -1.5491e-07, -1.1515e-07, -2.5359e-08,  4.9711e-08,  1.3979e-07,\n",
       "          -8.6575e-08, -8.9431e-08,  1.3025e-07, -2.3191e-07, -6.7193e-08,\n",
       "           1.1002e-07, -6.4796e-08, -1.4740e-07, -1.9011e-08,  2.2245e-07,\n",
       "          -1.4329e-07, -2.8459e-08,  7.1192e-08, -5.4687e-08, -1.0020e-07,\n",
       "          -1.4137e-07, -4.4915e-08, -2.6025e-08, -2.9728e-07, -8.7707e-08,\n",
       "          -3.9435e-08, -5.6708e-08, -4.0600e-08,  1.3973e-07, -2.7347e-07,\n",
       "          -1.3187e-07,  1.3556e-08, -1.4675e-08,  4.6401e-08, -5.2453e-08,\n",
       "          -6.1351e-08, -8.3668e-08,  1.2962e-07,  2.1639e-07,  7.9439e-08,\n",
       "          -4.9919e-08, -6.5474e-08,  1.9281e-08, -4.2483e-08,  1.6391e-07,\n",
       "          -3.8939e-07, -1.1450e-07, -3.8202e-07, -2.1552e-07,  3.9563e-07,\n",
       "           3.0277e-07,  1.6662e-07,  1.6972e-07, -1.3551e-07, -1.0182e-07,\n",
       "           2.8733e-07, -7.0817e-08, -3.8927e-07,  3.3062e-07,  1.4522e-07,\n",
       "          -1.2677e-07,  2.0442e-08, -1.0237e-08, -3.1870e-07,  4.2408e-08,\n",
       "           2.6782e-07,  1.2641e-08, -5.0443e-08,  1.5858e-07, -3.8922e-07,\n",
       "          -1.9792e-08, -3.1154e-08,  1.8157e-07,  6.6326e-09,  9.1542e-08,\n",
       "           4.1303e-09, -3.1614e-07, -1.3147e-07, -2.2811e-07,  1.1269e-07,\n",
       "          -2.5746e-07,  1.4798e-07,  4.5211e-08, -2.6614e-08,  3.9736e-07,\n",
       "          -2.6781e-07, -2.2600e-07,  3.2725e-07, -2.3893e-07,  2.4561e-07,\n",
       "          -2.0945e-07, -4.9683e-07, -6.8960e-08, -1.0985e-07, -1.9155e-08,\n",
       "           2.7619e-07, -7.8056e-08,  6.6799e-07,  3.3654e-07,  7.9260e-08,\n",
       "           1.8013e-07,  3.6409e-07,  5.8692e-08,  2.9595e-07,  1.3185e-07,\n",
       "          -1.5979e-07, -3.2274e-07,  1.1820e-07, -6.7861e-08,  6.2519e-08,\n",
       "           1.7694e-08,  1.9007e-07, -1.2253e-08,  9.4781e-08,  1.1964e-07,\n",
       "          -1.5201e-07, -2.0908e-07,  2.1751e-07,  1.3110e-07, -7.6005e-08,\n",
       "          -7.2351e-08, -1.1120e-07,  1.1059e-08, -1.1045e-08,  8.7099e-08,\n",
       "           1.6686e-07, -1.0117e-07, -2.8408e-07, -5.9637e-08,  8.9130e-09,\n",
       "          -2.2259e-08, -2.8968e-08,  7.7852e-08,  9.6652e-08, -9.8998e-08,\n",
       "          -1.9577e-07, -7.1940e-08,  1.5101e-08,  7.1112e-08,  3.1372e-08,\n",
       "           1.1861e-07,  2.4077e-08,  1.6269e-07,  3.9347e-09, -3.9538e-09,\n",
       "          -7.8518e-08,  1.1114e-07,  1.1625e-07,  1.1282e-08,  5.3560e-08,\n",
       "          -7.6504e-08, -6.0267e-08, -1.1799e-08,  8.6065e-08,  3.1305e-08,\n",
       "          -4.4863e-08, -2.1979e-07,  2.2306e-08, -5.4133e-09, -2.5605e-07,\n",
       "           6.2860e-08, -7.5285e-08, -1.1065e-07, -1.5208e-07, -3.0479e-08,\n",
       "           4.9316e-08,  6.7766e-08,  1.0247e-07, -9.5905e-08, -8.5965e-08,\n",
       "           2.1083e-08,  2.2197e-07,  5.0754e-08, -2.0659e-07, -1.0336e-07,\n",
       "          -9.2374e-08, -5.1872e-08, -4.9566e-07,  2.9554e-08,  2.0362e-07,\n",
       "          -2.6180e-09,  3.6847e-08, -1.4453e-07, -2.3543e-07,  1.4833e-07,\n",
       "           9.3070e-08,  7.9025e-08,  1.1281e-08,  1.0711e-07,  9.2672e-08,\n",
       "           4.4602e-08,  6.2573e-08,  1.8780e-08,  3.1025e-08, -4.8195e-08,\n",
       "          -1.5173e-08,  1.1228e-07, -1.7452e-07, -2.5543e-08, -1.2804e-07,\n",
       "          -3.5630e-07,  1.3448e-08, -2.9886e-08,  1.5605e-08, -9.5864e-08,\n",
       "           4.6114e-08,  1.2234e-07, -2.8767e-07,  1.9457e-08,  9.4918e-08,\n",
       "           1.6320e-07, -5.9371e-08, -1.8035e-07,  5.4930e-08, -7.1661e-08,\n",
       "           6.3309e-08,  2.8829e-08,  7.0693e-08,  4.5149e-08,  5.5471e-08,\n",
       "          -3.9404e-08,  7.4631e-08, -7.3951e-08,  4.0582e-08,  3.2118e-07,\n",
       "          -2.2044e-07, -7.3377e-09, -6.1323e-08,  1.4230e-07,  6.0617e-09,\n",
       "           7.6287e-08,  2.1655e-07,  8.8323e-09,  1.6086e-08, -4.0542e-08,\n",
       "          -1.1028e-07,  7.0081e-08,  1.3175e-07,  1.4077e-07, -1.0854e-07,\n",
       "           5.4646e-08,  2.7159e-08,  3.7363e-08,  5.6481e-09, -1.8462e-07,\n",
       "           5.1507e-08, -2.4222e-07, -1.7078e-07,  2.6885e-08, -3.7777e-08,\n",
       "          -5.4775e-08, -3.3475e-08,  1.8043e-08, -2.6002e-08,  1.0623e-07,\n",
       "          -2.9295e-08,  1.1175e-08,  3.4062e-07, -9.4083e-08, -1.3122e-08,\n",
       "           1.2139e-08,  4.2698e-08,  5.5309e-08, -3.2358e-08, -1.2346e-07,\n",
       "           2.6396e-09,  1.1075e-07, -4.0735e-08, -1.7003e-07,  4.3231e-08,\n",
       "          -1.5420e-07, -1.5166e-08,  2.9520e-08, -3.1258e-10,  1.0987e-07,\n",
       "           9.1626e-08, -7.3850e-08, -4.4047e-08,  1.2439e-07,  1.1803e-07,\n",
       "           1.7810e-07, -3.2616e-09, -7.2168e-08, -9.8500e-08,  1.8917e-07,\n",
       "           6.1860e-09, -4.9969e-09, -8.3460e-10, -1.4427e-08, -1.4171e-07,\n",
       "          -1.8042e-07,  5.7994e-08,  8.9171e-08, -5.3812e-08, -1.1186e-08,\n",
       "           1.2069e-07, -4.1306e-08,  3.8549e-08, -2.6021e-08,  5.5914e-08,\n",
       "           2.0648e-07,  2.2912e-08, -7.7152e-08,  5.2057e-08, -2.8716e-07,\n",
       "          -9.0171e-08, -2.3865e-08,  2.3779e-09, -1.6981e-07, -7.5877e-08,\n",
       "           2.6154e-07, -6.5519e-08, -1.1477e-07, -9.7396e-08, -2.5694e-08,\n",
       "          -1.0117e-07, -7.5133e-09, -1.1856e-07,  2.6017e-07, -5.2667e-08,\n",
       "           1.0610e-08,  1.9343e-08, -3.6094e-08,  1.1599e-07, -5.0230e-08,\n",
       "          -3.3382e-07,  2.1465e-07, -2.1061e-07, -3.4796e-09,  1.8067e-08,\n",
       "          -3.9974e-08,  5.6306e-08,  9.8680e-08,  1.4153e-07, -8.4846e-08,\n",
       "           2.4724e-07, -4.6981e-08,  2.4887e-07, -2.6745e-07,  9.5903e-08,\n",
       "          -2.8251e-07, -2.5444e-07, -8.0403e-08, -3.3439e-08,  1.4418e-07,\n",
       "           1.8163e-08, -1.1089e-07, -1.7096e-07,  4.3642e-09,  1.1574e-07,\n",
       "           6.3896e-08, -2.0590e-07,  2.8891e-08,  9.9786e-08,  1.1999e-07,\n",
       "           2.5415e-07, -7.2567e-08, -6.5372e-09,  1.6629e-07, -1.2370e-07,\n",
       "          -2.9782e-08,  1.8479e-07, -3.4119e-07, -4.5975e-08, -2.2810e-07,\n",
       "           3.2521e-07,  3.5040e-07, -2.5999e-07,  5.7259e-08, -9.0412e-08,\n",
       "          -1.6055e-07, -2.1732e-08,  1.2432e-07, -1.1365e-07,  4.9454e-08,\n",
       "           1.7514e-07,  8.0093e-08, -2.0677e-07,  1.1858e-07, -2.5169e-07,\n",
       "          -1.3862e-07, -3.5102e-08,  4.9936e-07,  1.2548e-07,  2.8573e-07,\n",
       "          -2.6150e-07,  2.4200e-07,  1.4818e-07,  8.9595e-08, -1.0800e-08,\n",
       "           1.1552e-08,  3.3775e-08,  4.6794e-07, -1.4711e-07, -9.5149e-08,\n",
       "          -6.9137e-08,  1.7641e-07,  1.9165e-07,  1.4456e-07, -6.6487e-08,\n",
       "          -1.7922e-07,  7.5856e-10,  1.7300e-07,  2.0469e-08, -4.2650e-07,\n",
       "          -3.5908e-08, -6.5810e-08, -3.7828e-08,  6.3509e-08,  3.0708e-07,\n",
       "           1.0413e-07, -5.5479e-08, -1.8942e-07,  9.0802e-08, -9.6585e-09,\n",
       "          -2.8615e-08,  4.9223e-08,  5.2463e-08, -7.0254e-08,  2.7888e-07,\n",
       "           4.1418e-08, -3.4391e-08,  5.9028e-09,  1.3962e-07,  2.4130e-07,\n",
       "          -3.6720e-07, -4.4258e-08,  1.4455e-07, -1.6950e-09, -8.9174e-08,\n",
       "          -4.3999e-08, -1.0885e-07,  2.7086e-07, -2.1972e-07,  1.0028e-07,\n",
       "          -2.0821e-07, -9.5543e-08, -5.7404e-08,  1.2352e-07, -6.8864e-08,\n",
       "           1.2697e-07,  1.2360e-07,  2.7013e-07, -8.9481e-09, -1.3829e-07,\n",
       "           8.3031e-08, -2.8425e-07, -2.9014e-07, -1.2196e-07, -1.0869e-07,\n",
       "          -2.1149e-07,  3.4489e-08, -1.4093e-07, -7.7525e-08,  2.1968e-07,\n",
       "          -5.0631e-08, -1.9246e-07,  1.3715e-07,  1.3899e-07,  1.1157e-07,\n",
       "          -3.3254e-08,  2.9268e-07,  1.3354e-07, -2.5957e-07, -6.3511e-09,\n",
       "           2.7985e-07,  2.5703e-07,  1.1644e-07,  3.3019e-07,  4.4714e-07,\n",
       "           2.6136e-07,  3.5940e-07, -1.8085e-07,  3.3423e-08, -3.8664e-07,\n",
       "          -1.5279e-07, -1.2050e-07,  4.2058e-07,  3.2963e-07,  9.5716e-08,\n",
       "           3.4245e-07,  1.7248e-07, -3.2550e-07, -1.5293e-07, -5.5632e-07,\n",
       "          -3.7536e-07,  9.1865e-08,  4.3624e-07, -3.6344e-07, -5.6842e-08,\n",
       "           3.4795e-07, -6.1592e-08, -3.4271e-08,  6.1030e-08, -1.0758e-07,\n",
       "          -1.8395e-07, -9.9803e-08, -5.4088e-09, -5.4233e-08, -9.2269e-09,\n",
       "           1.2381e-07, -2.5438e-07,  9.8197e-08, -9.5377e-08, -1.8550e-07,\n",
       "           1.3236e-08, -8.2342e-09, -9.9295e-08, -1.6971e-07, -1.0325e-07,\n",
       "          -9.7483e-08,  1.2700e-08, -1.7809e-08, -1.3853e-07,  3.0244e-07,\n",
       "          -7.3157e-09, -1.8094e-07,  1.0635e-07, -1.7925e-07, -1.3679e-07,\n",
       "           1.0447e-07, -6.1985e-08, -5.2934e-08,  2.6372e-08,  8.0918e-08,\n",
       "          -2.1866e-07,  1.0449e-07, -5.6597e-08,  3.6664e-08,  6.6950e-08,\n",
       "          -2.4457e-07, -8.0017e-08, -6.0411e-08, -8.5286e-08,  1.9282e-07,\n",
       "          -4.1565e-08, -1.9870e-07,  9.5402e-08,  2.8738e-08,  7.7193e-08,\n",
       "           9.0123e-08,  1.4046e-07, -1.0890e-07, -2.4012e-07, -1.7943e-07,\n",
       "          -3.9861e-08,  1.0663e-07, -2.3715e-07, -2.1413e-08,  7.6687e-08,\n",
       "           1.6217e-07, -4.2227e-07, -1.6835e-07, -1.8471e-07,  3.1524e-08,\n",
       "           2.0459e-07, -1.3775e-07, -1.6749e-08, -2.6716e-07,  2.5601e-08,\n",
       "          -1.4088e-07,  2.6604e-07, -1.6206e-07,  1.4477e-07, -1.3148e-07,\n",
       "           1.6999e-07, -1.3299e-07,  1.3983e-07,  4.5710e-08,  9.2621e-08,\n",
       "           3.8669e-08, -5.3204e-08, -4.5011e-09,  6.9478e-08, -7.7081e-08,\n",
       "          -1.6432e-07,  1.0455e-07, -1.9308e-07, -9.5926e-08,  4.8385e-08,\n",
       "           1.2267e-07, -9.1898e-08,  5.9563e-08, -1.3781e-07,  8.6945e-09,\n",
       "          -1.3107e-07, -7.0969e-08,  1.0931e-07, -1.1680e-07, -4.9075e-08,\n",
       "           3.6988e-08,  5.1331e-08,  2.0270e-08,  1.5435e-08,  6.8575e-09,\n",
       "          -1.4435e-07,  2.8441e-07, -1.9271e-07,  1.2306e-07, -1.3534e-07,\n",
       "          -9.7325e-08,  1.7742e-07,  2.1128e-08,  1.6122e-07,  7.7396e-08,\n",
       "           3.6802e-08,  2.2261e-08,  9.2789e-08,  6.7343e-09,  1.1408e-07,\n",
       "          -6.3088e-08,  1.0185e-08,  1.1353e-07,  1.5471e-07, -1.0849e-07,\n",
       "           6.8370e-08,  1.1572e-07,  1.1224e-07, -1.8016e-07,  3.3457e-07,\n",
       "           7.3493e-08, -2.0950e-07,  1.0402e-07,  4.4845e-08,  1.3230e-07,\n",
       "           3.0573e-09, -8.2084e-08,  8.3373e-08, -5.0208e-08, -1.0386e-07,\n",
       "          -2.6320e-07, -7.6536e-08, -3.7370e-07, -9.8152e-08,  9.9287e-08,\n",
       "          -2.1932e-07, -3.4836e-09,  1.0537e-07, -8.9699e-08,  2.3610e-07,\n",
       "          -3.4294e-08,  6.3627e-08, -2.9412e-07, -2.5985e-07,  7.2934e-08,\n",
       "           5.5913e-08,  2.4384e-08,  1.7635e-09,  6.7586e-08, -2.1892e-07,\n",
       "           3.4711e-08,  3.8239e-08, -2.2601e-07, -4.3452e-08,  3.3282e-07,\n",
       "           6.9132e-08, -1.2726e-07,  6.9563e-08, -6.7884e-09, -3.0566e-07,\n",
       "          -1.1005e-07,  3.8111e-08, -4.2025e-08, -3.9029e-08, -7.7983e-08,\n",
       "           2.1044e-07,  1.9684e-07,  1.3060e-07, -3.5063e-08, -6.6682e-08,\n",
       "          -1.0049e-07,  1.8091e-07,  2.7957e-07, -1.2904e-08, -6.2500e-09,\n",
       "           3.6601e-07,  6.7805e-09,  1.0812e-08, -1.8425e-07, -3.1444e-07,\n",
       "           1.7924e-07, -2.8672e-08,  5.2108e-08,  2.2712e-07,  1.3329e-07,\n",
       "           4.3251e-08,  6.7528e-08, -2.7907e-08, -1.0495e-08,  8.0822e-08,\n",
       "           1.2289e-08,  4.0666e-08,  6.9507e-08,  9.8026e-08, -2.8796e-08,\n",
       "           8.2916e-08, -4.6638e-09, -4.2722e-08,  4.5378e-08, -1.3197e-07,\n",
       "           1.7895e-08,  1.2429e-07, -1.0082e-07,  1.1847e-08, -3.8076e-08,\n",
       "          -9.2748e-08, -2.2646e-07,  3.6654e-08, -3.7660e-08, -3.4484e-09,\n",
       "          -1.8338e-07,  6.5280e-08, -6.8788e-08, -1.0414e-07, -1.5916e-07,\n",
       "           1.2688e-07,  2.1599e-09, -2.3045e-07, -5.5640e-08,  7.3935e-08,\n",
       "           4.5443e-08,  1.2464e-07, -1.1818e-07,  1.2796e-07, -3.8161e-08,\n",
       "          -1.8330e-07, -1.7057e-07,  2.0435e-08, -6.2736e-08,  3.6503e-08,\n",
       "           7.8009e-08,  1.1035e-07,  1.0935e-07,  4.8004e-08,  1.6678e-08,\n",
       "          -3.6434e-08, -1.6552e-07, -6.1854e-08, -8.3859e-08,  1.6588e-08,\n",
       "           2.4265e-07,  1.1019e-07,  5.2357e-08, -5.1577e-08, -1.1036e-07,\n",
       "          -1.4878e-07, -2.7722e-08, -8.8569e-08], requires_grad=True)),\n",
       " ('weight', Parameter containing:\n",
       "  tensor([[ 0.0267,  0.0000, -0.0713,  ..., -0.0344,  0.0055,  0.0277],\n",
       "          [ 0.0228,  0.0000, -0.0000,  ...,  0.0000,  0.1789,  0.0289],\n",
       "          [-0.0000, -0.0000, -0.0990,  ..., -0.0316,  0.0335, -0.0635],\n",
       "          ...,\n",
       "          [ 0.0040, -0.0000, -0.0924,  ..., -0.0338, -0.0250, -0.1278],\n",
       "          [ 0.0134,  0.0000,  0.0694,  ..., -0.0096, -0.0297,  0.0294],\n",
       "          [-0.0243, -0.0000, -0.0000,  ..., -0.0318, -0.0000,  0.0376]],\n",
       "         requires_grad=True))]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prune.remove(module, 'weight')\n",
    "list(module.named_parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('bias_mask',\n",
       "  tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 0., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 0., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 0., 1., 1.,\n",
       "          0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 0.,\n",
       "          0., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 0.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 0., 1., 1.,\n",
       "          1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 0., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 0., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]))]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(module.named_buffers())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([(1, <torch.nn.utils.prune.L1Unstructured at 0x26e661049b0>)])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "module._forward_pre_hooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 `weight`에 Pruning을 적용함으로 인해 생겼던 부산물들: `weight_orig`, `weight_mask` 그리고 **pre_hook**이 모두 제거되고, \n",
    "\n",
    "Pruning이 영구 적용된 텐서가 모듈의 파라미터 `weight`를 대체하게 되었습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 여러 개의 파라미터 Pruning\n",
    "\n",
    "Pruning 기법과 파라미터를 명세하여 여러 개의 텐서에 동시다발적으로 Pruning을 적용할 수도 있습니다. \n",
    "\n",
    "해당 유즈케이스는 아래 코드와 같습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_model = BertModel.from_pretrained('monologg/kobert')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "for name, module in new_model.named_modules():\n",
    "    if isinstance(module, torch.nn.Embedding):\n",
    "        prune.l1_unstructured(module, name='weight', amount=0.2)\n",
    "    elif isinstance(module, torch.nn.Linear):\n",
    "        prune.l1_unstructured(module, name='weight', amount=0.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['embeddings.word_embeddings.weight_mask', 'embeddings.position_embeddings.weight_mask', 'embeddings.token_type_embeddings.weight_mask', 'encoder.layer.0.attention.self.query.weight_mask', 'encoder.layer.0.attention.self.key.weight_mask', 'encoder.layer.0.attention.self.value.weight_mask', 'encoder.layer.0.attention.output.dense.weight_mask', 'encoder.layer.0.intermediate.dense.weight_mask', 'encoder.layer.0.output.dense.weight_mask', 'encoder.layer.1.attention.self.query.weight_mask', 'encoder.layer.1.attention.self.key.weight_mask', 'encoder.layer.1.attention.self.value.weight_mask', 'encoder.layer.1.attention.output.dense.weight_mask', 'encoder.layer.1.intermediate.dense.weight_mask', 'encoder.layer.1.output.dense.weight_mask', 'encoder.layer.2.attention.self.query.weight_mask', 'encoder.layer.2.attention.self.key.weight_mask', 'encoder.layer.2.attention.self.value.weight_mask', 'encoder.layer.2.attention.output.dense.weight_mask', 'encoder.layer.2.intermediate.dense.weight_mask', 'encoder.layer.2.output.dense.weight_mask', 'encoder.layer.3.attention.self.query.weight_mask', 'encoder.layer.3.attention.self.key.weight_mask', 'encoder.layer.3.attention.self.value.weight_mask', 'encoder.layer.3.attention.output.dense.weight_mask', 'encoder.layer.3.intermediate.dense.weight_mask', 'encoder.layer.3.output.dense.weight_mask', 'encoder.layer.4.attention.self.query.weight_mask', 'encoder.layer.4.attention.self.key.weight_mask', 'encoder.layer.4.attention.self.value.weight_mask', 'encoder.layer.4.attention.output.dense.weight_mask', 'encoder.layer.4.intermediate.dense.weight_mask', 'encoder.layer.4.output.dense.weight_mask', 'encoder.layer.5.attention.self.query.weight_mask', 'encoder.layer.5.attention.self.key.weight_mask', 'encoder.layer.5.attention.self.value.weight_mask', 'encoder.layer.5.attention.output.dense.weight_mask', 'encoder.layer.5.intermediate.dense.weight_mask', 'encoder.layer.5.output.dense.weight_mask', 'encoder.layer.6.attention.self.query.weight_mask', 'encoder.layer.6.attention.self.key.weight_mask', 'encoder.layer.6.attention.self.value.weight_mask', 'encoder.layer.6.attention.output.dense.weight_mask', 'encoder.layer.6.intermediate.dense.weight_mask', 'encoder.layer.6.output.dense.weight_mask', 'encoder.layer.7.attention.self.query.weight_mask', 'encoder.layer.7.attention.self.key.weight_mask', 'encoder.layer.7.attention.self.value.weight_mask', 'encoder.layer.7.attention.output.dense.weight_mask', 'encoder.layer.7.intermediate.dense.weight_mask', 'encoder.layer.7.output.dense.weight_mask', 'encoder.layer.8.attention.self.query.weight_mask', 'encoder.layer.8.attention.self.key.weight_mask', 'encoder.layer.8.attention.self.value.weight_mask', 'encoder.layer.8.attention.output.dense.weight_mask', 'encoder.layer.8.intermediate.dense.weight_mask', 'encoder.layer.8.output.dense.weight_mask', 'encoder.layer.9.attention.self.query.weight_mask', 'encoder.layer.9.attention.self.key.weight_mask', 'encoder.layer.9.attention.self.value.weight_mask', 'encoder.layer.9.attention.output.dense.weight_mask', 'encoder.layer.9.intermediate.dense.weight_mask', 'encoder.layer.9.output.dense.weight_mask', 'encoder.layer.10.attention.self.query.weight_mask', 'encoder.layer.10.attention.self.key.weight_mask', 'encoder.layer.10.attention.self.value.weight_mask', 'encoder.layer.10.attention.output.dense.weight_mask', 'encoder.layer.10.intermediate.dense.weight_mask', 'encoder.layer.10.output.dense.weight_mask', 'encoder.layer.11.attention.self.query.weight_mask', 'encoder.layer.11.attention.self.key.weight_mask', 'encoder.layer.11.attention.self.value.weight_mask', 'encoder.layer.11.attention.output.dense.weight_mask', 'encoder.layer.11.intermediate.dense.weight_mask', 'encoder.layer.11.output.dense.weight_mask', 'pooler.dense.weight_mask'])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dict(new_model.named_buffers()).keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 글로벌 Pruning\n",
    "\n",
    "지금까지 저희가 살펴본 예제들은 모델 내 존재하는 텐서들을 개별적으로 가지치기하는 **로컬 Pruning**의 예였습니다. \n",
    "\n",
    "그러나 가장 효과적이고 대중적인 Pruning 방법은 모델 전체에 Pruning을 한 번에 적용하는 **글로벌 Pruning** 입니다. \n",
    "\n",
    "아래 예제와 같이 모델 전체에 **글로벌 Pruning**을 적용하게 되면 앞선 예제들에서처럼 개별 텐서에서 영향력이 작은 파라미터가 가지치기 하는 것이 아닌, <br/>**모듈 간 연결**에 있어 영향력이 작은 파라미터들이 가지치기 하게 됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_model = BertModel.from_pretrained('monologg/kobert')\n",
    "\n",
    "parameters_to_prune = ()\n",
    "for i in range(12):\n",
    "    parameters_to_prune += (\n",
    "        (final_model.encoder.layer[i].attention.self.key, 'weight'),\n",
    "        (final_model.encoder.layer[i].attention.self.query, 'weight'),\n",
    "        (final_model.encoder.layer[i].attention.self.value, 'weight'),\n",
    "    )\n",
    "\n",
    "prune.global_unstructured(\n",
    "    parameters_to_prune,\n",
    "    pruning_method=prune.L1Unstructured,\n",
    "    amount=0.2,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sparsity in Layer 1-th key weight: 18.36%\n",
      "Sparsity in Layer 1-th query weightt: 18.93%\n",
      "Sparsity in Layer 1-th value weight: 28.16%\n",
      "\n",
      "Sparsity in Layer 2-th key weight: 16.87%\n",
      "Sparsity in Layer 2-th query weightt: 17.30%\n",
      "Sparsity in Layer 2-th value weight: 30.48%\n",
      "\n",
      "Sparsity in Layer 3-th key weight: 16.98%\n",
      "Sparsity in Layer 3-th query weightt: 16.89%\n",
      "Sparsity in Layer 3-th value weight: 27.35%\n",
      "\n",
      "Sparsity in Layer 4-th key weight: 17.22%\n",
      "Sparsity in Layer 4-th query weightt: 17.20%\n",
      "Sparsity in Layer 4-th value weight: 27.11%\n",
      "\n",
      "Sparsity in Layer 5-th key weight: 17.35%\n",
      "Sparsity in Layer 5-th query weightt: 17.37%\n",
      "Sparsity in Layer 5-th value weight: 26.13%\n",
      "\n",
      "Sparsity in Layer 6-th key weight: 17.03%\n",
      "Sparsity in Layer 6-th query weightt: 17.20%\n",
      "Sparsity in Layer 6-th value weight: 26.90%\n",
      "\n",
      "Sparsity in Layer 7-th key weight: 16.89%\n",
      "Sparsity in Layer 7-th query weightt: 16.98%\n",
      "Sparsity in Layer 7-th value weight: 24.79%\n",
      "\n",
      "Sparsity in Layer 8-th key weight: 16.84%\n",
      "Sparsity in Layer 8-th query weightt: 16.76%\n",
      "Sparsity in Layer 8-th value weight: 25.74%\n",
      "\n",
      "Sparsity in Layer 9-th key weight: 17.02%\n",
      "Sparsity in Layer 9-th query weightt: 16.86%\n",
      "Sparsity in Layer 9-th value weight: 24.70%\n",
      "\n",
      "Sparsity in Layer 10-th key weight: 16.59%\n",
      "Sparsity in Layer 10-th query weightt: 16.55%\n",
      "Sparsity in Layer 10-th value weight: 25.15%\n",
      "\n",
      "Sparsity in Layer 11-th key weight: 16.35%\n",
      "Sparsity in Layer 11-th query weightt: 16.17%\n",
      "Sparsity in Layer 11-th value weight: 24.10%\n",
      "\n",
      "Sparsity in Layer 12-th key weight: 16.18%\n",
      "Sparsity in Layer 12-th query weightt: 16.19%\n",
      "Sparsity in Layer 12-th value weight: 21.34%\n",
      "\n",
      "Global sparsity: 20.00%\n"
     ]
    }
   ],
   "source": [
    "for i in range(12):\n",
    "    print(\n",
    "        \"Sparsity in Layer {}-th key weight: {:.2f}%\".format(\n",
    "            i+1,\n",
    "            100. * float(torch.sum(final_model.encoder.layer[i].attention.self.key.weight == 0))\n",
    "            / float(final_model.encoder.layer[i].attention.self.key.weight.nelement())\n",
    "        )\n",
    "    )\n",
    "    print(\n",
    "        \"Sparsity in Layer {}-th query weightt: {:.2f}%\".format(\n",
    "            i+1,\n",
    "            100. * float(torch.sum(final_model.encoder.layer[i].attention.self.query.weight == 0))\n",
    "            / float(final_model.encoder.layer[i].attention.self.query.weight.nelement())\n",
    "        )\n",
    "    )\n",
    "    print(\n",
    "        \"Sparsity in Layer {}-th value weight: {:.2f}%\".format(\n",
    "            i+1,\n",
    "            100. * float(torch.sum(final_model.encoder.layer[i].attention.self.value.weight == 0))\n",
    "            / float(final_model.encoder.layer[i].attention.self.value.weight.nelement())\n",
    "        )\n",
    "    )\n",
    "    print()\n",
    "\n",
    "    \n",
    "numerator, denominator = 0, 0\n",
    "for i in range(12):\n",
    "    numerator += torch.sum(final_model.encoder.layer[i].attention.self.key.weight == 0)\n",
    "    numerator += torch.sum(final_model.encoder.layer[i].attention.self.query.weight == 0)\n",
    "    numerator += torch.sum(final_model.encoder.layer[i].attention.self.value.weight == 0)\n",
    "\n",
    "    denominator += final_model.encoder.layer[i].attention.self.key.weight.nelement()\n",
    "    denominator += final_model.encoder.layer[i].attention.self.query.weight.nelement()\n",
    "    denominator += final_model.encoder.layer[i].attention.self.value.weight.nelement()\n",
    "    \n",
    "print(\"Global sparsity: {:.2f}%\".format(100. * float(numerator) / float(denominator)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "위 결과를 통해 모듈 내 각 파라미터의 **Pruning** 비율은 **20%**가 되지 않지만 전체 Sparsity가 20%가 되는 것을 확인할 수 있습니다.\n",
    "\n",
    "본 튜토리얼에서 살펴본 바와 같이 `utils.prune` 모듈을 활용하면 모델 내 여러 모듈들에 다양한 Pruning 기법을 적용해볼 수 있습니다.\n",
    "\n",
    "그리고 `utils.prune` 모듈에서 제공하는 추상 클래스를 활용해 본인의 커스텀 Pruning 메서드 또한 손쉽게 작성할 수 있습니다.\n",
    "\n",
    "해당 모듈은 연구자분들도 많이 사용하는 모듈이라고 하니, 실험 시 활용해보심을 고려해봐도 좋을 것 같아 이번 기회를 통해 소개드렸습니다. 많은 분들께 도움이 되는 튜토리얼이 되었기를 바랍니다!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}