dog_app.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional Neural Networks\n",
    "\n",
    "## Project: Write an Algorithm for a Dog Identification App \n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, some template code has already been provided for you, and you will need to implement additional functionality to successfully complete this project. You will not need to modify the included code beyond what is requested. Sections that begin with **'(IMPLEMENTATION)'** in the header indicate that the following block of code will require additional functionality which you must provide. Instructions will be provided for each section, and the specifics of the implementation are marked in the code block with a 'TODO' statement. Please be sure to read the instructions carefully! \n",
    "\n",
    "> **Note**: Once you have completed all of the code implementations, you need to finalize your work by exporting the Jupyter Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You can then export the notebook by using the menu above and navigating to **File -> Download as -> HTML (.html)**. Include the finished document along with this notebook as your submission.\n",
    "\n",
    "In addition to implementing code, there will be questions that you must answer which relate to the project and your implementation. Each section where you will answer a question is preceded by a **'Question X'** header. Carefully read each question and provide thorough answers in the following text boxes that begin with **'Answer:'**. Your project submission will be evaluated based on your answers to each of the questions and the implementation you provide.\n",
    "\n",
    ">**Note:** Code and Markdown cells can be executed using the **Shift + Enter** keyboard shortcut.  Markdown cells can be edited by double-clicking the cell to enter edit mode.\n",
    "\n",
    "The rubric contains _optional_ \"Stand Out Suggestions\" for enhancing the project beyond the minimum requirements. If you decide to pursue the \"Stand Out Suggestions\", you should include the code in this Jupyter notebook.\n",
    "\n",
    "\n",
    "\n",
    "---\n",
    "### Why We're Here \n",
    "\n",
    "In this notebook, you will make the first steps towards developing an algorithm that could be used as part of a mobile or web app.  At the end of this project, your code will accept any user-supplied image as input.  If a dog is detected in the image, it will provide an estimate of the dog's breed.  If a human is detected, it will provide an estimate of the dog breed that is most resembling.  The image below displays potential sample output of your finished project (... but we expect that each student's algorithm will behave differently!). \n",
    "\n",
    "![Sample Dog Output](images/sample_dog_output.png)\n",
    "\n",
    "In this real-world setting, you will need to piece together a series of models to perform different tasks; for instance, the algorithm that detects humans in an image will be different from the CNN that infers dog breed.  There are many points of possible failure, and no perfect algorithm exists.  Your imperfect solution will nonetheless create a fun user experience!\n",
    "\n",
    "### The Road Ahead\n",
    "\n",
    "We break the notebook into separate steps.  Feel free to use the links below to navigate the notebook.\n",
    "\n",
    "* [Step 0](#step0): Import Datasets\n",
    "* [Step 1](#step1): Detect Humans\n",
    "* [Step 2](#step2): Detect Dogs\n",
    "* [Step 3](#step3): Create a CNN to Classify Dog Breeds (from Scratch)\n",
    "* [Step 4](#step4): Create a CNN to Classify Dog Breeds (using Transfer Learning)\n",
    "* [Step 5](#step5): Write your Algorithm\n",
    "* [Step 6](#step6): Test Your Algorithm\n",
    "\n",
    "---\n",
    "<a id='step0'></a>\n",
    "## Step 0: Import Datasets\n",
    "\n",
    "Make sure that you've downloaded the required human and dog datasets:\n",
    "* Download the [dog dataset](https://s3-us-west-1.amazonaws.com/udacity-aind/dog-project/dogImages.zip).  Unzip the folder and place it in this project's home directory, at the location `/dog_images`. \n",
    "\n",
    "* Download the [human dataset](https://s3-us-west-1.amazonaws.com/udacity-aind/dog-project/lfw.zip).  Unzip the folder and place it in the home directory, at location `/lfw`.  \n",
    "\n",
    "*Note: If you are using a Windows machine, you are encouraged to use [7zip](http://www.7-zip.org/) to extract the folder.*\n",
    "\n",
    "In the code cell below, we save the file paths for both the human (LFW) dataset and dog dataset in the numpy arrays `human_files` and `dog_files`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 13233 total human images.\n",
      "There are 8351 total dog images.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from glob import glob\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "# load filenames for human and dog images\n",
    "human_files = np.array(glob(\"data/lfw/*/*\"))\n",
    "dog_files = np.array(glob(\"data/dogImages/*/*/*\"))\n",
    "\n",
    "# print number of images in each dataset\n",
    "print('There are %d total human images.' % len(human_files))\n",
    "print('There are %d total dog images.' % len(dog_files))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='step1'></a>\n",
    "## Step 1: Detect Humans\n",
    "\n",
    "In this section, we use OpenCV's implementation of [Haar feature-based cascade classifiers](http://docs.opencv.org/trunk/d7/d8b/tutorial_py_face_detection.html) to detect human faces in images.  \n",
    "\n",
    "OpenCV provides many pre-trained face detectors, stored as XML files on [github](https://github.com/opencv/opencv/tree/master/data/haarcascades).  We have downloaded one of these detectors and stored it in the `haarcascades` directory.  In the next code cell, we demonstrate how to use this detector to find human faces in a sample image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2                \n",
    "import matplotlib.pyplot as plt                        \n",
    "%matplotlib inline                               \n",
    "\n",
    "# extract pre-trained face detector\n",
    "# face_cascade = cv2.CascadeClassifier('haarcascades/haarcascade_frontalface_alt.xml')\n",
    "face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_alt.xml')\n",
    "\n",
    "# load color (BGR) image\n",
    "img = cv2.imread(human_files[0])\n",
    "# convert BGR image to grayscale\n",
    "gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
    "\n",
    "# find faces in image\n",
    "faces = face_cascade.detectMultiScale(gray)\n",
    "\n",
    "# print number of faces detected in the image\n",
    "print('Number of faces detected:', len(faces))\n",
    "\n",
    "# get bounding box for each detected face\n",
    "for (x,y,w,h) in faces:\n",
    "    # add bounding box to color image\n",
    "    cv2.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)\n",
    "    \n",
    "# convert BGR image to RGB for plotting\n",
    "cv_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n",
    "\n",
    "# display the image, along with bounding box\n",
    "plt.imshow(cv_rgb)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before using any of the face detectors, it is standard procedure to convert the images to grayscale.  The `detectMultiScale` function executes the classifier stored in `face_cascade` and takes the grayscale image as a parameter.  \n",
    "\n",
    "In the above code, `faces` is a numpy array of detected faces, where each row corresponds to a detected face.  Each detected face is a 1D array with four entries that specifies the bounding box of the detected face.  The first two entries in the array (extracted in the above code as `x` and `y`) specify the horizontal and vertical positions of the top left corner of the bounding box.  The last two entries in the array (extracted here as `w` and `h`) specify the width and height of the box.\n",
    "\n",
    "### Write a Human Face Detector\n",
    "\n",
    "We can use this procedure to write a function that returns `True` if a human face is detected in an image and `False` otherwise.  This function, aptly named `face_detector`, takes a string-valued file path to an image as input and appears in the code block below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# returns \"True\" if face is detected in image stored at img_path\n",
    "def face_detector(img_path):\n",
    "    img = cv2.imread(img_path)\n",
    "    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
    "    faces = face_cascade.detectMultiScale(gray)\n",
    "    return len(faces) > 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_face(img_path):\n",
    "    img = cv2.imread(img_path)\n",
    "    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
    "    faces = face_cascade.detectMultiScale(gray)\n",
    "    for (x,y,w,h) in faces:\n",
    "        # add bounding box to color image\n",
    "        cv2.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)\n",
    "\n",
    "    # convert BGR image to RGB for plotting\n",
    "    cv_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n",
    "\n",
    "    # display the image, along with bounding box\n",
    "    plt.imshow(cv_rgb)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (IMPLEMENTATION) Assess the Human Face Detector\n",
    "\n",
    "__Question 1:__ Use the code cell below to test the performance of the `face_detector` function.  \n",
    "- What percentage of the first 100 images in `human_files` have a detected human face?  \n",
    "- What percentage of the first 100 images in `dog_files` have a detected human face? \n",
    "\n",
    "Ideally, we would like 100% of human images with a detected face and 0% of dog images with a detected face.  You will see that our algorithm falls short of this goal, but still gives acceptable performance.  We extract the file paths for the first 100 images from each of the datasets and store them in the numpy arrays `human_files_short` and `dog_files_short`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Answer:__ \n",
    "(You can print out your results and/or write your percentages in this cell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "human_files_short = human_files[:100]\n",
    "dog_files_short = dog_files[:100]\n",
    "\n",
    "#-#-# Do NOT modify the code above this line. #-#-#\n",
    "\n",
    "## TODO: Test the performance of the face_detector algorithm \n",
    "## on the images in human_files_short and dog_files_short.\n",
    "face_det_res = [face_detector(human_files_short[i]) for i in tqdm(range(len(human_files_short)))]\n",
    "print(\"Face detector:\", len(np.nonzero(face_det_res)[0]), '%')\n",
    "\n",
    "face_det_dogs = [face_detector(dog_files_short[i]) for i in tqdm(range(len(dog_files_short)))]\n",
    "print(\"Face detector on dogs:\", len(np.nonzero(face_det_dogs)[0]), '%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.nonzero(face_det_dogs)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# show all face positive cases on dog images\n",
    "for idx in np.nonzero(face_det_dogs)[0]:\n",
    "    print(dog_files_short[idx])\n",
    "    show_face(dog_files_short[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We suggest the face detector from OpenCV as a potential way to detect human images in your algorithm, but you are free to explore other approaches, especially approaches that make use of deep learning :).  Please use the code cell below to design and test your own face detection algorithm.  If you decide to pursue this _optional_ task, report performance on `human_files_short` and `dog_files_short`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### (Optional) \n",
    "### TODO: Test performance of anotherface detection algorithm.\n",
    "### Feel free to use as many code cells as needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a id='step2'></a>\n",
    "## Step 2: Detect Dogs\n",
    "\n",
    "In this section, we use a [pre-trained model](http://pytorch.org/docs/master/torchvision/models.html) to detect dogs in images.  \n",
    "\n",
    "### Obtain Pre-trained VGG-16 Model\n",
    "\n",
    "The code cell below downloads the VGG-16 model, along with weights that have been trained on [ImageNet](http://www.image-net.org/), a very large, very popular dataset used for image classification and other vision tasks.  ImageNet contains over 10 million URLs, each linking to an image containing an object from one of [1000 categories](https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision.models as models\n",
    "\n",
    "# define VGG16 model\n",
    "VGG16 = models.vgg16(pretrained=True)\n",
    "\n",
    "# check if CUDA is available\n",
    "use_cuda = torch.cuda.is_available()\n",
    "\n",
    "# move model to GPU if CUDA is available\n",
    "if use_cuda:\n",
    "    VGG16 = VGG16.cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given an image, this pre-trained VGG-16 model returns a prediction (derived from the 1000 possible categories in ImageNet) for the object that is contained in the image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (IMPLEMENTATION) Making Predictions with a Pre-trained Model\n",
    "\n",
    "In the next code cell, you will write a function that accepts a path to an image (such as `'dogImages/train/001.Affenpinscher/Affenpinscher_00001.jpg'`) as input and returns the index corresponding to the ImageNet class that is predicted by the pre-trained VGG-16 model.  The output should always be an integer between 0 and 999, inclusive.\n",
    "\n",
    "Before writing the function, make sure that you take the time to learn  how to appropriately pre-process tensors for pre-trained models in the [PyTorch documentation](http://pytorch.org/docs/stable/torchvision/models.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "import torchvision.transforms as transforms\n",
    "import torch.nn.functional as F\n",
    "\n",
    "# Set PIL to be tolerant of image files that are truncated.\n",
    "from PIL import ImageFile\n",
    "ImageFile.LOAD_TRUNCATED_IMAGES = True\n",
    "\n",
    "normalize = transforms.Normalize(\n",
    "   mean=[0.485, 0.456, 0.406],\n",
    "   std=[0.229, 0.224, 0.225]\n",
    ")\n",
    "preprocess = transforms.Compose([\n",
    "   transforms.Resize(256),\n",
    "   transforms.CenterCrop(224),\n",
    "   transforms.ToTensor(),\n",
    "   normalize\n",
    "])\n",
    "\n",
    "def VGG16_predict(img_path):\n",
    "    '''\n",
    "    Use pre-trained VGG-16 model to obtain index corresponding to \n",
    "    predicted ImageNet class for image at specified path\n",
    "    \n",
    "    Args:\n",
    "        img_path: path to an image\n",
    "        \n",
    "    Returns:\n",
    "        Index corresponding to VGG-16 model's prediction\n",
    "    '''\n",
    "    \n",
    "    ## TODO: Complete the function.\n",
    "    ## Load and pre-process an image from the given img_path\n",
    "    ## Return the *index* of the predicted class for that image\n",
    "    \n",
    "    img_pil = Image.open(img_path)\n",
    "    img_tensor = preprocess(img_pil)\n",
    "    img_tensor.unsqueeze_(0)\n",
    "    fc_out = VGG16(img_tensor.cuda())   \n",
    "    output = F.softmax(fc_out)    \n",
    "    values, indices = output.max(1)\n",
    "    \n",
    "    if use_cuda:\n",
    "        return indices[0].cpu().numpy()#res # predicted class index\n",
    "    else:\n",
    "        return indices[0].numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(VGG16_predict(dog_files[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (IMPLEMENTATION) Write a Dog Detector\n",
    "\n",
    "While looking at the [dictionary](https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a), you will notice that the categories corresponding to dogs appear in an uninterrupted sequence and correspond to dictionary keys 151-268, inclusive, to include all categories from `'Chihuahua'` to `'Mexican hairless'`.  Thus, in order to check to see if an image is predicted to contain a dog by the pre-trained VGG-16 model, we need only check if the pre-trained model predicts an index between 151 and 268 (inclusive).\n",
    "\n",
    "Use these ideas to complete the `dog_detector` function below, which returns `True` if a dog is detected in an image (and `False` if not)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### returns \"True\" if a dog is detected in the image stored at img_path\n",
    "def dog_detector(img_path):\n",
    "    ## TODO: Complete the function.\n",
    "    class_id = VGG16_predict(img_path)    \n",
    "    return class_id in range(151, 269) # true/false"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (IMPLEMENTATION) Assess the Dog Detector\n",
    "\n",
    "__Question 2:__ Use the code cell below to test the performance of your `dog_detector` function.  \n",
    "- What percentage of the images in `human_files_short` have a detected dog?  \n",
    "- What percentage of the images in `dog_files_short` have a detected dog?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Answer:__ \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### TODO: Test the performance of the dog_detector function\n",
    "### on the images in human_files_short and dog_files_short.\n",
    "dog_det_res = [dog_detector(dog_files_short[i]) for i in tqdm(range(len(dog_files_short)))]\n",
    "print('Dog detector on dogs:', len(np.nonzero(dog_det_res)[0]), '%')\n",
    "\n",
    "dog_det_res = [dog_detector(human_files_short[i]) for i in tqdm(range(len(human_files_short)))]\n",
    "print('Dog detector on humans:', len(np.nonzero(dog_det_res)[0]), '%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We suggest VGG-16 as a potential network to detect dog images in your algorithm, but you are free to explore other pre-trained networks (such as [Inception-v3](http://pytorch.org/docs/master/torchvision/models.html#inception-v3), [ResNet-50](http://pytorch.org/docs/master/torchvision/models.html#id3), etc).  Please use the code cell below to test other pre-trained PyTorch models.  If you decide to pursue this _optional_ task, report performance on `human_files_short` and `dog_files_short`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### (Optional) \n",
    "### TODO: Report the performance of another pre-trained network.\n",
    "### Feel free to use as many code cells as needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a id='step3'></a>\n",
    "## Step 3: Create a CNN to Classify Dog Breeds (from Scratch)\n",
    "\n",
    "Now that we have functions for detecting humans and dogs in images, we need a way to predict breed from images.  In this step, you will create a CNN that classifies dog breeds.  You must create your CNN _from scratch_ (so, you can't use transfer learning _yet_!), and you must attain a test accuracy of at least 10%.  In Step 4 of this notebook, you will have the opportunity to use transfer learning to create a CNN that attains greatly improved accuracy.\n",
    "\n",
    "We mention that the task of assigning breed to dogs from images is considered exceptionally challenging.  To see why, consider that *even a human* would have trouble distinguishing between a Brittany and a Welsh Springer Spaniel.  \n",
    "\n",
    "Brittany | Welsh Springer Spaniel\n",
    "- | - \n",
    "<img src=\"images/Brittany_02625.jpg\" width=\"100\"> | <img src=\"images/Welsh_springer_spaniel_08203.jpg\" width=\"200\">\n",
    "\n",
    "It is not difficult to find other dog breed pairs with minimal inter-class variation (for instance, Curly-Coated Retrievers and American Water Spaniels).  \n",
    "\n",
    "Curly-Coated Retriever | American Water Spaniel\n",
    "- | -\n",
    "<img src=\"images/Curly-coated_retriever_03896.jpg\" width=\"200\"> | <img src=\"images/American_water_spaniel_00648.jpg\" width=\"200\">\n",
    "\n",
    "\n",
    "Likewise, recall that labradors come in yellow, chocolate, and black.  Your vision-based algorithm will have to conquer this high intra-class variation to determine how to classify all of these different shades as the same breed.  \n",
    "\n",
    "Yellow Labrador | Chocolate Labrador | Black Labrador\n",
    "- | -\n",
    "<img src=\"images/Labrador_retriever_06457.jpg\" width=\"150\"> | <img src=\"images/Labrador_retriever_06455.jpg\" width=\"240\"> | <img src=\"images/Labrador_retriever_06449.jpg\" width=\"220\">\n",
    "\n",
    "We also mention that random chance presents an exceptionally low bar: setting aside the fact that the classes are slightly imabalanced, a random guess will provide a correct answer roughly 1 in 133 times, which corresponds to an accuracy of less than 1%.  \n",
    "\n",
    "Remember that the practice is far ahead of the theory in deep learning.  Experiment with many different architectures, and trust your intuition.  And, of course, have fun!\n",
    "\n",
    "### (IMPLEMENTATION) Specify Data Loaders for the Dog Dataset\n",
    "\n",
    "Use the code cell below to write three separate [data loaders](http://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader) for the training, validation, and test datasets of dog images (located at `dog_images/train`, `dog_images/valid`, and `dog_images/test`, respectively).  You may find [this documentation on custom datasets](http://pytorch.org/docs/stable/torchvision/datasets.html) to be a useful resource.  If you are interested in augmenting your training and/or validation data, check out the wide variety of [transforms](http://pytorch.org/docs/stable/torchvision/transforms.html?highlight=transform)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import torch\n",
    "from torchvision import datasets\n",
    "import numpy as np\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "# check if CUDA is available\n",
    "use_cuda = torch.cuda.is_available()\n",
    "\n",
    "### TODO: Write data loaders for training, validation, and test sets\n",
    "## Specify appropriate transforms, and batch_sizes\n",
    "\n",
    "normalize = transforms.Normalize(\n",
    "   mean=[0.485, 0.456, 0.406],\n",
    "   std=[0.229, 0.224, 0.225]\n",
    ")\n",
    "preprocess = transforms.Compose([\n",
    "   transforms.Resize(256),\n",
    "   transforms.CenterCrop(224),\n",
    "   transforms.ToTensor(),\n",
    "   normalize\n",
    "])\n",
    "\n",
    "train_data = datasets.ImageFolder(\"data/dogImages/train/\", transform=preprocess)\n",
    "valid_data = datasets.ImageFolder(\"data/dogImages/valid/\", transform=preprocess)\n",
    "test_data = datasets.ImageFolder(\"data/dogImages/test/\", transform=preprocess)\n",
    "\n",
    "# define dataloader parameters\n",
    "batch_size = 32\n",
    "num_workers=0\n",
    "\n",
    "# prepare data loaders\n",
    "loaders_scratch = {}\n",
    "loaders_scratch['train'] = torch.utils.data.DataLoader(train_data, batch_size=batch_size, \n",
    "                                           num_workers=num_workers, shuffle=True)\n",
    "loaders_scratch['valid'] = torch.utils.data.DataLoader(test_data, batch_size=batch_size, \n",
    "                                          num_workers=num_workers, shuffle=True)\n",
    "loaders_scratch['test'] = torch.utils.data.DataLoader(test_data, batch_size=batch_size, \n",
    "                                          num_workers=num_workers, shuffle=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 3:** Describe your chosen procedure for preprocessing the data. \n",
    "- How does your code resize the images (by cropping, stretching, etc)?  What size did you pick for the input tensor, and why?\n",
    "- Did you decide to augment the dataset?  If so, how (through translations, flips, rotations, etc)?  If not, why not?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer**:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (IMPLEMENTATION) Model Architecture\n",
    "\n",
    "Create a CNN to classify dog breed.  Use the template in the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{151: 'Chihuahua', 152: 'Japanese spaniel', 153: 'Maltese dog, Maltese terrier, Maltese', 154: 'Pekinese, Pekingese, Peke', 155: 'Shih-Tzu', 156: 'Blenheim spaniel', 157: 'papillon', 158: 'toy terrier', 159: 'Rhodesian ridgeback', 160: 'Afghan hound, Afghan', 161: 'basset, basset hound', 162: 'beagle', 163: 'bloodhound, sleuthhound', 164: 'bluetick', 165: 'black-and-tan coonhound', 166: 'Walker hound, Walker foxhound', 167: 'English foxhound', 168: 'redbone', 169: 'borzoi, Russian wolfhound', 170: 'Irish wolfhound', 171: 'Italian greyhound', 172: 'whippet', 173: 'Ibizan hound, Ibizan Podenco', 174: 'Norwegian elkhound, elkhound', 175: 'otterhound, otter hound', 176: 'Saluki, gazelle hound', 177: 'Scottish deerhound, deerhound', 178: 'Weimaraner', 179: 'Staffordshire bullterrier, Staffordshire bull terrier', 180: 'American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier', 181: 'Bedlington terrier', 182: 'Border terrier', 183: 'Kerry blue terrier', 184: 'Irish terrier', 185: 'Norfolk terrier', 186: 'Norwich terrier', 187: 'Yorkshire terrier', 188: 'wire-haired fox terrier', 189: 'Lakeland terrier', 190: 'Sealyham terrier, Sealyham', 191: 'Airedale, Airedale terrier', 192: 'cairn, cairn terrier', 193: 'Australian terrier', 194: 'Dandie Dinmont, Dandie Dinmont terrier', 195: 'Boston bull, Boston terrier', 196: 'miniature schnauzer', 197: 'giant schnauzer', 198: 'standard schnauzer', 199: 'Scotch terrier, Scottish terrier, Scottie', 200: 'Tibetan terrier, chrysanthemum dog', 201: 'silky terrier, Sydney silky', 202: 'soft-coated wheaten terrier', 203: 'West Highland white terrier', 204: 'Lhasa, Lhasa apso', 205: 'flat-coated retriever', 206: 'curly-coated retriever', 207: 'golden retriever', 208: 'Labrador retriever', 209: 'Chesapeake Bay retriever', 210: 'German short-haired pointer', 211: 'vizsla, Hungarian pointer', 212: 'English setter', 213: 'Irish setter, red setter', 214: 'Gordon setter', 215: 'Brittany spaniel', 216: 'clumber, clumber spaniel', 217: 'English springer, English springer spaniel', 218: 'Welsh springer spaniel', 219: 'cocker spaniel, English cocker spaniel, cocker', 220: 'Sussex spaniel', 221: 'Irish water spaniel', 222: 'kuvasz', 223: 'schipperke', 224: 'groenendael', 225: 'malinois', 226: 'briard', 227: 'kelpie', 228: 'komondor', 229: 'Old English sheepdog, bobtail', 230: 'Shetland sheepdog, Shetland sheep dog, Shetland', 231: 'collie', 232: 'Border collie', 233: 'Bouvier des Flandres, Bouviers des Flandres', 234: 'Rottweiler', 235: 'German shepherd, German shepherd dog, German police dog, alsatian', 236: 'Doberman, Doberman pinscher', 237: 'miniature pinscher', 238: 'Greater Swiss Mountain dog', 239: 'Bernese mountain dog', 240: 'Appenzeller', 241: 'EntleBucher', 242: 'boxer', 243: 'bull mastiff', 244: 'Tibetan mastiff', 245: 'French bulldog', 246: 'Great Dane', 247: 'Saint Bernard, St Bernard', 248: 'Eskimo dog, husky', 249: 'malamute, malemute, Alaskan malamute', 250: 'Siberian husky', 251: 'dalmatian, coach dog, carriage dog', 252: 'affenpinscher, monkey pinscher, monkey dog', 253: 'basenji', 254: 'pug, pug-dog', 255: 'Leonberg', 256: 'Newfoundland, Newfoundland dog', 257: 'Great Pyrenees', 258: 'Samoyed, Samoyede', 259: 'Pomeranian', 260: 'chow, chow chow', 261: 'keeshond', 262: 'Brabancon griffon', 263: 'Pembroke, Pembroke Welsh corgi', 264: 'Cardigan, Cardigan Welsh corgi', 265: 'toy poodle', 266: 'miniature poodle', 267: 'standard poodle', 268: 'Mexican hairless', 269: 'timber wolf, grey wolf, gray wolf, Canis lupus'}\n"
     ]
    }
   ],
   "source": [
    "# load imagenet 1000 classes dict\n",
    "import json\n",
    "classes_path = \"classes_imagenet.json\"\n",
    "with open(classes_path) as json_file:\n",
    "    classes_dict = json.load(json_file)\n",
    "classes_dict = {int(k): v for k, v in classes_dict.items()}\n",
    "# print(classes_dict)\n",
    "classes_dogs = {k : v for k, v in classes_dict.items() if k in list(range(151, 270))}\n",
    "print(classes_dogs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Net(\n",
      "  (conv1): Conv2d(3, 48, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))\n",
      "  (conv2): Conv2d(48, 128, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))\n",
      "  (conv3): Conv2d(128, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "  (conv4): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "  (conv5): Conv2d(192, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (fc1): Linear(in_features=4608, out_features=2048, bias=True)\n",
      "  (fc2): Linear(in_features=2048, out_features=2048, bias=True)\n",
      "  (fc3): Linear(in_features=2048, out_features=133, bias=True)\n",
      "  (dropout): Dropout(p=0.5, inplace=False)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch import sigmoid\n",
    "\n",
    "# NCLASSES = 269 - 151\n",
    "NCLASSES = 133\n",
    "DEBUG = 0\n",
    "\n",
    "# define the CNN architecture\n",
    "class Net(nn.Module):\n",
    "    ### TODO: choose an architecture, and complete the class\n",
    "    # A simple AlexNet architecture as baseline model\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 48, 11, 4, padding=2)\n",
    "        self.conv2 = nn.Conv2d(48, 128, 5, 1, padding=2)\n",
    "        self.conv3 = nn.Conv2d(128, 192, 3, 1, padding=1)\n",
    "        self.conv4 = nn.Conv2d(192, 192, 3, 1, padding=1)\n",
    "        self.conv5 = nn.Conv2d(192, 128, 3, 1, padding=1)\n",
    "        \n",
    "        self.pool = nn.MaxPool2d(2, 2)\n",
    "        \n",
    "        self.fc1 = nn.Linear(6 * 6 * 128, 2048, bias=True)\n",
    "        self.fc2 = nn.Linear(2048, 2048, bias=True)\n",
    "        self.fc3 = nn.Linear(2048, NCLASSES, bias=True)\n",
    "        \n",
    "        self.dropout = nn.Dropout(0.5)\n",
    "        \n",
    "    \n",
    "    def forward(self, x):\n",
    "        ## Define forward behavior\n",
    "        x = F.relu(self.conv1(x))\n",
    "        x = self.pool(x)\n",
    "        x = F.relu(self.conv2(x))\n",
    "        x = self.pool(x)\n",
    "        x = F.relu(self.conv3(x))\n",
    "        x = F.relu(self.conv4(x))\n",
    "        x = F.relu(self.conv5(x))\n",
    "        x = self.pool(x)\n",
    "        x = x.view(-1, 6 * 6 * 128)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = self.dropout(x)\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.dropout(x)\n",
    "        x = self.fc3(x)\n",
    "        x = sigmoid(x)\n",
    "        \n",
    "        return x\n",
    "\n",
    "#-#-# You do NOT have to modify the code below this line. #-#-#\n",
    "\n",
    "# instantiate the CNN\n",
    "model_scratch = Net()\n",
    "\n",
    "# move tensors to GPU if CUDA is available\n",
    "if use_cuda:\n",
    "    model_scratch.cuda()\n",
    "print(model_scratch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of memory allocated on GPU: 400.39 MB\n"
     ]
    }
   ],
   "source": [
    "print(\"Total number of memory allocated on GPU: {:.2f} MB\".format(torch.cuda.max_memory_allocated() * 1.0e-6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Question 4:__ Outline the steps you took to get to your final CNN architecture and your reasoning at each step.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Answer:__ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (IMPLEMENTATION) Specify Loss Function and Optimizer\n",
    "\n",
    "Use the next code cell to specify a [loss function](http://pytorch.org/docs/stable/nn.html#loss-functions) and [optimizer](http://pytorch.org/docs/stable/optim.html).  Save the chosen loss function as `criterion_scratch`, and the optimizer as `optimizer_scratch` below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "### TODO: select loss function\n",
    "criterion_scratch = nn.CrossEntropyLoss()\n",
    "\n",
    "### TODO: select optimizer\n",
    "optimizer_scratch = optim.SGD(model_scratch.parameters(), lr=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorboardX import SummaryWriter\n",
    "from pathlib import Path\n",
    "\n",
    "lr = optimizer_scratch.param_groups[0]['lr']\n",
    "log_path = 'tensorboard_logs'\n",
    "checkpoints_dir = 'checkpoints'\n",
    "comment = \"scratch-\"\n",
    "checkpoint_name = '{}model-batch-{}-lr-{}'.format(\n",
    "                comment, batch_size, lr)\n",
    "current_checkpoint_dir = Path(checkpoints_dir) / checkpoint_name\n",
    "checkpoints_path = str(current_checkpoint_dir / '{}.pt'.format(checkpoint_name))\n",
    "Path(checkpoints_path).parent.mkdir(parents=True, exist_ok=True)  # create checkpoint folder\n",
    "tensorboard_dir = str(Path(log_path) / checkpoint_name)\n",
    "writer = SummaryWriter(tensorboard_dir)  # creating TensorBoard writer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (IMPLEMENTATION) Train and Validate the Model\n",
    "\n",
    "Train and validate your model in the code cell below.  [Save the final model parameters](http://pytorch.org/docs/master/notes/serialization.html) at filepath `'model_scratch.pt'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_checkpoint(checkpoint: dict, save_path: str, checkpoint_type: str = 'best'):\n",
    "    checkpoint_type = checkpoint_type.replace(' ', '_')\n",
    "    if len(checkpoint_type) != 0:\n",
    "        if checkpoint_type[-1] != '-':\n",
    "            checkpoint_type += '-'  # 'best' -> 'best-' OR 'epoch_10' -> 'epoch_10-'\n",
    "\n",
    "    save_path = str(Path(save_path).parent / '{}{}'.format(checkpoint_type, Path(save_path).name))\n",
    "\n",
    "    torch.save(checkpoint, save_path)  # save checkpoint\n",
    "    print(\"SUCCESSFULLY SAVED '{}' checkpoint to: {}\".format(checkpoint_type, save_path))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_checkpoint(checkpoint_path, model=None, optimizer=None):\n",
    "    \n",
    "    \"\"\"\n",
    "    :param checkpoint_path: path to save checkpoint\n",
    "    :param model: model that we want to load checkpoint parameters into\n",
    "    :param optimizer: optimizer we defined in previous training\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    assert Path(checkpoint_path).exists(), \"NOT EXISTS the checkpoint_path: {}\".format(checkpoint_path)\n",
    "    checkpoint_dict = torch.load(checkpoint_path)  # load checkpoint file\n",
    "    print(\"\\nSUCCESSFULLY LOADED checkpoint from {}\".format(checkpoint_path))\n",
    "    print(\"Available checkpoint's dictionary keys: {}\".format(checkpoint_dict.keys()))\n",
    "\n",
    "    result_dict = dict()\n",
    "    if 'datetime_saved' in checkpoint_dict.keys():\n",
    "        result_dict['datetime_saved'] = checkpoint_dict['datetime_saved']\n",
    "    else:\n",
    "        result_dict['datetime_saved'] = None\n",
    "\n",
    "    if 'epoch' in checkpoint_dict.keys():\n",
    "        result_dict['epoch'] = checkpoint_dict['epoch']\n",
    "    else:\n",
    "        result_dict['epoch'] = None\n",
    "\n",
    "    if 'loss_train' in checkpoint_dict.keys():\n",
    "        result_dict['loss_train'] = checkpoint_dict['loss_train']\n",
    "    else:\n",
    "        result_dict['loss_train'] = None\n",
    "\n",
    "    if 'loss_val' in checkpoint_dict.keys():\n",
    "        result_dict['loss_val'] = checkpoint_dict['loss_val']\n",
    "    else:\n",
    "        result_dict['loss_val'] = None\n",
    "        \n",
    "    if 'lr' in checkpoint_dict.keys():\n",
    "        result_dict['lr'] = checkpoint_dict['lr']\n",
    "    else:\n",
    "        result_dict['lr'] = None\n",
    "\n",
    "    if 'num_classes' in checkpoint_dict.keys():\n",
    "        result_dict['num_classes'] = checkpoint_dict['num_classes']\n",
    "    else:\n",
    "        result_dict['num_classes'] = None\n",
    "\n",
    "    if 'model_state_dict' in checkpoint_dict.keys():\n",
    "        _model_state_dict = checkpoint_dict['model_state_dict']\n",
    "    else:\n",
    "        _model_state_dict = None\n",
    "\n",
    "    if 'optimizer_state_dict' in checkpoint_dict.keys():\n",
    "        _optimizer_state_dict = checkpoint_dict['optimizer_state_dict']\n",
    "    else:\n",
    "        _optimizer_state_dict = None\n",
    "\n",
    "    if 'end_training_time' in checkpoint_dict.keys():\n",
    "        result_dict['end_training_time'] = checkpoint_dict['end_training_time']\n",
    "    else:\n",
    "        result_dict['end_training_time'] = None\n",
    "\n",
    "    if 'total_train_time_seconds' in checkpoint_dict.keys():\n",
    "        result_dict['total_train_time_seconds'] = checkpoint_dict['total_train_time_seconds']\n",
    "    else:\n",
    "        result_dict['total_train_time_seconds'] = None\n",
    "        \n",
    "    if 'classes_dict' in checkpoint_dict.keys():\n",
    "        result_dict['classes_dict'] = checkpoint_dict['classes_dict']\n",
    "    else:\n",
    "        result_dict['classes_dict'] = None\n",
    "\n",
    "    if 'tensorboard_dir' in checkpoint_dict.keys():\n",
    "        result_dict['tensorboard_dir'] = checkpoint_dict['tensorboard_dir']\n",
    "    else:\n",
    "        result_dict['tensorboard_dir'] = None\n",
    "\n",
    "    if model is None:  # if model not defined, than model should be initialized to load in the checkpoint parameters\n",
    "        assert _model_state_dict is not None, 'NOT FOUNDED model_state_dict in checkpoint dict'\n",
    "        \n",
    "        model = Net()\n",
    "        model.load_state_dict(_model_state_dict)  # loading parameters from saved state_dict to initialized model\n",
    "    else:\n",
    "        assert _model_state_dict is not None, 'NOT FOUNDED model_state_dict in checkpoint dict'\n",
    "        model.load_state_dict(_model_state_dict)  # loading parameters from saved state_dict to defined model\n",
    "        \n",
    "    if optimizer is None:\n",
    "        assert _optimizer_state_dict is not None, 'NOT FOUNDED optimizer_state_dict in checkoint dict'\n",
    "        optimizer = optim.SGD(model.parameters(), lr=lr)\n",
    "        optimizer.load_state_dict(_optimizer_state_dict)\n",
    "    else:\n",
    "        assert _optimizer_state_dict is not None, 'NOT FOUNDED optimizer_state_dict in checkoint dict'\n",
    "        optimizer.load_state_dict(_optimizer_state_dict)\n",
    "\n",
    "    return model, optimizer, result_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "SUCCESSFULLY LOADED checkpoint from checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n",
      "Available checkpoint's dictionary keys: dict_keys(['datetime_saved', 'epoch', 'loss_train', 'loss_val', 'lr', 'num_classes', 'model_state_dict', 'optimizer_state_dict', 'end_training_time', 'total_train_time_seconds', 'classes_dict', 'tensorboard_dir'])\n",
      "Loaded model. Epoch: 94, val_batch_loss: 4.673764726191616\n"
     ]
    }
   ],
   "source": [
    "# if load model\n",
    "load_model = True\n",
    "load_last = False\n",
    "if load_model:\n",
    "    if load_last:\n",
    "        # loading last checkpoint\n",
    "        checkpoint_path = Path(checkpoints_path).parent / 'last-{}.pt'.format(checkpoint_name)\n",
    "    else:\n",
    "        # loading best checkpoint\n",
    "        checkpoint_path = Path(checkpoints_path).parent / 'best-{}.pt'.format(checkpoint_name)\n",
    "    model_scratch, optimizer_scratch, result_dict = load_checkpoint(checkpoint_path)\n",
    "    if use_cuda:\n",
    "        model_scratch.cuda()\n",
    "    print(\"Loaded model. Epoch: {}, val_batch_loss: {}\".format(result_dict['epoch'], result_dict['loss_val']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from PIL import ImageFile\n",
    "ImageFile.LOAD_TRUNCATED_IMAGES = True\n",
    "from tqdm import tqdm\n",
    "\n",
    "def train(n_epochs, loaders, model, optimizer, criterion, use_cuda,\n",
    "         resume_train=False, result_dict=None):\n",
    "    \"\"\"returns trained model\"\"\"\n",
    "    # initialize tracker for minimum validation loss\n",
    "    \n",
    "    if resume_train:\n",
    "        assert result_dict is not None\n",
    "        \n",
    "    valid_loss_min = np.Inf \n",
    "    total_train_time_seconds = 0.0\n",
    "    epoch_begin = 1\n",
    "    \n",
    "    if resume_train:\n",
    "        print(\"Resuming training process. Last epoch: {}, valid loss min: {}\".\n",
    "             format(result_dict['epoch'], result_dict['loss_val']))\n",
    "        valid_loss_min = result_dict['loss_val']\n",
    "        total_train_time_seconds = result_dict['total_train_time_seconds']\n",
    "        epoch_begin = result_dict['epoch'] + 1\n",
    "    else:\n",
    "        print(\"Starting train from epoch 1, from scratch\")\n",
    "        \n",
    "    \n",
    "    start_training_time = datetime.now()\n",
    "    \n",
    "    for epoch in range(epoch_begin, epoch_begin + n_epochs):\n",
    "        # initialize variables to monitor training and validation loss\n",
    "        train_loss = 0.0\n",
    "        valid_loss = 0.0\n",
    "        \n",
    "        if writer is not None:\n",
    "            writer.add_scalar('Learning Rate', lr, epoch)\n",
    "        \n",
    "        \n",
    "        ###################\n",
    "        # train the model #\n",
    "        ###################\n",
    "        model.train()\n",
    "        with tqdm(loaders['train'], unit=\"batch\") as tepoch:\n",
    "            for data, target in tepoch:\n",
    "                tepoch.set_description(f\"Train. Epoch {epoch}\")\n",
    "    #             print(target)\n",
    "    #             print(\"data shape: {}\".format(data.shape))\n",
    "    #             print(\"target shape: {}\".format(target.shape))\n",
    "                # move to GPU\n",
    "                if use_cuda:\n",
    "                    data, target = data.cuda(), target.cuda()\n",
    "                ## find the loss and update the model parameters accordingly\n",
    "                ## record the average training loss, using something like\n",
    "                ## train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))\n",
    "                \n",
    "                optimizer_scratch.zero_grad()\n",
    "                output = model_scratch(data)\n",
    "                loss = criterion_scratch(output, target)\n",
    "                loss.backward()\n",
    "                optimizer_scratch.step()\n",
    "                train_loss += loss.item() * data.size(0)\n",
    "                tepoch.set_postfix(loss=train_loss)\n",
    "            \n",
    "        ######################    \n",
    "        # validate the model #\n",
    "        ######################\n",
    "        model.eval()\n",
    "        with tqdm(loaders['valid'], unit=\"batch\") as vepoch:\n",
    "            for data, target in vepoch:\n",
    "                vepoch.set_description(f\"Valid. Epoch {epoch}\")\n",
    "                # move to GPU\n",
    "                if use_cuda:\n",
    "                    data, target = data.cuda(), target.cuda()\n",
    "                \n",
    "                output = model_scratch(data)\n",
    "                loss = criterion_scratch(output, target)\n",
    "                ## update the average validation loss\n",
    "                valid_loss += loss.item()*data.size(0)\n",
    "                vepoch.set_postfix(loss=valid_loss)\n",
    "\n",
    "        ## calculate average batch train and validation loss\n",
    "        train_loss /= len(loaders['train'].sampler)\n",
    "        valid_loss /= len(loaders['valid'].sampler)\n",
    "    \n",
    "        if writer is not None:\n",
    "            writer.add_scalar('{}/TrainLoss'.format(\"train\"), train_loss, epoch)\n",
    "            writer.add_scalar('{}/ValidLoss'.format(\"val\"), valid_loss, epoch)\n",
    "            \n",
    "        # print training/validation statistics \n",
    "        print('Epoch: {} \\tTraining Loss: {:.6f} \\tValidation Loss: {:.6f}'.format(\n",
    "            epoch, \n",
    "            train_loss,\n",
    "            valid_loss\n",
    "            ))\n",
    "        \n",
    "        ## TODO: save the model if validation loss has decreased\n",
    "        if valid_loss < valid_loss_min:\n",
    "            print(\"Epoch {}: Upgrate of model quality: ({:.5f})->({:.5f})\".format(epoch, valid_loss_min, valid_loss))\n",
    "            \n",
    "            delta_time = datetime.now() - start_training_time\n",
    "            checkpoint = {\n",
    "                    'datetime_saved': datetime.now(),\n",
    "                    'epoch': epoch,\n",
    "                    'loss_train': train_loss, \n",
    "                    'loss_val': valid_loss,\n",
    "                    'lr': lr, \n",
    "                    'num_classes': model.fc3.out_features,  \n",
    "                    'model_state_dict': model_scratch.state_dict(), \n",
    "                    'optimizer_state_dict': optimizer_scratch.state_dict(), \n",
    "                    'end_training_time': delta_time, \n",
    "                    'total_train_time_seconds': total_train_time_seconds + \n",
    "                                                delta_time.seconds + \n",
    "                                                delta_time.days * 24 * 3600,\n",
    "                    'classes_dict': classes_dict, \n",
    "                    'tensorboard_dir': tensorboard_dir\n",
    "                }\n",
    "            save_checkpoint(checkpoint=checkpoint, save_path=checkpoints_path, \n",
    "                           checkpoint_type='best')\n",
    "#             torch.save(model_scratch.state_dict(), 'model_scratch.pt')\n",
    "            valid_loss_min = valid_loss\n",
    "    \n",
    "        # save last checkpoint:\n",
    "        delta_time = datetime.now() - start_training_time\n",
    "        checkpoint = {\n",
    "                'datetime_saved': datetime.now(),\n",
    "                'epoch': epoch,\n",
    "                'loss_train': train_loss, \n",
    "                'loss_val': valid_loss,\n",
    "                'lr': lr, \n",
    "                'num_classes': model.fc3.out_features,  \n",
    "                'model_state_dict': model_scratch.state_dict(), \n",
    "                'optimizer_state_dict': optimizer_scratch.state_dict(), \n",
    "                'end_training_time': delta_time, \n",
    "                'total_train_time_seconds': total_train_time_seconds + \n",
    "                                                delta_time.seconds + \n",
    "                                                delta_time.days * 24 * 3600,\n",
    "                'classes_dict': classes_dict, \n",
    "                'tensorboard_dir': tensorboard_dir\n",
    "        }\n",
    "        save_checkpoint(checkpoint=checkpoint, save_path=checkpoints_path, \n",
    "                           checkpoint_type='last')\n",
    "    \n",
    "            \n",
    "    # return trained model\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting train from epoch 1, from scratch\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 1: 100%|██████████| 209/209 [01:04<00:00,  3.25batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 1: 100%|██████████| 27/27 [00:07<00:00,  3.71batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1 \tTraining Loss: 4.890361 \tValidation Loss: 4.890307\n",
      "Epoch 1: Upgrate of model quality: (inf)->(4.89031)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 2: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 2: 100%|██████████| 27/27 [00:07<00:00,  3.66batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 2 \tTraining Loss: 4.890280 \tValidation Loss: 4.890206\n",
      "Epoch 2: Upgrate of model quality: (4.89031)->(4.89021)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 3: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 3: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 3 \tTraining Loss: 4.890170 \tValidation Loss: 4.890104\n",
      "Epoch 3: Upgrate of model quality: (4.89021)->(4.89010)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 4: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 4: 100%|██████████| 27/27 [00:07<00:00,  3.57batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 4 \tTraining Loss: 4.890133 \tValidation Loss: 4.890001\n",
      "Epoch 4: Upgrate of model quality: (4.89010)->(4.89000)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 5: 100%|██████████| 209/209 [01:08<00:00,  3.07batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 5: 100%|██████████| 27/27 [00:07<00:00,  3.70batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 5 \tTraining Loss: 4.890011 \tValidation Loss: 4.889899\n",
      "Epoch 5: Upgrate of model quality: (4.89000)->(4.88990)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 6: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 6: 100%|██████████| 27/27 [00:07<00:00,  3.58batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 6 \tTraining Loss: 4.889917 \tValidation Loss: 4.889795\n",
      "Epoch 6: Upgrate of model quality: (4.88990)->(4.88980)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 7: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 7: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 7 \tTraining Loss: 4.889826 \tValidation Loss: 4.889689\n",
      "Epoch 7: Upgrate of model quality: (4.88980)->(4.88969)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 8: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 8: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 8 \tTraining Loss: 4.889743 \tValidation Loss: 4.889582\n",
      "Epoch 8: Upgrate of model quality: (4.88969)->(4.88958)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 9: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 9: 100%|██████████| 27/27 [00:07<00:00,  3.67batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 9 \tTraining Loss: 4.889679 \tValidation Loss: 4.889476\n",
      "Epoch 9: Upgrate of model quality: (4.88958)->(4.88948)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 10: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 10: 100%|██████████| 27/27 [00:07<00:00,  3.85batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 10 \tTraining Loss: 4.889525 \tValidation Loss: 4.889363\n",
      "Epoch 10: Upgrate of model quality: (4.88948)->(4.88936)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 11: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 11: 100%|██████████| 27/27 [00:07<00:00,  3.64batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 11 \tTraining Loss: 4.889504 \tValidation Loss: 4.889253\n",
      "Epoch 11: Upgrate of model quality: (4.88936)->(4.88925)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 12: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 12: 100%|██████████| 27/27 [00:07<00:00,  3.58batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 12 \tTraining Loss: 4.889384 \tValidation Loss: 4.889135\n",
      "Epoch 12: Upgrate of model quality: (4.88925)->(4.88914)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 13: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 13: 100%|██████████| 27/27 [00:07<00:00,  3.72batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 13 \tTraining Loss: 4.889205 \tValidation Loss: 4.889007\n",
      "Epoch 13: Upgrate of model quality: (4.88914)->(4.88901)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 14: 100%|██████████| 209/209 [01:06<00:00,  3.15batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 14: 100%|██████████| 27/27 [00:07<00:00,  3.80batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 14 \tTraining Loss: 4.889159 \tValidation Loss: 4.888877\n",
      "Epoch 14: Upgrate of model quality: (4.88901)->(4.88888)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 15: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 15: 100%|██████████| 27/27 [00:07<00:00,  3.64batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 15 \tTraining Loss: 4.888982 \tValidation Loss: 4.888728\n",
      "Epoch 15: Upgrate of model quality: (4.88888)->(4.88873)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 16: 100%|██████████| 209/209 [01:06<00:00,  3.12batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 16: 100%|██████████| 27/27 [00:07<00:00,  3.56batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 16 \tTraining Loss: 4.888907 \tValidation Loss: 4.888573\n",
      "Epoch 16: Upgrate of model quality: (4.88873)->(4.88857)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 17: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 17: 100%|██████████| 27/27 [00:07<00:00,  3.55batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 17 \tTraining Loss: 4.888730 \tValidation Loss: 4.888397\n",
      "Epoch 17: Upgrate of model quality: (4.88857)->(4.88840)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 18: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 18: 100%|██████████| 27/27 [00:07<00:00,  3.59batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 18 \tTraining Loss: 4.888575 \tValidation Loss: 4.888181\n",
      "Epoch 18: Upgrate of model quality: (4.88840)->(4.88818)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 19: 100%|██████████| 209/209 [01:05<00:00,  3.18batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 19: 100%|██████████| 27/27 [00:07<00:00,  3.77batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 19 \tTraining Loss: 4.888361 \tValidation Loss: 4.887921\n",
      "Epoch 19: Upgrate of model quality: (4.88818)->(4.88792)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 20: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 20: 100%|██████████| 27/27 [00:07<00:00,  3.58batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 20 \tTraining Loss: 4.888072 \tValidation Loss: 4.887566\n",
      "Epoch 20: Upgrate of model quality: (4.88792)->(4.88757)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 21: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=3.27e+4]\n",
      "Valid. Epoch 21: 100%|██████████| 27/27 [00:07<00:00,  3.79batch/s, loss=4.09e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 21 \tTraining Loss: 4.887770 \tValidation Loss: 4.887050\n",
      "Epoch 21: Upgrate of model quality: (4.88757)->(4.88705)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 22: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=3.26e+4]\n",
      "Valid. Epoch 22: 100%|██████████| 27/27 [00:07<00:00,  3.67batch/s, loss=4.08e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 22 \tTraining Loss: 4.887133 \tValidation Loss: 4.886140\n",
      "Epoch 22: Upgrate of model quality: (4.88705)->(4.88614)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 23: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=3.26e+4]\n",
      "Valid. Epoch 23: 100%|██████████| 27/27 [00:07<00:00,  3.77batch/s, loss=4.08e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 23 \tTraining Loss: 4.885859 \tValidation Loss: 4.884079\n",
      "Epoch 23: Upgrate of model quality: (4.88614)->(4.88408)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 24: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=3.26e+4]\n",
      "Valid. Epoch 24: 100%|██████████| 27/27 [00:07<00:00,  3.66batch/s, loss=4.08e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 24 \tTraining Loss: 4.882648 \tValidation Loss: 4.877193\n",
      "Epoch 24: Upgrate of model quality: (4.88408)->(4.87719)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 25: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=3.26e+4]\n",
      "Valid. Epoch 25: 100%|██████████| 27/27 [00:07<00:00,  3.79batch/s, loss=4.06e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 25 \tTraining Loss: 4.873605 \tValidation Loss: 4.858085\n",
      "Epoch 25: Upgrate of model quality: (4.87719)->(4.85809)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 26: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=3.25e+4]\n",
      "Valid. Epoch 26: 100%|██████████| 27/27 [00:07<00:00,  3.80batch/s, loss=4.06e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 26 \tTraining Loss: 4.865431 \tValidation Loss: 4.850740\n",
      "Epoch 26: Upgrate of model quality: (4.85809)->(4.85074)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 27: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=3.25e+4]\n",
      "Valid. Epoch 27: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=4.05e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 27 \tTraining Loss: 4.860453 \tValidation Loss: 4.846228\n",
      "Epoch 27: Upgrate of model quality: (4.85074)->(4.84623)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 28: 100%|██████████| 209/209 [01:06<00:00,  3.12batch/s, loss=3.24e+4]\n",
      "Valid. Epoch 28: 100%|██████████| 27/27 [00:07<00:00,  3.59batch/s, loss=4.05e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 28 \tTraining Loss: 4.857316 \tValidation Loss: 4.843226\n",
      "Epoch 28: Upgrate of model quality: (4.84623)->(4.84323)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 29: 100%|██████████| 209/209 [01:08<00:00,  3.06batch/s, loss=3.24e+4]\n",
      "Valid. Epoch 29: 100%|██████████| 27/27 [00:07<00:00,  3.66batch/s, loss=4.04e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 29 \tTraining Loss: 4.850686 \tValidation Loss: 4.836051\n",
      "Epoch 29: Upgrate of model quality: (4.84323)->(4.83605)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 30: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=3.24e+4]\n",
      "Valid. Epoch 30: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=4.04e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 30 \tTraining Loss: 4.843686 \tValidation Loss: 4.827525\n",
      "Epoch 30: Upgrate of model quality: (4.83605)->(4.82752)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 31: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=3.23e+4]\n",
      "Valid. Epoch 31: 100%|██████████| 27/27 [00:07<00:00,  3.55batch/s, loss=4.02e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 31 \tTraining Loss: 4.834924 \tValidation Loss: 4.814266\n",
      "Epoch 31: Upgrate of model quality: (4.82752)->(4.81427)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 32: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=3.22e+4]\n",
      "Valid. Epoch 32: 100%|██████████| 27/27 [00:07<00:00,  3.85batch/s, loss=4.01e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 32 \tTraining Loss: 4.823556 \tValidation Loss: 4.800680\n",
      "Epoch 32: Upgrate of model quality: (4.81427)->(4.80068)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 33: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=3.21e+4]\n",
      "Valid. Epoch 33: 100%|██████████| 27/27 [00:07<00:00,  3.53batch/s, loss=4e+3]   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 33 \tTraining Loss: 4.809777 \tValidation Loss: 4.788923\n",
      "Epoch 33: Upgrate of model quality: (4.80068)->(4.78892)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 34: 100%|██████████| 209/209 [01:08<00:00,  3.06batch/s, loss=3.21e+4]\n",
      "Valid. Epoch 34: 100%|██████████| 27/27 [00:07<00:00,  3.60batch/s, loss=4e+3]   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 34 \tTraining Loss: 4.799266 \tValidation Loss: 4.779295\n",
      "Epoch 34: Upgrate of model quality: (4.78892)->(4.77930)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 35: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=3.2e+4] \n",
      "Valid. Epoch 35: 100%|██████████| 27/27 [00:07<00:00,  3.56batch/s, loss=3.99e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 35 \tTraining Loss: 4.788276 \tValidation Loss: 4.771012\n",
      "Epoch 35: Upgrate of model quality: (4.77930)->(4.77101)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 36: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=3.19e+4]\n",
      "Valid. Epoch 36: 100%|██████████| 27/27 [00:07<00:00,  3.61batch/s, loss=3.99e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 36 \tTraining Loss: 4.778551 \tValidation Loss: 4.769551\n",
      "Epoch 36: Upgrate of model quality: (4.77101)->(4.76955)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 37: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=3.18e+4]\n",
      "Valid. Epoch 37: 100%|██████████| 27/27 [00:07<00:00,  3.65batch/s, loss=3.98e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 37 \tTraining Loss: 4.765150 \tValidation Loss: 4.761139\n",
      "Epoch 37: Upgrate of model quality: (4.76955)->(4.76114)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 38: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=3.18e+4]\n",
      "Valid. Epoch 38: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.98e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 38 \tTraining Loss: 4.756204 \tValidation Loss: 4.756461\n",
      "Epoch 38: Upgrate of model quality: (4.76114)->(4.75646)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 39: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=3.17e+4]\n",
      "Valid. Epoch 39: 100%|██████████| 27/27 [00:07<00:00,  3.69batch/s, loss=3.97e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 39 \tTraining Loss: 4.750811 \tValidation Loss: 4.747220\n",
      "Epoch 39: Upgrate of model quality: (4.75646)->(4.74722)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 40: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=3.17e+4]\n",
      "Valid. Epoch 40: 100%|██████████| 27/27 [00:07<00:00,  3.61batch/s, loss=3.97e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 40 \tTraining Loss: 4.741160 \tValidation Loss: 4.748500\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 41: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=3.16e+4]\n",
      "Valid. Epoch 41: 100%|██████████| 27/27 [00:07<00:00,  3.67batch/s, loss=3.97e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 41 \tTraining Loss: 4.729669 \tValidation Loss: 4.748848\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 42: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=3.16e+4]\n",
      "Valid. Epoch 42: 100%|██████████| 27/27 [00:07<00:00,  3.49batch/s, loss=3.97e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 42 \tTraining Loss: 4.727039 \tValidation Loss: 4.744410\n",
      "Epoch 42: Upgrate of model quality: (4.74722)->(4.74441)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 43:   0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 43: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=3.15e+4]\n",
      "Valid. Epoch 43: 100%|██████████| 27/27 [00:07<00:00,  3.65batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 43 \tTraining Loss: 4.715753 \tValidation Loss: 4.740765\n",
      "Epoch 43: Upgrate of model quality: (4.74441)->(4.74077)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 44: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=3.14e+4]\n",
      "Valid. Epoch 44: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 44 \tTraining Loss: 4.705297 \tValidation Loss: 4.729262\n",
      "Epoch 44: Upgrate of model quality: (4.74077)->(4.72926)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 45: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=3.14e+4]\n",
      "Valid. Epoch 45: 100%|██████████| 27/27 [00:07<00:00,  3.78batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 45 \tTraining Loss: 4.695612 \tValidation Loss: 4.721776\n",
      "Epoch 45: Upgrate of model quality: (4.72926)->(4.72178)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 46: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=3.13e+4]\n",
      "Valid. Epoch 46: 100%|██████████| 27/27 [00:06<00:00,  3.91batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 46 \tTraining Loss: 4.689684 \tValidation Loss: 4.722678\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 47: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=3.13e+4]\n",
      "Valid. Epoch 47: 100%|██████████| 27/27 [00:07<00:00,  3.72batch/s, loss=3.94e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 47 \tTraining Loss: 4.681267 \tValidation Loss: 4.715889\n",
      "Epoch 47: Upgrate of model quality: (4.72178)->(4.71589)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 48: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=3.12e+4]\n",
      "Valid. Epoch 48: 100%|██████████| 27/27 [00:07<00:00,  3.70batch/s, loss=3.94e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 48 \tTraining Loss: 4.677570 \tValidation Loss: 4.716206\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 49: 100%|██████████| 209/209 [01:07<00:00,  3.07batch/s, loss=3.12e+4]\n",
      "Valid. Epoch 49: 100%|██████████| 27/27 [00:07<00:00,  3.69batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 49 \tTraining Loss: 4.669536 \tValidation Loss: 4.699258\n",
      "Epoch 49: Upgrate of model quality: (4.71589)->(4.69926)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 50: 100%|██████████| 209/209 [01:06<00:00,  3.12batch/s, loss=3.12e+4]\n",
      "Valid. Epoch 50: 100%|██████████| 27/27 [00:06<00:00,  3.88batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 50 \tTraining Loss: 4.663298 \tValidation Loss: 4.697594\n",
      "Epoch 50: Upgrate of model quality: (4.69926)->(4.69759)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 51: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=3.11e+4]\n",
      "Valid. Epoch 51: 100%|██████████| 27/27 [00:07<00:00,  3.60batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 51 \tTraining Loss: 4.660093 \tValidation Loss: 4.700853\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 52: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=3.11e+4]\n",
      "Valid. Epoch 52: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 52 \tTraining Loss: 4.656224 \tValidation Loss: 4.704864\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 53: 100%|██████████| 209/209 [01:05<00:00,  3.17batch/s, loss=3.11e+4]\n",
      "Valid. Epoch 53: 100%|██████████| 27/27 [00:07<00:00,  3.68batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 53 \tTraining Loss: 4.648398 \tValidation Loss: 4.695730\n",
      "Epoch 53: Upgrate of model quality: (4.69759)->(4.69573)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 54: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=3.1e+4] \n",
      "Valid. Epoch 54: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 54 \tTraining Loss: 4.642554 \tValidation Loss: 4.697342\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 55: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=3.1e+4] \n",
      "Valid. Epoch 55: 100%|██████████| 27/27 [00:06<00:00,  3.87batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 55 \tTraining Loss: 4.641467 \tValidation Loss: 4.703937\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 56: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=3.1e+4] \n",
      "Valid. Epoch 56: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 56 \tTraining Loss: 4.638464 \tValidation Loss: 4.694655\n",
      "Epoch 56: Upgrate of model quality: (4.69573)->(4.69465)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 57: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=3.1e+4] \n",
      "Valid. Epoch 57: 100%|██████████| 27/27 [00:07<00:00,  3.61batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 57 \tTraining Loss: 4.633786 \tValidation Loss: 4.689875\n",
      "Epoch 57: Upgrate of model quality: (4.69465)->(4.68987)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 58: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=3.09e+4]\n",
      "Valid. Epoch 58: 100%|██████████| 27/27 [00:07<00:00,  3.48batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 58 \tTraining Loss: 4.625226 \tValidation Loss: 4.695076\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 59: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=3.09e+4]\n",
      "Valid. Epoch 59: 100%|██████████| 27/27 [00:07<00:00,  3.83batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 59 \tTraining Loss: 4.627557 \tValidation Loss: 4.691949\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 60: 100%|██████████| 209/209 [01:06<00:00,  3.12batch/s, loss=3.09e+4]\n",
      "Valid. Epoch 60: 100%|██████████| 27/27 [00:07<00:00,  3.60batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 60 \tTraining Loss: 4.620971 \tValidation Loss: 4.694576\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 61: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=3.08e+4]\n",
      "Valid. Epoch 61: 100%|██████████| 27/27 [00:07<00:00,  3.56batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 61 \tTraining Loss: 4.616201 \tValidation Loss: 4.685842\n",
      "Epoch 61: Upgrate of model quality: (4.68987)->(4.68584)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 62: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=3.08e+4]\n",
      "Valid. Epoch 62: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 62 \tTraining Loss: 4.616562 \tValidation Loss: 4.687884\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 63: 100%|██████████| 209/209 [01:06<00:00,  3.12batch/s, loss=3.08e+4]\n",
      "Valid. Epoch 63: 100%|██████████| 27/27 [00:07<00:00,  3.67batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 63 \tTraining Loss: 4.610530 \tValidation Loss: 4.695020\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 64: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=3.08e+4]\n",
      "Valid. Epoch 64: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 64 \tTraining Loss: 4.603980 \tValidation Loss: 4.690830\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 65: 100%|██████████| 209/209 [01:05<00:00,  3.18batch/s, loss=3.07e+4]\n",
      "Valid. Epoch 65: 100%|██████████| 27/27 [00:07<00:00,  3.61batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 65 \tTraining Loss: 4.599030 \tValidation Loss: 4.688208\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 66: 100%|██████████| 209/209 [01:06<00:00,  3.16batch/s, loss=3.07e+4]\n",
      "Valid. Epoch 66: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 66 \tTraining Loss: 4.597408 \tValidation Loss: 4.694548\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 67: 100%|██████████| 209/209 [01:06<00:00,  3.16batch/s, loss=3.07e+4]\n",
      "Valid. Epoch 67: 100%|██████████| 27/27 [00:07<00:00,  3.79batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 67 \tTraining Loss: 4.598368 \tValidation Loss: 4.689216\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 68: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=3.07e+4]\n",
      "Valid. Epoch 68: 100%|██████████| 27/27 [00:07<00:00,  3.72batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 68 \tTraining Loss: 4.594613 \tValidation Loss: 4.694610\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 69:   0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 69: 100%|██████████| 209/209 [01:08<00:00,  3.07batch/s, loss=3.07e+4]\n",
      "Valid. Epoch 69: 100%|██████████| 27/27 [00:07<00:00,  3.64batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 69 \tTraining Loss: 4.591058 \tValidation Loss: 4.685800\n",
      "Epoch 69: Upgrate of model quality: (4.68584)->(4.68580)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 70: 100%|██████████| 209/209 [01:06<00:00,  3.15batch/s, loss=3.06e+4]\n",
      "Valid. Epoch 70: 100%|██████████| 27/27 [00:07<00:00,  3.59batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 70 \tTraining Loss: 4.586873 \tValidation Loss: 4.688469\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 71: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=3.06e+4]\n",
      "Valid. Epoch 71: 100%|██████████| 27/27 [00:07<00:00,  3.70batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 71 \tTraining Loss: 4.576346 \tValidation Loss: 4.684979\n",
      "Epoch 71: Upgrate of model quality: (4.68580)->(4.68498)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 72: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=3.06e+4]\n",
      "Valid. Epoch 72: 100%|██████████| 27/27 [00:07<00:00,  3.58batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 72 \tTraining Loss: 4.573922 \tValidation Loss: 4.688504\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 73: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=3.05e+4]\n",
      "Valid. Epoch 73: 100%|██████████| 27/27 [00:07<00:00,  3.67batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 73 \tTraining Loss: 4.571506 \tValidation Loss: 4.688796\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 74: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=3.05e+4]\n",
      "Valid. Epoch 74: 100%|██████████| 27/27 [00:07<00:00,  3.65batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 74 \tTraining Loss: 4.570968 \tValidation Loss: 4.693777\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 75: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=3.05e+4]\n",
      "Valid. Epoch 75: 100%|██████████| 27/27 [00:07<00:00,  3.68batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 75 \tTraining Loss: 4.559022 \tValidation Loss: 4.687072\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 76: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=3.05e+4]\n",
      "Valid. Epoch 76: 100%|██████████| 27/27 [00:07<00:00,  3.57batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 76 \tTraining Loss: 4.562216 \tValidation Loss: 4.690836\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 77: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=3.04e+4]\n",
      "Valid. Epoch 77: 100%|██████████| 27/27 [00:07<00:00,  3.69batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 77 \tTraining Loss: 4.557769 \tValidation Loss: 4.681603\n",
      "Epoch 77: Upgrate of model quality: (4.68498)->(4.68160)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 78: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=3.04e+4]\n",
      "Valid. Epoch 78: 100%|██████████| 27/27 [00:07<00:00,  3.68batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 78 \tTraining Loss: 4.552603 \tValidation Loss: 4.687426\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 79: 100%|██████████| 209/209 [01:06<00:00,  3.15batch/s, loss=3.04e+4]\n",
      "Valid. Epoch 79: 100%|██████████| 27/27 [00:07<00:00,  3.64batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 79 \tTraining Loss: 4.552407 \tValidation Loss: 4.682031\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 80: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=3.04e+4]\n",
      "Valid. Epoch 80: 100%|██████████| 27/27 [00:07<00:00,  3.53batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 80 \tTraining Loss: 4.547525 \tValidation Loss: 4.681817\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 81: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=3.03e+4]\n",
      "Valid. Epoch 81: 100%|██████████| 27/27 [00:07<00:00,  3.69batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 81 \tTraining Loss: 4.542003 \tValidation Loss: 4.677395\n",
      "Epoch 81: Upgrate of model quality: (4.68160)->(4.67739)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 82: 100%|██████████| 209/209 [01:07<00:00,  3.07batch/s, loss=3.03e+4]\n",
      "Valid. Epoch 82: 100%|██████████| 27/27 [00:07<00:00,  3.66batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 82 \tTraining Loss: 4.536803 \tValidation Loss: 4.675982\n",
      "Epoch 82: Upgrate of model quality: (4.67739)->(4.67598)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 83: 100%|██████████| 209/209 [01:05<00:00,  3.20batch/s, loss=3.03e+4]\n",
      "Valid. Epoch 83: 100%|██████████| 27/27 [00:07<00:00,  3.60batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 83 \tTraining Loss: 4.531068 \tValidation Loss: 4.678774\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 84: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=3.03e+4]\n",
      "Valid. Epoch 84: 100%|██████████| 27/27 [00:07<00:00,  3.72batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 84 \tTraining Loss: 4.532832 \tValidation Loss: 4.684025\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 85: 100%|██████████| 209/209 [01:08<00:00,  3.06batch/s, loss=3.02e+4]\n",
      "Valid. Epoch 85: 100%|██████████| 27/27 [00:07<00:00,  3.60batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 85 \tTraining Loss: 4.522408 \tValidation Loss: 4.680805\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 86: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=3.02e+4]\n",
      "Valid. Epoch 86: 100%|██████████| 27/27 [00:07<00:00,  3.84batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 86 \tTraining Loss: 4.520978 \tValidation Loss: 4.676800\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 87: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=3.01e+4]\n",
      "Valid. Epoch 87: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 87 \tTraining Loss: 4.513353 \tValidation Loss: 4.677070\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 88: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=3.01e+4]\n",
      "Valid. Epoch 88: 100%|██████████| 27/27 [00:07<00:00,  3.58batch/s, loss=3.91e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 88 \tTraining Loss: 4.513164 \tValidation Loss: 4.682253\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 89: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=3.01e+4]\n",
      "Valid. Epoch 89: 100%|██████████| 27/27 [00:07<00:00,  3.64batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 89 \tTraining Loss: 4.507605 \tValidation Loss: 4.673943\n",
      "Epoch 89: Upgrate of model quality: (4.67598)->(4.67394)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 90: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=3.01e+4]\n",
      "Valid. Epoch 90: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 90 \tTraining Loss: 4.511842 \tValidation Loss: 4.685815\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 91: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=3.01e+4]\n",
      "Valid. Epoch 91: 100%|██████████| 27/27 [00:07<00:00,  3.76batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 91 \tTraining Loss: 4.500693 \tValidation Loss: 4.694979\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 92: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=3.01e+4]\n",
      "Valid. Epoch 92: 100%|██████████| 27/27 [00:07<00:00,  3.65batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 92 \tTraining Loss: 4.499249 \tValidation Loss: 4.683918\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 93: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=3e+4]   \n",
      "Valid. Epoch 93: 100%|██████████| 27/27 [00:07<00:00,  3.61batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 93 \tTraining Loss: 4.488649 \tValidation Loss: 4.677910\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 94: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=3e+4]   \n",
      "Valid. Epoch 94: 100%|██████████| 27/27 [00:07<00:00,  3.69batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 94 \tTraining Loss: 4.485791 \tValidation Loss: 4.673765\n",
      "Epoch 94: Upgrate of model quality: (4.67394)->(4.67376)\n",
      "SUCCESSFULLY SAVED 'best-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/best-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 95: 100%|██████████| 209/209 [01:05<00:00,  3.17batch/s, loss=3e+4]   \n",
      "Valid. Epoch 95: 100%|██████████| 27/27 [00:07<00:00,  3.57batch/s, loss=3.91e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 95 \tTraining Loss: 4.490060 \tValidation Loss: 4.676527\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 96: 100%|██████████| 209/209 [01:08<00:00,  3.06batch/s, loss=3e+4]   \n",
      "Valid. Epoch 96: 100%|██████████| 27/27 [00:07<00:00,  3.54batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 96 \tTraining Loss: 4.485168 \tValidation Loss: 4.684236\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 97: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.99e+4]\n",
      "Valid. Epoch 97: 100%|██████████| 27/27 [00:07<00:00,  3.85batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 97 \tTraining Loss: 4.476609 \tValidation Loss: 4.685959\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 98: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.99e+4]\n",
      "Valid. Epoch 98: 100%|██████████| 27/27 [00:07<00:00,  3.51batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 98 \tTraining Loss: 4.476778 \tValidation Loss: 4.676997\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 99: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=2.99e+4]\n",
      "Valid. Epoch 99: 100%|██████████| 27/27 [00:06<00:00,  3.88batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 99 \tTraining Loss: 4.470708 \tValidation Loss: 4.681223\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 100: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.98e+4]\n",
      "Valid. Epoch 100: 100%|██████████| 27/27 [00:07<00:00,  3.68batch/s, loss=3.91e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 100 \tTraining Loss: 4.464786 \tValidation Loss: 4.677303\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 101: 100%|██████████| 209/209 [01:08<00:00,  3.07batch/s, loss=2.98e+4]\n",
      "Valid. Epoch 101: 100%|██████████| 27/27 [00:07<00:00,  3.51batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 101 \tTraining Loss: 4.460474 \tValidation Loss: 4.691601\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 102: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.98e+4]\n",
      "Valid. Epoch 102: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 102 \tTraining Loss: 4.457927 \tValidation Loss: 4.679241\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 103: 100%|██████████| 209/209 [01:06<00:00,  3.12batch/s, loss=2.97e+4]\n",
      "Valid. Epoch 103: 100%|██████████| 27/27 [00:07<00:00,  3.61batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 103 \tTraining Loss: 4.453283 \tValidation Loss: 4.692203\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 104: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.97e+4]\n",
      "Valid. Epoch 104: 100%|██████████| 27/27 [00:07<00:00,  3.65batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 104 \tTraining Loss: 4.449827 \tValidation Loss: 4.688268\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 105: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.97e+4]\n",
      "Valid. Epoch 105: 100%|██████████| 27/27 [00:07<00:00,  3.78batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 105 \tTraining Loss: 4.444532 \tValidation Loss: 4.687269\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 106: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.97e+4]\n",
      "Valid. Epoch 106: 100%|██████████| 27/27 [00:07<00:00,  3.67batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 106 \tTraining Loss: 4.438862 \tValidation Loss: 4.674627\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 107: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.96e+4]\n",
      "Valid. Epoch 107: 100%|██████████| 27/27 [00:07<00:00,  3.74batch/s, loss=3.91e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 107 \tTraining Loss: 4.436149 \tValidation Loss: 4.678952\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 108: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.96e+4]\n",
      "Valid. Epoch 108: 100%|██████████| 27/27 [00:07<00:00,  3.54batch/s, loss=3.91e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 108 \tTraining Loss: 4.433634 \tValidation Loss: 4.681213\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 109: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=2.96e+4]\n",
      "Valid. Epoch 109: 100%|██████████| 27/27 [00:07<00:00,  3.71batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 109 \tTraining Loss: 4.426822 \tValidation Loss: 4.687597\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 110: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.95e+4]\n",
      "Valid. Epoch 110: 100%|██████████| 27/27 [00:07<00:00,  3.66batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 110 \tTraining Loss: 4.419858 \tValidation Loss: 4.690460\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 111: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.95e+4]\n",
      "Valid. Epoch 111: 100%|██████████| 27/27 [00:07<00:00,  3.60batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 111 \tTraining Loss: 4.421523 \tValidation Loss: 4.693965\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 112: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=2.95e+4]\n",
      "Valid. Epoch 112: 100%|██████████| 27/27 [00:07<00:00,  3.56batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 112 \tTraining Loss: 4.415167 \tValidation Loss: 4.689252\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 113: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.95e+4]\n",
      "Valid. Epoch 113: 100%|██████████| 27/27 [00:07<00:00,  3.75batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 113 \tTraining Loss: 4.413592 \tValidation Loss: 4.693153\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 114: 100%|██████████| 209/209 [01:05<00:00,  3.19batch/s, loss=2.94e+4]\n",
      "Valid. Epoch 114: 100%|██████████| 27/27 [00:07<00:00,  3.70batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 114 \tTraining Loss: 4.400623 \tValidation Loss: 4.691845\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 115: 100%|██████████| 209/209 [01:08<00:00,  3.05batch/s, loss=2.94e+4]\n",
      "Valid. Epoch 115: 100%|██████████| 27/27 [00:07<00:00,  3.68batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 115 \tTraining Loss: 4.402778 \tValidation Loss: 4.697612\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 116:   0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 116: 100%|██████████| 209/209 [01:06<00:00,  3.15batch/s, loss=2.94e+4]\n",
      "Valid. Epoch 116: 100%|██████████| 27/27 [00:07<00:00,  3.61batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 116 \tTraining Loss: 4.398140 \tValidation Loss: 4.690088\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 117: 100%|██████████| 209/209 [01:06<00:00,  3.12batch/s, loss=2.94e+4]\n",
      "Valid. Epoch 117: 100%|██████████| 27/27 [00:07<00:00,  3.69batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 117 \tTraining Loss: 4.395332 \tValidation Loss: 4.687571\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 118: 100%|██████████| 209/209 [01:06<00:00,  3.12batch/s, loss=2.94e+4]\n",
      "Valid. Epoch 118: 100%|██████████| 27/27 [00:07<00:00,  3.73batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 118 \tTraining Loss: 4.394783 \tValidation Loss: 4.690366\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 119: 100%|██████████| 209/209 [01:08<00:00,  3.06batch/s, loss=2.93e+4]\n",
      "Valid. Epoch 119: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 119 \tTraining Loss: 4.385205 \tValidation Loss: 4.692025\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 120: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=2.93e+4]\n",
      "Valid. Epoch 120: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 120 \tTraining Loss: 4.386617 \tValidation Loss: 4.684752\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 121: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.93e+4]\n",
      "Valid. Epoch 121: 100%|██████████| 27/27 [00:07<00:00,  3.58batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 121 \tTraining Loss: 4.381295 \tValidation Loss: 4.689739\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 122: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.92e+4]\n",
      "Valid. Epoch 122: 100%|██████████| 27/27 [00:07<00:00,  3.65batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 122 \tTraining Loss: 4.373446 \tValidation Loss: 4.684590\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 123: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.92e+4]\n",
      "Valid. Epoch 123: 100%|██████████| 27/27 [00:07<00:00,  3.75batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 123 \tTraining Loss: 4.367914 \tValidation Loss: 4.684420\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 124: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.92e+4]\n",
      "Valid. Epoch 124: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 124 \tTraining Loss: 4.372928 \tValidation Loss: 4.689312\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 125: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.92e+4]\n",
      "Valid. Epoch 125: 100%|██████████| 27/27 [00:07<00:00,  3.58batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 125 \tTraining Loss: 4.365919 \tValidation Loss: 4.686558\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 126: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.91e+4]\n",
      "Valid. Epoch 126: 100%|██████████| 27/27 [00:07<00:00,  3.71batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 126 \tTraining Loss: 4.357860 \tValidation Loss: 4.697543\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 127: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.91e+4]\n",
      "Valid. Epoch 127: 100%|██████████| 27/27 [00:07<00:00,  3.66batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 127 \tTraining Loss: 4.354027 \tValidation Loss: 4.694030\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 128: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.9e+4] \n",
      "Valid. Epoch 128: 100%|██████████| 27/27 [00:07<00:00,  3.60batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 128 \tTraining Loss: 4.347289 \tValidation Loss: 4.694012\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 129: 100%|██████████| 209/209 [01:06<00:00,  3.16batch/s, loss=2.9e+4] \n",
      "Valid. Epoch 129: 100%|██████████| 27/27 [00:07<00:00,  3.66batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 129 \tTraining Loss: 4.346365 \tValidation Loss: 4.694399\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 130: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.9e+4] \n",
      "Valid. Epoch 130: 100%|██████████| 27/27 [00:07<00:00,  3.56batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 130 \tTraining Loss: 4.340389 \tValidation Loss: 4.687886\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 131: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.9e+4] \n",
      "Valid. Epoch 131: 100%|██████████| 27/27 [00:07<00:00,  3.53batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 131 \tTraining Loss: 4.340494 \tValidation Loss: 4.692506\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 132: 100%|██████████| 209/209 [01:08<00:00,  3.07batch/s, loss=2.9e+4] \n",
      "Valid. Epoch 132: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 132 \tTraining Loss: 4.335242 \tValidation Loss: 4.695957\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 133: 100%|██████████| 209/209 [01:06<00:00,  3.15batch/s, loss=2.89e+4]\n",
      "Valid. Epoch 133: 100%|██████████| 27/27 [00:07<00:00,  3.68batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 133 \tTraining Loss: 4.328253 \tValidation Loss: 4.692361\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 134: 100%|██████████| 209/209 [01:08<00:00,  3.07batch/s, loss=2.89e+4]\n",
      "Valid. Epoch 134: 100%|██████████| 27/27 [00:07<00:00,  3.57batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 134 \tTraining Loss: 4.329943 \tValidation Loss: 4.686843\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 135: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.89e+4]\n",
      "Valid. Epoch 135: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 135 \tTraining Loss: 4.327476 \tValidation Loss: 4.694768\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 136: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.89e+4]\n",
      "Valid. Epoch 136: 100%|██████████| 27/27 [00:07<00:00,  3.77batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 136 \tTraining Loss: 4.322046 \tValidation Loss: 4.695753\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 137: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=2.89e+4]\n",
      "Valid. Epoch 137: 100%|██████████| 27/27 [00:07<00:00,  3.53batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 137 \tTraining Loss: 4.322120 \tValidation Loss: 4.693659\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 138: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.88e+4]\n",
      "Valid. Epoch 138: 100%|██████████| 27/27 [00:07<00:00,  3.58batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 138 \tTraining Loss: 4.313780 \tValidation Loss: 4.688922\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 139: 100%|██████████| 209/209 [01:06<00:00,  3.15batch/s, loss=2.88e+4]\n",
      "Valid. Epoch 139: 100%|██████████| 27/27 [00:07<00:00,  3.54batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 139 \tTraining Loss: 4.311272 \tValidation Loss: 4.693772\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 140: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=2.88e+4]\n",
      "Valid. Epoch 140: 100%|██████████| 27/27 [00:07<00:00,  3.68batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 140 \tTraining Loss: 4.308878 \tValidation Loss: 4.683415\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 141: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=2.88e+4]\n",
      "Valid. Epoch 141: 100%|██████████| 27/27 [00:07<00:00,  3.51batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 141 \tTraining Loss: 4.306436 \tValidation Loss: 4.691083\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 142: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.87e+4]\n",
      "Valid. Epoch 142: 100%|██████████| 27/27 [00:07<00:00,  3.65batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 142 \tTraining Loss: 4.301911 \tValidation Loss: 4.696682\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 143: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.87e+4]\n",
      "Valid. Epoch 143: 100%|██████████| 27/27 [00:07<00:00,  3.65batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 143 \tTraining Loss: 4.298902 \tValidation Loss: 4.703166\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 144: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.87e+4]\n",
      "Valid. Epoch 144: 100%|██████████| 27/27 [00:07<00:00,  3.58batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 144 \tTraining Loss: 4.292394 \tValidation Loss: 4.693221\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 145: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.87e+4]\n",
      "Valid. Epoch 145: 100%|██████████| 27/27 [00:07<00:00,  3.67batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 145 \tTraining Loss: 4.291742 \tValidation Loss: 4.689258\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 146: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.86e+4]\n",
      "Valid. Epoch 146: 100%|██████████| 27/27 [00:07<00:00,  3.70batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 146 \tTraining Loss: 4.288680 \tValidation Loss: 4.693616\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 147: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.86e+4]\n",
      "Valid. Epoch 147: 100%|██████████| 27/27 [00:06<00:00,  3.89batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 147 \tTraining Loss: 4.285154 \tValidation Loss: 4.693064\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 148: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.86e+4]\n",
      "Valid. Epoch 148: 100%|██████████| 27/27 [00:07<00:00,  3.58batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 148 \tTraining Loss: 4.284919 \tValidation Loss: 4.699423\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 149: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.86e+4]\n",
      "Valid. Epoch 149: 100%|██████████| 27/27 [00:07<00:00,  3.73batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 149 \tTraining Loss: 4.282212 \tValidation Loss: 4.688961\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 150: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.86e+4]\n",
      "Valid. Epoch 150: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 150 \tTraining Loss: 4.282291 \tValidation Loss: 4.691018\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 151: 100%|██████████| 209/209 [01:07<00:00,  3.07batch/s, loss=2.86e+4]\n",
      "Valid. Epoch 151: 100%|██████████| 27/27 [00:07<00:00,  3.67batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 151 \tTraining Loss: 4.276676 \tValidation Loss: 4.693094\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 152: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.86e+4]\n",
      "Valid. Epoch 152: 100%|██████████| 27/27 [00:07<00:00,  3.65batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 152 \tTraining Loss: 4.277874 \tValidation Loss: 4.697564\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 153: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.85e+4]\n",
      "Valid. Epoch 153: 100%|██████████| 27/27 [00:07<00:00,  3.57batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 153 \tTraining Loss: 4.272832 \tValidation Loss: 4.688191\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 154: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=2.85e+4]\n",
      "Valid. Epoch 154: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 154 \tTraining Loss: 4.264248 \tValidation Loss: 4.685075\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 155: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=2.85e+4]\n",
      "Valid. Epoch 155: 100%|██████████| 27/27 [00:07<00:00,  3.64batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 155 \tTraining Loss: 4.265490 \tValidation Loss: 4.690589\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 156: 100%|██████████| 209/209 [01:06<00:00,  3.15batch/s, loss=2.85e+4]\n",
      "Valid. Epoch 156: 100%|██████████| 27/27 [00:07<00:00,  3.82batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 156 \tTraining Loss: 4.264391 \tValidation Loss: 4.697789\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 157: 100%|██████████| 209/209 [01:08<00:00,  3.07batch/s, loss=2.85e+4]\n",
      "Valid. Epoch 157: 100%|██████████| 27/27 [00:07<00:00,  3.64batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 157 \tTraining Loss: 4.259029 \tValidation Loss: 4.697725\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 158: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=2.84e+4]\n",
      "Valid. Epoch 158: 100%|██████████| 27/27 [00:07<00:00,  3.56batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 158 \tTraining Loss: 4.257527 \tValidation Loss: 4.694578\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 159: 100%|██████████| 209/209 [01:08<00:00,  3.07batch/s, loss=2.84e+4]\n",
      "Valid. Epoch 159: 100%|██████████| 27/27 [00:07<00:00,  3.78batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 159 \tTraining Loss: 4.251944 \tValidation Loss: 4.695550\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 160: 100%|██████████| 209/209 [01:08<00:00,  3.06batch/s, loss=2.83e+4]\n",
      "Valid. Epoch 160: 100%|██████████| 27/27 [00:07<00:00,  3.70batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 160 \tTraining Loss: 4.243334 \tValidation Loss: 4.695150\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 161: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.84e+4]\n",
      "Valid. Epoch 161: 100%|██████████| 27/27 [00:07<00:00,  3.54batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 161 \tTraining Loss: 4.248165 \tValidation Loss: 4.698407\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 162: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.84e+4]\n",
      "Valid. Epoch 162: 100%|██████████| 27/27 [00:07<00:00,  3.57batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 162 \tTraining Loss: 4.245198 \tValidation Loss: 4.704319\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 163: 100%|██████████| 209/209 [01:07<00:00,  3.07batch/s, loss=2.83e+4]\n",
      "Valid. Epoch 163: 100%|██████████| 27/27 [00:07<00:00,  3.59batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 163 \tTraining Loss: 4.243921 \tValidation Loss: 4.702868\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 164: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.84e+4]\n",
      "Valid. Epoch 164: 100%|██████████| 27/27 [00:07<00:00,  3.55batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 164 \tTraining Loss: 4.246844 \tValidation Loss: 4.692017\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 165: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.83e+4]\n",
      "Valid. Epoch 165: 100%|██████████| 27/27 [00:07<00:00,  3.71batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 165 \tTraining Loss: 4.243086 \tValidation Loss: 4.700937\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 166: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.83e+4]\n",
      "Valid. Epoch 166: 100%|██████████| 27/27 [00:07<00:00,  3.60batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 166 \tTraining Loss: 4.234826 \tValidation Loss: 4.693401\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 167:   0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 167: 100%|██████████| 209/209 [01:06<00:00,  3.15batch/s, loss=2.83e+4]\n",
      "Valid. Epoch 167: 100%|██████████| 27/27 [00:07<00:00,  3.79batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 167 \tTraining Loss: 4.240547 \tValidation Loss: 4.694154\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 168: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.83e+4]\n",
      "Valid. Epoch 168: 100%|██████████| 27/27 [00:07<00:00,  3.61batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 168 \tTraining Loss: 4.236335 \tValidation Loss: 4.690016\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 169: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.82e+4]\n",
      "Valid. Epoch 169: 100%|██████████| 27/27 [00:07<00:00,  3.57batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 169 \tTraining Loss: 4.228607 \tValidation Loss: 4.690443\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 170: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.82e+4]\n",
      "Valid. Epoch 170: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 170 \tTraining Loss: 4.228169 \tValidation Loss: 4.697242\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 171: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.82e+4]\n",
      "Valid. Epoch 171: 100%|██████████| 27/27 [00:07<00:00,  3.86batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 171 \tTraining Loss: 4.226137 \tValidation Loss: 4.698196\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 172:   0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 172: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.82e+4]\n",
      "Valid. Epoch 172: 100%|██████████| 27/27 [00:07<00:00,  3.48batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 172 \tTraining Loss: 4.226366 \tValidation Loss: 4.700656\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 173: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.82e+4]\n",
      "Valid. Epoch 173: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 173 \tTraining Loss: 4.223990 \tValidation Loss: 4.706312\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 174: 100%|██████████| 209/209 [01:08<00:00,  3.06batch/s, loss=2.82e+4]\n",
      "Valid. Epoch 174: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 174 \tTraining Loss: 4.225889 \tValidation Loss: 4.704112\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 175: 100%|██████████| 209/209 [01:05<00:00,  3.21batch/s, loss=2.81e+4]\n",
      "Valid. Epoch 175: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 175 \tTraining Loss: 4.210118 \tValidation Loss: 4.700677\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 176: 100%|██████████| 209/209 [01:06<00:00,  3.16batch/s, loss=2.82e+4]\n",
      "Valid. Epoch 176: 100%|██████████| 27/27 [00:07<00:00,  3.64batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 176 \tTraining Loss: 4.216311 \tValidation Loss: 4.705316\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 177: 100%|██████████| 209/209 [01:06<00:00,  3.12batch/s, loss=2.81e+4]\n",
      "Valid. Epoch 177: 100%|██████████| 27/27 [00:07<00:00,  3.69batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 177 \tTraining Loss: 4.210757 \tValidation Loss: 4.698105\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 178:   0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 178: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.81e+4]\n",
      "Valid. Epoch 178: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 178 \tTraining Loss: 4.210517 \tValidation Loss: 4.696174\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 179: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.81e+4]\n",
      "Valid. Epoch 179: 100%|██████████| 27/27 [00:07<00:00,  3.67batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 179 \tTraining Loss: 4.209214 \tValidation Loss: 4.694364\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 180: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=2.81e+4]\n",
      "Valid. Epoch 180: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.94e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 180 \tTraining Loss: 4.200379 \tValidation Loss: 4.711141\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 181: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.81e+4]\n",
      "Valid. Epoch 181: 100%|██████████| 27/27 [00:07<00:00,  3.67batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 181 \tTraining Loss: 4.203010 \tValidation Loss: 4.702340\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 182: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.8e+4] \n",
      "Valid. Epoch 182: 100%|██████████| 27/27 [00:07<00:00,  3.59batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 182 \tTraining Loss: 4.198677 \tValidation Loss: 4.702565\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 183: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.81e+4]\n",
      "Valid. Epoch 183: 100%|██████████| 27/27 [00:07<00:00,  3.74batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 183 \tTraining Loss: 4.201843 \tValidation Loss: 4.699024\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 184: 100%|██████████| 209/209 [01:06<00:00,  3.16batch/s, loss=2.8e+4] \n",
      "Valid. Epoch 184: 100%|██████████| 27/27 [00:07<00:00,  3.70batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 184 \tTraining Loss: 4.190923 \tValidation Loss: 4.702440\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 185: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.8e+4] \n",
      "Valid. Epoch 185: 100%|██████████| 27/27 [00:07<00:00,  3.57batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 185 \tTraining Loss: 4.196194 \tValidation Loss: 4.702928\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 186: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.8e+4] \n",
      "Valid. Epoch 186: 100%|██████████| 27/27 [00:07<00:00,  3.72batch/s, loss=3.92e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 186 \tTraining Loss: 4.195723 \tValidation Loss: 4.694455\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 187: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.8e+4] \n",
      "Valid. Epoch 187: 100%|██████████| 27/27 [00:07<00:00,  3.68batch/s, loss=3.92e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 187 \tTraining Loss: 4.190030 \tValidation Loss: 4.694527\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 188: 100%|██████████| 209/209 [01:05<00:00,  3.21batch/s, loss=2.8e+4] \n",
      "Valid. Epoch 188: 100%|██████████| 27/27 [00:07<00:00,  3.64batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 188 \tTraining Loss: 4.189001 \tValidation Loss: 4.703826\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 189: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.8e+4] \n",
      "Valid. Epoch 189: 100%|██████████| 27/27 [00:07<00:00,  3.78batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 189 \tTraining Loss: 4.191095 \tValidation Loss: 4.697735\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 190: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.8e+4] \n",
      "Valid. Epoch 190: 100%|██████████| 27/27 [00:07<00:00,  3.60batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 190 \tTraining Loss: 4.190307 \tValidation Loss: 4.698559\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 191: 100%|██████████| 209/209 [01:08<00:00,  3.07batch/s, loss=2.8e+4] \n",
      "Valid. Epoch 191: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 191 \tTraining Loss: 4.186487 \tValidation Loss: 4.703086\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 192: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.79e+4]\n",
      "Valid. Epoch 192: 100%|██████████| 27/27 [00:07<00:00,  3.64batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 192 \tTraining Loss: 4.184103 \tValidation Loss: 4.698923\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 193: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=2.79e+4]\n",
      "Valid. Epoch 193: 100%|██████████| 27/27 [00:06<00:00,  3.90batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 193 \tTraining Loss: 4.181907 \tValidation Loss: 4.699669\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 194: 100%|██████████| 209/209 [01:06<00:00,  3.15batch/s, loss=2.8e+4] \n",
      "Valid. Epoch 194: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 194 \tTraining Loss: 4.184551 \tValidation Loss: 4.701245\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 195: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=2.79e+4]\n",
      "Valid. Epoch 195: 100%|██████████| 27/27 [00:07<00:00,  3.61batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 195 \tTraining Loss: 4.181992 \tValidation Loss: 4.702001\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 196: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.79e+4]\n",
      "Valid. Epoch 196: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.94e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 196 \tTraining Loss: 4.181155 \tValidation Loss: 4.709567\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 197: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=2.79e+4]\n",
      "Valid. Epoch 197: 100%|██████████| 27/27 [00:07<00:00,  3.56batch/s, loss=3.94e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 197 \tTraining Loss: 4.179125 \tValidation Loss: 4.710854\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 198: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.79e+4]\n",
      "Valid. Epoch 198: 100%|██████████| 27/27 [00:07<00:00,  3.68batch/s, loss=3.94e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 198 \tTraining Loss: 4.170377 \tValidation Loss: 4.708844\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 199:   0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 199: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.78e+4]\n",
      "Valid. Epoch 199: 100%|██████████| 27/27 [00:07<00:00,  3.57batch/s, loss=3.94e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 199 \tTraining Loss: 4.165540 \tValidation Loss: 4.714741\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 200: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.79e+4]\n",
      "Valid. Epoch 200: 100%|██████████| 27/27 [00:07<00:00,  3.60batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 200 \tTraining Loss: 4.173892 \tValidation Loss: 4.703891\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 201: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=2.78e+4]\n",
      "Valid. Epoch 201: 100%|██████████| 27/27 [00:07<00:00,  3.61batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 201 \tTraining Loss: 4.168731 \tValidation Loss: 4.720355\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 202: 100%|██████████| 209/209 [01:08<00:00,  3.07batch/s, loss=2.78e+4]\n",
      "Valid. Epoch 202: 100%|██████████| 27/27 [00:06<00:00,  3.90batch/s, loss=3.94e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 202 \tTraining Loss: 4.166334 \tValidation Loss: 4.714984\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 203: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.78e+4]\n",
      "Valid. Epoch 203: 100%|██████████| 27/27 [00:07<00:00,  3.53batch/s, loss=3.94e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 203 \tTraining Loss: 4.160061 \tValidation Loss: 4.708600\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 204: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.78e+4]\n",
      "Valid. Epoch 204: 100%|██████████| 27/27 [00:07<00:00,  3.58batch/s, loss=3.94e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 204 \tTraining Loss: 4.158580 \tValidation Loss: 4.711186\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 205: 100%|██████████| 209/209 [01:06<00:00,  3.15batch/s, loss=2.78e+4]\n",
      "Valid. Epoch 205: 100%|██████████| 27/27 [00:07<00:00,  3.66batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 205 \tTraining Loss: 4.161963 \tValidation Loss: 4.706644\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 206: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.78e+4]\n",
      "Valid. Epoch 206: 100%|██████████| 27/27 [00:06<00:00,  3.90batch/s, loss=3.94e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 206 \tTraining Loss: 4.165502 \tValidation Loss: 4.707795\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 207: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.78e+4]\n",
      "Valid. Epoch 207: 100%|██████████| 27/27 [00:07<00:00,  3.59batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 207 \tTraining Loss: 4.157894 \tValidation Loss: 4.704514\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 208: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.78e+4]\n",
      "Valid. Epoch 208: 100%|██████████| 27/27 [00:07<00:00,  3.57batch/s, loss=3.94e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 208 \tTraining Loss: 4.157359 \tValidation Loss: 4.712716\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 209: 100%|██████████| 209/209 [01:05<00:00,  3.18batch/s, loss=2.77e+4]\n",
      "Valid. Epoch 209: 100%|██████████| 27/27 [00:06<00:00,  3.86batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 209 \tTraining Loss: 4.150961 \tValidation Loss: 4.704954\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 210:   0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 210: 100%|██████████| 209/209 [01:05<00:00,  3.19batch/s, loss=2.78e+4]\n",
      "Valid. Epoch 210: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 210 \tTraining Loss: 4.155435 \tValidation Loss: 4.706352\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 211: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=2.77e+4]\n",
      "Valid. Epoch 211: 100%|██████████| 27/27 [00:07<00:00,  3.64batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 211 \tTraining Loss: 4.153251 \tValidation Loss: 4.702725\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 212: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.77e+4]\n",
      "Valid. Epoch 212: 100%|██████████| 27/27 [00:07<00:00,  3.64batch/s, loss=3.93e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 212 \tTraining Loss: 4.148889 \tValidation Loss: 4.704620\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 213:   0%|          | 0/209 [00:00<?, ?batch/s, loss=131]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 213: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.77e+4]\n",
      "Valid. Epoch 213: 100%|██████████| 27/27 [00:07<00:00,  3.69batch/s, loss=3.93e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 213 \tTraining Loss: 4.146470 \tValidation Loss: 4.705036\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 214: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.77e+4]\n",
      "Valid. Epoch 214: 100%|██████████| 27/27 [00:07<00:00,  3.58batch/s, loss=3.94e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 214 \tTraining Loss: 4.149190 \tValidation Loss: 4.715051\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 215: 100%|██████████| 209/209 [01:05<00:00,  3.17batch/s, loss=2.77e+4]\n",
      "Valid. Epoch 215: 100%|██████████| 27/27 [00:07<00:00,  3.57batch/s, loss=3.94e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 215 \tTraining Loss: 4.144606 \tValidation Loss: 4.711238\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 216: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=2.77e+4]\n",
      "Valid. Epoch 216: 100%|██████████| 27/27 [00:07<00:00,  3.73batch/s, loss=3.94e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 216 \tTraining Loss: 4.146671 \tValidation Loss: 4.715573\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 217: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.77e+4]\n",
      "Valid. Epoch 217: 100%|██████████| 27/27 [00:07<00:00,  3.57batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 217 \tTraining Loss: 4.142456 \tValidation Loss: 4.726346\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 218: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.77e+4]\n",
      "Valid. Epoch 218: 100%|██████████| 27/27 [00:07<00:00,  3.70batch/s, loss=3.94e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 218 \tTraining Loss: 4.140070 \tValidation Loss: 4.714533\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 219: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.77e+4]\n",
      "Valid. Epoch 219: 100%|██████████| 27/27 [00:07<00:00,  3.56batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 219 \tTraining Loss: 4.141195 \tValidation Loss: 4.719244\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 220: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.77e+4]\n",
      "Valid. Epoch 220: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.94e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 220 \tTraining Loss: 4.143628 \tValidation Loss: 4.711465\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 221: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.76e+4]\n",
      "Valid. Epoch 221: 100%|██████████| 27/27 [00:07<00:00,  3.56batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 221 \tTraining Loss: 4.133699 \tValidation Loss: 4.720233\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 222: 100%|██████████| 209/209 [01:08<00:00,  3.07batch/s, loss=2.76e+4]\n",
      "Valid. Epoch 222: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.94e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 222 \tTraining Loss: 4.136245 \tValidation Loss: 4.717081\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 223: 100%|██████████| 209/209 [01:06<00:00,  3.15batch/s, loss=2.76e+4]\n",
      "Valid. Epoch 223: 100%|██████████| 27/27 [00:07<00:00,  3.81batch/s, loss=3.94e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 223 \tTraining Loss: 4.132090 \tValidation Loss: 4.715069\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 224: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.76e+4]\n",
      "Valid. Epoch 224: 100%|██████████| 27/27 [00:07<00:00,  3.59batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 224 \tTraining Loss: 4.131494 \tValidation Loss: 4.719722\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 225: 100%|██████████| 209/209 [01:06<00:00,  3.12batch/s, loss=2.76e+4]\n",
      "Valid. Epoch 225: 100%|██████████| 27/27 [00:07<00:00,  3.53batch/s, loss=3.94e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 225 \tTraining Loss: 4.131531 \tValidation Loss: 4.716767\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 226: 100%|██████████| 209/209 [01:08<00:00,  3.07batch/s, loss=2.76e+4]\n",
      "Valid. Epoch 226: 100%|██████████| 27/27 [00:07<00:00,  3.59batch/s, loss=3.94e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 226 \tTraining Loss: 4.128416 \tValidation Loss: 4.713854\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 227: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.76e+4]\n",
      "Valid. Epoch 227: 100%|██████████| 27/27 [00:07<00:00,  3.83batch/s, loss=3.94e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 227 \tTraining Loss: 4.125831 \tValidation Loss: 4.715677\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 228: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.76e+4]\n",
      "Valid. Epoch 228: 100%|██████████| 27/27 [00:07<00:00,  3.71batch/s, loss=3.94e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 228 \tTraining Loss: 4.131269 \tValidation Loss: 4.713006\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 229: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.76e+4]\n",
      "Valid. Epoch 229: 100%|██████████| 27/27 [00:07<00:00,  3.55batch/s, loss=3.94e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 229 \tTraining Loss: 4.127741 \tValidation Loss: 4.712527\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 230: 100%|██████████| 209/209 [01:06<00:00,  3.15batch/s, loss=2.76e+4]\n",
      "Valid. Epoch 230: 100%|██████████| 27/27 [00:07<00:00,  3.59batch/s, loss=3.94e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 230 \tTraining Loss: 4.125431 \tValidation Loss: 4.714334\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 231: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.75e+4]\n",
      "Valid. Epoch 231: 100%|██████████| 27/27 [00:07<00:00,  3.67batch/s, loss=3.94e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 231 \tTraining Loss: 4.123374 \tValidation Loss: 4.717834\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 232: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.76e+4]\n",
      "Valid. Epoch 232: 100%|██████████| 27/27 [00:07<00:00,  3.68batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 232 \tTraining Loss: 4.125322 \tValidation Loss: 4.724246\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 233: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.75e+4]\n",
      "Valid. Epoch 233: 100%|██████████| 27/27 [00:07<00:00,  3.69batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 233 \tTraining Loss: 4.123385 \tValidation Loss: 4.726544\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 234: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.75e+4]\n",
      "Valid. Epoch 234: 100%|██████████| 27/27 [00:07<00:00,  3.55batch/s, loss=3.94e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 234 \tTraining Loss: 4.123425 \tValidation Loss: 4.718168\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 235: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.75e+4]\n",
      "Valid. Epoch 235: 100%|██████████| 27/27 [00:06<00:00,  3.88batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 235 \tTraining Loss: 4.121211 \tValidation Loss: 4.728933\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 236: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=2.75e+4]\n",
      "Valid. Epoch 236: 100%|██████████| 27/27 [00:07<00:00,  3.64batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 236 \tTraining Loss: 4.121101 \tValidation Loss: 4.724308\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 237: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.75e+4]\n",
      "Valid. Epoch 237: 100%|██████████| 27/27 [00:07<00:00,  3.69batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 237 \tTraining Loss: 4.117356 \tValidation Loss: 4.725534\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 238: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.75e+4]\n",
      "Valid. Epoch 238: 100%|██████████| 27/27 [00:07<00:00,  3.61batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 238 \tTraining Loss: 4.117013 \tValidation Loss: 4.724283\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 239: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=2.75e+4]\n",
      "Valid. Epoch 239: 100%|██████████| 27/27 [00:07<00:00,  3.61batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 239 \tTraining Loss: 4.113018 \tValidation Loss: 4.728166\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 240: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.75e+4]\n",
      "Valid. Epoch 240: 100%|██████████| 27/27 [00:07<00:00,  3.58batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 240 \tTraining Loss: 4.111961 \tValidation Loss: 4.723663\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 241: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.75e+4]\n",
      "Valid. Epoch 241: 100%|██████████| 27/27 [00:07<00:00,  3.76batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 241 \tTraining Loss: 4.113654 \tValidation Loss: 4.732463\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 242: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.75e+4]\n",
      "Valid. Epoch 242: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 242 \tTraining Loss: 4.114716 \tValidation Loss: 4.731104\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 243: 100%|██████████| 209/209 [01:05<00:00,  3.17batch/s, loss=2.75e+4]\n",
      "Valid. Epoch 243: 100%|██████████| 27/27 [00:07<00:00,  3.58batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 243 \tTraining Loss: 4.113906 \tValidation Loss: 4.733771\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 244: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.74e+4]\n",
      "Valid. Epoch 244: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 244 \tTraining Loss: 4.105651 \tValidation Loss: 4.732473\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 245: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.75e+4]\n",
      "Valid. Epoch 245: 100%|██████████| 27/27 [00:07<00:00,  3.77batch/s, loss=3.96e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 245 \tTraining Loss: 4.112928 \tValidation Loss: 4.738187\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 246: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.75e+4]\n",
      "Valid. Epoch 246: 100%|██████████| 27/27 [00:07<00:00,  3.65batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 246 \tTraining Loss: 4.111606 \tValidation Loss: 4.728867\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 247: 100%|██████████| 209/209 [01:06<00:00,  3.15batch/s, loss=2.75e+4]\n",
      "Valid. Epoch 247: 100%|██████████| 27/27 [00:07<00:00,  3.68batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 247 \tTraining Loss: 4.112938 \tValidation Loss: 4.727820\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 248: 100%|██████████| 209/209 [01:06<00:00,  3.16batch/s, loss=2.74e+4]\n",
      "Valid. Epoch 248: 100%|██████████| 27/27 [00:07<00:00,  3.60batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 248 \tTraining Loss: 4.108628 \tValidation Loss: 4.725475\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 249: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=2.74e+4]\n",
      "Valid. Epoch 249: 100%|██████████| 27/27 [00:07<00:00,  3.60batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 249 \tTraining Loss: 4.103721 \tValidation Loss: 4.723694\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 250: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.74e+4]\n",
      "Valid. Epoch 250: 100%|██████████| 27/27 [00:07<00:00,  3.66batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 250 \tTraining Loss: 4.106060 \tValidation Loss: 4.719773\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 251: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.74e+4]\n",
      "Valid. Epoch 251: 100%|██████████| 27/27 [00:07<00:00,  3.61batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 251 \tTraining Loss: 4.101162 \tValidation Loss: 4.722623\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 252: 100%|██████████| 209/209 [01:05<00:00,  3.18batch/s, loss=2.74e+4]\n",
      "Valid. Epoch 252: 100%|██████████| 27/27 [00:07<00:00,  3.82batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 252 \tTraining Loss: 4.103801 \tValidation Loss: 4.723060\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 253: 100%|██████████| 209/209 [01:06<00:00,  3.12batch/s, loss=2.74e+4]\n",
      "Valid. Epoch 253: 100%|██████████| 27/27 [00:07<00:00,  3.75batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 253 \tTraining Loss: 4.102861 \tValidation Loss: 4.722045\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 254: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.74e+4]\n",
      "Valid. Epoch 254: 100%|██████████| 27/27 [00:07<00:00,  3.61batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 254 \tTraining Loss: 4.104823 \tValidation Loss: 4.723591\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 255: 100%|██████████| 209/209 [01:05<00:00,  3.18batch/s, loss=2.74e+4]\n",
      "Valid. Epoch 255: 100%|██████████| 27/27 [00:06<00:00,  3.87batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 255 \tTraining Loss: 4.099656 \tValidation Loss: 4.726251\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 256: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.74e+4]\n",
      "Valid. Epoch 256: 100%|██████████| 27/27 [00:07<00:00,  3.64batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 256 \tTraining Loss: 4.096896 \tValidation Loss: 4.724645\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 257: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.74e+4]\n",
      "Valid. Epoch 257: 100%|██████████| 27/27 [00:06<00:00,  3.92batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 257 \tTraining Loss: 4.098523 \tValidation Loss: 4.725941\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 258: 100%|██████████| 209/209 [01:06<00:00,  3.16batch/s, loss=2.74e+4]\n",
      "Valid. Epoch 258: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 258 \tTraining Loss: 4.099338 \tValidation Loss: 4.726283\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 259: 100%|██████████| 209/209 [01:07<00:00,  3.07batch/s, loss=2.74e+4]\n",
      "Valid. Epoch 259: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 259 \tTraining Loss: 4.095286 \tValidation Loss: 4.735974\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 260: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.74e+4]\n",
      "Valid. Epoch 260: 100%|██████████| 27/27 [00:07<00:00,  3.61batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 260 \tTraining Loss: 4.094902 \tValidation Loss: 4.730000\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 261: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.74e+4]\n",
      "Valid. Epoch 261: 100%|██████████| 27/27 [00:07<00:00,  3.81batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 261 \tTraining Loss: 4.094428 \tValidation Loss: 4.729771\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 262: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=2.73e+4]\n",
      "Valid. Epoch 262: 100%|██████████| 27/27 [00:07<00:00,  3.75batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 262 \tTraining Loss: 4.090336 \tValidation Loss: 4.733500\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 263: 100%|██████████| 209/209 [01:06<00:00,  3.13batch/s, loss=2.73e+4]\n",
      "Valid. Epoch 263: 100%|██████████| 27/27 [00:07<00:00,  3.56batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 263 \tTraining Loss: 4.093880 \tValidation Loss: 4.726579\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 264: 100%|██████████| 209/209 [01:06<00:00,  3.12batch/s, loss=2.73e+4]\n",
      "Valid. Epoch 264: 100%|██████████| 27/27 [00:07<00:00,  3.70batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 264 \tTraining Loss: 4.093711 \tValidation Loss: 4.728403\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 265: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.73e+4]\n",
      "Valid. Epoch 265: 100%|██████████| 27/27 [00:07<00:00,  3.63batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 265 \tTraining Loss: 4.090097 \tValidation Loss: 4.730072\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 266: 100%|██████████| 209/209 [01:08<00:00,  3.07batch/s, loss=2.73e+4]\n",
      "Valid. Epoch 266: 100%|██████████| 27/27 [00:07<00:00,  3.55batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 266 \tTraining Loss: 4.088137 \tValidation Loss: 4.732268\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 267: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=2.73e+4]\n",
      "Valid. Epoch 267: 100%|██████████| 27/27 [00:07<00:00,  3.60batch/s, loss=3.96e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 267 \tTraining Loss: 4.085478 \tValidation Loss: 4.738687\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 268: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.73e+4]\n",
      "Valid. Epoch 268: 100%|██████████| 27/27 [00:07<00:00,  3.65batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 268 \tTraining Loss: 4.085371 \tValidation Loss: 4.729684\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 269: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.73e+4]\n",
      "Valid. Epoch 269: 100%|██████████| 27/27 [00:07<00:00,  3.70batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 269 \tTraining Loss: 4.083128 \tValidation Loss: 4.732665\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 270: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.73e+4]\n",
      "Valid. Epoch 270: 100%|██████████| 27/27 [00:07<00:00,  3.61batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 270 \tTraining Loss: 4.088477 \tValidation Loss: 4.740406\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 271: 100%|██████████| 209/209 [01:05<00:00,  3.21batch/s, loss=2.73e+4]\n",
      "Valid. Epoch 271: 100%|██████████| 27/27 [00:07<00:00,  3.57batch/s, loss=3.96e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 271 \tTraining Loss: 4.082819 \tValidation Loss: 4.736836\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 272: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.73e+4]\n",
      "Valid. Epoch 272: 100%|██████████| 27/27 [00:07<00:00,  3.57batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 272 \tTraining Loss: 4.082206 \tValidation Loss: 4.730747\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 273: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.73e+4]\n",
      "Valid. Epoch 273: 100%|██████████| 27/27 [00:07<00:00,  3.72batch/s, loss=3.96e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 273 \tTraining Loss: 4.084480 \tValidation Loss: 4.734172\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 274: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.73e+4]\n",
      "Valid. Epoch 274: 100%|██████████| 27/27 [00:07<00:00,  3.50batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 274 \tTraining Loss: 4.079576 \tValidation Loss: 4.736182\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 275: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.73e+4]\n",
      "Valid. Epoch 275: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 275 \tTraining Loss: 4.083653 \tValidation Loss: 4.729566\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 276: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 276: 100%|██████████| 27/27 [00:07<00:00,  3.78batch/s, loss=3.96e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 276 \tTraining Loss: 4.078554 \tValidation Loss: 4.733967\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 277: 100%|██████████| 209/209 [01:07<00:00,  3.12batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 277: 100%|██████████| 27/27 [00:07<00:00,  3.57batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 277 \tTraining Loss: 4.077224 \tValidation Loss: 4.727992\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 278: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 278: 100%|██████████| 27/27 [00:07<00:00,  3.52batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 278 \tTraining Loss: 4.077905 \tValidation Loss: 4.730400\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 279: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 279: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 279 \tTraining Loss: 4.076847 \tValidation Loss: 4.730613\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 280: 100%|██████████| 209/209 [01:06<00:00,  3.12batch/s, loss=2.73e+4]\n",
      "Valid. Epoch 280: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 280 \tTraining Loss: 4.080672 \tValidation Loss: 4.722816\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 281: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 281: 100%|██████████| 27/27 [00:07<00:00,  3.85batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 281 \tTraining Loss: 4.077863 \tValidation Loss: 4.728639\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 282: 100%|██████████| 209/209 [01:07<00:00,  3.07batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 282: 100%|██████████| 27/27 [00:07<00:00,  3.60batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 282 \tTraining Loss: 4.075582 \tValidation Loss: 4.729630\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 283: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 283: 100%|██████████| 27/27 [00:07<00:00,  3.74batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 283 \tTraining Loss: 4.077048 \tValidation Loss: 4.728042\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 284: 100%|██████████| 209/209 [01:06<00:00,  3.12batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 284: 100%|██████████| 27/27 [00:07<00:00,  3.68batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 284 \tTraining Loss: 4.077802 \tValidation Loss: 4.734362\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 285: 100%|██████████| 209/209 [01:08<00:00,  3.06batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 285: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 285 \tTraining Loss: 4.078472 \tValidation Loss: 4.733177\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 286: 100%|██████████| 209/209 [01:08<00:00,  3.07batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 286: 100%|██████████| 27/27 [00:07<00:00,  3.57batch/s, loss=3.96e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 286 \tTraining Loss: 4.076596 \tValidation Loss: 4.736462\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 287: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 287: 100%|██████████| 27/27 [00:07<00:00,  3.70batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 287 \tTraining Loss: 4.073434 \tValidation Loss: 4.732443\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 288: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 288: 100%|██████████| 27/27 [00:07<00:00,  3.74batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 288 \tTraining Loss: 4.074758 \tValidation Loss: 4.724780\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 289: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 289: 100%|██████████| 27/27 [00:07<00:00,  3.75batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 289 \tTraining Loss: 4.073534 \tValidation Loss: 4.728836\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 290: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 290: 100%|██████████| 27/27 [00:07<00:00,  3.64batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 290 \tTraining Loss: 4.074055 \tValidation Loss: 4.729736\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 291: 100%|██████████| 209/209 [01:07<00:00,  3.10batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 291: 100%|██████████| 27/27 [00:07<00:00,  3.66batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 291 \tTraining Loss: 4.070655 \tValidation Loss: 4.732070\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 292: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 292: 100%|██████████| 27/27 [00:07<00:00,  3.62batch/s, loss=3.94e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 292 \tTraining Loss: 4.074772 \tValidation Loss: 4.718142\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 293: 100%|██████████| 209/209 [01:07<00:00,  3.09batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 293: 100%|██████████| 27/27 [00:07<00:00,  3.64batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 293 \tTraining Loss: 4.070978 \tValidation Loss: 4.724296\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 294: 100%|██████████| 209/209 [01:06<00:00,  3.14batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 294: 100%|██████████| 27/27 [00:06<00:00,  3.90batch/s, loss=3.95e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 294 \tTraining Loss: 4.069394 \tValidation Loss: 4.726571\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 295: 100%|██████████| 209/209 [01:06<00:00,  3.12batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 295: 100%|██████████| 27/27 [00:07<00:00,  3.83batch/s, loss=3.96e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 295 \tTraining Loss: 4.066405 \tValidation Loss: 4.735570\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 296: 100%|██████████| 209/209 [01:08<00:00,  3.06batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 296: 100%|██████████| 27/27 [00:07<00:00,  3.60batch/s, loss=3.95e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 296 \tTraining Loss: 4.067986 \tValidation Loss: 4.730159\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 297: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 297: 100%|██████████| 27/27 [00:07<00:00,  3.58batch/s, loss=3.96e+3]\n",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 297 \tTraining Loss: 4.072881 \tValidation Loss: 4.737721\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 298: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 298: 100%|██████████| 27/27 [00:07<00:00,  3.56batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 298 \tTraining Loss: 4.067602 \tValidation Loss: 4.738943\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 299: 100%|██████████| 209/209 [01:07<00:00,  3.08batch/s, loss=2.72e+4]\n",
      "Valid. Epoch 299: 100%|██████████| 27/27 [00:07<00:00,  3.58batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 299 \tTraining Loss: 4.069098 \tValidation Loss: 4.736155\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/209 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train. Epoch 300: 100%|██████████| 209/209 [01:07<00:00,  3.11batch/s, loss=2.71e+4]\n",
      "Valid. Epoch 300: 100%|██████████| 27/27 [00:07<00:00,  3.69batch/s, loss=3.96e+3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 300 \tTraining Loss: 4.064142 \tValidation Loss: 4.737166\n",
      "SUCCESSFULLY SAVED 'last-' checkpoint to: checkpoints/scratch-model-batch-32-lr-0.1/last-scratch-model-batch-32-lr-0.1.pt\n"
     ]
    }
   ],
   "source": [
    "# train the model\n",
    "num_epochs = 300\n",
    "# model_scratch = train(num_epochs, loaders_scratch, model_scratch, optimizer_scratch, \n",
    "#                       criterion_scratch, use_cuda, resume_train=True, result_dict=result_dict)\n",
    "\n",
    "model_scratch = train(num_epochs, loaders_scratch, model_scratch, optimizer_scratch, \n",
    "                      criterion_scratch, use_cuda, resume_train=False, result_dict=None)\n",
    "\n",
    "# load the model that got the best validation accuracy\n",
    "# model_scratch.load_state_dict(torch.load('model_scratch.pt'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (IMPLEMENTATION) Test the Model\n",
    "\n",
    "Try out your model on the test dataset of dog images.  Use the code cell below to calculate and print the test loss and accuracy.  Ensure that your test accuracy is greater than 10%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test(loaders, model, criterion, use_cuda):\n",
    "\n",
    "    # monitor test loss and accuracy\n",
    "    test_loss = 0.\n",
    "    correct = 0.\n",
    "    total = 0.\n",
    "\n",
    "    model.eval()\n",
    "    \n",
    "    with tqdm(loaders['test'], unit=\"batch\") as ttepoch:\n",
    "        for data, target in ttepoch:\n",
    "            # move to GPU\n",
    "            if use_cuda:\n",
    "                data, target = data.cuda(), target.cuda()\n",
    "            # forward pass: compute predicted outputs by passing inputs to the model\n",
    "            output = model(data)\n",
    "            # calculate the loss\n",
    "            loss = criterion(output, target)\n",
    "            # update average test loss \n",
    "            test_loss += loss.item() * data.size(0)\n",
    "    #         test_loss = test_loss + ((1 / (batch_idx + 1)) * (loss.data - test_loss))\n",
    "            # convert output probabilities to predicted class\n",
    "            pred = output.data.max(1, keepdim=True)[1]\n",
    "            # compare predictions to true label\n",
    "            correct += np.sum(np.squeeze(pred.eq(target.data.view_as(pred))).cpu().numpy())\n",
    "            total += data.size(0)\n",
    "            ttepoch.set_postfix(loss=test_loss, accuracy=f\"{correct*100.0/total:.2f}%\")\n",
    "            \n",
    "    print('Test Loss: {:.6f}\\n'.format(test_loss))\n",
    "\n",
    "    print('\\nTest Accuracy: %2d%% (%2d/%2d)' % (\n",
    "        100. * correct / total, correct, total))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 27/27 [00:07<00:00,  3.59batch/s, accuracy=3.59%, loss=3.91e+3]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Loss: 3907.267307\n",
      "\n",
      "\n",
      "Test Accuracy:  3% (30/836)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# call test function    \n",
    "test(loaders_scratch, model_scratch, criterion_scratch, use_cuda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a id='step4'></a>\n",
    "## Step 4: Create a CNN to Classify Dog Breeds (using Transfer Learning)\n",
    "\n",
    "You will now use transfer learning to create a CNN that can identify dog breed from images.  Your CNN must attain at least 60% accuracy on the test set.\n",
    "\n",
    "### (IMPLEMENTATION) Specify Data Loaders for the Dog Dataset\n",
    "\n",
    "Use the code cell below to write three separate [data loaders](http://pytorch.org/docs/master/data.html#torch.utils.data.DataLoader) for the training, validation, and test datasets of dog images (located at `dogImages/train`, `dogImages/valid`, and `dogImages/test`, respectively). \n",
    "\n",
    "If you like, **you are welcome to use the same data loaders from the previous step**, when you created a CNN from scratch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO: Specify data loaders\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (IMPLEMENTATION) Model Architecture\n",
    "\n",
    "Use transfer learning to create a CNN to classify dog breed.  Use the code cell below, and save your initialized model as the variable `model_transfer`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torchvision.models as models\n",
    "import torch.nn as nn\n",
    "\n",
    "## TODO: Specify model architecture \n",
    "\n",
    "\n",
    "if use_cuda:\n",
    "    model_transfer = model_transfer.cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Question 5:__ Outline the steps you took to get to your final CNN architecture and your reasoning at each step.  Describe why you think the architecture is suitable for the current problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Answer:__ \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (IMPLEMENTATION) Specify Loss Function and Optimizer\n",
    "\n",
    "Use the next code cell to specify a [loss function](http://pytorch.org/docs/master/nn.html#loss-functions) and [optimizer](http://pytorch.org/docs/master/optim.html).  Save the chosen loss function as `criterion_transfer`, and the optimizer as `optimizer_transfer` below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion_transfer = None\n",
    "optimizer_transfer = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (IMPLEMENTATION) Train and Validate the Model\n",
    "\n",
    "Train and validate your model in the code cell below.  [Save the final model parameters](http://pytorch.org/docs/master/notes/serialization.html) at filepath `'model_transfer.pt'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train the model\n",
    "model_transfer = # train(n_epochs, loaders_transfer, model_transfer, optimizer_transfer, criterion_transfer, use_cuda, 'model_transfer.pt')\n",
    "\n",
    "# load the model that got the best validation accuracy (uncomment the line below)\n",
    "#model_transfer.load_state_dict(torch.load('model_transfer.pt'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (IMPLEMENTATION) Test the Model\n",
    "\n",
    "Try out your model on the test dataset of dog images. Use the code cell below to calculate and print the test loss and accuracy.  Ensure that your test accuracy is greater than 60%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test(loaders_transfer, model_transfer, criterion_transfer, use_cuda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (IMPLEMENTATION) Predict Dog Breed with the Model\n",
    "\n",
    "Write a function that takes an image path as input and returns the dog breed (`Affenpinscher`, `Afghan hound`, etc) that is predicted by your model.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### TODO: Write a function that takes a path to an image as input\n",
    "### and returns the dog breed that is predicted by the model.\n",
    "\n",
    "# list of class names by index, i.e. a name can be accessed like class_names[0]\n",
    "class_names = [item[4:].replace(\"_\", \" \") for item in data_transfer['train'].classes]\n",
    "\n",
    "def predict_breed_transfer(img_path):\n",
    "    # load the image and return the predicted breed\n",
    "    return None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a id='step5'></a>\n",
    "## Step 5: Write your Algorithm\n",
    "\n",
    "Write an algorithm that accepts a file path to an image and first determines whether the image contains a human, dog, or neither.  Then,\n",
    "- if a __dog__ is detected in the image, return the predicted breed.\n",
    "- if a __human__ is detected in the image, return the resembling dog breed.\n",
    "- if __neither__ is detected in the image, provide output that indicates an error.\n",
    "\n",
    "You are welcome to write your own functions for detecting humans and dogs in images, but feel free to use the `face_detector` and `human_detector` functions developed above.  You are __required__ to use your CNN from Step 4 to predict dog breed.  \n",
    "\n",
    "Some sample output for our algorithm is provided below, but feel free to design your own user experience!\n",
    "\n",
    "![Sample Human Output](images/sample_human_output.png)\n",
    "\n",
    "\n",
    "### (IMPLEMENTATION) Write your Algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### TODO: Write your algorithm.\n",
    "### Feel free to use as many code cells as needed.\n",
    "\n",
    "def run_app(img_path):\n",
    "    ## handle cases for a human face, dog, and neither\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a id='step6'></a>\n",
    "## Step 6: Test Your Algorithm\n",
    "\n",
    "In this section, you will take your new algorithm for a spin!  What kind of dog does the algorithm think that _you_ look like?  If you have a dog, does it predict your dog's breed accurately?  If you have a cat, does it mistakenly think that your cat is a dog?\n",
    "\n",
    "### (IMPLEMENTATION) Test Your Algorithm on Sample Images!\n",
    "\n",
    "Test your algorithm at least six images on your computer.  Feel free to use any images you like.  Use at least two human and two dog images.  \n",
    "\n",
    "__Question 6:__ Is the output better than you expected :) ?  Or worse :( ?  Provide at least three possible points of improvement for your algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Answer:__ (Three possible points for improvement)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO: Execute your algorithm from Step 6 on\n",
    "## at least 6 images on your computer.\n",
    "## Feel free to use as many code cells as needed.\n",
    "\n",
    "## suggested code, below\n",
    "for file in np.hstack((human_files[:3], dog_files[:3])):\n",
    "    run_app(file)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}