Recommendation System on Amazon SageMaker (ObjectToVec).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommendation System on Amazon SageMaker (ObjectToVec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We process the data with Spark, train it with XGBoost and deploy as Inference Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ObjectToVec\n",
    "Object2Vec is a highly customizable multi-purpose algorithm that can learn embeddings of pairs of objects. The embeddings are learned such that it preserves their pairwise similarities in the original space.\n",
    "\n",
    "- Similarity is user-defined: users need to provide the algorithm with pairs of objects that they define as similar (1) or dissimilar (0); alternatively, the users can define similarity in a continuous sense (provide a real-valued similarity score)\n",
    "- The learned embeddings can be used to efficiently compute nearest neighbors of objects, as well as to visualize natural clusters of related objects in the embedding space. In addition, the embeddings can also be used as features of the corresponding objects in downstream supervised tasks such as classification or regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np                                # For matrix operations and numerical processing\n",
    "import pandas as pd                               # For munging tabular data\n",
    "import matplotlib.pyplot as plt                   # For charts and visualizations\n",
    "%matplotlib inline\n",
    "from IPython.display import Image                 # For displaying images in the notebook\n",
    "from IPython.display import display               # For displaying outputs in the notebook\n",
    "from time import gmtime, strftime                 # For labeling SageMaker models, endpoints, etc.\n",
    "import sys                                        # For writing outputs to notebook\n",
    "import math                                       # For ceiling function\n",
    "import json                                       # For parsing hosting outputs\n",
    "import os                                         # For manipulating filepath names\n",
    "import jsonlines\n",
    "import boto3 \n",
    "import csv\n",
    "import io\n",
    "import re\n",
    "import s3fs\n",
    "import copy\n",
    "import random\n",
    "import time\n",
    "\n",
    "\n",
    "from urllib.request import urlretrieve\n",
    "import zipfile\n",
    "from dateutil.parser import parse\n",
    "from random import shuffle\n",
    "import random\n",
    "import datetime\n",
    "\n",
    "import sagemaker                                 \n",
    "from sagemaker.predictor import csv_serializer \n",
    "from sagemaker.predictor import json_deserializer\n",
    "import sagemaker.amazon.common as smac\n",
    "from sagemaker import get_execution_role"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Preparation (Specifying Sagemaker roles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sagemaker session : <sagemaker.session.Session object at 0x000001A2F33656C8>\n",
      "S3 bucket : sagemaker-us-west-2-570447867175\n",
      "Prefix : recomm-object2vec\n",
      "Region selected : us-west-2\n",
      "IAM role : arn:aws:iam::570447867175:role/SageMakerNotebookRole\n"
     ]
    }
   ],
   "source": [
    "sess   = sagemaker.Session()\n",
    "bucket = sess.default_bucket()\n",
    "s3_prefix = 'recomm-object2vec'     \n",
    "region = boto3.Session().region_name\n",
    "role = 'arn:aws:iam::570447867175:role/SageMakerNotebookRole' # pass your IAM role name\n",
    "\n",
    "print('Sagemaker session :', sess)\n",
    "print('S3 bucket :', bucket)\n",
    "print('Prefix :', s3_prefix)\n",
    "print('Region selected :', region)\n",
    "print('IAM role :', role)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Load Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset including number of users/moview are in official [Movielens website](https://grouplens.org/datasets/movielens/100k/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "## some utility functions\n",
    "\n",
    "def load_csv_data(filename, delimiter, verbose=True):\n",
    "    \"\"\"\n",
    "    input: a file readable as csv and separated by a delimiter\n",
    "    and has format users - movies - ratings - etc\n",
    "    output: a list, where each row of the list is of the form\n",
    "    {'in0':userID, 'in1':movieID, 'label':rating}\n",
    "    \"\"\"\n",
    "    to_data_list = list()\n",
    "    users = list()\n",
    "    movies = list()\n",
    "    ratings = list()\n",
    "    unique_users = set()\n",
    "    unique_movies = set()\n",
    "    with open(filename, 'r') as csvfile:\n",
    "        reader = csv.reader(csvfile, delimiter=delimiter)\n",
    "        for count, row in enumerate(reader):\n",
    "            #if count!=0:\n",
    "            to_data_list.append({'in0':[int(row[0])], 'in1':[int(row[1])], 'label':float(row[2])})\n",
    "            users.append(row[0])\n",
    "            movies.append(row[1])\n",
    "            ratings.append(float(row[2]))\n",
    "            unique_users.add(row[0])\n",
    "            unique_movies.add(row[1])\n",
    "    if verbose:\n",
    "        print(\"In file {}, there are {} ratings\".format(filename, len(ratings)))\n",
    "        print(\"The ratings have mean: {}, median: {}, and variance: {}\".format(\n",
    "                                            round(np.mean(ratings), 2), \n",
    "                                            round(np.median(ratings), 2), \n",
    "                                            round(np.var(ratings), 2)))\n",
    "        print(\"There are {} unique users and {} unique movies\".format(len(unique_users), len(unique_movies)))\n",
    "    return to_data_list\n",
    "\n",
    "\n",
    "def csv_to_augmented_data_dict(filename, delimiter):\n",
    "    \"\"\"\n",
    "    Input: a file that must be readable as csv and separated by delimiter (to make columns)\n",
    "    has format users - movies - ratings - etc\n",
    "    Output:\n",
    "      Users dictionary: keys as user ID's; each key corresponds to a list of movie ratings by that user\n",
    "      Movies dictionary: keys as movie ID's; each key corresponds a list of ratings of that movie by different users\n",
    "    \"\"\"\n",
    "    to_users_dict = dict() \n",
    "    to_movies_dict = dict()\n",
    "    with open(filename, 'r') as csvfile:\n",
    "        reader = csv.reader(csvfile, delimiter=delimiter)\n",
    "        for count, row in enumerate(reader):\n",
    "            #if count!=0:\n",
    "            if row[0] not in to_users_dict:\n",
    "                to_users_dict[row[0]] = [(row[1], row[2])]\n",
    "            else:\n",
    "                to_users_dict[row[0]].append((row[1], row[2]))\n",
    "            if row[1] not in to_movies_dict:\n",
    "                to_movies_dict[row[1]] = list(row[0])\n",
    "            else:\n",
    "                to_movies_dict[row[1]].append(row[0])\n",
    "    return to_users_dict, to_movies_dict\n",
    "\n",
    "\n",
    "def user_dict_to_data_list(user_dict):\n",
    "    # turn user_dict format to data list format (acceptable to the algorithm)\n",
    "    data_list = list()\n",
    "    for user, movie_rating_list in user_dict.items():\n",
    "        for movie, rating in movie_rating_list:\n",
    "            data_list.append({'in0':[int(user)], 'in1':[int(movie)], 'label':float(rating)})\n",
    "    return data_list\n",
    "\n",
    "def divide_user_dicts(user_dict, sp_ratio_dict):\n",
    "    \"\"\"\n",
    "    Input: A user dictionary, a ration dictionary\n",
    "         - format of sp_ratio_dict = {'train':0.8, \"test\":0.2}\n",
    "    Output: \n",
    "        A dictionary of dictionaries, with key corresponding to key provided by sp_ratio_dict\n",
    "        and each key corresponds to a subdivded user dictionary\n",
    "    \"\"\"\n",
    "    ratios = [val for _, val in sp_ratio_dict.items()]\n",
    "    assert np.sum(ratios) == 1, \"the sampling ratios must sum to 1!\"\n",
    "    divided_dict = {}\n",
    "    for user, movie_rating_list in user_dict.items():\n",
    "        sub_movies_ptr = 0\n",
    "        sub_movies_list = []\n",
    "        #movie_list, _ = zip(*movie_rating_list)\n",
    "        #print(movie_list)\n",
    "        for i, ratio in enumerate(ratios):\n",
    "            if i < len(ratios)-1:\n",
    "                sub_movies_ptr_end = sub_movies_ptr + int(len(movie_rating_list)*ratio)\n",
    "                sub_movies_list.append(movie_rating_list[sub_movies_ptr:sub_movies_ptr_end])\n",
    "                sub_movies_ptr = sub_movies_ptr_end\n",
    "            else:\n",
    "                sub_movies_list.append(movie_rating_list[sub_movies_ptr:])\n",
    "        for subset_name in sp_ratio_dict.keys():\n",
    "            if subset_name not in divided_dict:\n",
    "                divided_dict[subset_name] = {user: sub_movies_list.pop(0)}\n",
    "            else:\n",
    "                #access sub-dictionary\n",
    "                divided_dict[subset_name][user] = sub_movies_list.pop(0)\n",
    "    \n",
    "    return divided_dict\n",
    "\n",
    "def write_csv_to_jsonl(jsonl_fname, csv_fname, csv_delimiter):\n",
    "    \"\"\"\n",
    "    Input: a file readable as csv and separated by delimiter (to make columns)\n",
    "        - has format users - movies - ratings - etc\n",
    "    Output: a jsonline file converted from the csv file\n",
    "    \"\"\"\n",
    "    with jsonlines.open(jsonl_fname, mode='w') as writer:\n",
    "        with open(csv_fname, 'r') as csvfile:\n",
    "            reader = csv.reader(csvfile, delimiter=csv_delimiter)\n",
    "            for count, row in enumerate(reader):\n",
    "                #print(row)\n",
    "                #if count!=0:\n",
    "                writer.write({'in0':[int(row[0])], 'in1':[int(row[1])], 'label':float(row[2])})\n",
    "        print('Created {} jsonline file'.format(jsonl_fname))\n",
    "                    \n",
    "    \n",
    "def write_data_list_to_jsonl(data_list, to_fname):\n",
    "    \"\"\"\n",
    "    Input: a data list, where each row of the list is a Python dictionary taking form\n",
    "    {'in0':userID, 'in1':movieID, 'label':rating}\n",
    "    Output: save the list as a jsonline file\n",
    "    \"\"\"\n",
    "    with jsonlines.open(to_fname, mode='w') as writer:\n",
    "        for row in data_list:\n",
    "            #print(row)\n",
    "            writer.write({'in0':row['in0'], 'in1':row['in1'], 'label':row['label']})\n",
    "    print(\"Created {} jsonline file\".format(to_fname))\n",
    "\n",
    "def data_list_to_inference_format(data_list, binarize=True, label_thres=3):\n",
    "    \"\"\"\n",
    "    Input: a data list\n",
    "    Output: test data and label, acceptable by SageMaker for inference\n",
    "    \"\"\"\n",
    "    data_ = [({\"in0\":row['in0'], 'in1':row['in1']}, row['label']) for row in data_list]\n",
    "    data, label = zip(*data_)\n",
    "    infer_data = {\"instances\":data}\n",
    "    if binarize:\n",
    "        label = get_binarized_label(list(label), label_thres)\n",
    "    return infer_data, label\n",
    "\n",
    "\n",
    "def get_binarized_label(data_list, thres):\n",
    "    \"\"\"\n",
    "    Input: data list\n",
    "    Output: a binarized data list for recommendation task\n",
    "    \"\"\"\n",
    "    for i, row in enumerate(data_list):\n",
    "        if type(row) is dict:\n",
    "            #if i < 10:\n",
    "                #print(row['label'])\n",
    "            if row['label'] > thres:\n",
    "                #print(row)\n",
    "                data_list[i]['label'] = 1\n",
    "            else:\n",
    "                data_list[i]['label'] = 0\n",
    "        else:\n",
    "            if row > thres:\n",
    "                data_list[i] = 1\n",
    "            else:\n",
    "                data_list[i] = 0\n",
    "    return data_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_data : C:\\Users\\bokhy\\Desktop\\Python\\github\\amazon-sagemaker\\data\\ml-100k\\ua.base\n",
      "valid_data : C:\\Users\\bokhy\\Desktop\\Python\\github\\amazon-sagemaker\\data\\ml-100k\\ua.test\n",
      "test_data : C:\\Users\\bokhy\\Desktop\\Python\\github\\amazon-sagemaker\\data\\ml-100k\\ub.test\n"
     ]
    }
   ],
   "source": [
    "## Load data and shuffle\n",
    "path = os.path.abspath('./data/ml-100k')\n",
    "train_path = os.path.join(path, 'ua.base')\n",
    "valid_path = os.path.join(path, 'ua.test')\n",
    "test_path = os.path.join(path, 'ub.test')\n",
    "\n",
    "print('train_data :', train_path)\n",
    "print('valid_data :', valid_path)\n",
    "print('test_data :', test_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In file C:\\Users\\bokhy\\Desktop\\Python\\github\\amazon-sagemaker\\data\\ml-100k\\ua.base, there are 90570 ratings\n",
      "The ratings have mean: 3.52, median: 4.0, and variance: 1.27\n",
      "There are 943 unique users and 1680 unique movies\n",
      "In file C:\\Users\\bokhy\\Desktop\\Python\\github\\amazon-sagemaker\\data\\ml-100k\\ua.test, there are 9430 ratings\n",
      "The ratings have mean: 3.59, median: 4.0, and variance: 1.25\n",
      "There are 943 unique users and 1129 unique movies\n"
     ]
    }
   ],
   "source": [
    "train_data_list = load_csv_data(train_path, '\\t')\n",
    "random.shuffle(train_data_list)\n",
    "validation_data_list = load_csv_data(valid_path, '\\t')\n",
    "random.shuffle(validation_data_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_users_dict, to_movies_dict = csv_to_augmented_data_dict(train_path, '\\t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Data Pre-processing and EDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The min, max, and median 'movies per user' is 10, 727, and 55.0\n",
      "The min, max, and median 'users per movie' is 1, 495, and 25.0\n",
      "In the training set\n",
      "There are 213 users with no more than 20 movies\n",
      "There are 12 movies with no more than 2 user\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Users per movie')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAFCdJREFUeJzt3X2Q3dV93/H3x8hgjDHiQRAsYWRq1bXjNphRMC6160KSGuwaJmMSqBsUV62ahrZ2nZlYuG1Sd5IUOo2xyXTsKsap8CMEm0LBCaGA06QzYAsbMFhQiSejCCMRnoyfEuDbP+5ZfL1aae9qV7t3D+/XzJ37+51z7u/33b2rz549996fUlVIkvr1ooUuQJK0bxn0ktQ5g16SOmfQS1LnDHpJ6pxBL0mdM+jVhSRvTnLPQtchjSODXvMiyQNJ/irJEZPab0tSSVbO5vhV9WdV9ZrZHEPqlUGv+XQ/cM7ETpK/DRy4cOXMnyRLXkjn1Xgx6DWfPgWcO7S/Brh0eECSQ5JcmmRnkgeT/PskL0pyQJInkrx+aOyyJN9PcmSStybZNtT3iiRfaMe5P8m/Geo7McmmJE8leSTJh6cqduKYST6Y5NH2V8m7h/oPSPJfk3yrHefjSQ6c9NgPJPk28AdTHP8/Jvn00P7K9tfNkrb/y0nuS/Kd9jUMn/ufJtmc5PEk1yU5dqivkpyXZAuwZY/PiF4QDHrNp5uBlyd5bZL9gF8EPj1pzO8BhwDHAX+fwS+G91TVD4EvMvQXAfALwJ9W1Y7hAyR5EfC/gNuB5cCpwPuS/MM25KPAR6vq5cDfAC7fQ80/ARzRjrMG2JBkYonoQuBvAscDr25jfmPSYw8DjgXW7eEcu0hyEHAxcFpVHQz8XeC21ncm8EHg54FlwJ8Bn5t0iDOBNwKvm8l51SeDXvNtYlb/s8DdwF9MdAyF//lV9Z2qegD4XeCX2pDP8uNB/49b22Q/DSyrqv9UVX9VVfcBvw+c3fr/Gnh1kiOq6umqunmamv9DVf2wqv4UuBb4hSQB/jnwb6vqsar6DvA7Q+cAeA74zfbY709zjqk8B7w+yYFV9XBV3dXa/wXwn6tqc1U90857/PCsvvU/tpfnVWcMes23TzEI6F9m0rINg5nz/sCDQ20PMpgpA9wIHJjkjS3UjgeunOIcxwKvaEs9TyR5gsEM+KjWv5bBTPzuJF9N8o491Pt4VX13Uj2vYDCTfilw69A5/ri1T9hZVT/Yw7F3q53zF4FfAR5Ocm2SvzX09X106LyPAeFH3yeAh/bmvOqTL9RoXlXVg0nuB05nELjDHmUw2z4W+GZreyVt1l9VzyW5nMGs/hHgmjaTnuwh4P6qWrWbGrYA57Qlnp8Hrkhy+KRAn3BokoOG+l4J3Nlq/T7wk1X1F1M8DmC6S8N+l8Eviwk/ManO64Dr2rr/bzH4q+TN7ev77ar6zB6O7WVp9Txn9FoIa4FTJgdrVT3LYL38t5Mc3Gbt7+fH1/E/y2Cm+26mXrYB+ArwVHsh9MAk+yV5fZKfBkjyT5Isq6rngCfaY57dQ70fSrJ/kjcD7wD+sD3294GLkhzZjrt86HWAUdwGvCXJK5McApw/0ZHkqCTvbGv1PwSeHqrx48D5SX6yjT0kyVkzOK9eYAx6zbuqureqNu2m+18zmOneB/w5gzD/5NBjb2n9rwD+aDfHfxb4RwyWdu5nMPv+BIMXeQHeBtyV5GkGL8yevYcllm8DjwPbgc8Av1JVd7e+DwBbgZuTPAX8b2Dk9/JX1fXAZcAdwK3ANUPdLwJ+rZ33MQYvTP9qe9yVDF4I/nw7753AaaOeVy888T8ekaaW5K3Ap6tqxULXIs2GM3pJ6pxBL0mdc+lGkjrnjF6SOjcW76M/4ogjauXKlQtdhiQtKrfeeuujVbVsunFjEfQrV65k06bdvdtOkjSVJA9OP8qlG0nqnkEvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6txYfDJ2Nlauv3bBzv3ABW9fsHNL0qic0UtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6txIQZ/kgSTfSHJbkk2t7bAk1yfZ0u4Pbe1JcnGSrUnuSHLCvvwCJEl7NpMZ/T+oquOranXbXw/cUFWrgBvaPsBpwKp2Wwd8bK6KlSTN3GyWbs4ANrbtjcCZQ+2X1sDNwNIkR8/iPJKkWRg16Av4kyS3JlnX2o6qqocB2v2RrX058NDQY7e1th+TZF2STUk27dy5c++qlyRNa8mI406uqu1JjgSuT3L3HsZmirbapaFqA7ABYPXq1bv0S5Lmxkgz+qra3u53AFcCJwKPTCzJtPsdbfg24Jihh68Ats9VwZKkmZk26JMclOTgiW3g54A7gauBNW3YGuCqtn01cG57981JwJMTSzySpPk3ytLNUcCVSSbGf7aq/jjJV4HLk6wFvgWc1cZ/CTgd2Ap8D3jPnFctSRrZtEFfVfcBPzVF+18Cp07RXsB5c1KdJGnW/GSsJHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6pxBL0mdM+glqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOjdy0CfZL8nXk1zT9l+V5JYkW5JclmT/1n5A29/a+lfum9IlSaOYyYz+vcDmof0LgYuqahXwOLC2ta8FHq+qVwMXtXGSpAUyUtAnWQG8HfhE2w9wCnBFG7IROLNtn9H2af2ntvGSpAUw6oz+I8CvA8+1/cOBJ6rqmba/DVjetpcDDwG0/ifb+B+TZF2STUk27dy5cy/LlyRNZ9qgT/IOYEdV3TrcPMXQGqHvRw1VG6pqdVWtXrZs2UjFSpJmbskIY04G3pnkdOAlwMsZzPCXJlnSZu0rgO1t/DbgGGBbkiXAIcBjc165JGkk087oq+r8qlpRVSuBs4Ebq+rdwE3Au9qwNcBVbfvqtk/rv7GqdpnRS5Lmx2zeR/8B4P1JtjJYg7+ktV8CHN7a3w+sn12JkqTZGGXp5nlV9WXgy237PuDEKcb8ADhrDmqTJM0BPxkrSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6pxBL0mdM+glqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9JHXOoJekzk0b9ElekuQrSW5PcleSD7X2VyW5JcmWJJcl2b+1H9D2t7b+lfv2S5Ak7ckoM/ofAqdU1U8BxwNvS3IScCFwUVWtAh4H1rbxa4HHq+rVwEVtnCRpgUwb9DXwdNt9cbsVcApwRWvfCJzZts9o+7T+U5NkziqWJM3ISGv0SfZLchuwA7geuBd4oqqeaUO2Acvb9nLgIYDW/yRw+BTHXJdkU5JNO3funN1XIUnarZGCvqqerarjgRXAicBrpxrW7qeavdcuDVUbqmp1Va1etmzZqPVKkmZoRu+6qaongC8DJwFLkyxpXSuA7W17G3AMQOs/BHhsLoqVJM3cKO+6WZZkads+EPgZYDNwE/CuNmwNcFXbvrrt0/pvrKpdZvSSpPmxZPohHA1sTLIfg18Ml1fVNUm+CXw+yW8BXwcuaeMvAT6VZCuDmfzZ+6BuSdKIpg36qroDeMMU7fcxWK+f3P4D4Kw5qU6SNGt+MlaSOmfQS1LnDHpJ6pxBL0mdM+glqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6pxBL0mdmzbokxyT5KYkm5PcleS9rf2wJNcn2dLuD23tSXJxkq1J7khywr7+IiRJuzfKjP4Z4Neq6rXAScB5SV4HrAduqKpVwA1tH+A0YFW7rQM+NudVS5JGNm3QV9XDVfW1tv0dYDOwHDgD2NiGbQTObNtnAJfWwM3A0iRHz3nlkqSRzGiNPslK4A3ALcBRVfUwDH4ZAEe2YcuBh4Yetq21TT7WuiSbkmzauXPnzCuXJI1k5KBP8jLgC8D7quqpPQ2doq12aajaUFWrq2r1smXLRi1DkjRDIwV9khczCPnPVNUXW/MjE0sy7X5Ha98GHDP08BXA9rkpV5I0U6O86ybAJcDmqvrwUNfVwJq2vQa4aqj93Pbum5OAJyeWeCRJ82/JCGNOBn4J+EaS21rbB4ELgMuTrAW+BZzV+r4EnA5sBb4HvGdOK5Ykzci0QV9Vf87U6+4Ap04xvoDzZlmXJGmOjDKj126sXH/tgpz3gQveviDnlbQ4eQkESeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6pxBL0mdM+glqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalz0wZ9kk8m2ZHkzqG2w5Jcn2RLuz+0tSfJxUm2JrkjyQn7snhJ0vRGmdH/D+Btk9rWAzdU1SrghrYPcBqwqt3WAR+bmzIlSXtr2qCvqv8DPDap+QxgY9veCJw51H5pDdwMLE1y9FwVK0maub1doz+qqh4GaPdHtvblwEND47a1tl0kWZdkU5JNO3fu3MsyJEnTWTLHx8sUbTXVwKraAGwAWL169ZRjNLWV669dsHM/cMHbF+zckvbO3s7oH5lYkmn3O1r7NuCYoXErgO17X54kabb2NuivBta07TXAVUPt57Z335wEPDmxxCNJWhjTLt0k+RzwVuCIJNuA3wQuAC5Pshb4FnBWG/4l4HRgK/A94D37oGZJ0gxMG/RVdc5uuk6dYmwB5822KEnS3PGTsZLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI6N9eXKVbnFuoSyV4eWdp7zuglqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOea0bLQpeY0fae87oJalzBr0kdc6lG2kPFmrJCFw20txxRi9JnTPoJalz+yTok7wtyT1JtiZZvy/OIUkazZyv0SfZD/hvwM8C24CvJrm6qr451+eSeuZbSjVX9sWLsScCW6vqPoAknwfOAAx6aRFYyBegX4jm4xfrvgj65cBDQ/vbgDdOHpRkHbCu7T6d5J4Rj38E8OisKpwfi6VOWDy1LpY6YfHUuljqhMVT64zqzIWzOtexowzaF0GfKdpql4aqDcCGGR882VRVq/emsPm0WOqExVPrYqkTFk+ti6VOWDy1jmOd++LF2G3AMUP7K4Dt++A8kqQR7Iug/yqwKsmrkuwPnA1cvQ/OI0kawZwv3VTVM0n+FXAdsB/wyaq6aw5PMePlngWyWOqExVPrYqkTFk+ti6VOWDy1jl2dqdpl+VyS1BE/GStJnTPoJalziyrox+nSCkk+mWRHkjuH2g5Lcn2SLe3+0NaeJBe3uu9IcsI81nlMkpuSbE5yV5L3jnGtL0nylSS3t1o/1NpfleSWVutl7UV+khzQ9re2/pXzVWs7/35Jvp7kmjGv84Ek30hyW5JNrW0cn/+lSa5Icnf7eX3TmNb5mva9nLg9leR941jr86pqUdwYvLB7L3AcsD9wO/C6BaznLcAJwJ1Dbf8FWN+21wMXtu3TgT9i8BmDk4Bb5rHOo4ET2vbBwP8DXjemtQZ4Wdt+MXBLq+Fy4OzW/nHgX7btXwU+3rbPBi6b55+B9wOfBa5p++Na5wPAEZPaxvH53wj8s7a9P7B0HOucVPN+wLcZfHBpbGud92/MLL6hbwKuG9o/Hzh/gWtaOSno7wGObttHA/e07f8OnDPVuAWo+SoG1yEa61qBlwJfY/Cp6keBJZN/Dhi8s+tNbXtJG5d5qm8FcANwCnBN+0c8dnW2c04V9GP1/AMvB+6f/H0ZtzqnqPvngP877rUupqWbqS6tsHyBatmdo6rqYYB2f2RrH4va25LBGxjMlMey1rYcchuwA7iewV9xT1TVM1PU83ytrf9J4PB5KvUjwK8Dz7X9w8e0Thh8Mv1PktyawaVHYPye/+OAncAftOWwTyQ5aAzrnOxs4HNte2xrXUxBP9KlFcbUgtee5GXAF4D3VdVTexo6Rdu81VpVz1bV8QxmzCcCr91DPQtSa5J3ADuq6tbh5j3UstDP/8lVdQJwGnBekrfsYexC1bqEwVLox6rqDcB3GSx/7M5Cf09pr8G8E/jD6YZO0TavtS6moF8Ml1Z4JMnRAO1+R2tf0NqTvJhByH+mqr44zrVOqKongC8zWNNcmmTiw33D9Txfa+s/BHhsHso7GXhnkgeAzzNYvvnIGNYJQFVtb/c7gCsZ/AIdt+d/G7Ctqm5p+1cwCP5xq3PYacDXquqRtj+2tS6moF8Ml1a4GljTttcwWA+faD+3vfp+EvDkxJ94+1qSAJcAm6vqw2Ne67IkS9v2gcDPAJuBm4B37abWia/hXcCN1RZB96WqOr+qVlTVSgY/hzdW1bvHrU6AJAclOXhim8Ga8p2M2fNfVd8GHkrymtZ0KoNLm49VnZOcw4+WbSZqGs9a5/vFi1m+8HE6g3eN3Av8uwWu5XPAw8BfM/iNvZbBuusNwJZ2f1gbGwb/Gcu9wDeA1fNY599j8GfiHcBt7Xb6mNb6d4Cvt1rvBH6jtR8HfAXYyuDP5ANa+0va/tbWf9wC/By8lR+962bs6mw13d5ud038uxnT5/94YFN7/v8ncOg41tnO/1LgL4FDhtrGstaq8hIIktS7xbR0I0naCwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6tz/B/hh5KbU6Wq7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEICAYAAABWJCMKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAE5BJREFUeJzt3X+w5XV93/HnS1YQNbL8uFDY3bJYtlbjNMqsiLFpqVgDaLJMR1IIho2z053O0EQDM3GxaVA67UCbiNqktBvB4PgLgqZs0UbJgqlpA7ooIriS3QCy10X2MizgbwXe/eN8rh6Wuz+85+497P08HzN3zvf7+X6+38/nczic1/l+vud7NlWFJKk/zxl3ByRJ42EASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQFqAkv5TknnH3Q89u8T4AjUuSAlZU1dahsncBJ1bVW8bWMakTngFowUuyqKd2pX1lAOhZK8lRSW5M8miSR5J8Pslz2rbjknwiyVSS+5L89tB+70pyfZIPJ3kc+M0kJyfZlOTxJA8lec9u2jw1yWSSdyZ5OMn9Sc4b2n5Ikj9I8kA7zn9Pcugu+74jybeAD85w/N9M8n+TXNHGdW+SX2zl25LsSLJ6qP5hST7UxvmNJL+X5DmtH48meflQ3Ykk309y9HRfhrbt9vlSvwwAPZtdBEwCE8AxwDuBaiHwv4CvAEuA04C3J/nloX1XAdcDi4GPAO8D3ldVLwL+AXDdHtr9e8BR7dirgfVJXtK2XQ78Q+AVwImtzu/vsu8RwPHA2t0c/9XAncCRwEeBjwOvasd7C/BHSV7Y6v5X4DDgxcA/A84H3lpVPwQ+CZw7dNxfA/6qqnYMN7aPz5c6ZADo2ezHwLHA8VX146r6fA0uWr0KmKiqS6vqR1V1L/AnwDlD+/5NVf3Pqnqqqr7fjnVikqOq6jtVdete2v73VfXDqvor4FPAryUJ8K+B36mqR6rq28B/2qXdp4BL2r7f382x76uqD1bVk8C1wDLg0rbPZ4Eftb4eBPwr4OKq+nZV3Q/8IfAb7Tgf5ekB8OutbFf78nypQ85RapyeBJ67S9lzGbxZA/wX4F3AZwfvvayvqssYfLo+LsmjQ/sdBHx+aH3bLsddA1wKfD3JfcC7q+rG3fRrZ1V9d2j9G8BxDM5Eng/c3voDkNb2tKmq+sFujjvtoaHl7wNU1a5lL2RwFnJwa3+4L0va8s3AoUleDXyLwVnJn8/Q3r48X+qQAaBxegBYDmweKjsB+FuA9gn7IuCiJD8P3JLkiwze3O+rqhV7OPbTvt5WVVuAc9t0yL8Erk9y5C5v9NMOT/KCoW1/H7gLeJjBm/PPV9U396XdET3MIAyPB7421JdvAlTVU0muY3AW8BBwY3vOdrUvz5c65BSQxula4PeSLG0XNl8P/AqDuXuSvCnJiW3q5XEGZwxPAl8AHm8XWw9NclCSlyd51e4aSvKWJBNV9RQw/Un4yT307d1JDk7yS8CbgD9r+/4JcEWSo9txl+yvufQ2RXQd8B+T/FyS44ELgQ8PVfsog2mi85h5+gdm8XypDwaAxulS4P8Bfw3sBP4zcF5V3dW2rwD+EvgO8DfAf6uqz7U3xl9hMOVxH4NPyh9gcLF0d04H7k7yHQYXhM/Zw1TNt1p/tjO4gPxvqurrbds7gK3Are0bRn8JvGTGo8yN3wK+C9zL4Hn6KHD19Maquq1tPw743zMdYJbPlzrgjWDSkCSnAh+uqqXj7ou0v3kGIEmdMgAkqVNOAUlSpzwDkKROPavvAzjqqKNq+fLl4+6GJB1Qbr/99oeramJv9Z7VAbB8+XI2bdo07m5I0gElyTf2XsspIEnqlgEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6tSz+k7gUS1f96mxtHv/ZW8cS7uS9LPwDECSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1Km9BkCSq5PsSHLXUNkRSW5KsqU9Ht7Kk+T9SbYmuTPJSUP7rG71tyRZvX+GI0naV/tyBvCnwOm7lK0DNlbVCmBjWwc4A1jR/tYCV8IgMIBLgFcDJwOXTIeGJGk89hoAVfV/gEd2KV4FXNOWrwHOGir/UA3cCixOcizwy8BNVfVIVe0EbuKZoSJJmkezvQZwTFU9CNAej27lS4BtQ/UmW9nuyiVJYzLXF4EzQ1ntofyZB0jWJtmUZNPU1NScdk6S9FOzDYCH2tQO7XFHK58Elg3VWwps30P5M1TV+qpaWVUrJyYmZtk9SdLezDYANgDT3+RZDdwwVH5++zbQKcBjbYroM8AbkhzeLv6+oZVJksZk0d4qJPkYcCpwVJJJBt/muQy4Lska4AHg7Fb908CZwFbge8BbAarqkST/Afhiq3dpVe16YVmSNI/2GgBVde5uNp02Q90CLtjNca4Grv6ZeidJ2m+8E1iSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMjBUCS30lyd5K7knwsyfOSnJDktiRbklyb5OBW95C2vrVtXz4XA5Akzc6sAyDJEuC3gZVV9XLgIOAc4HLgiqpaAewE1rRd1gA7q+pE4IpWT5I0JqNOAS0CDk2yCHg+8CDwOuD6tv0a4Ky2vKqt07afliQjti9JmqVZB0BVfRP4A+ABBm/8jwG3A49W1ROt2iSwpC0vAba1fZ9o9Y/c9bhJ1ibZlGTT1NTUbLsnSdqLUaaADmfwqf4E4DjgBcAZM1St6V32sO2nBVXrq2plVa2cmJiYbfckSXsxyhTQ64H7qmqqqn4MfBL4RWBxmxICWApsb8uTwDKAtv0w4JER2pckjWCUAHgAOCXJ89tc/mnA14BbgDe3OquBG9ryhrZO235zVT3jDECSND9GuQZwG4OLuV8CvtqOtR54B3Bhkq0M5vivartcBRzZyi8E1o3Qb0nSiBbtvcruVdUlwCW7FN8LnDxD3R8AZ4/SniRp7ngnsCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjo1UgAkWZzk+iRfT7I5yWuSHJHkpiRb2uPhrW6SvD/J1iR3JjlpboYgSZqNUc8A3gf8RVX9I+AXgM3AOmBjVa0ANrZ1gDOAFe1vLXDliG1LkkYw6wBI8iLgnwJXAVTVj6rqUWAVcE2rdg1wVlteBXyoBm4FFic5dtY9lySNZJQzgBcDU8AHk3w5yQeSvAA4pqoeBGiPR7f6S4BtQ/tPtrKnSbI2yaYkm6ampkboniRpT0YJgEXAScCVVfVK4Lv8dLpnJpmhrJ5RULW+qlZW1cqJiYkRuidJ2pNRAmASmKyq29r69QwC4aHpqZ32uGOo/rKh/ZcC20doX5I0glkHQFV9C9iW5CWt6DTga8AGYHUrWw3c0JY3AOe3bwOdAjw2PVUkSZp/i0bc/7eAjyQ5GLgXeCuDULkuyRrgAeDsVvfTwJnAVuB7ra4kaUxGCoCqugNYOcOm02aoW8AFo7QnSZo73gksSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkTo0cAEkOSvLlJDe29ROS3JZkS5Jrkxzcyg9p61vb9uWjti1Jmr25OAN4G7B5aP1y4IqqWgHsBNa08jXAzqo6Ebii1ZMkjclIAZBkKfBG4ANtPcDrgOtblWuAs9ryqrZO235aqy9JGoNRzwDeC/wu8FRbPxJ4tKqeaOuTwJK2vATYBtC2P9bqP02StUk2Jdk0NTU1YvckSbsz6wBI8iZgR1XdPlw8Q9Xah20/LahaX1Urq2rlxMTEbLsnSdqLRSPs+1rgV5OcCTwPeBGDM4LFSRa1T/lLge2t/iSwDJhMsgg4DHhkhPYlSSOY9RlAVV1cVUurajlwDnBzVZ0H3AK8uVVbDdzQlje0ddr2m6vqGWcAkqT5sT/uA3gHcGGSrQzm+K9q5VcBR7byC4F1+6FtSdI+GmUK6Ceq6nPA59ryvcDJM9T5AXD2XLQnSRqddwJLUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdWrRuDuwEC1f96mxtHv/ZW8cS7uSDkyeAUhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROzToAkixLckuSzUnuTvK2Vn5EkpuSbGmPh7fyJHl/kq1J7kxy0lwNQpL0sxvlDOAJ4KKqeilwCnBBkpcB64CNVbUC2NjWAc4AVrS/tcCVI7QtSRrRrAOgqh6sqi+15W8Dm4ElwCrgmlbtGuCstrwK+FAN3AosTnLsrHsuSRrJnFwDSLIceCVwG3BMVT0Ig5AAjm7VlgDbhnabbGWSpDEYOQCSvBD4BPD2qnp8T1VnKKsZjrc2yaYkm6ampkbtniRpN0YKgCTPZfDm/5Gq+mQrfmh6aqc97mjlk8Cyod2XAtt3PWZVra+qlVW1cmJiYpTuSZL2YJRvAQW4CthcVe8Z2rQBWN2WVwM3DJWf374NdArw2PRUkSRp/o3ya6CvBX4D+GqSO1rZO4HLgOuSrAEeAM5u2z4NnAlsBb4HvHWEtiVJI5p1AFTVXzPzvD7AaTPUL+CC2bYnSZpb3gksSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnRrkRTM8yy9d9amxt33/ZG8fWtqTZ8QxAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI65Z3AmhPjugvZO5Cl2fMMQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKr4HqgOY/giPNnmcAktQpA0CSOuUUkDRL3v2sA51nAJLUqXkPgCSnJ7knydYk6+a7fUnSwLxOASU5CPhj4F8Ak8AXk2yoqq/NZz+kA9k4v/k0Lk577R/zfQ3gZGBrVd0LkOTjwCrAAJC0W15v2T/mOwCWANuG1ieBVw9XSLIWWNtWv5Pknlm2dRTw8Cz3PZD1OO4exwx9jntex5zL56ulvfpZx338vlSa7wDIDGX1tJWq9cD6kRtKNlXVylGPc6Dpcdw9jhn6HHePY4b9N+75vgg8CSwbWl8KbJ/nPkiSmP8A+CKwIskJSQ4GzgE2zHMfJEnM8xRQVT2R5N8CnwEOAq6uqrv3U3MjTyMdoHocd49jhj7H3eOYYT+NO1W191qSpAXHO4ElqVMGgCR1asEFwEL+qYkkVyfZkeSuobIjktyUZEt7PLyVJ8n72/NwZ5KTxtfz2UuyLMktSTYnuTvJ21r5Qh/385J8IclX2rjf3cpPSHJbG/e17csUJDmkrW9t25ePs/+jSHJQki8nubGt9zDm+5N8NckdSTa1sv3+Gl9QATD0UxNnAC8Dzk3ysvH2ak79KXD6LmXrgI1VtQLY2NZh8BysaH9rgSvnqY9z7Qngoqp6KXAKcEH7b7rQx/1D4HVV9QvAK4DTk5wCXA5c0ca9E1jT6q8BdlbVicAVrd6B6m3A5qH1HsYM8M+r6hVD3/ff/6/xqlowf8BrgM8MrV8MXDzufs3xGJcDdw2t3wMc25aPBe5py/8DOHemegfyH3ADg9+S6mbcwPOBLzG4a/5hYFEr/8nrncE3617Tlhe1ehl332cx1qXtze51wI0Mbh5d0GNu/b8fOGqXsv3+Gl9QZwDM/FMTS8bUl/lyTFU9CNAej27lC+65aKf4rwRuo4Nxt6mQO4AdwE3A3wGPVtUTrcrw2H4y7rb9MeDI+e3xnHgv8LvAU239SBb+mGHwiwifTXJ7+zkcmIfX+EL7B2H2+lMTHVlQz0WSFwKfAN5eVY8nMw1vUHWGsgNy3FX1JPCKJIuBPwdeOlO19njAjzvJm4AdVXV7klOni2eoumDGPOS1VbU9ydHATUm+voe6czbuhXYG0ONPTTyU5FiA9rijlS+Y5yLJcxm8+X+kqj7Zihf8uKdV1aPA5xhcA1mcZPqD2/DYfjLutv0w4JH57enIXgv8apL7gY8zmAZ6Lwt7zABU1fb2uINB2J/MPLzGF1oA9PhTExuA1W15NYM58uny89s3Bk4BHps+nTyQZPBR/ypgc1W9Z2jTQh/3RPvkT5JDgdczuDB6C/DmVm3XcU8/H28Gbq42QXygqKqLq2ppVS1n8P/uzVV1Hgt4zABJXpDk56aXgTcAdzEfr/FxX/zYDxdTzgT+lsF86b8bd3/meGwfAx4EfszgU8AaBnOeG4Et7fGIVjcMvhH1d8BXgZXj7v8sx/xPGJze3gnc0f7O7GDc/xj4chv3XcDvt/IXA18AtgJ/BhzSyp/X1re27S8e9xhGHP+pwI09jLmN7yvt7+7p9635eI37UxCS1KmFNgUkSdpHBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnq1P8HIU4dS4Fo1nUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Calculate min, max, median of number of movies per user\n",
    "movies_per_user = [len(val) for key, val in to_users_dict.items()]\n",
    "\n",
    "print(\"The min, max, and median 'movies per user' is {}, {}, and {}\".format(np.amin(movies_per_user),\n",
    "                                                                         np.amax(movies_per_user),\n",
    "                                                                         np.median(movies_per_user)))\n",
    "users_per_movie = [len(val) for key, val in to_movies_dict.items()]\n",
    "print(\"The min, max, and median 'users per movie' is {}, {}, and {}\".format(np.amin(users_per_movie),\n",
    "                                                                         np.amax(users_per_movie),\n",
    "                                                                          np.median(users_per_movie)))\n",
    "\n",
    "\n",
    "count = 0\n",
    "n_movies_lower_bound = 20\n",
    "for n_movies in movies_per_user:\n",
    "    if n_movies <= n_movies_lower_bound:\n",
    "        count += 1\n",
    "print(\"In the training set\")\n",
    "print('There are {} users with no more than {} movies'.format(count, n_movies_lower_bound))\n",
    "#\n",
    "count = 0\n",
    "n_users_lower_bound = 2\n",
    "for n_users in users_per_movie:\n",
    "    if n_users <= n_users_lower_bound:\n",
    "        count += 1\n",
    "print('There are {} movies with no more than {} user'.format(count, n_users_lower_bound))\n",
    "\n",
    "\n",
    "## figures\n",
    "\n",
    "f = plt.figure(1)\n",
    "plt.hist(movies_per_user)\n",
    "plt.title(\"Movies per user\")\n",
    "##\n",
    "g = plt.figure(2)\n",
    "plt.hist(users_per_movie)\n",
    "plt.title(\"Users per movie\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created ./data/ml-100k/train_r.jsonl jsonline file\n",
      "Created ./data/ml-100k/validation_r.jsonl jsonline file\n"
     ]
    }
   ],
   "source": [
    "## Save training and validation data locally for rating-prediction (regression) task\n",
    "\n",
    "write_data_list_to_jsonl(copy.deepcopy(train_data_list), './data/ml-100k/train_r.jsonl')\n",
    "write_data_list_to_jsonl(copy.deepcopy(validation_data_list), './data/ml-100k/validation_r.jsonl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created ./data/ml-100k/train_c.jsonl jsonline file\n",
      "Created ./data/ml-100k/validation_c.jsonl jsonline file\n"
     ]
    }
   ],
   "source": [
    "## Save training and validation data locally for recommendation (classification) task\n",
    "\n",
    "### binarize the data \n",
    "\n",
    "train_c = get_binarized_label(copy.deepcopy(train_data_list), 3.0)\n",
    "valid_c = get_binarized_label(copy.deepcopy(validation_data_list), 3.0)\n",
    "\n",
    "write_data_list_to_jsonl(train_c, './data/ml-100k/train_c.jsonl')\n",
    "write_data_list_to_jsonl(valid_c, './data/ml-100k/validation_c.jsonl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We check whether the two classes are balanced after binarization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 0.5510213094843768 fraction of positive ratings in train_c.jsonl\n",
      "There are 0.5799575821845175 fraction of positive ratings in validation_c.jsonl\n"
     ]
    }
   ],
   "source": [
    "train_c_label = [row['label'] for row in train_c]\n",
    "valid_c_label = [row['label'] for row in valid_c]\n",
    "\n",
    "print(\"There are {} fraction of positive ratings in train_c.jsonl\".format(\n",
    "                                np.count_nonzero(train_c_label)/len(train_c_label)))\n",
    "print(\"There are {} fraction of positive ratings in validation_c.jsonl\".format(\n",
    "                                np.sum(valid_c_label)/len(valid_c_label)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that it is pretty balanced "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. (Optional) Baseline model to compare with our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_mse_loss(res, labels):\n",
    "    if type(res) is dict:\n",
    "        res = res['predictions']\n",
    "    assert len(res)==len(labels), 'result and label length mismatch!'\n",
    "    loss = 0\n",
    "    for row, label in zip(res, labels):\n",
    "        if type(row)is dict:\n",
    "            loss += (row['scores'][0]-label)**2\n",
    "        else:\n",
    "            loss += (row-label)**2\n",
    "    return round(loss/float(len(labels)), 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_r_data, valid_r_label = data_list_to_inference_format(copy.deepcopy(validation_data_list), binarize=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first test the problem on two baseline algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### (1) predict using the global average of the user predictions in the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Baseline 1 (global rating average) prediction is 3.52\n",
      "The validation mse loss of the Baseline 1 is 1.26\n"
     ]
    }
   ],
   "source": [
    "train_r_label = [row['label'] for row in copy.deepcopy(train_data_list)]\n",
    "\n",
    "bs1_prediction = round(np.mean(train_r_label), 2)\n",
    "print('The Baseline 1 (global rating average) prediction is {}'.format(bs1_prediction))\n",
    "print(\"The validation mse loss of the Baseline 1 is {}\".format(\n",
    "                                     get_mse_loss(len(valid_r_label)*[bs1_prediction], valid_r_label)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### (2) predict based on the user-averaged ratings of movies on training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bs2_predictor(test_data, user_dict, is_classification=False, thres=3):\n",
    "    test_data = copy.deepcopy(test_data['instances'])\n",
    "    predictions = list()\n",
    "    for row in test_data:\n",
    "        userID = str(row[\"in0\"][0])\n",
    "        # predict movie ID based on local average of user's prediction\n",
    "        local_movies, local_ratings = zip(*user_dict[userID])\n",
    "        local_ratings = [float(score) for score in local_ratings]\n",
    "        predictions.append(np.mean(local_ratings))\n",
    "        if is_classification:\n",
    "            predictions[-1] = int(predictions[-1] > 3)\n",
    "    return predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The validation loss of the Baseline 2 (user-based rating average) is 1.09\n"
     ]
    }
   ],
   "source": [
    "bs2_prediction = bs2_predictor(valid_r_data, to_users_dict, is_classification=False)\n",
    "print(\"The validation loss of the Baseline 2 (user-based rating average) is {}\".format(\n",
    "                                     get_mse_loss(bs2_prediction, valid_r_label)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Upload data to S3 bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_data_path = \"s3://{}/{}/data\".format(bucket, s3_prefix)\n",
    "s3_output_path = \"s3://{}/{}/output\".format(bucket, s3_prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3 = boto3.resource('s3')\n",
    "def copy_to_s3(local_file, s3_path, override=False):\n",
    "    assert s3_path.startswith('s3://')\n",
    "    split = s3_path.split('/')\n",
    "    bucket = split[2]\n",
    "    path = '/'.join(split[3:])\n",
    "    buk = s3.Bucket(bucket)\n",
    "    \n",
    "    if len(list(buk.objects.filter(Prefix=path))) > 0:\n",
    "        if not override:\n",
    "            print('File s3://{}/{} already exists.\\nSet override to upload anyway.\\n'.format(s3_bucket, s3_path))\n",
    "            return\n",
    "        else:\n",
    "            print('Overwriting existing file')\n",
    "    with open(local_file, 'rb') as data:\n",
    "        print('Uploading file to {}'.format(s3_path))\n",
    "        buk.put_object(Key=path, Body=data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading file to s3://sagemaker-us-west-2-570447867175/recomm-object2vec/data/train/train_r.jsonl\n",
      "Uploading file to s3://sagemaker-us-west-2-570447867175/recomm-object2vec/data/validation/validation_r.jsonl\n",
      "Wall time: 4.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "copy_to_s3(\"train_r.jsonl\", s3_data_path + \"/train/train_r.jsonl\")\n",
    "copy_to_s3(\"validation_r.jsonl\", s3_data_path + \"/validation/validation_r.jsonl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Start Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker.amazon.amazon_estimator:'get_image_uri' method will be deprecated in favor of 'ImageURIProvider' class in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Algorithm : 174872318107.dkr.ecr.us-west-2.amazonaws.com/object2vec:latest\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.tuner import IntegerParameter, CategoricalParameter, ContinuousParameter, HyperparameterTuner\n",
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "\n",
    "container = get_image_uri(region, 'object2vec','latest')\n",
    "print('Algorithm :', container)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### (1) Define training hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feel Free to play around with hyperparameters to get the best record"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters = {\n",
    "    \"_kvstore\": \"device\",\n",
    "    \"_num_gpus\": \"auto\",\n",
    "    \"_num_kv_servers\": \"auto\",\n",
    "    \"bucket_width\": 0,\n",
    "    \"early_stopping_patience\": 3,\n",
    "    \"early_stopping_tolerance\": 0.01,\n",
    "    \"enc0_cnn_filter_width\": 3,\n",
    "    \"enc0_layers\": \"auto\",\n",
    "    \"enc0_max_seq_len\": 1,\n",
    "    \"enc0_network\": \"pooled_embedding\",\n",
    "    \"enc0_token_embedding_dim\": 300,\n",
    "    \"enc0_vocab_size\": 944,\n",
    "    \"enc1_layers\": \"auto\",\n",
    "    \"enc1_max_seq_len\": 1,\n",
    "    \"enc1_network\": \"pooled_embedding\",\n",
    "    \"enc1_token_embedding_dim\": 300,\n",
    "    \"enc1_vocab_size\": 1684,\n",
    "    \"enc_dim\": 1024,\n",
    "    \"epochs\": 10,\n",
    "    \"learning_rate\": 0.001,\n",
    "    \"mini_batch_size\": 64,\n",
    "    \"mlp_activation\": \"tanh\",\n",
    "    \"mlp_dim\": 256,\n",
    "    \"mlp_layers\": 1,\n",
    "    \"num_classes\": 2,\n",
    "    \"optimizer\": \"adam\",\n",
    "    \"output_layer\": \"mean_squared_error\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### (2) Create 'Estimator' to start training job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.\n",
      "WARNING:sagemaker:'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n",
      "WARNING:sagemaker:'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-07-01 00:51:27 Starting - Starting the training job...\n",
      "2020-07-01 00:51:29 Starting - Launching requested ML instances......\n",
      "2020-07-01 00:52:38 Starting - Preparing the instances for training...\n",
      "2020-07-01 00:53:31 Downloading - Downloading input data...\n",
      "2020-07-01 00:53:56 Training - Downloading the training image...\n",
      "2020-07-01 00:54:30 Training - Training image download completed. Training in progress..\u001b[34mDocker entrypoint called with argument(s): train\u001b[0m\n",
      "\u001b[34mRunning default environment configuration script\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:33 INFO 139885740795712] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/default-input.json: {u'comparator_list': u'hadamard, concat, abs_diff', u'enc0_vocab_file': u'', u'output_layer': u'softmax', u'enc0_cnn_filter_width': 3, u'epochs': 30, u'mlp_dim': 512, u'enc0_freeze_pretrained_embedding': u'true', u'mlp_layers': 2, u'_num_kv_servers': u'auto', u'weight_decay': 0, u'enc0_pretrained_embedding_file': u'', u'token_embedding_storage_type': u'dense', u'enc0_token_embedding_dim': 300, u'tied_token_embedding_weight': u'false', u'learning_rate': 0.0004, u'enc1_cnn_filter_width': 3, u'negative_sampling_rate': 0, u'enc0_network': u'hcnn', u'enc1_layers': u'auto', u'early_stopping_patience': 3, u'optimizer': u'adam', u'_tuning_objective_metric': u'', u'early_stopping_tolerance': 0.01, u'dropout': 0, u'bucket_width': 0, u'enc_dim': 4096, u'enc1_vocab_file': u'', u'enc1_freeze_pretrained_embedding': u'true', u'enc0_layers': u'auto', u'mini_batch_size': 32, u'enc1_pretrained_embedding_file': u'', u'num_classes': 2, u'_num_gpus': u'auto', u'enc1_token_embedding_dim': 300, u'mlp_activation': u'linear', u'enc1_network': u'enc0', u'_kvstore': u'auto_gpu'}\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:33 INFO 139885740795712] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'output_layer': u'mean_squared_error', u'enc0_cnn_filter_width': u'3', u'epochs': u'10', u'mlp_dim': u'256', u'mlp_layers': u'1', u'_num_kv_servers': u'auto', u'mini_batch_size': u'64', u'enc0_token_embedding_dim': u'300', u'enc1_network': u'pooled_embedding', u'enc0_network': u'pooled_embedding', u'enc1_layers': u'auto', u'early_stopping_patience': u'3', u'optimizer': u'adam', u'enc1_max_seq_len': u'1', u'early_stopping_tolerance': u'0.01', u'learning_rate': u'0.001', u'bucket_width': u'0', u'enc_dim': u'1024', u'enc0_layers': u'auto', u'enc0_max_seq_len': u'1', u'num_classes': u'2', u'_num_gpus': u'auto', u'mlp_activation': u'tanh', u'enc1_token_embedding_dim': u'300', u'_kvstore': u'device', u'enc1_vocab_size': u'1684', u'enc0_vocab_size': u'944'}\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:33 INFO 139885740795712] Final configuration: {u'comparator_list': u'hadamard, concat, abs_diff', u'enc0_vocab_file': u'', u'output_layer': u'mean_squared_error', u'enc0_cnn_filter_width': u'3', u'epochs': u'10', u'mlp_dim': u'256', u'enc0_freeze_pretrained_embedding': u'true', u'mlp_layers': u'1', u'_num_kv_servers': u'auto', u'weight_decay': 0, u'enc0_pretrained_embedding_file': u'', u'enc0_max_seq_len': u'1', u'token_embedding_storage_type': u'dense', u'enc0_token_embedding_dim': u'300', u'tied_token_embedding_weight': u'false', u'learning_rate': u'0.001', u'enc1_cnn_filter_width': 3, u'negative_sampling_rate': 0, u'enc0_network': u'pooled_embedding', u'enc1_layers': u'auto', u'early_stopping_patience': u'3', u'optimizer': u'adam', u'enc1_max_seq_len': u'1', u'_tuning_objective_metric': u'', u'early_stopping_tolerance': u'0.01', u'dropout': 0, u'bucket_width': u'0', u'enc_dim': u'1024', u'enc1_vocab_file': u'', u'enc1_freeze_pretrained_embedding': u'true', u'enc0_layers': u'auto', u'mini_batch_size': u'64', u'enc1_pretrained_embedding_file': u'', u'num_classes': u'2', u'_num_gpus': u'auto', u'enc1_token_embedding_dim': u'300', u'mlp_activation': u'tanh', u'enc1_network': u'pooled_embedding', u'_kvstore': u'device', u'enc1_vocab_size': u'1684', u'enc0_vocab_size': u'944'}\u001b[0m\n",
      "\u001b[34mProcess 1 is a worker.\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:33 INFO 139885740795712] Using default worker.\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:33 INFO 139885740795712] Loaded iterator creator application/jsonlines for content type ('application/jsonlines', '1.0')\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:33 INFO 139885740795712] create_iter params {u'comparator_list': u'hadamard, concat, abs_diff', u'enc0_vocab_file': u'', u'output_layer': u'mean_squared_error', u'enc0_cnn_filter_width': u'3', u'epochs': u'10', u'mlp_dim': u'256', u'enc0_freeze_pretrained_embedding': u'true', u'mlp_layers': u'1', u'_num_kv_servers': u'auto', u'weight_decay': 0, u'enc0_pretrained_embedding_file': u'', u'enc0_max_seq_len': u'1', u'token_embedding_storage_type': u'dense', u'enc0_token_embedding_dim': u'300', u'tied_token_embedding_weight': u'false', u'learning_rate': u'0.001', u'enc1_cnn_filter_width': 3, u'negative_sampling_rate': 0, u'enc0_network': u'pooled_embedding', u'enc1_layers': u'auto', u'early_stopping_patience': u'3', u'optimizer': u'adam', u'enc1_max_seq_len': u'1', u'_tuning_objective_metric': u'', u'early_stopping_tolerance': u'0.01', u'dropout': 0, u'bucket_width': u'0', u'enc_dim': u'1024', u'enc1_vocab_file': u'', u'enc1_freeze_pretrained_embedding': u'true', u'enc0_layers': u'auto', u'mini_batch_size': u'64', u'enc1_pretrained_embedding_file': u'', u'num_classes': u'2', u'_num_gpus': u'auto', u'enc1_token_embedding_dim': u'300', u'mlp_activation': u'tanh', u'enc1_network': u'pooled_embedding', u'_kvstore': u'device', u'enc1_vocab_size': u'1684', u'enc0_vocab_size': u'944'}\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:33 INFO 139885740795712] create_iter content_params {}\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:33 INFO 139885740795712] Parameters of encoders: [{u'layers': u'auto', u'vocab_size': u'944', u'network': u'pooled_embedding', u'vocab_file': u'', u'cnn_filter_width': u'3', u'token_embedding_dim': u'300', u'max_seq_len': u'1', u'freeze_pretrained_embedding': u'true', u'pretrained_embedding_file': u''}, {u'layers': u'auto', u'vocab_size': u'1684', u'network': u'pooled_embedding', u'vocab_file': u'', u'cnn_filter_width': 3, u'token_embedding_dim': u'300', u'max_seq_len': u'1', u'freeze_pretrained_embedding': u'true', u'pretrained_embedding_file': u''}]\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:33 INFO 139885740795712] output_layer is set to mean_squared_error. Ignoring the hyperparameter 'num_classes'.\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:33 INFO 139885740795712] Inconsistent enc_dim and token_embedding_dim found for token-embedding-encoder: 1024 vs 300. Setting token embedding dim to be 1024\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:33 INFO 139885740795712] Inconsistent enc_dim and token_embedding_dim found for token-embedding-encoder: 1024 vs 300. Setting token embedding dim to be 1024\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:33 INFO 139885740795712] Encoder configs: [{'vocab_size': 944, 'enc_index': 0, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 1024, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}, {'vocab_size': 1684, 'enc_index': 1, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 1024, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}]\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:33 INFO 139885740795712] Config: {'comparator_list': ['hadamard', 'concat', 'abs_diff'], 'epochs': 10, 'mini_batch_size': 64, 'optimizer': 'adam', 'output_layer': 'mean_squared_error', 'token_embedding_storage_type': 'dense', 'mlp_dim': 256, 'early_stopping_tolerance': 0.01, 'dropout': 0.0, 'bucket_width': 0, 'enc_dim': 1024, 'negative_sampling_rate': 0, 'early_stopping_patience': 3, 'learning_rate': 0.001, 'max_seq_lens': [1, 1], 'tied_token_embedding_weight': False, 'enc_configs': [{'vocab_size': 944, 'enc_index': 0, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 1024, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}, {'vocab_size': 1684, 'enc_index': 1, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 1024, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}], 'mlp_layers': 1, 'weight_decay': 0.0, 'mlp_activation': 'tanh', 'num_classes': None}\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:33 INFO 139885740795712] use bucketing: False\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:33 INFO 139885740795712] Creating data iterator for /opt/ml/input/data/train\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Source words: 90570\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Target words: 90570\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Total: 90570 samples in 1 buckets\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Bucket of (1, 1) : 90570 samples in 1415 batches of 64, approx 64.0 words/batch\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] 0 sentence pairs discarded\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] fill up mode: replicate\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Negative sampling not used\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Replicating 54 random sentences from bucket (1, 1) to size it to multiple of 64\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Bucket batch sizes: [BucketBatchSize(batch_size=64, average_words_per_batch=64)]\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] create_iter params {u'comparator_list': u'hadamard, concat, abs_diff', u'enc0_vocab_file': u'', u'output_layer': u'mean_squared_error', u'enc0_cnn_filter_width': u'3', u'epochs': u'10', u'mlp_dim': u'256', u'enc0_freeze_pretrained_embedding': u'true', u'mlp_layers': u'1', u'_num_kv_servers': u'auto', u'weight_decay': 0, u'enc0_pretrained_embedding_file': u'', u'enc0_max_seq_len': u'1', u'token_embedding_storage_type': u'dense', u'enc0_token_embedding_dim': u'300', u'tied_token_embedding_weight': u'false', u'learning_rate': u'0.001', u'enc1_cnn_filter_width': 3, u'negative_sampling_rate': 0, u'enc0_network': u'pooled_embedding', u'enc1_layers': u'auto', u'early_stopping_patience': u'3', u'optimizer': u'adam', u'enc1_max_seq_len': u'1', u'_tuning_objective_metric': u'', u'early_stopping_tolerance': u'0.01', u'dropout': 0, u'bucket_width': u'0', u'enc_dim': u'1024', u'enc1_vocab_file': u'', u'enc1_freeze_pretrained_embedding': u'true', u'enc0_layers': u'auto', u'mini_batch_size': u'64', u'enc1_pretrained_embedding_file': u'', u'num_classes': u'2', u'_num_gpus': u'auto', u'enc1_token_embedding_dim': u'300', u'mlp_activation': u'tanh', u'enc1_network': u'pooled_embedding', u'_kvstore': u'device', u'enc1_vocab_size': u'1684', u'enc0_vocab_size': u'944'}\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] create_iter content_params {}\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Parameters of encoders: [{u'layers': u'auto', u'vocab_size': u'944', u'network': u'pooled_embedding', u'vocab_file': u'', u'cnn_filter_width': u'3', u'token_embedding_dim': u'300', u'max_seq_len': u'1', u'freeze_pretrained_embedding': u'true', u'pretrained_embedding_file': u''}, {u'layers': u'auto', u'vocab_size': u'1684', u'network': u'pooled_embedding', u'vocab_file': u'', u'cnn_filter_width': 3, u'token_embedding_dim': u'300', u'max_seq_len': u'1', u'freeze_pretrained_embedding': u'true', u'pretrained_embedding_file': u''}]\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] output_layer is set to mean_squared_error. Ignoring the hyperparameter 'num_classes'.\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Inconsistent enc_dim and token_embedding_dim found for token-embedding-encoder: 1024 vs 300. Setting token embedding dim to be 1024\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Inconsistent enc_dim and token_embedding_dim found for token-embedding-encoder: 1024 vs 300. Setting token embedding dim to be 1024\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Encoder configs: [{'vocab_size': 944, 'enc_index': 0, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 1024, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}, {'vocab_size': 1684, 'enc_index': 1, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 1024, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}]\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Config: {'comparator_list': ['hadamard', 'concat', 'abs_diff'], 'epochs': 10, 'mini_batch_size': 64, 'optimizer': 'adam', 'output_layer': 'mean_squared_error', 'token_embedding_storage_type': 'dense', 'mlp_dim': 256, 'early_stopping_tolerance': 0.01, 'dropout': 0.0, 'bucket_width': 0, 'enc_dim': 1024, 'negative_sampling_rate': 0, 'early_stopping_patience': 3, 'learning_rate': 0.001, 'max_seq_lens': [1, 1], 'tied_token_embedding_weight': False, 'enc_configs': [{'vocab_size': 944, 'enc_index': 0, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 1024, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}, {'vocab_size': 1684, 'enc_index': 1, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 1024, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}], 'mlp_layers': 1, 'weight_decay': 0.0, 'mlp_activation': 'tanh', 'num_classes': None}\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] use bucketing: False\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Creating data iterator for /opt/ml/input/data/test\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Source words: 9430\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Target words: 9430\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Total: 9430 samples in 1 buckets\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Bucket of (1, 1) : 9430 samples in 147 batches of 64, approx 64.0 words/batch\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] 0 sentence pairs discarded\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] fill up mode: replicate\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Negative sampling not used\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Replicating 42 random sentences from bucket (1, 1) to size it to multiple of 64\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Parameters of encoders: [{u'layers': u'auto', u'vocab_size': u'944', u'network': u'pooled_embedding', u'vocab_file': u'', u'cnn_filter_width': u'3', u'token_embedding_dim': u'300', u'max_seq_len': u'1', u'freeze_pretrained_embedding': u'true', u'pretrained_embedding_file': u''}, {u'layers': u'auto', u'vocab_size': u'1684', u'network': u'pooled_embedding', u'vocab_file': u'', u'cnn_filter_width': 3, u'token_embedding_dim': u'300', u'max_seq_len': u'1', u'freeze_pretrained_embedding': u'true', u'pretrained_embedding_file': u''}]\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] output_layer is set to mean_squared_error. Ignoring the hyperparameter 'num_classes'.\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Inconsistent enc_dim and token_embedding_dim found for token-embedding-encoder: 1024 vs 300. Setting token embedding dim to be 1024\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Inconsistent enc_dim and token_embedding_dim found for token-embedding-encoder: 1024 vs 300. Setting token embedding dim to be 1024\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Encoder configs: [{'vocab_size': 944, 'enc_index': 0, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 1024, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}, {'vocab_size': 1684, 'enc_index': 1, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 1024, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}]\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Config: {'comparator_list': ['hadamard', 'concat', 'abs_diff'], 'epochs': 10, 'mini_batch_size': 64, 'optimizer': 'adam', 'output_layer': 'mean_squared_error', 'token_embedding_storage_type': 'dense', 'mlp_dim': 256, 'early_stopping_tolerance': 0.01, 'dropout': 0.0, 'bucket_width': 0, 'enc_dim': 1024, 'negative_sampling_rate': 0, 'early_stopping_patience': 3, 'learning_rate': 0.001, 'max_seq_lens': [1, 1], 'tied_token_embedding_weight': False, 'enc_configs': [{'vocab_size': 944, 'enc_index': 0, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 1024, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}, {'vocab_size': 1684, 'enc_index': 1, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 1024, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}], 'mlp_layers': 1, 'weight_decay': 0.0, 'mlp_activation': 'tanh', 'num_classes': None}\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Creating new state\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] params {u'comparator_list': u'hadamard, concat, abs_diff', u'enc0_vocab_file': u'', u'output_layer': u'mean_squared_error', u'enc0_cnn_filter_width': u'3', u'epochs': u'10', u'mlp_dim': u'256', u'enc0_freeze_pretrained_embedding': u'true', u'mlp_layers': u'1', u'_num_kv_servers': u'auto', u'weight_decay': 0, u'enc0_pretrained_embedding_file': u'', u'enc0_max_seq_len': u'1', u'token_embedding_storage_type': u'dense', u'enc0_token_embedding_dim': u'300', u'tied_token_embedding_weight': u'false', u'learning_rate': u'0.001', u'enc1_cnn_filter_width': 3, u'negative_sampling_rate': 0, u'enc0_network': u'pooled_embedding', u'enc1_layers': u'auto', u'early_stopping_patience': u'3', u'optimizer': u'adam', u'enc1_max_seq_len': u'1', u'_tuning_objective_metric': u'', u'early_stopping_tolerance': u'0.01', u'dropout': 0, u'bucket_width': u'0', u'enc_dim': u'1024', 'default_bucket_key': (1, 1), u'enc1_vocab_file': u'', u'enc1_freeze_pretrained_embedding': u'true', u'enc0_layers': u'auto', u'mini_batch_size': u'64', u'enc1_pretrained_embedding_file': u'', u'num_classes': u'2', u'_num_gpus': u'auto', u'enc1_token_embedding_dim': u'300', u'mlp_activation': u'tanh', u'enc1_network': u'pooled_embedding', u'_kvstore': u'device', u'enc1_vocab_size': u'1684', u'enc0_vocab_size': u'944'}\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] default_bucket_key (1, 1)\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] nvidia-smi took: 0.0251839160919 secs to identify 0 gpus\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] context [cpu(0)]\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Create Store: device\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 WARNING 139885740795712] dense token embedding is used in a multi-gpu setting...consider changing 'token_embedding_storage_type' to 'row_sparse'\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] data_names: ['source', 'target']\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] label_names: ['out_layer_label']\u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mLayer (type)                                        Output Shape            Param #     Previous Layer                  \u001b[0m\n",
      "\u001b[34m========================================================================================================================\u001b[0m\n",
      "\u001b[34msource(null)                                        1                       0                                           \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34membed_0(Embedding)                                  1x1024                  0           source                          \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m_not_equal_scalar0(_not_equal_scalar)               1                       0           source                          \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mreshape0(Reshape)                                   1x1                     0           _not_equal_scalar0              \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mbroadcast_mul0(broadcast_mul)                       1x1024                  0           embed_0                         \n",
      "                                                                                        reshape0                        \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34msum0(sum)                                           1024                    0           broadcast_mul0                  \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34msum1(sum)                                           1                       0           reshape0                        \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mzeros_like0(zeros_like)                             1                       0           sum1                            \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m_equal0(_equal)                                     1                       0           sum1                            \n",
      "                                                                                        zeros_like0                     \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m_plus0(elemwise_add)                                1                       0           sum1                            \n",
      "                                                                                        _equal0                         \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mbroadcast_div0(broadcast_div)                       1024                    0           sum0                            \n",
      "                                                                                        _plus0                          \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mdropout0(Dropout)                                   1024                    0           broadcast_div0                  \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34membed_1(Embedding)                                  1x1024                  0                                           \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m_not_equal_scalar1(_not_equal_scalar)               1                       0                                           \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mreshape1(Reshape)                                   1x1                     0           _not_equal_scalar1              \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mbroadcast_mul1(broadcast_mul)                       1x1024                  0           embed_1                         \n",
      "                                                                                        reshape1                        \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34msum2(sum)                                           1024                    0           broadcast_mul1                  \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34msum3(sum)                                           1                       0           reshape1                        \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mzeros_like1(zeros_like)                             1                       0           sum3                            \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m_equal1(_equal)                                     1                       0           sum3                            \n",
      "                                                                                        zeros_like1                     \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m_plus1(elemwise_add)                                1                       0           sum3                            \n",
      "                                                                                        _equal1                         \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mbroadcast_div1(broadcast_div)                       1024                    0           sum2                            \n",
      "                                                                                        _plus1                          \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mdropout1(Dropout)                                   1024                    0           broadcast_div1                  \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m_mul0(elemwise_mul)                                 1024                    0           dropout0                        \n",
      "                                                                                        dropout1                        \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m_minus0(elemwise_sub)                               1024                    0           dropout0                        \n",
      "                                                                                        dropout1                        \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mabs0(abs)                                           1024                    0           _minus0                         \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mconcat0(Concat)                                     4096                    0           _mul0                           \n",
      "                                                                                        dropout0                        \n",
      "                                                                                        dropout1                        \n",
      "                                                                                        abs0                            \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mmlp_fc0(FullyConnected)                             256                     1048832     concat0                         \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mactivation0(Activation)                             256                     0           mlp_fc0                         \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mdropout2(Dropout)                                   256                     0           activation0                     \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34moutput_layer(FullyConnected)                        1                       257         dropout2                        \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mout_layer(LinearRegressionOutput)                   1                       0           output_layer                    \u001b[0m\n",
      "\u001b[34m========================================================================================================================\u001b[0m\n",
      "\u001b[34mTotal params: 1049089\u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] data_shapes [DataDesc[source,(64, 1),<type 'numpy.float32'>,NTC], DataDesc[target,(64, 1),<type 'numpy.float32'>,NTC]]\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] label_shapes [DataDesc[out_layer_label,(64,),<type 'numpy.float32'>,NTC]]\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] fixed_param_names: []\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] Initialized BucketingPlus Module\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] arg_params keys for module initialization: []\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:36 INFO 139885740795712] all params:['output_layer_weight', 'mlp_fc0_weight', 'mlp_fc0_bias', 'output_layer_bias', 'embed_1_weight', 'embed_0_weight']\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"initialize.time\": {\"count\": 1, \"max\": 96.81200981140137, \"sum\": 96.81200981140137, \"min\": 96.81200981140137}}, \"EndTime\": 1593564876.918406, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593564873.846331}\n",
      "\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Records Seen\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Reset Count\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}}, \"EndTime\": 1593564876.918636, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"init_train_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593564876.918574}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:40 INFO 139885740795712] Epoch: 0, batches: 100, num_examples: 6400, 2032.8 samples/sec, epoch time so far: 0:00:03.148409\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:40 INFO 139885740795712] #011Training metrics: mean_squared_error: 1.632 mean_absolute_error: 0.991 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:43 INFO 139885740795712] Epoch: 0, batches: 200, num_examples: 12800, 2048.1 samples/sec, epoch time so far: 0:00:06.249563\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:43 INFO 139885740795712] #011Training metrics: mean_squared_error: 1.374 mean_absolute_error: 0.915 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:46 INFO 139885740795712] Epoch: 0, batches: 300, num_examples: 19200, 2056.3 samples/sec, epoch time so far: 0:00:09.337084\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:46 INFO 139885740795712] #011Training metrics: mean_squared_error: 1.256 mean_absolute_error: 0.879 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:49 INFO 139885740795712] Epoch: 0, batches: 400, num_examples: 25600, 2061.5 samples/sec, epoch time so far: 0:00:12.417864\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:49 INFO 139885740795712] #011Training metrics: mean_squared_error: 1.203 mean_absolute_error: 0.863 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:52 INFO 139885740795712] Epoch: 0, batches: 500, num_examples: 32000, 2064.6 samples/sec, epoch time so far: 0:00:15.499508\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:52 INFO 139885740795712] #011Training metrics: mean_squared_error: 1.162 mean_absolute_error: 0.849 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:55 INFO 139885740795712] Epoch: 0, batches: 600, num_examples: 38400, 2068.4 samples/sec, epoch time so far: 0:00:18.564917\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:55 INFO 139885740795712] #011Training metrics: mean_squared_error: 1.134 mean_absolute_error: 0.840 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:58 INFO 139885740795712] Epoch: 0, batches: 700, num_examples: 44800, 2060.2 samples/sec, epoch time so far: 0:00:21.745572\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:54:58 INFO 139885740795712] #011Training metrics: mean_squared_error: 1.113 mean_absolute_error: 0.833 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:02 INFO 139885740795712] Epoch: 0, batches: 800, num_examples: 51200, 2042.0 samples/sec, epoch time so far: 0:00:25.073115\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:02 INFO 139885740795712] #011Training metrics: mean_squared_error: 1.095 mean_absolute_error: 0.826 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:05 INFO 139885740795712] Epoch: 0, batches: 900, num_examples: 57600, 2009.1 samples/sec, epoch time so far: 0:00:28.669768\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:05 INFO 139885740795712] #011Training metrics: mean_squared_error: 1.079 mean_absolute_error: 0.821 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:09 INFO 139885740795712] Epoch: 0, batches: 1000, num_examples: 64000, 1980.5 samples/sec, epoch time so far: 0:00:32.314732\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:09 INFO 139885740795712] #011Training metrics: mean_squared_error: 1.063 mean_absolute_error: 0.815 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:13 INFO 139885740795712] Epoch: 0, batches: 1100, num_examples: 70400, 1945.6 samples/sec, epoch time so far: 0:00:36.183852\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:13 INFO 139885740795712] #011Training metrics: mean_squared_error: 1.053 mean_absolute_error: 0.811 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:17 INFO 139885740795712] Epoch: 0, batches: 1200, num_examples: 76800, 1912.6 samples/sec, epoch time so far: 0:00:40.154722\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:17 INFO 139885740795712] #011Training metrics: mean_squared_error: 1.045 mean_absolute_error: 0.808 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:21 INFO 139885740795712] Epoch: 0, batches: 1300, num_examples: 83200, 1883.0 samples/sec, epoch time so far: 0:00:44.185253\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:21 INFO 139885740795712] #011Training metrics: mean_squared_error: 1.035 mean_absolute_error: 0.805 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:25 INFO 139885740795712] Epoch: 0, batches: 1400, num_examples: 89600, 1855.6 samples/sec, epoch time so far: 0:00:48.287065\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:25 INFO 139885740795712] #011Training metrics: mean_squared_error: 1.025 mean_absolute_error: 0.801 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:25 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:25 INFO 139885740795712] Completed Epoch: 0, time taken: 0:00:48.938895\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:25 INFO 139885740795712] Epoch 0 Training metrics:   mean_squared_error: 1.023 mean_absolute_error: 0.800 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:25 INFO 139885740795712] #quality_metric: host=algo-1, epoch=0, train mean_squared_error <loss>=1.02309812313\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:25 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"epochs\": {\"count\": 1, \"max\": 10, \"sum\": 10.0, \"min\": 10}, \"early_stop.time\": {\"count\": 1, \"max\": 0.49805641174316406, \"sum\": 0.49805641174316406, \"min\": 0.49805641174316406}, \"update.time\": {\"count\": 1, \"max\": 48943.82381439209, \"sum\": 48943.82381439209, \"min\": 48943.82381439209}}, \"EndTime\": 1593564925.900092, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593564876.91852}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:25 INFO 139885740795712] #progress_metric: host=algo-1, completed 10 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Total Batches Seen\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Total Records Seen\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Reset Count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1593564925.900593, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 0}, \"StartTime\": 1593564876.956241}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:25 INFO 139885740795712] #throughput_metric: host=algo-1, train throughput=1851.56138534 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:30 INFO 139885740795712] Epoch: 1, batches: 100, num_examples: 6400, 1355.4 samples/sec, epoch time so far: 0:00:04.721801\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:30 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.747 mean_absolute_error: 0.684 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:34 INFO 139885740795712] Epoch: 1, batches: 200, num_examples: 12800, 1420.5 samples/sec, epoch time so far: 0:00:09.011132\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:34 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.718 mean_absolute_error: 0.671 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:39 INFO 139885740795712] Epoch: 1, batches: 300, num_examples: 19200, 1438.0 samples/sec, epoch time so far: 0:00:13.352116\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:39 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.707 mean_absolute_error: 0.665 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:43 INFO 139885740795712] Epoch: 1, batches: 400, num_examples: 25600, 1446.7 samples/sec, epoch time so far: 0:00:17.695919\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:43 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.704 mean_absolute_error: 0.663 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:47 INFO 139885740795712] Epoch: 1, batches: 500, num_examples: 32000, 1459.8 samples/sec, epoch time so far: 0:00:21.920614\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:47 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.698 mean_absolute_error: 0.659 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:51 INFO 139885740795712] Epoch: 1, batches: 600, num_examples: 38400, 1474.5 samples/sec, epoch time so far: 0:00:26.042368\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:51 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.693 mean_absolute_error: 0.656 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:55 INFO 139885740795712] Epoch: 1, batches: 700, num_examples: 44800, 1492.9 samples/sec, epoch time so far: 0:00:30.008110\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:55 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.686 mean_absolute_error: 0.652 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:59 INFO 139885740795712] Epoch: 1, batches: 800, num_examples: 51200, 1504.8 samples/sec, epoch time so far: 0:00:34.025009\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:55:59 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.677 mean_absolute_error: 0.648 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:04 INFO 139885740795712] Epoch: 1, batches: 900, num_examples: 57600, 1509.4 samples/sec, epoch time so far: 0:00:38.160558\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:04 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.673 mean_absolute_error: 0.645 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:08 INFO 139885740795712] Epoch: 1, batches: 1000, num_examples: 64000, 1518.5 samples/sec, epoch time so far: 0:00:42.148128\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:08 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.670 mean_absolute_error: 0.643 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:12 INFO 139885740795712] Epoch: 1, batches: 1100, num_examples: 70400, 1525.1 samples/sec, epoch time so far: 0:00:46.161853\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:12 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.666 mean_absolute_error: 0.642 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:16 INFO 139885740795712] Epoch: 1, batches: 1200, num_examples: 76800, 1530.5 samples/sec, epoch time so far: 0:00:50.178870\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:16 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.666 mean_absolute_error: 0.642 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:20 INFO 139885740795712] Epoch: 1, batches: 1300, num_examples: 83200, 1534.5 samples/sec, epoch time so far: 0:00:54.220966\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:20 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.663 mean_absolute_error: 0.640 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:24 INFO 139885740795712] Epoch: 1, batches: 1400, num_examples: 89600, 1537.7 samples/sec, epoch time so far: 0:00:58.268661\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:24 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.661 mean_absolute_error: 0.639 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:24 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:24 INFO 139885740795712] Completed Epoch: 1, time taken: 0:00:58.913693\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:24 INFO 139885740795712] Epoch 1 Training metrics:   mean_squared_error: 0.660 mean_absolute_error: 0.638 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:24 INFO 139885740795712] #quality_metric: host=algo-1, epoch=1, train mean_squared_error <loss>=0.659926070963\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:24 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 4.160881042480469, \"sum\": 4.160881042480469, \"min\": 4.160881042480469}, \"update.time\": {\"count\": 1, \"max\": 58918.6270236969, \"sum\": 58918.6270236969, \"min\": 58918.6270236969}}, \"EndTime\": 1593564984.863238, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593564925.900205}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:24 INFO 139885740795712] #progress_metric: host=algo-1, completed 20 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Total Batches Seen\": {\"count\": 1, \"max\": 2832, \"sum\": 2832.0, \"min\": 2832}, \"Total Records Seen\": {\"count\": 1, \"max\": 181248, \"sum\": 181248.0, \"min\": 181248}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Reset Count\": {\"count\": 1, \"max\": 2, \"sum\": 2.0, \"min\": 2}}, \"EndTime\": 1593564984.863626, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 1}, \"StartTime\": 1593564925.944577}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:24 INFO 139885740795712] #throughput_metric: host=algo-1, train throughput=1538.10556005 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:29 INFO 139885740795712] Epoch: 2, batches: 100, num_examples: 6400, 1484.9 samples/sec, epoch time so far: 0:00:04.309977\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:29 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.287 mean_absolute_error: 0.412 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:33 INFO 139885740795712] Epoch: 2, batches: 200, num_examples: 12800, 1491.0 samples/sec, epoch time so far: 0:00:08.585036\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:33 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.272 mean_absolute_error: 0.399 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:37 INFO 139885740795712] Epoch: 2, batches: 300, num_examples: 19200, 1500.8 samples/sec, epoch time so far: 0:00:12.792781\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:37 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.270 mean_absolute_error: 0.398 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:41 INFO 139885740795712] Epoch: 2, batches: 400, num_examples: 25600, 1502.1 samples/sec, epoch time so far: 0:00:17.042547\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:41 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.268 mean_absolute_error: 0.397 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:46 INFO 139885740795712] Epoch: 2, batches: 500, num_examples: 32000, 1508.4 samples/sec, epoch time so far: 0:00:21.215233\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:46 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.266 mean_absolute_error: 0.395 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:50 INFO 139885740795712] Epoch: 2, batches: 600, num_examples: 38400, 1511.9 samples/sec, epoch time so far: 0:00:25.398405\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:50 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.265 mean_absolute_error: 0.395 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:54 INFO 139885740795712] Epoch: 2, batches: 700, num_examples: 44800, 1521.8 samples/sec, epoch time so far: 0:00:29.439554\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:54 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.266 mean_absolute_error: 0.395 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:58 INFO 139885740795712] Epoch: 2, batches: 800, num_examples: 51200, 1518.9 samples/sec, epoch time so far: 0:00:33.709297\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:56:58 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.267 mean_absolute_error: 0.396 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:02 INFO 139885740795712] Epoch: 2, batches: 900, num_examples: 57600, 1519.7 samples/sec, epoch time so far: 0:00:37.902051\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:02 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.268 mean_absolute_error: 0.396 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:07 INFO 139885740795712] Epoch: 2, batches: 1000, num_examples: 64000, 1520.1 samples/sec, epoch time so far: 0:00:42.102104\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:07 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.269 mean_absolute_error: 0.397 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:11 INFO 139885740795712] Epoch: 2, batches: 1100, num_examples: 70400, 1520.8 samples/sec, epoch time so far: 0:00:46.292383\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:11 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.270 mean_absolute_error: 0.399 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:15 INFO 139885740795712] Epoch: 2, batches: 1200, num_examples: 76800, 1523.7 samples/sec, epoch time so far: 0:00:50.402539\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:15 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.273 mean_absolute_error: 0.401 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:19 INFO 139885740795712] Epoch: 2, batches: 1300, num_examples: 83200, 1526.2 samples/sec, epoch time so far: 0:00:54.515917\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:19 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.274 mean_absolute_error: 0.402 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:23 INFO 139885740795712] Epoch: 2, batches: 1400, num_examples: 89600, 1531.0 samples/sec, epoch time so far: 0:00:58.523171\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:23 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.275 mean_absolute_error: 0.403 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:24 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:24 INFO 139885740795712] Completed Epoch: 2, time taken: 0:00:59.153882\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:24 INFO 139885740795712] Epoch 2 Training metrics:   mean_squared_error: 0.275 mean_absolute_error: 0.403 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:24 INFO 139885740795712] #quality_metric: host=algo-1, epoch=2, train mean_squared_error <loss>=0.274666644275\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:24 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 2.998828887939453, \"sum\": 2.998828887939453, \"min\": 2.998828887939453}, \"update.time\": {\"count\": 1, \"max\": 59157.65690803528, \"sum\": 59157.65690803528, \"min\": 59157.65690803528}}, \"EndTime\": 1593565044.062814, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593564984.863368}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:24 INFO 139885740795712] #progress_metric: host=algo-1, completed 30 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Total Batches Seen\": {\"count\": 1, \"max\": 4248, \"sum\": 4248.0, \"min\": 4248}, \"Total Records Seen\": {\"count\": 1, \"max\": 271872, \"sum\": 271872.0, \"min\": 271872}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Reset Count\": {\"count\": 1, \"max\": 3, \"sum\": 3.0, \"min\": 3}}, \"EndTime\": 1593565044.063166, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 2}, \"StartTime\": 1593564984.905125}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:24 INFO 139885740795712] #throughput_metric: host=algo-1, train throughput=1531.8901229 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:28 INFO 139885740795712] Epoch: 3, batches: 100, num_examples: 6400, 1581.7 samples/sec, epoch time so far: 0:00:04.046337\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:28 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.162 mean_absolute_error: 0.294 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:32 INFO 139885740795712] Epoch: 3, batches: 200, num_examples: 12800, 1530.9 samples/sec, epoch time so far: 0:00:08.361344\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:32 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.164 mean_absolute_error: 0.295 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:36 INFO 139885740795712] Epoch: 3, batches: 300, num_examples: 19200, 1520.7 samples/sec, epoch time so far: 0:00:12.625832\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:36 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.163 mean_absolute_error: 0.296 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:41 INFO 139885740795712] Epoch: 3, batches: 400, num_examples: 25600, 1513.3 samples/sec, epoch time so far: 0:00:16.916404\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:41 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.161 mean_absolute_error: 0.295 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:45 INFO 139885740795712] Epoch: 3, batches: 500, num_examples: 32000, 1510.6 samples/sec, epoch time so far: 0:00:21.183497\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:45 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.161 mean_absolute_error: 0.296 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:49 INFO 139885740795712] Epoch: 3, batches: 600, num_examples: 38400, 1512.3 samples/sec, epoch time so far: 0:00:25.391793\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:49 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.161 mean_absolute_error: 0.296 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:53 INFO 139885740795712] Epoch: 3, batches: 700, num_examples: 44800, 1519.0 samples/sec, epoch time so far: 0:00:29.492601\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:53 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.161 mean_absolute_error: 0.297 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:57 INFO 139885740795712] Epoch: 3, batches: 800, num_examples: 51200, 1526.7 samples/sec, epoch time so far: 0:00:33.536986\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:57:57 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.161 mean_absolute_error: 0.297 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:01 INFO 139885740795712] Epoch: 3, batches: 900, num_examples: 57600, 1527.2 samples/sec, epoch time so far: 0:00:37.717059\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:01 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.161 mean_absolute_error: 0.297 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:05 INFO 139885740795712] Epoch: 3, batches: 1000, num_examples: 64000, 1531.6 samples/sec, epoch time so far: 0:00:41.787310\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:05 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.161 mean_absolute_error: 0.298 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:09 INFO 139885740795712] Epoch: 3, batches: 1100, num_examples: 70400, 1534.1 samples/sec, epoch time so far: 0:00:45.890871\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:09 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.162 mean_absolute_error: 0.299 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:14 INFO 139885740795712] Epoch: 3, batches: 1200, num_examples: 76800, 1537.7 samples/sec, epoch time so far: 0:00:49.943941\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:14 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.161 mean_absolute_error: 0.299 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:18 INFO 139885740795712] Epoch: 3, batches: 1300, num_examples: 83200, 1539.1 samples/sec, epoch time so far: 0:00:54.056771\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:18 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.162 mean_absolute_error: 0.299 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:22 INFO 139885740795712] Epoch: 3, batches: 1400, num_examples: 89600, 1540.6 samples/sec, epoch time so far: 0:00:58.159715\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:22 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.162 mean_absolute_error: 0.300 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:22 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:22 INFO 139885740795712] Completed Epoch: 3, time taken: 0:00:58.806689\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:22 INFO 139885740795712] Epoch 3 Training metrics:   mean_squared_error: 0.162 mean_absolute_error: 0.300 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:22 INFO 139885740795712] #quality_metric: host=algo-1, epoch=3, train mean_squared_error <loss>=0.162367937729\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:22 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:22 INFO 139885740795712] patience losses: [1.0230981231282996, 0.6599260709632588, 0.2746666442750201]\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:22 INFO 139885740795712] min patience losses: 0.274666644275\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:22 INFO 139885740795712] current loss: 0.162367937729\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:22 INFO 139885740795712] absolute loss difference: 0.112298706546\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 4.21905517578125, \"sum\": 4.21905517578125, \"min\": 4.21905517578125}, \"update.time\": {\"count\": 1, \"max\": 58811.689138412476, \"sum\": 58811.689138412476, \"min\": 58811.689138412476}}, \"EndTime\": 1593565102.913819, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593565044.062942}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:22 INFO 139885740795712] #progress_metric: host=algo-1, completed 40 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Total Batches Seen\": {\"count\": 1, \"max\": 5664, \"sum\": 5664.0, \"min\": 5664}, \"Total Records Seen\": {\"count\": 1, \"max\": 362496, \"sum\": 362496.0, \"min\": 362496}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Reset Count\": {\"count\": 1, \"max\": 4, \"sum\": 4.0, \"min\": 4}}, \"EndTime\": 1593565102.914074, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 3}, \"StartTime\": 1593565044.102098}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:22 INFO 139885740795712] #throughput_metric: host=algo-1, train throughput=1540.90595706 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:27 INFO 139885740795712] Epoch: 4, batches: 100, num_examples: 6400, 1502.0 samples/sec, epoch time so far: 0:00:04.261078\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:27 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.131 mean_absolute_error: 0.270 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:31 INFO 139885740795712] Epoch: 4, batches: 200, num_examples: 12800, 1502.7 samples/sec, epoch time so far: 0:00:08.518226\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:31 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.124 mean_absolute_error: 0.263 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:35 INFO 139885740795712] Epoch: 4, batches: 300, num_examples: 19200, 1505.1 samples/sec, epoch time so far: 0:00:12.756771\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:35 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.120 mean_absolute_error: 0.259 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:39 INFO 139885740795712] Epoch: 4, batches: 400, num_examples: 25600, 1503.3 samples/sec, epoch time so far: 0:00:17.029118\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:39 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.118 mean_absolute_error: 0.257 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:44 INFO 139885740795712] Epoch: 4, batches: 500, num_examples: 32000, 1507.9 samples/sec, epoch time so far: 0:00:21.221249\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:44 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.117 mean_absolute_error: 0.256 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:48 INFO 139885740795712] Epoch: 4, batches: 600, num_examples: 38400, 1520.0 samples/sec, epoch time so far: 0:00:25.263586\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:48 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.116 mean_absolute_error: 0.255 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:52 INFO 139885740795712] Epoch: 4, batches: 700, num_examples: 44800, 1530.1 samples/sec, epoch time so far: 0:00:29.279539\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:52 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.114 mean_absolute_error: 0.254 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:56 INFO 139885740795712] Epoch: 4, batches: 800, num_examples: 51200, 1533.3 samples/sec, epoch time so far: 0:00:33.392277\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:58:56 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.114 mean_absolute_error: 0.254 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:00 INFO 139885740795712] Epoch: 4, batches: 900, num_examples: 57600, 1534.5 samples/sec, epoch time so far: 0:00:37.537659\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:00 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.114 mean_absolute_error: 0.254 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:04 INFO 139885740795712] Epoch: 4, batches: 1000, num_examples: 64000, 1536.5 samples/sec, epoch time so far: 0:00:41.652578\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:04 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.114 mean_absolute_error: 0.254 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:08 INFO 139885740795712] Epoch: 4, batches: 1100, num_examples: 70400, 1538.9 samples/sec, epoch time so far: 0:00:45.746582\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:08 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.114 mean_absolute_error: 0.254 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:12 INFO 139885740795712] Epoch: 4, batches: 1200, num_examples: 76800, 1538.6 samples/sec, epoch time so far: 0:00:49.915243\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:12 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.113 mean_absolute_error: 0.254 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:16 INFO 139885740795712] Epoch: 4, batches: 1300, num_examples: 83200, 1539.9 samples/sec, epoch time so far: 0:00:54.028422\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:16 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.113 mean_absolute_error: 0.254 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:21 INFO 139885740795712] Epoch: 4, batches: 1400, num_examples: 89600, 1542.0 samples/sec, epoch time so far: 0:00:58.107561\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:21 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.113 mean_absolute_error: 0.254 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:21 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:21 INFO 139885740795712] Completed Epoch: 4, time taken: 0:00:58.740027\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:21 INFO 139885740795712] Epoch 4 Training metrics:   mean_squared_error: 0.113 mean_absolute_error: 0.254 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:21 INFO 139885740795712] #quality_metric: host=algo-1, epoch=4, train mean_squared_error <loss>=0.112873401619\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:21 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:21 INFO 139885740795712] patience losses: [0.6599260709632588, 0.2746666442750201, 0.16236793772859426]\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:21 INFO 139885740795712] min patience losses: 0.162367937729\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:21 INFO 139885740795712] current loss: 0.112873401619\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:21 INFO 139885740795712] absolute loss difference: 0.0494945361095\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 4.255056381225586, \"sum\": 4.255056381225586, \"min\": 4.255056381225586}, \"update.time\": {\"count\": 1, \"max\": 58744.93217468262, \"sum\": 58744.93217468262, \"min\": 58744.93217468262}}, \"EndTime\": 1593565161.712657, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593565102.913912}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:21 INFO 139885740795712] #progress_metric: host=algo-1, completed 50 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Total Batches Seen\": {\"count\": 1, \"max\": 7080, \"sum\": 7080.0, \"min\": 7080}, \"Total Records Seen\": {\"count\": 1, \"max\": 453120, \"sum\": 453120.0, \"min\": 453120}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Reset Count\": {\"count\": 1, \"max\": 5, \"sum\": 5.0, \"min\": 5}}, \"EndTime\": 1593565161.712949, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 4}, \"StartTime\": 1593565102.967696}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:21 INFO 139885740795712] #throughput_metric: host=algo-1, train throughput=1542.65625075 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:25 INFO 139885740795712] Epoch: 5, batches: 100, num_examples: 6400, 1580.7 samples/sec, epoch time so far: 0:00:04.048922\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:25 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.095 mean_absolute_error: 0.237 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:29 INFO 139885740795712] Epoch: 5, batches: 200, num_examples: 12800, 1573.0 samples/sec, epoch time so far: 0:00:08.137078\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:29 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.091 mean_absolute_error: 0.233 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:34 INFO 139885740795712] Epoch: 5, batches: 300, num_examples: 19200, 1556.1 samples/sec, epoch time so far: 0:00:12.338562\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:34 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.089 mean_absolute_error: 0.230 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:38 INFO 139885740795712] Epoch: 5, batches: 400, num_examples: 25600, 1549.8 samples/sec, epoch time so far: 0:00:16.517819\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:38 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.087 mean_absolute_error: 0.228 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:42 INFO 139885740795712] Epoch: 5, batches: 500, num_examples: 32000, 1553.3 samples/sec, epoch time so far: 0:00:20.601106\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:42 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.087 mean_absolute_error: 0.227 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:46 INFO 139885740795712] Epoch: 5, batches: 600, num_examples: 38400, 1557.4 samples/sec, epoch time so far: 0:00:24.657130\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:46 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.087 mean_absolute_error: 0.227 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:50 INFO 139885740795712] Epoch: 5, batches: 700, num_examples: 44800, 1554.5 samples/sec, epoch time so far: 0:00:28.820304\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:50 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.087 mean_absolute_error: 0.227 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:54 INFO 139885740795712] Epoch: 5, batches: 800, num_examples: 51200, 1561.4 samples/sec, epoch time so far: 0:00:32.790994\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:54 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.087 mean_absolute_error: 0.227 \u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:58 INFO 139885740795712] Epoch: 5, batches: 900, num_examples: 57600, 1566.6 samples/sec, epoch time so far: 0:00:36.767683\u001b[0m\n",
      "\u001b[34m[07/01/2020 00:59:58 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.087 mean_absolute_error: 0.227 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:02 INFO 139885740795712] Epoch: 5, batches: 1000, num_examples: 64000, 1565.6 samples/sec, epoch time so far: 0:00:40.879632\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:02 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.087 mean_absolute_error: 0.227 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:06 INFO 139885740795712] Epoch: 5, batches: 1100, num_examples: 70400, 1566.0 samples/sec, epoch time so far: 0:00:44.956487\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:06 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.087 mean_absolute_error: 0.227 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:10 INFO 139885740795712] Epoch: 5, batches: 1200, num_examples: 76800, 1565.6 samples/sec, epoch time so far: 0:00:49.053357\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:10 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.087 mean_absolute_error: 0.228 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:14 INFO 139885740795712] Epoch: 5, batches: 1300, num_examples: 83200, 1566.8 samples/sec, epoch time so far: 0:00:53.101438\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:14 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.087 mean_absolute_error: 0.228 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:18 INFO 139885740795712] Epoch: 5, batches: 1400, num_examples: 89600, 1570.7 samples/sec, epoch time so far: 0:00:57.042988\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:18 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.087 mean_absolute_error: 0.228 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:19 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:19 INFO 139885740795712] Completed Epoch: 5, time taken: 0:00:57.742527\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:19 INFO 139885740795712] Epoch 5 Training metrics:   mean_squared_error: 0.087 mean_absolute_error: 0.229 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:19 INFO 139885740795712] #quality_metric: host=algo-1, epoch=5, train mean_squared_error <loss>=0.0874581936363\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:19 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:19 INFO 139885740795712] patience losses: [0.2746666442750201, 0.16236793772859426, 0.11287340161911512]\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:19 INFO 139885740795712] min patience losses: 0.112873401619\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:19 INFO 139885740795712] current loss: 0.0874581936363\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:19 INFO 139885740795712] absolute loss difference: 0.0254152079828\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 4.397869110107422, \"sum\": 4.397869110107422, \"min\": 4.397869110107422}, \"update.time\": {\"count\": 1, \"max\": 57747.52902984619, \"sum\": 57747.52902984619, \"min\": 57747.52902984619}}, \"EndTime\": 1593565219.507239, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593565161.712756}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:19 INFO 139885740795712] #progress_metric: host=algo-1, completed 60 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Total Batches Seen\": {\"count\": 1, \"max\": 8496, \"sum\": 8496.0, \"min\": 8496}, \"Total Records Seen\": {\"count\": 1, \"max\": 543744, \"sum\": 543744.0, \"min\": 543744}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Reset Count\": {\"count\": 1, \"max\": 6, \"sum\": 6.0, \"min\": 6}}, \"EndTime\": 1593565219.507553, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 5}, \"StartTime\": 1593565161.759676}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:19 INFO 139885740795712] #throughput_metric: host=algo-1, train throughput=1569.29754042 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:23 INFO 139885740795712] Epoch: 6, batches: 100, num_examples: 6400, 1542.5 samples/sec, epoch time so far: 0:00:04.149079\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:23 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.088 mean_absolute_error: 0.229 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:28 INFO 139885740795712] Epoch: 6, batches: 200, num_examples: 12800, 1503.5 samples/sec, epoch time so far: 0:00:08.513612\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:28 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.087 mean_absolute_error: 0.227 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:32 INFO 139885740795712] Epoch: 6, batches: 300, num_examples: 19200, 1474.3 samples/sec, epoch time so far: 0:00:13.023196\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:32 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.086 mean_absolute_error: 0.227 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:37 INFO 139885740795712] Epoch: 6, batches: 400, num_examples: 25600, 1463.3 samples/sec, epoch time so far: 0:00:17.494612\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:37 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.085 mean_absolute_error: 0.225 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:41 INFO 139885740795712] Epoch: 6, batches: 500, num_examples: 32000, 1473.2 samples/sec, epoch time so far: 0:00:21.721698\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:41 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.084 mean_absolute_error: 0.224 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:45 INFO 139885740795712] Epoch: 6, batches: 600, num_examples: 38400, 1486.1 samples/sec, epoch time so far: 0:00:25.839486\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:45 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.083 mean_absolute_error: 0.223 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:49 INFO 139885740795712] Epoch: 6, batches: 700, num_examples: 44800, 1502.4 samples/sec, epoch time so far: 0:00:29.818786\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:49 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.083 mean_absolute_error: 0.223 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:53 INFO 139885740795712] Epoch: 6, batches: 800, num_examples: 51200, 1515.0 samples/sec, epoch time so far: 0:00:33.795002\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:53 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.083 mean_absolute_error: 0.223 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:57 INFO 139885740795712] Epoch: 6, batches: 900, num_examples: 57600, 1521.9 samples/sec, epoch time so far: 0:00:37.848104\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:00:57 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.082 mean_absolute_error: 0.222 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:01 INFO 139885740795712] Epoch: 6, batches: 1000, num_examples: 64000, 1525.8 samples/sec, epoch time so far: 0:00:41.944711\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:01 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.082 mean_absolute_error: 0.222 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:05 INFO 139885740795712] Epoch: 6, batches: 1100, num_examples: 70400, 1528.9 samples/sec, epoch time so far: 0:00:46.047418\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:05 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.082 mean_absolute_error: 0.222 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:09 INFO 139885740795712] Epoch: 6, batches: 1200, num_examples: 76800, 1530.8 samples/sec, epoch time so far: 0:00:50.168903\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:09 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.082 mean_absolute_error: 0.221 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:13 INFO 139885740795712] Epoch: 6, batches: 1300, num_examples: 83200, 1531.3 samples/sec, epoch time so far: 0:00:54.333376\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:13 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.082 mean_absolute_error: 0.222 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:18 INFO 139885740795712] Epoch: 6, batches: 1400, num_examples: 89600, 1530.1 samples/sec, epoch time so far: 0:00:58.556414\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:18 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.082 mean_absolute_error: 0.222 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:18 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:18 INFO 139885740795712] Completed Epoch: 6, time taken: 0:00:59.217306\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:18 INFO 139885740795712] Epoch 6 Training metrics:   mean_squared_error: 0.082 mean_absolute_error: 0.222 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:18 INFO 139885740795712] #quality_metric: host=algo-1, epoch=6, train mean_squared_error <loss>=0.0820817179468\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:18 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:18 INFO 139885740795712] patience losses: [0.16236793772859426, 0.11287340161911512, 0.0874581936362731]\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:18 INFO 139885740795712] min patience losses: 0.0874581936363\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:18 INFO 139885740795712] current loss: 0.0820817179468\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:18 INFO 139885740795712] absolute loss difference: 0.00537647568952\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:18 INFO 139885740795712] Bad epoch: loss has not improved (enough). Bad count:1\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 3.4508705139160156, \"sum\": 3.4508705139160156, \"min\": 3.4508705139160156}, \"update.time\": {\"count\": 1, \"max\": 59221.59004211426, \"sum\": 59221.59004211426, \"min\": 59221.59004211426}}, \"EndTime\": 1593565278.772687, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593565219.507352}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:18 INFO 139885740795712] #progress_metric: host=algo-1, completed 70 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Total Batches Seen\": {\"count\": 1, \"max\": 9912, \"sum\": 9912.0, \"min\": 9912}, \"Total Records Seen\": {\"count\": 1, \"max\": 634368, \"sum\": 634368.0, \"min\": 634368}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Reset Count\": {\"count\": 1, \"max\": 7, \"sum\": 7.0, \"min\": 7}}, \"EndTime\": 1593565278.772961, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 6}, \"StartTime\": 1593565219.551063}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:18 INFO 139885740795712] #throughput_metric: host=algo-1, train throughput=1530.23942597 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:22 INFO 139885740795712] Epoch: 7, batches: 100, num_examples: 6400, 1541.3 samples/sec, epoch time so far: 0:00:04.152334\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:22 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.083 mean_absolute_error: 0.222 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:27 INFO 139885740795712] Epoch: 7, batches: 200, num_examples: 12800, 1534.5 samples/sec, epoch time so far: 0:00:08.341599\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:27 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.080 mean_absolute_error: 0.216 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:31 INFO 139885740795712] Epoch: 7, batches: 300, num_examples: 19200, 1530.2 samples/sec, epoch time so far: 0:00:12.547221\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:31 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.080 mean_absolute_error: 0.217 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:35 INFO 139885740795712] Epoch: 7, batches: 400, num_examples: 25600, 1529.6 samples/sec, epoch time so far: 0:00:16.736737\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:35 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.080 mean_absolute_error: 0.217 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:39 INFO 139885740795712] Epoch: 7, batches: 500, num_examples: 32000, 1530.2 samples/sec, epoch time so far: 0:00:20.911757\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:39 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.079 mean_absolute_error: 0.216 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:43 INFO 139885740795712] Epoch: 7, batches: 600, num_examples: 38400, 1535.5 samples/sec, epoch time so far: 0:00:25.008083\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:43 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.079 mean_absolute_error: 0.215 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:47 INFO 139885740795712] Epoch: 7, batches: 700, num_examples: 44800, 1544.1 samples/sec, epoch time so far: 0:00:29.013885\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:47 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.079 mean_absolute_error: 0.215 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:51 INFO 139885740795712] Epoch: 7, batches: 800, num_examples: 51200, 1550.7 samples/sec, epoch time so far: 0:00:33.017266\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:51 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.078 mean_absolute_error: 0.215 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:55 INFO 139885740795712] Epoch: 7, batches: 900, num_examples: 57600, 1554.5 samples/sec, epoch time so far: 0:00:37.052742\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:55 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.078 mean_absolute_error: 0.215 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:59 INFO 139885740795712] Epoch: 7, batches: 1000, num_examples: 64000, 1558.0 samples/sec, epoch time so far: 0:00:41.078374\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:01:59 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.078 mean_absolute_error: 0.215 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:04 INFO 139885740795712] Epoch: 7, batches: 1100, num_examples: 70400, 1557.1 samples/sec, epoch time so far: 0:00:45.212982\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:04 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.078 mean_absolute_error: 0.215 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:08 INFO 139885740795712] Epoch: 7, batches: 1200, num_examples: 76800, 1558.0 samples/sec, epoch time so far: 0:00:49.293976\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:08 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.078 mean_absolute_error: 0.215 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:12 INFO 139885740795712] Epoch: 7, batches: 1300, num_examples: 83200, 1558.0 samples/sec, epoch time so far: 0:00:53.402256\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:12 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.078 mean_absolute_error: 0.215 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:16 INFO 139885740795712] Epoch: 7, batches: 1400, num_examples: 89600, 1559.0 samples/sec, epoch time so far: 0:00:57.472554\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:16 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.078 mean_absolute_error: 0.215 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:16 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:16 INFO 139885740795712] Completed Epoch: 7, time taken: 0:00:58.118594\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:16 INFO 139885740795712] Epoch 7 Training metrics:   mean_squared_error: 0.078 mean_absolute_error: 0.215 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:16 INFO 139885740795712] #quality_metric: host=algo-1, epoch=7, train mean_squared_error <loss>=0.0778349056382\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:16 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:16 INFO 139885740795712] patience losses: [0.11287340161911512, 0.0874581936362731, 0.0820817179467508]\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:16 INFO 139885740795712] min patience losses: 0.0820817179468\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:16 INFO 139885740795712] current loss: 0.0778349056382\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:16 INFO 139885740795712] absolute loss difference: 0.00424681230853\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:16 INFO 139885740795712] Bad epoch: loss has not improved (enough). Bad count:2\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 5.325078964233398, \"sum\": 5.325078964233398, \"min\": 5.325078964233398}, \"update.time\": {\"count\": 1, \"max\": 58124.93181228638, \"sum\": 58124.93181228638, \"min\": 58124.93181228638}}, \"EndTime\": 1593565336.942338, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593565278.772792}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:16 INFO 139885740795712] #progress_metric: host=algo-1, completed 80 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Total Batches Seen\": {\"count\": 1, \"max\": 11328, \"sum\": 11328.0, \"min\": 11328}, \"Total Records Seen\": {\"count\": 1, \"max\": 724992, \"sum\": 724992.0, \"min\": 724992}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Reset Count\": {\"count\": 1, \"max\": 8, \"sum\": 8.0, \"min\": 8}}, \"EndTime\": 1593565336.942712, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 7}, \"StartTime\": 1593565278.81737}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:16 INFO 139885740795712] #throughput_metric: host=algo-1, train throughput=1559.10801241 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:21 INFO 139885740795712] Epoch: 8, batches: 100, num_examples: 6400, 1543.1 samples/sec, epoch time so far: 0:00:04.147588\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:21 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.070 mean_absolute_error: 0.203 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:25 INFO 139885740795712] Epoch: 8, batches: 200, num_examples: 12800, 1536.6 samples/sec, epoch time so far: 0:00:08.330169\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:25 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.068 mean_absolute_error: 0.199 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:29 INFO 139885740795712] Epoch: 8, batches: 300, num_examples: 19200, 1515.6 samples/sec, epoch time so far: 0:00:12.668092\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:29 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.067 mean_absolute_error: 0.198 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:34 INFO 139885740795712] Epoch: 8, batches: 400, num_examples: 25600, 1502.2 samples/sec, epoch time so far: 0:00:17.042059\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:34 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.068 mean_absolute_error: 0.199 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:38 INFO 139885740795712] Epoch: 8, batches: 500, num_examples: 32000, 1490.7 samples/sec, epoch time so far: 0:00:21.466522\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:38 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.067 mean_absolute_error: 0.198 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:42 INFO 139885740795712] Epoch: 8, batches: 600, num_examples: 38400, 1496.3 samples/sec, epoch time so far: 0:00:25.662472\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:42 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.067 mean_absolute_error: 0.198 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:46 INFO 139885740795712] Epoch: 8, batches: 700, num_examples: 44800, 1507.2 samples/sec, epoch time so far: 0:00:29.724628\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:46 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.066 mean_absolute_error: 0.197 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:50 INFO 139885740795712] Epoch: 8, batches: 800, num_examples: 51200, 1515.9 samples/sec, epoch time so far: 0:00:33.775844\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:50 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.066 mean_absolute_error: 0.197 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:54 INFO 139885740795712] Epoch: 8, batches: 900, num_examples: 57600, 1519.9 samples/sec, epoch time so far: 0:00:37.897124\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:54 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.066 mean_absolute_error: 0.197 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:58 INFO 139885740795712] Epoch: 8, batches: 1000, num_examples: 64000, 1524.5 samples/sec, epoch time so far: 0:00:41.981463\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:02:58 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.065 mean_absolute_error: 0.196 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:03 INFO 139885740795712] Epoch: 8, batches: 1100, num_examples: 70400, 1525.9 samples/sec, epoch time so far: 0:00:46.136312\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:03 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.065 mean_absolute_error: 0.196 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:07 INFO 139885740795712] Epoch: 8, batches: 1200, num_examples: 76800, 1528.7 samples/sec, epoch time so far: 0:00:50.239179\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:07 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.065 mean_absolute_error: 0.196 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:11 INFO 139885740795712] Epoch: 8, batches: 1300, num_examples: 83200, 1528.7 samples/sec, epoch time so far: 0:00:54.426354\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:11 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.065 mean_absolute_error: 0.196 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:15 INFO 139885740795712] Epoch: 8, batches: 1400, num_examples: 89600, 1529.2 samples/sec, epoch time so far: 0:00:58.592936\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:15 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.065 mean_absolute_error: 0.196 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:16 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:16 INFO 139885740795712] Completed Epoch: 8, time taken: 0:00:59.272671\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:16 INFO 139885740795712] Epoch 8 Training metrics:   mean_squared_error: 0.065 mean_absolute_error: 0.196 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:16 INFO 139885740795712] #quality_metric: host=algo-1, epoch=8, train mean_squared_error <loss>=0.065496226457\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:16 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:16 INFO 139885740795712] patience losses: [0.0874581936362731, 0.0820817179467508, 0.07783490563821742]\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:16 INFO 139885740795712] min patience losses: 0.0778349056382\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:16 INFO 139885740795712] current loss: 0.065496226457\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:16 INFO 139885740795712] absolute loss difference: 0.0123386791813\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 4.6901702880859375, \"sum\": 4.6901702880859375, \"min\": 4.6901702880859375}, \"update.time\": {\"count\": 1, \"max\": 59278.41114997864, \"sum\": 59278.41114997864, \"min\": 59278.41114997864}}, \"EndTime\": 1593565396.26577, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593565336.942441}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:16 INFO 139885740795712] #progress_metric: host=algo-1, completed 90 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Total Batches Seen\": {\"count\": 1, \"max\": 12744, \"sum\": 12744.0, \"min\": 12744}, \"Total Records Seen\": {\"count\": 1, \"max\": 815616, \"sum\": 815616.0, \"min\": 815616}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Reset Count\": {\"count\": 1, \"max\": 9, \"sum\": 9.0, \"min\": 9}}, \"EndTime\": 1593565396.266175, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 8}, \"StartTime\": 1593565336.987329}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:16 INFO 139885740795712] #throughput_metric: host=algo-1, train throughput=1528.75051054 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:20 INFO 139885740795712] Epoch: 9, batches: 100, num_examples: 6400, 1555.3 samples/sec, epoch time so far: 0:00:04.114854\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:20 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.063 mean_absolute_error: 0.190 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:24 INFO 139885740795712] Epoch: 9, batches: 200, num_examples: 12800, 1543.0 samples/sec, epoch time so far: 0:00:08.295459\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:24 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.060 mean_absolute_error: 0.186 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:28 INFO 139885740795712] Epoch: 9, batches: 300, num_examples: 19200, 1529.0 samples/sec, epoch time so far: 0:00:12.556855\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:28 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.058 mean_absolute_error: 0.183 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:33 INFO 139885740795712] Epoch: 9, batches: 400, num_examples: 25600, 1514.6 samples/sec, epoch time so far: 0:00:16.902335\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:33 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.057 mean_absolute_error: 0.182 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:37 INFO 139885740795712] Epoch: 9, batches: 500, num_examples: 32000, 1506.3 samples/sec, epoch time so far: 0:00:21.244146\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:37 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.057 mean_absolute_error: 0.183 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:41 INFO 139885740795712] Epoch: 9, batches: 600, num_examples: 38400, 1505.7 samples/sec, epoch time so far: 0:00:25.503579\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:41 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.057 mean_absolute_error: 0.182 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:45 INFO 139885740795712] Epoch: 9, batches: 700, num_examples: 44800, 1511.9 samples/sec, epoch time so far: 0:00:29.631848\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:45 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.057 mean_absolute_error: 0.182 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:50 INFO 139885740795712] Epoch: 9, batches: 800, num_examples: 51200, 1517.5 samples/sec, epoch time so far: 0:00:33.740122\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:50 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.057 mean_absolute_error: 0.182 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:54 INFO 139885740795712] Epoch: 9, batches: 900, num_examples: 57600, 1525.5 samples/sec, epoch time so far: 0:00:37.757156\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:54 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.057 mean_absolute_error: 0.182 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:58 INFO 139885740795712] Epoch: 9, batches: 1000, num_examples: 64000, 1531.6 samples/sec, epoch time so far: 0:00:41.785799\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:03:58 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.057 mean_absolute_error: 0.182 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:02 INFO 139885740795712] Epoch: 9, batches: 1100, num_examples: 70400, 1531.3 samples/sec, epoch time so far: 0:00:45.974306\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:02 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.056 mean_absolute_error: 0.181 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:06 INFO 139885740795712] Epoch: 9, batches: 1200, num_examples: 76800, 1533.8 samples/sec, epoch time so far: 0:00:50.070239\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:06 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.056 mean_absolute_error: 0.181 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:10 INFO 139885740795712] Epoch: 9, batches: 1300, num_examples: 83200, 1538.8 samples/sec, epoch time so far: 0:00:54.067838\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:10 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.057 mean_absolute_error: 0.182 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:14 INFO 139885740795712] Epoch: 9, batches: 1400, num_examples: 89600, 1543.4 samples/sec, epoch time so far: 0:00:58.053093\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:14 INFO 139885740795712] #011Training metrics: mean_squared_error: 0.057 mean_absolute_error: 0.182 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] Completed Epoch: 9, time taken: 0:00:58.711457\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] Epoch 9 Training metrics:   mean_squared_error: 0.057 mean_absolute_error: 0.182 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] #quality_metric: host=algo-1, epoch=9, train mean_squared_error <loss>=0.0566795017454\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] patience losses: [0.0820817179467508, 0.07783490563821742, 0.06549622645696342]\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] min patience losses: 0.065496226457\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] current loss: 0.0566795017454\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] absolute loss difference: 0.00881672471151\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] Bad epoch: loss has not improved (enough). Bad count:1\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 4.94384765625, \"sum\": 4.94384765625, \"min\": 4.94384765625}, \"update.time\": {\"count\": 1, \"max\": 58716.989040374756, \"sum\": 58716.989040374756, \"min\": 58716.989040374756}}, \"EndTime\": 1593565455.026671, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593565396.265901}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] #progress_metric: host=algo-1, completed 100 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 1416, \"sum\": 1416.0, \"min\": 1416}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Total Batches Seen\": {\"count\": 1, \"max\": 14160, \"sum\": 14160.0, \"min\": 14160}, \"Total Records Seen\": {\"count\": 1, \"max\": 906240, \"sum\": 906240.0, \"min\": 906240}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90624, \"sum\": 90624.0, \"min\": 90624}, \"Reset Count\": {\"count\": 1, \"max\": 10, \"sum\": 10.0, \"min\": 10}}, \"EndTime\": 1593565455.026969, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 9}, \"StartTime\": 1593565396.309654}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] #throughput_metric: host=algo-1, train throughput=1543.3905377 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 WARNING 139885740795712] wait_for_all_workers will not sync workers since the kv store is not running distributed\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] Best model based on epoch 9. Best loss: 0.057\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"finalize.time\": {\"count\": 1, \"max\": 1.5740394592285156, \"sum\": 1.5740394592285156, \"min\": 1.5740394592285156}}, \"EndTime\": 1593565455.028921, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593565455.026795}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] Serializing model to /opt/ml/model/model_algo-1\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] Saved checkpoint to \"/tmp/tmpdUoYcM/state-0001.params\"\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python2.7/site-packages/mxnet/module/module.py:865: UserWarning: embed_1_weight.stype is not 'row_sparse'. No need to perform row_sparse_pull.\n",
      "  \"perform row_sparse_pull.\" % param_name))\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python2.7/site-packages/mxnet/module/module.py:865: UserWarning: embed_0_weight.stype is not 'row_sparse'. No need to perform row_sparse_pull.\n",
      "  \"perform row_sparse_pull.\" % param_name))\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] Finished scoring on 9472 examples from 148 batches, each of size 64.\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] Test Metrics:mean_squared_error: 0.924 mean_absolute_error: 0.756 \u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] Test Metric names:['mean_squared_error', 'mean_absolute_error']\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] Test Metric values:[0.9240536717949687, 0.756318515216982]\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] Test Metric names and values:[('mean_squared_error', 0.9240536717949687), ('mean_absolute_error', 0.756318515216982)]\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 148, \"sum\": 148.0, \"min\": 148}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 148, \"sum\": 148.0, \"min\": 148}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 9472, \"sum\": 9472.0, \"min\": 9472}, \"Total Batches Seen\": {\"count\": 1, \"max\": 148, \"sum\": 148.0, \"min\": 148}, \"Total Records Seen\": {\"count\": 1, \"max\": 9472, \"sum\": 9472.0, \"min\": 9472}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 9472, \"sum\": 9472.0, \"min\": 9472}, \"Reset Count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1593565455.921176, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"test_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593565455.153117}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] #test_score (algo-1) : ('mean_squared_error', 0.9240536717949687)\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] #test_score (algo-1) : ('mean_absolute_error', 0.756318515216982)\u001b[0m\n",
      "\u001b[34m[07/01/2020 01:04:15 INFO 139885740795712] #quality_metric: host=algo-1, test mean_squared_error <loss>=0.924053671795\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"totaltime\": {\"count\": 1, \"max\": 582383.5048675537, \"sum\": 582383.5048675537, \"min\": 582383.5048675537}, \"model.score.time\": {\"count\": 1, \"max\": 767.9738998413086, \"sum\": 767.9738998413086, \"min\": 767.9738998413086}, \"model.serialize.time\": {\"count\": 1, \"max\": 123.72803688049316, \"sum\": 123.72803688049316, \"min\": 123.72803688049316}, \"setuptime\": {\"count\": 1, \"max\": 267.4379348754883, \"sum\": 267.4379348754883, \"min\": 267.4379348754883}}, \"EndTime\": 1593565455.92522, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593565455.029042}\n",
      "\u001b[0m\n",
      "\n",
      "2020-07-01 01:04:43 Uploading - Uploading generated training model\n",
      "2020-07-01 01:04:43 Completed - Training job completed\n",
      "Training seconds: 672\n",
      "Billable seconds: 672\n"
     ]
    }
   ],
   "source": [
    "regressor = sagemaker.estimator.Estimator(container,\n",
    "                                          role, \n",
    "                                          train_instance_count=1, \n",
    "                                          train_instance_type='ml.m4.xlarge',\n",
    "                                          output_path=s3_output_path,\n",
    "                                          sagemaker_session=sess)\n",
    "\n",
    "## set hyperparameters\n",
    "regressor.set_hyperparameters(**hyperparameters)\n",
    "\n",
    "## train the model\n",
    "data_channels = {\n",
    "    \"train\": \"{}/train/\".format(s3_data_path),\n",
    "    \"test\": \"{}/validation/\".format(s3_data_path)\n",
    "}\n",
    "\n",
    "regressor.fit(inputs=data_channels, wait=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now our model data will start training with the given algorithm and data  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But, what if we want to send additional unlabeled data to the algorithm and get predictions from the trained model? This step is called **inference** in the Sagemaker. Next, we demonstrate how to use a trained model to perform inference on unseen data points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Deploy the model (Inference)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker:Parameter image will be renamed to image_uri in SageMaker Python SDK v2.\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.predictor import json_serializer, json_deserializer\n",
    "\n",
    "# create a model using the trained algorithm\n",
    "regression_model = regressor.create_model(\n",
    "                        serializer=json_serializer,\n",
    "                        deserializer=json_deserializer,\n",
    "                        content_type='application/json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------!"
     ]
    }
   ],
   "source": [
    "# deploy the model\n",
    "predictor = regression_model.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we send validation data (without labels) to the deployed endpoint for inference. We will see that the resulting prediction error we get from post-training inference matches the best validation error from the training log in the console above (up to floating point error). If you follow the training instruction and parameter setup, you should get mean squared error on the validation set approximately 0.91."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The mean squared error on validation set is 0.930\n"
     ]
    }
   ],
   "source": [
    "# Send data to the endpoint to get predictions\n",
    "prediction = predictor.predict(valid_r_data)\n",
    "\n",
    "print(\"The mean squared error on validation set is %.3f\" %get_mse_loss(prediction, valid_r_label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Predicting and evaluating with the test set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we showcase how to use Object2Vec to recommend movies, using the binarized rating labels. Here, if a movie rating label for a given user is binarized to 1, then it means that the movie should be recommended to the user; otherwise, the label is binarized to 0. The binarized data set is already obtained in the preprocessing section, so we will proceed to apply the algorithm.\n",
    "\n",
    "We upload the binarized datasets for classification task to S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading file to s3://sagemaker-us-west-2-570447867175/recomm-object2vec/data/train/train_c.jsonl\n",
      "Uploading file to s3://sagemaker-us-west-2-570447867175/recomm-object2vec/data/validation/validation_c.jsonl\n",
      "Wall time: 4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "copy_to_s3(\"./data/ml-100k/train_c.jsonl\", s3_data_path + \"/train/train_c.jsonl\")\n",
    "copy_to_s3(\"./data/ml-100k/validation_c.jsonl\", s3_data_path + \"/validation/validation_c.jsonl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Below steps resembles above training steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'s3://sagemaker-us-west-2-570447867175/recomm-object2vec/data'"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s3_data_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.\n",
      "WARNING:sagemaker:'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n",
      "WARNING:sagemaker:'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-07-01 03:02:43 Starting - Starting the training job...\n",
      "2020-07-01 03:02:46 Starting - Launching requested ML instances......\n",
      "2020-07-01 03:03:55 Starting - Preparing the instances for training......\n",
      "2020-07-01 03:05:15 Downloading - Downloading input data\n",
      "2020-07-01 03:05:15 Training - Downloading the training image......\n",
      "2020-07-01 03:06:08 Training - Training image download completed. Training in progress.\u001b[34mDocker entrypoint called with argument(s): train\u001b[0m\n",
      "\u001b[34mRunning default environment configuration script\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:13 INFO 140313708324672] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/default-input.json: {u'comparator_list': u'hadamard, concat, abs_diff', u'enc0_vocab_file': u'', u'output_layer': u'softmax', u'enc0_cnn_filter_width': 3, u'epochs': 30, u'mlp_dim': 512, u'enc0_freeze_pretrained_embedding': u'true', u'mlp_layers': 2, u'_num_kv_servers': u'auto', u'weight_decay': 0, u'enc0_pretrained_embedding_file': u'', u'token_embedding_storage_type': u'dense', u'enc0_token_embedding_dim': 300, u'tied_token_embedding_weight': u'false', u'learning_rate': 0.0004, u'enc1_cnn_filter_width': 3, u'negative_sampling_rate': 0, u'enc0_network': u'hcnn', u'enc1_layers': u'auto', u'early_stopping_patience': 3, u'optimizer': u'adam', u'_tuning_objective_metric': u'', u'early_stopping_tolerance': 0.01, u'dropout': 0, u'bucket_width': 0, u'enc_dim': 4096, u'enc1_vocab_file': u'', u'enc1_freeze_pretrained_embedding': u'true', u'enc0_layers': u'auto', u'mini_batch_size': 32, u'enc1_pretrained_embedding_file': u'', u'num_classes': 2, u'_num_gpus': u'auto', u'enc1_token_embedding_dim': 300, u'mlp_activation': u'linear', u'enc1_network': u'enc0', u'_kvstore': u'auto_gpu'}\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:13 INFO 140313708324672] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'output_layer': u'softmax', u'enc0_cnn_filter_width': u'3', u'epochs': u'10', u'mlp_dim': u'1024', u'mlp_layers': u'1', u'_num_kv_servers': u'auto', u'mini_batch_size': u'2048', u'enc0_token_embedding_dim': u'300', u'enc1_network': u'pooled_embedding', u'early_stopping_tolerance': u'0.01', u'enc0_network': u'pooled_embedding', u'enc1_layers': u'auto', u'early_stopping_patience': u'3', u'optimizer': u'adam', u'enc1_max_seq_len': u'1', u'enc1_cnn_filter_width': u'3', u'learning_rate': u'0.001', u'bucket_width': u'0', u'enc_dim': u'2048', u'enc0_layers': u'auto', u'enc0_max_seq_len': u'1', u'num_classes': u'2', u'_num_gpus': u'auto', u'mlp_activation': u'relu', u'enc1_token_embedding_dim': u'300', u'_kvstore': u'device', u'enc1_vocab_size': u'1684', u'enc0_vocab_size': u'944'}\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:13 INFO 140313708324672] Final configuration: {u'comparator_list': u'hadamard, concat, abs_diff', u'enc0_vocab_file': u'', u'output_layer': u'softmax', u'enc0_cnn_filter_width': u'3', u'epochs': u'10', u'mlp_dim': u'1024', u'enc0_freeze_pretrained_embedding': u'true', u'mlp_layers': u'1', u'_num_kv_servers': u'auto', u'weight_decay': 0, u'enc0_pretrained_embedding_file': u'', u'enc0_max_seq_len': u'1', u'token_embedding_storage_type': u'dense', u'enc0_token_embedding_dim': u'300', u'tied_token_embedding_weight': u'false', u'learning_rate': u'0.001', u'enc1_cnn_filter_width': u'3', u'negative_sampling_rate': 0, u'enc0_network': u'pooled_embedding', u'enc1_layers': u'auto', u'early_stopping_patience': u'3', u'optimizer': u'adam', u'enc1_max_seq_len': u'1', u'_tuning_objective_metric': u'', u'early_stopping_tolerance': u'0.01', u'dropout': 0, u'bucket_width': u'0', u'enc_dim': u'2048', u'enc1_vocab_file': u'', u'enc1_freeze_pretrained_embedding': u'true', u'enc0_layers': u'auto', u'mini_batch_size': u'2048', u'enc1_pretrained_embedding_file': u'', u'num_classes': u'2', u'_num_gpus': u'auto', u'enc1_token_embedding_dim': u'300', u'mlp_activation': u'relu', u'enc1_network': u'pooled_embedding', u'_kvstore': u'device', u'enc1_vocab_size': u'1684', u'enc0_vocab_size': u'944'}\u001b[0m\n",
      "\u001b[34mProcess 1 is a worker.\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:13 INFO 140313708324672] Using default worker.\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:13 INFO 140313708324672] Loaded iterator creator application/jsonlines for content type ('application/jsonlines', '1.0')\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:13 INFO 140313708324672] create_iter params {u'comparator_list': u'hadamard, concat, abs_diff', u'enc0_vocab_file': u'', u'output_layer': u'softmax', u'enc0_cnn_filter_width': u'3', u'epochs': u'10', u'mlp_dim': u'1024', u'enc0_freeze_pretrained_embedding': u'true', u'mlp_layers': u'1', u'_num_kv_servers': u'auto', u'weight_decay': 0, u'enc0_pretrained_embedding_file': u'', u'enc0_max_seq_len': u'1', u'token_embedding_storage_type': u'dense', u'enc0_token_embedding_dim': u'300', u'tied_token_embedding_weight': u'false', u'learning_rate': u'0.001', u'enc1_cnn_filter_width': u'3', u'negative_sampling_rate': 0, u'enc0_network': u'pooled_embedding', u'enc1_layers': u'auto', u'early_stopping_patience': u'3', u'optimizer': u'adam', u'enc1_max_seq_len': u'1', u'_tuning_objective_metric': u'', u'early_stopping_tolerance': u'0.01', u'dropout': 0, u'bucket_width': u'0', u'enc_dim': u'2048', u'enc1_vocab_file': u'', u'enc1_freeze_pretrained_embedding': u'true', u'enc0_layers': u'auto', u'mini_batch_size': u'2048', u'enc1_pretrained_embedding_file': u'', u'num_classes': u'2', u'_num_gpus': u'auto', u'enc1_token_embedding_dim': u'300', u'mlp_activation': u'relu', u'enc1_network': u'pooled_embedding', u'_kvstore': u'device', u'enc1_vocab_size': u'1684', u'enc0_vocab_size': u'944'}\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:13 INFO 140313708324672] create_iter content_params {}\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:13 INFO 140313708324672] Parameters of encoders: [{u'layers': u'auto', u'vocab_size': u'944', u'network': u'pooled_embedding', u'vocab_file': u'', u'cnn_filter_width': u'3', u'token_embedding_dim': u'300', u'max_seq_len': u'1', u'freeze_pretrained_embedding': u'true', u'pretrained_embedding_file': u''}, {u'layers': u'auto', u'vocab_size': u'1684', u'network': u'pooled_embedding', u'vocab_file': u'', u'cnn_filter_width': u'3', u'token_embedding_dim': u'300', u'max_seq_len': u'1', u'freeze_pretrained_embedding': u'true', u'pretrained_embedding_file': u''}]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:13 INFO 140313708324672] Inconsistent enc_dim and token_embedding_dim found for token-embedding-encoder: 2048 vs 300. Setting token embedding dim to be 2048\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:13 INFO 140313708324672] Inconsistent enc_dim and token_embedding_dim found for token-embedding-encoder: 2048 vs 300. Setting token embedding dim to be 2048\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:13 INFO 140313708324672] Encoder configs: [{'vocab_size': 944, 'enc_index': 0, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 2048, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}, {'vocab_size': 1684, 'enc_index': 1, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 2048, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:13 INFO 140313708324672] Config: {'comparator_list': ['hadamard', 'concat', 'abs_diff'], 'epochs': 10, 'mini_batch_size': 2048, 'optimizer': 'adam', 'output_layer': 'softmax', 'token_embedding_storage_type': 'dense', 'mlp_dim': 1024, 'early_stopping_tolerance': 0.01, 'dropout': 0.0, 'bucket_width': 0, 'enc_dim': 2048, 'negative_sampling_rate': 0, 'early_stopping_patience': 3, 'learning_rate': 0.001, 'max_seq_lens': [1, 1], 'tied_token_embedding_weight': False, 'enc_configs': [{'vocab_size': 944, 'enc_index': 0, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 2048, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}, {'vocab_size': 1684, 'enc_index': 1, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 2048, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}], 'mlp_layers': 1, 'weight_decay': 0.0, 'mlp_activation': 'relu', 'num_classes': 2}\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:13 INFO 140313708324672] use bucketing: False\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:13 INFO 140313708324672] Creating data iterator for /opt/ml/input/data/train\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:15 INFO 140313708324672] Source words: 90570\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:15 INFO 140313708324672] Target words: 90570\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:15 INFO 140313708324672] Total: 90570 samples in 1 buckets\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:15 INFO 140313708324672] Bucket of (1, 1) : 90570 samples in 44 batches of 2048, approx 2048.0 words/batch\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:15 INFO 140313708324672] 0 sentence pairs discarded\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:15 INFO 140313708324672] fill up mode: replicate\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:15 INFO 140313708324672] \u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:15 INFO 140313708324672] Negative sampling not used\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Replicating 1590 random sentences from bucket (1, 1) to size it to multiple of 2048\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Bucket batch sizes: [BucketBatchSize(batch_size=2048, average_words_per_batch=2048)]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] create_iter params {u'comparator_list': u'hadamard, concat, abs_diff', u'enc0_vocab_file': u'', u'output_layer': u'softmax', u'enc0_cnn_filter_width': u'3', u'epochs': u'10', u'mlp_dim': u'1024', u'enc0_freeze_pretrained_embedding': u'true', u'mlp_layers': u'1', u'_num_kv_servers': u'auto', u'weight_decay': 0, u'enc0_pretrained_embedding_file': u'', u'enc0_max_seq_len': u'1', u'token_embedding_storage_type': u'dense', u'enc0_token_embedding_dim': u'300', u'tied_token_embedding_weight': u'false', u'learning_rate': u'0.001', u'enc1_cnn_filter_width': u'3', u'negative_sampling_rate': 0, u'enc0_network': u'pooled_embedding', u'enc1_layers': u'auto', u'early_stopping_patience': u'3', u'optimizer': u'adam', u'enc1_max_seq_len': u'1', u'_tuning_objective_metric': u'', u'early_stopping_tolerance': u'0.01', u'dropout': 0, u'bucket_width': u'0', u'enc_dim': u'2048', u'enc1_vocab_file': u'', u'enc1_freeze_pretrained_embedding': u'true', u'enc0_layers': u'auto', u'mini_batch_size': u'2048', u'enc1_pretrained_embedding_file': u'', u'num_classes': u'2', u'_num_gpus': u'auto', u'enc1_token_embedding_dim': u'300', u'mlp_activation': u'relu', u'enc1_network': u'pooled_embedding', u'_kvstore': u'device', u'enc1_vocab_size': u'1684', u'enc0_vocab_size': u'944'}\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] create_iter content_params {}\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Parameters of encoders: [{u'layers': u'auto', u'vocab_size': u'944', u'network': u'pooled_embedding', u'vocab_file': u'', u'cnn_filter_width': u'3', u'token_embedding_dim': u'300', u'max_seq_len': u'1', u'freeze_pretrained_embedding': u'true', u'pretrained_embedding_file': u''}, {u'layers': u'auto', u'vocab_size': u'1684', u'network': u'pooled_embedding', u'vocab_file': u'', u'cnn_filter_width': u'3', u'token_embedding_dim': u'300', u'max_seq_len': u'1', u'freeze_pretrained_embedding': u'true', u'pretrained_embedding_file': u''}]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Inconsistent enc_dim and token_embedding_dim found for token-embedding-encoder: 2048 vs 300. Setting token embedding dim to be 2048\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Inconsistent enc_dim and token_embedding_dim found for token-embedding-encoder: 2048 vs 300. Setting token embedding dim to be 2048\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Encoder configs: [{'vocab_size': 944, 'enc_index': 0, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 2048, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}, {'vocab_size': 1684, 'enc_index': 1, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 2048, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Config: {'comparator_list': ['hadamard', 'concat', 'abs_diff'], 'epochs': 10, 'mini_batch_size': 2048, 'optimizer': 'adam', 'output_layer': 'softmax', 'token_embedding_storage_type': 'dense', 'mlp_dim': 1024, 'early_stopping_tolerance': 0.01, 'dropout': 0.0, 'bucket_width': 0, 'enc_dim': 2048, 'negative_sampling_rate': 0, 'early_stopping_patience': 3, 'learning_rate': 0.001, 'max_seq_lens': [1, 1], 'tied_token_embedding_weight': False, 'enc_configs': [{'vocab_size': 944, 'enc_index': 0, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 2048, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}, {'vocab_size': 1684, 'enc_index': 1, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 2048, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}], 'mlp_layers': 1, 'weight_decay': 0.0, 'mlp_activation': 'relu', 'num_classes': 2}\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] use bucketing: False\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Creating data iterator for /opt/ml/input/data/test\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Source words: 9430\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Target words: 9430\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Total: 9430 samples in 1 buckets\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Bucket of (1, 1) : 9430 samples in 4 batches of 2048, approx 2048.0 words/batch\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] 0 sentence pairs discarded\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] fill up mode: replicate\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] \u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Negative sampling not used\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Replicating 810 random sentences from bucket (1, 1) to size it to multiple of 2048\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Parameters of encoders: [{u'layers': u'auto', u'vocab_size': u'944', u'network': u'pooled_embedding', u'vocab_file': u'', u'cnn_filter_width': u'3', u'token_embedding_dim': u'300', u'max_seq_len': u'1', u'freeze_pretrained_embedding': u'true', u'pretrained_embedding_file': u''}, {u'layers': u'auto', u'vocab_size': u'1684', u'network': u'pooled_embedding', u'vocab_file': u'', u'cnn_filter_width': u'3', u'token_embedding_dim': u'300', u'max_seq_len': u'1', u'freeze_pretrained_embedding': u'true', u'pretrained_embedding_file': u''}]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Inconsistent enc_dim and token_embedding_dim found for token-embedding-encoder: 2048 vs 300. Setting token embedding dim to be 2048\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Inconsistent enc_dim and token_embedding_dim found for token-embedding-encoder: 2048 vs 300. Setting token embedding dim to be 2048\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Encoder configs: [{'vocab_size': 944, 'enc_index': 0, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 2048, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}, {'vocab_size': 1684, 'enc_index': 1, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 2048, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Config: {'comparator_list': ['hadamard', 'concat', 'abs_diff'], 'epochs': 10, 'mini_batch_size': 2048, 'optimizer': 'adam', 'output_layer': 'softmax', 'token_embedding_storage_type': 'dense', 'mlp_dim': 1024, 'early_stopping_tolerance': 0.01, 'dropout': 0.0, 'bucket_width': 0, 'enc_dim': 2048, 'negative_sampling_rate': 0, 'early_stopping_patience': 3, 'learning_rate': 0.001, 'max_seq_lens': [1, 1], 'tied_token_embedding_weight': False, 'enc_configs': [{'vocab_size': 944, 'enc_index': 0, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 2048, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}, {'vocab_size': 1684, 'enc_index': 1, 'pretrained_embedding_file_path': None, 'vocab_dict': None, 'token_embedding_dim': 2048, 'vocab_file': '', 'freeze_pretrained_embedding': True, 'num_layers': 1, 'is_train': True, 'dropout': 0.0, 'pretrained_embedding_file': ''}], 'mlp_layers': 1, 'weight_decay': 0.0, 'mlp_activation': 'relu', 'num_classes': 2}\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Creating new state\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] params {u'comparator_list': u'hadamard, concat, abs_diff', u'enc0_vocab_file': u'', u'output_layer': u'softmax', u'enc0_cnn_filter_width': u'3', u'epochs': u'10', u'mlp_dim': u'1024', u'enc0_freeze_pretrained_embedding': u'true', u'mlp_layers': u'1', u'_num_kv_servers': u'auto', u'weight_decay': 0, u'enc0_pretrained_embedding_file': u'', u'enc0_max_seq_len': u'1', u'token_embedding_storage_type': u'dense', u'enc0_token_embedding_dim': u'300', u'tied_token_embedding_weight': u'false', u'learning_rate': u'0.001', u'enc1_cnn_filter_width': u'3', u'negative_sampling_rate': 0, u'enc0_network': u'pooled_embedding', u'enc1_layers': u'auto', u'early_stopping_patience': u'3', u'optimizer': u'adam', u'enc1_max_seq_len': u'1', u'_tuning_objective_metric': u'', u'early_stopping_tolerance': u'0.01', u'dropout': 0, u'bucket_width': u'0', u'enc_dim': u'2048', 'default_bucket_key': (1, 1), u'enc1_vocab_file': u'', u'enc1_freeze_pretrained_embedding': u'true', u'enc0_layers': u'auto', u'mini_batch_size': u'2048', u'enc1_pretrained_embedding_file': u'', u'num_classes': u'2', u'_num_gpus': u'auto', u'enc1_token_embedding_dim': u'300', u'mlp_activation': u'relu', u'enc1_network': u'pooled_embedding', u'_kvstore': u'device', u'enc1_vocab_size': u'1684', u'enc0_vocab_size': u'944'}\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] default_bucket_key (1, 1)\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] nvidia-smi took: 0.0252258777618 secs to identify 1 gpus\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Number of GPUs being used: 1\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] context [gpu(0)]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Create Store: device\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Number of GPUs being used: 1\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 WARNING 140313708324672] dense token embedding is used in a multi-gpu setting...consider changing 'token_embedding_storage_type' to 'row_sparse'\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] data_names: ['source', 'target']\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] label_names: ['out_layer_label']\u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mLayer (type)                                        Output Shape            Param #     Previous Layer                  \u001b[0m\n",
      "\u001b[34m========================================================================================================================\u001b[0m\n",
      "\u001b[34msource(null)                                        1                       0                                           \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34membed_0(Embedding)                                  1x2048                  0           source                          \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m_not_equal_scalar0(_not_equal_scalar)               1                       0           source                          \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mreshape0(Reshape)                                   1x1                     0           _not_equal_scalar0              \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mbroadcast_mul0(broadcast_mul)                       1x2048                  0           embed_0                         \n",
      "                                                                                        reshape0                        \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34msum0(sum)                                           2048                    0           broadcast_mul0                  \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34msum1(sum)                                           1                       0           reshape0                        \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mzeros_like0(zeros_like)                             1                       0           sum1                            \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m_equal0(_equal)                                     1                       0           sum1                            \n",
      "                                                                                        zeros_like0                     \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m_plus0(elemwise_add)                                1                       0           sum1                            \n",
      "                                                                                        _equal0                         \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mbroadcast_div0(broadcast_div)                       2048                    0           sum0                            \n",
      "                                                                                        _plus0                          \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mdropout0(Dropout)                                   2048                    0           broadcast_div0                  \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34membed_1(Embedding)                                  1x2048                  0                                           \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m_not_equal_scalar1(_not_equal_scalar)               1                       0                                           \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mreshape1(Reshape)                                   1x1                     0           _not_equal_scalar1              \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mbroadcast_mul1(broadcast_mul)                       1x2048                  0           embed_1                         \n",
      "                                                                                        reshape1                        \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34msum2(sum)                                           2048                    0           broadcast_mul1                  \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34msum3(sum)                                           1                       0           reshape1                        \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mzeros_like1(zeros_like)                             1                       0           sum3                            \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m_equal1(_equal)                                     1                       0           sum3                            \n",
      "                                                                                        zeros_like1                     \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m_plus1(elemwise_add)                                1                       0           sum3                            \n",
      "                                                                                        _equal1                         \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mbroadcast_div1(broadcast_div)                       2048                    0           sum2                            \n",
      "                                                                                        _plus1                          \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mdropout1(Dropout)                                   2048                    0           broadcast_div1                  \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m_mul0(elemwise_mul)                                 2048                    0           dropout0                        \n",
      "                                                                                        dropout1                        \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m_minus0(elemwise_sub)                               2048                    0           dropout0                        \n",
      "                                                                                        dropout1                        \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mabs0(abs)                                           2048                    0           _minus0                         \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mconcat0(Concat)                                     8192                    0           _mul0                           \n",
      "                                                                                        dropout0                        \n",
      "                                                                                        dropout1                        \n",
      "                                                                                        abs0                            \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mmlp_fc0(FullyConnected)                             1024                    8389632     concat0                         \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mactivation0(Activation)                             1024                    0           mlp_fc0                         \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mdropout2(Dropout)                                   1024                    0           activation0                     \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34moutput_layer(FullyConnected)                        2                       2050        dropout2                        \u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34mout_layer(SoftmaxOutput)                            2                       0           output_layer                    \u001b[0m\n",
      "\u001b[34m========================================================================================================================\u001b[0m\n",
      "\u001b[34mTotal params: 8391682\u001b[0m\n",
      "\u001b[34m________________________________________________________________________________________________________________________\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] data_shapes [DataDesc[source,(2048, 1),<type 'numpy.float32'>,NTC], DataDesc[target,(2048, 1),<type 'numpy.float32'>,NTC]]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] label_shapes [DataDesc[out_layer_label,(2048,),<type 'numpy.float32'>,NTC]]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] fixed_param_names: []\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:16 INFO 140313708324672] Initialized BucketingPlus Module\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:21 INFO 140313708324672] arg_params keys for module initialization: []\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:21 INFO 140313708324672] all params:['output_layer_weight', 'mlp_fc0_weight', 'mlp_fc0_bias', 'output_layer_bias', 'embed_1_weight', 'embed_0_weight']\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"initialize.time\": {\"count\": 1, \"max\": 5602.940797805786, \"sum\": 5602.940797805786, \"min\": 5602.940797805786}}, \"EndTime\": 1593572781.959609, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593572773.291883}\n",
      "\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Records Seen\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Reset Count\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}}, \"EndTime\": 1593572781.959807, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"init_train_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593572781.959756}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:26 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:26 INFO 140313708324672] Completed Epoch: 0, time taken: 0:00:04.187507\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:26 INFO 140313708324672] Epoch 0 Training metrics:   perplexity: 1.859 cross_entropy: 0.620 accuracy: 0.650 \u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:26 INFO 140313708324672] #quality_metric: host=algo-1, epoch=0, train cross_entropy <loss>=0.62012501955\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:26 INFO 140313708324672] #quality_metric: host=algo-1, epoch=0, train accuracy <score>=0.650336371528\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:26 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"epochs\": {\"count\": 1, \"max\": 10, \"sum\": 10.0, \"min\": 10}, \"early_stop.time\": {\"count\": 1, \"max\": 0.5888938903808594, \"sum\": 0.5888938903808594, \"min\": 0.5888938903808594}, \"update.time\": {\"count\": 1, \"max\": 4192.384958267212, \"sum\": 4192.384958267212, \"min\": 4192.384958267212}}, \"EndTime\": 1593572786.381406, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593572781.959706}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:26 INFO 140313708324672] #progress_metric: host=algo-1, completed 10 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Total Batches Seen\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Total Records Seen\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Reset Count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1593572786.381791, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 0}, \"StartTime\": 1593572782.18899}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:26 INFO 140313708324672] #throughput_metric: host=algo-1, train throughput=21979.6154439 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:30 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:30 INFO 140313708324672] Completed Epoch: 1, time taken: 0:00:04.508876\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:30 INFO 140313708324672] Epoch 1 Training metrics:   perplexity: 1.726 cross_entropy: 0.546 accuracy: 0.721 \u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:30 INFO 140313708324672] #quality_metric: host=algo-1, epoch=1, train cross_entropy <loss>=0.545878766643\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:30 INFO 140313708324672] #quality_metric: host=algo-1, epoch=1, train accuracy <score>=0.721180555556\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:30 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 4.152059555053711, \"sum\": 4.152059555053711, \"min\": 4.152059555053711}, \"update.time\": {\"count\": 1, \"max\": 4513.812780380249, \"sum\": 4513.812780380249, \"min\": 4513.812780380249}}, \"EndTime\": 1593572790.976997, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593572786.381536}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:30 INFO 140313708324672] #progress_metric: host=algo-1, completed 20 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Total Batches Seen\": {\"count\": 1, \"max\": 90, \"sum\": 90.0, \"min\": 90}, \"Total Records Seen\": {\"count\": 1, \"max\": 184320, \"sum\": 184320.0, \"min\": 184320}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Reset Count\": {\"count\": 1, \"max\": 2, \"sum\": 2.0, \"min\": 2}}, \"EndTime\": 1593572790.977291, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 1}, \"StartTime\": 1593572786.463163}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:30 INFO 140313708324672] #throughput_metric: host=algo-1, train throughput=20415.1387971 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:35 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:35 INFO 140313708324672] Completed Epoch: 2, time taken: 0:00:04.461564\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:35 INFO 140313708324672] Epoch 2 Training metrics:   perplexity: 1.665 cross_entropy: 0.510 accuracy: 0.747 \u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:35 INFO 140313708324672] #quality_metric: host=algo-1, epoch=2, train cross_entropy <loss>=0.510030962361\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:35 INFO 140313708324672] #quality_metric: host=algo-1, epoch=2, train accuracy <score>=0.747016059028\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:35 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 4.730939865112305, \"sum\": 4.730939865112305, \"min\": 4.730939865112305}, \"update.time\": {\"count\": 1, \"max\": 4467.144966125488, \"sum\": 4467.144966125488, \"min\": 4467.144966125488}}, \"EndTime\": 1593572795.556575, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593572790.977078}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:35 INFO 140313708324672] #progress_metric: host=algo-1, completed 30 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Total Batches Seen\": {\"count\": 1, \"max\": 135, \"sum\": 135.0, \"min\": 135}, \"Total Records Seen\": {\"count\": 1, \"max\": 276480, \"sum\": 276480.0, \"min\": 276480}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Reset Count\": {\"count\": 1, \"max\": 3, \"sum\": 3.0, \"min\": 3}}, \"EndTime\": 1593572795.556964, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 2}, \"StartTime\": 1593572791.089401}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:35 INFO 140313708324672] #throughput_metric: host=algo-1, train throughput=20627.8930108 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:39 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:39 INFO 140313708324672] Completed Epoch: 3, time taken: 0:00:04.214880\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:39 INFO 140313708324672] Epoch 3 Training metrics:   perplexity: 1.425 cross_entropy: 0.354 accuracy: 0.851 \u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:39 INFO 140313708324672] #quality_metric: host=algo-1, epoch=3, train cross_entropy <loss>=0.354139728679\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:39 INFO 140313708324672] #quality_metric: host=algo-1, epoch=3, train accuracy <score>=0.850716145833\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:39 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:39 INFO 140313708324672] patience losses: [0.6201250195503235, 0.5458787666426764, 0.5100309623612298]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:39 INFO 140313708324672] min patience losses: 0.510030962361\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:39 INFO 140313708324672] current loss: 0.354139728679\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:39 INFO 140313708324672] absolute loss difference: 0.155891233683\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 4.358053207397461, \"sum\": 4.358053207397461, \"min\": 4.358053207397461}, \"update.time\": {\"count\": 1, \"max\": 4220.009088516235, \"sum\": 4220.009088516235, \"min\": 4220.009088516235}}, \"EndTime\": 1593572799.907539, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593572795.55672}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:39 INFO 140313708324672] #progress_metric: host=algo-1, completed 40 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Total Batches Seen\": {\"count\": 1, \"max\": 180, \"sum\": 180.0, \"min\": 180}, \"Total Records Seen\": {\"count\": 1, \"max\": 368640, \"sum\": 368640.0, \"min\": 368640}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Reset Count\": {\"count\": 1, \"max\": 4, \"sum\": 4.0, \"min\": 4}}, \"EndTime\": 1593572799.907863, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 3}, \"StartTime\": 1593572795.687501}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:39 INFO 140313708324672] #throughput_metric: host=algo-1, train throughput=21836.0792305 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:44 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:44 INFO 140313708324672] Completed Epoch: 4, time taken: 0:00:04.579083\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:44 INFO 140313708324672] Epoch 4 Training metrics:   perplexity: 1.098 cross_entropy: 0.093 accuracy: 0.974 \u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:44 INFO 140313708324672] #quality_metric: host=algo-1, epoch=4, train cross_entropy <loss>=0.0930616597335\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:44 INFO 140313708324672] #quality_metric: host=algo-1, epoch=4, train accuracy <score>=0.973654513889\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:44 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:44 INFO 140313708324672] patience losses: [0.5458787666426764, 0.5100309623612298, 0.35413972867859733]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:44 INFO 140313708324672] min patience losses: 0.354139728679\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:44 INFO 140313708324672] current loss: 0.0930616597335\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:44 INFO 140313708324672] absolute loss difference: 0.261078068945\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 4.310131072998047, \"sum\": 4.310131072998047, \"min\": 4.310131072998047}, \"update.time\": {\"count\": 1, \"max\": 4584.192991256714, \"sum\": 4584.192991256714, \"min\": 4584.192991256714}}, \"EndTime\": 1593572804.517437, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593572799.907655}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:44 INFO 140313708324672] #progress_metric: host=algo-1, completed 50 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Total Batches Seen\": {\"count\": 1, \"max\": 225, \"sum\": 225.0, \"min\": 225}, \"Total Records Seen\": {\"count\": 1, \"max\": 460800, \"sum\": 460800.0, \"min\": 460800}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Reset Count\": {\"count\": 1, \"max\": 5, \"sum\": 5.0, \"min\": 5}}, \"EndTime\": 1593572804.517764, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 4}, \"StartTime\": 1593572799.933212}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:44 INFO 140313708324672] #throughput_metric: host=algo-1, train throughput=20101.4757388 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:48 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:48 INFO 140313708324672] Completed Epoch: 5, time taken: 0:00:04.406856\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:48 INFO 140313708324672] Epoch 5 Training metrics:   perplexity: 1.016 cross_entropy: 0.016 accuracy: 0.997 \u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:48 INFO 140313708324672] #quality_metric: host=algo-1, epoch=5, train cross_entropy <loss>=0.0162533772902\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:48 INFO 140313708324672] #quality_metric: host=algo-1, epoch=5, train accuracy <score>=0.997482638889\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:48 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:48 INFO 140313708324672] patience losses: [0.5100309623612298, 0.35413972867859733, 0.09306165973345439]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:48 INFO 140313708324672] min patience losses: 0.0930616597335\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:48 INFO 140313708324672] current loss: 0.0162533772902\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:48 INFO 140313708324672] absolute loss difference: 0.0768082824432\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 4.354953765869141, \"sum\": 4.354953765869141, \"min\": 4.354953765869141}, \"update.time\": {\"count\": 1, \"max\": 4412.029981613159, \"sum\": 4412.029981613159, \"min\": 4412.029981613159}}, \"EndTime\": 1593572808.950357, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593572804.517556}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:48 INFO 140313708324672] #progress_metric: host=algo-1, completed 60 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Total Batches Seen\": {\"count\": 1, \"max\": 270, \"sum\": 270.0, \"min\": 270}, \"Total Records Seen\": {\"count\": 1, \"max\": 552960, \"sum\": 552960.0, \"min\": 552960}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Reset Count\": {\"count\": 1, \"max\": 6, \"sum\": 6.0, \"min\": 6}}, \"EndTime\": 1593572808.950686, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 5}, \"StartTime\": 1593572804.538293}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:48 INFO 140313708324672] #throughput_metric: host=algo-1, train throughput=20885.6928559 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:53 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:53 INFO 140313708324672] Completed Epoch: 6, time taken: 0:00:04.208280\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:53 INFO 140313708324672] Epoch 6 Training metrics:   perplexity: 1.003 cross_entropy: 0.003 accuracy: 1.000 \u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:53 INFO 140313708324672] #quality_metric: host=algo-1, epoch=6, train cross_entropy <loss>=0.00333411886564\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:53 INFO 140313708324672] #quality_metric: host=algo-1, epoch=6, train accuracy <score>=0.99990234375\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:53 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:53 INFO 140313708324672] patience losses: [0.35413972867859733, 0.09306165973345439, 0.016253377290235627]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:53 INFO 140313708324672] min patience losses: 0.0162533772902\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:53 INFO 140313708324672] current loss: 0.00333411886564\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:53 INFO 140313708324672] absolute loss difference: 0.0129192584246\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 4.522085189819336, \"sum\": 4.522085189819336, \"min\": 4.522085189819336}, \"update.time\": {\"count\": 1, \"max\": 4213.570833206177, \"sum\": 4213.570833206177, \"min\": 4213.570833206177}}, \"EndTime\": 1593572813.184633, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593572808.950475}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:53 INFO 140313708324672] #progress_metric: host=algo-1, completed 70 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Total Batches Seen\": {\"count\": 1, \"max\": 315, \"sum\": 315.0, \"min\": 315}, \"Total Records Seen\": {\"count\": 1, \"max\": 645120, \"sum\": 645120.0, \"min\": 645120}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Reset Count\": {\"count\": 1, \"max\": 7, \"sum\": 7.0, \"min\": 7}}, \"EndTime\": 1593572813.184967, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 6}, \"StartTime\": 1593572808.970993}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:53 INFO 140313708324672] #throughput_metric: host=algo-1, train throughput=21869.1110381 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:57 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:57 INFO 140313708324672] Completed Epoch: 7, time taken: 0:00:04.599210\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:57 INFO 140313708324672] Epoch 7 Training metrics:   perplexity: 1.001 cross_entropy: 0.001 accuracy: 1.000 \u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:57 INFO 140313708324672] #quality_metric: host=algo-1, epoch=7, train cross_entropy <loss>=0.00122575404304\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:57 INFO 140313708324672] #quality_metric: host=algo-1, epoch=7, train accuracy <score>=0.999989149306\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:57 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:57 INFO 140313708324672] patience losses: [0.09306165973345439, 0.016253377290235627, 0.003334118865637316]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:57 INFO 140313708324672] min patience losses: 0.00333411886564\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:57 INFO 140313708324672] current loss: 0.00122575404304\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:57 INFO 140313708324672] absolute loss difference: 0.0021083648226\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:57 INFO 140313708324672] Bad epoch: loss has not improved (enough). Bad count:1\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 4.578113555908203, \"sum\": 4.578113555908203, \"min\": 4.578113555908203}, \"update.time\": {\"count\": 1, \"max\": 4604.564189910889, \"sum\": 4604.564189910889, \"min\": 4604.564189910889}}, \"EndTime\": 1593572817.874011, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593572813.184742}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:57 INFO 140313708324672] #progress_metric: host=algo-1, completed 80 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Total Batches Seen\": {\"count\": 1, \"max\": 360, \"sum\": 360.0, \"min\": 360}, \"Total Records Seen\": {\"count\": 1, \"max\": 737280, \"sum\": 737280.0, \"min\": 737280}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Reset Count\": {\"count\": 1, \"max\": 8, \"sum\": 8.0, \"min\": 8}}, \"EndTime\": 1593572817.87432, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 7}, \"StartTime\": 1593572813.26942}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:06:57 INFO 140313708324672] #throughput_metric: host=algo-1, train throughput=20012.7028506 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:02 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:02 INFO 140313708324672] Completed Epoch: 8, time taken: 0:00:04.490736\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:02 INFO 140313708324672] Epoch 8 Training metrics:   perplexity: 1.001 cross_entropy: 0.001 accuracy: 1.000 \u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:02 INFO 140313708324672] #quality_metric: host=algo-1, epoch=8, train cross_entropy <loss>=0.000706633933199\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:02 INFO 140313708324672] #quality_metric: host=algo-1, epoch=8, train accuracy <score>=1.0\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:02 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:02 INFO 140313708324672] patience losses: [0.016253377290235627, 0.003334118865637316, 0.0012257540430356231]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:02 INFO 140313708324672] min patience losses: 0.00122575404304\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:02 INFO 140313708324672] current loss: 0.000706633933199\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:02 INFO 140313708324672] absolute loss difference: 0.000519120109837\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:02 INFO 140313708324672] Bad epoch: loss has not improved (enough). Bad count:2\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 4.881858825683594, \"sum\": 4.881858825683594, \"min\": 4.881858825683594}, \"update.time\": {\"count\": 1, \"max\": 4496.409893035889, \"sum\": 4496.409893035889, \"min\": 4496.409893035889}}, \"EndTime\": 1593572822.448868, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593572817.874116}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:02 INFO 140313708324672] #progress_metric: host=algo-1, completed 90 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Total Batches Seen\": {\"count\": 1, \"max\": 405, \"sum\": 405.0, \"min\": 405}, \"Total Records Seen\": {\"count\": 1, \"max\": 829440, \"sum\": 829440.0, \"min\": 829440}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Reset Count\": {\"count\": 1, \"max\": 9, \"sum\": 9.0, \"min\": 9}}, \"EndTime\": 1593572822.449161, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 8}, \"StartTime\": 1593572817.952424}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:02 INFO 140313708324672] #throughput_metric: host=algo-1, train throughput=20493.973286 records/second\u001b[0m\n",
      "\n",
      "2020-07-01 03:07:21 Uploading - Uploading generated training model\n",
      "2020-07-01 03:07:21 Completed - Training job completed\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] Completed Epoch: 9, time taken: 0:00:04.736124\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] Epoch 9 Training metrics:   perplexity: 1.001 cross_entropy: 0.001 accuracy: 1.000 \u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] #quality_metric: host=algo-1, epoch=9, train cross_entropy <loss>=0.000517297562652\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] #quality_metric: host=algo-1, epoch=9, train accuracy <score>=1.0\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] **************\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] patience losses: [0.003334118865637316, 0.0012257540430356231, 0.0007066339331989487]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] min patience losses: 0.000706633933199\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] current loss: 0.000517297562652\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] absolute loss difference: 0.000189336370547\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] Bad epoch: loss has not improved (enough). Bad count:3\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"early_stop.time\": {\"count\": 1, \"max\": 4.829883575439453, \"sum\": 4.829883575439453, \"min\": 4.829883575439453}, \"update.time\": {\"count\": 1, \"max\": 4742.0079708099365, \"sum\": 4742.0079708099365, \"min\": 4742.0079708099365}}, \"EndTime\": 1593572827.279121, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593572822.448958}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] #progress_metric: host=algo-1, completed 100 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 45, \"sum\": 45.0, \"min\": 45}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Total Batches Seen\": {\"count\": 1, \"max\": 450, \"sum\": 450.0, \"min\": 450}, \"Total Records Seen\": {\"count\": 1, \"max\": 921600, \"sum\": 921600.0, \"min\": 921600}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 92160, \"sum\": 92160.0, \"min\": 92160}, \"Reset Count\": {\"count\": 1, \"max\": 10, \"sum\": 10.0, \"min\": 10}}, \"EndTime\": 1593572827.279426, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\", \"epoch\": 9}, \"StartTime\": 1593572822.537079}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] #throughput_metric: host=algo-1, train throughput=19432.5714618 records/second\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 WARNING 140313708324672] wait_for_all_workers will not sync workers since the kv store is not running distributed\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] Best model based on epoch 9. Best loss: 0.001\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"finalize.time\": {\"count\": 1, \"max\": 1.2729167938232422, \"sum\": 1.2729167938232422, \"min\": 1.2729167938232422}}, \"EndTime\": 1593572827.281118, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593572827.279224}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] Serializing model to /opt/ml/model/model_algo-1\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] Saved checkpoint to \"/tmp/tmp3Pb1rV/state-0001.params\"\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python2.7/site-packages/mxnet/module/module.py:865: UserWarning: embed_1_weight.stype is not 'row_sparse'. No need to perform row_sparse_pull.\n",
      "  \"perform row_sparse_pull.\" % param_name))\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python2.7/site-packages/mxnet/module/module.py:865: UserWarning: embed_0_weight.stype is not 'row_sparse'. No need to perform row_sparse_pull.\n",
      "  \"perform row_sparse_pull.\" % param_name))\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] Finished scoring on 10240 examples from 5 batches, each of size 2048.\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] Test Metrics:perplexity: 3.530 cross_entropy: 1.261 accuracy: 0.698 \u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] Test Metric names:['perplexity', 'cross_entropy', 'accuracy']\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] Test Metric values:[3.529990485276108, 1.26129469871521, 0.6984375]\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] Test Metric names and values:[('perplexity', 3.529990485276108), ('cross_entropy', 1.26129469871521), ('accuracy', 0.6984375)]\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 5, \"sum\": 5.0, \"min\": 5}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 5, \"sum\": 5.0, \"min\": 5}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 10240, \"sum\": 10240.0, \"min\": 10240}, \"Total Batches Seen\": {\"count\": 1, \"max\": 5, \"sum\": 5.0, \"min\": 5}, \"Total Records Seen\": {\"count\": 1, \"max\": 10240, \"sum\": 10240.0, \"min\": 10240}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 10240, \"sum\": 10240.0, \"min\": 10240}, \"Reset Count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1593572827.843491, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"test_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593572827.646998}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] #test_score (algo-1) : ('perplexity', 3.529990485276108)\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] #test_score (algo-1) : ('cross_entropy', 1.26129469871521)\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] #test_score (algo-1) : ('accuracy', 0.6984375)\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] #quality_metric: host=algo-1, test cross_entropy <loss>=1.26129469872\u001b[0m\n",
      "\u001b[34m[07/01/2020 03:07:07 INFO 140313708324672] #quality_metric: host=algo-1, test accuracy <score>=0.6984375\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"totaltime\": {\"count\": 1, \"max\": 54926.86700820923, \"sum\": 54926.86700820923, \"min\": 54926.86700820923}, \"model.score.time\": {\"count\": 1, \"max\": 196.39301300048828, \"sum\": 196.39301300048828, \"min\": 196.39301300048828}, \"model.serialize.time\": {\"count\": 1, \"max\": 365.3979301452637, \"sum\": 365.3979301452637, \"min\": 365.3979301452637}, \"setuptime\": {\"count\": 1, \"max\": 324.62096214294434, \"sum\": 324.62096214294434, \"min\": 324.62096214294434}}, \"EndTime\": 1593572827.850425, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ObjectToVec\"}, \"StartTime\": 1593572827.281174}\n",
      "\u001b[0m\n",
      "Training seconds: 138\n",
      "Billable seconds: 138\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker:Parameter image will be renamed to image_uri in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------!"
     ]
    }
   ],
   "source": [
    "hyperparameters_c = {\n",
    "    \"_kvstore\": \"device\",\n",
    "    \"_num_gpus\": \"auto\",\n",
    "    \"_num_kv_servers\": \"auto\",\n",
    "    \"bucket_width\": 0,\n",
    "    \"early_stopping_patience\": 3, \n",
    "    \"early_stopping_tolerance\": 0.01,\n",
    "    \"enc0_cnn_filter_width\": 3,\n",
    "    \"enc0_layers\": \"auto\",\n",
    "    \"enc0_max_seq_len\": 1,\n",
    "    \"enc0_network\": \"pooled_embedding\",\n",
    "    \"enc0_token_embedding_dim\": 300,\n",
    "    \"enc0_vocab_size\": 944,\n",
    "    \"enc1_cnn_filter_width\": 3,\n",
    "    \"enc1_layers\": \"auto\",\n",
    "    \"enc1_max_seq_len\": 1,\n",
    "    \"enc1_network\": \"pooled_embedding\",\n",
    "    \"enc1_token_embedding_dim\": 300,\n",
    "    \"enc1_vocab_size\": 1684,\n",
    "    \"enc_dim\": 2048,\n",
    "    \"epochs\": 10,\n",
    "    \"learning_rate\": 0.001,\n",
    "    \"mini_batch_size\": 2048,\n",
    "    \"mlp_activation\": \"relu\",\n",
    "    \"mlp_dim\": 1024,\n",
    "    \"mlp_layers\": 1,\n",
    "    \"num_classes\": 2,\n",
    "    \"optimizer\": \"adam\",\n",
    "    \"output_layer\": \"softmax\"\n",
    "}\n",
    "\n",
    "## get estimator\n",
    "classifier = sagemaker.estimator.Estimator(container,\n",
    "                                    role, \n",
    "                                    train_instance_count=1, \n",
    "                                    train_instance_type='ml.p2.xlarge',\n",
    "                                    output_path=s3_output_path,\n",
    "                                    sagemaker_session=sess)\n",
    "\n",
    "## set hyperparameters\n",
    "classifier.set_hyperparameters(**hyperparameters_c)\n",
    "\n",
    "\n",
    "## train the model\n",
    "data_channels = {\n",
    "    \"train\": \"{}/train/train_c.jsonl\".format(s3_data_path),\n",
    "    \"test\": \"{}/validation/validation_c.jsonl\".format(s3_data_path)\n",
    "}\n",
    "\n",
    "classifier.fit(inputs=data_channels, wait=True)\n",
    "\n",
    "\n",
    "# create the model\n",
    "classification_model = classifier.create_model(\n",
    "                        serializer=json_serializer,\n",
    "                        deserializer=json_deserializer,\n",
    "                        content_type='application/json')\n",
    "# Deploy the model\n",
    "predictor_2 = classification_model.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')\n",
    "\n",
    "# change test-set into correct format\n",
    "valid_c_data, valid_c_label = data_list_to_inference_format(copy.deepcopy(validation_data_list), \n",
    "                                                            label_thres=3, binarize=True)\n",
    "# prediction on test-set\n",
    "predictions = predictor_2.predict(valid_c_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The accuracy on the binarized validation set is 0.699\n"
     ]
    }
   ],
   "source": [
    "# Helper function to get accuracy\n",
    "def get_class_accuracy(res, labels, thres):\n",
    "    if type(res) is dict:\n",
    "        res = res['predictions']\n",
    "    assert len(res)==len(labels), 'result and label length mismatch!'\n",
    "    accuracy = 0\n",
    "    for row, label in zip(res, labels):\n",
    "        if type(row) is dict:\n",
    "            if row['scores'][1] > thres:\n",
    "                prediction = 1\n",
    "            else: \n",
    "                prediction = 0\n",
    "            if label > thres:\n",
    "                label = 1\n",
    "            else:\n",
    "                label = 0\n",
    "            accuracy += 1 - (prediction - label)**2\n",
    "    return accuracy / float(len(res))\n",
    "\n",
    "print(\"The accuracy on the binarized validation set is %.3f\" %get_class_accuracy(predictions, valid_c_label, 0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Movie retrieval in the embedding space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since Object2Vec transforms user and movie ID's into embeddings as part of the training process. After training, it obtains user and movie embeddings in the left and right encoders, respectively. Intuitively, the embeddings should be tuned by the algorithm in a way that facilitates the supervised learning task: since for a specific user, similar movies should have similar ratings, we expect that similar movies should be close-by in the embedding space.\n",
    "\n",
    "In this section, we demonstrate how to find the nearest-neighbor (in Euclidean distance) of a given movie ID, among all movie ID's."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_movie_embedding_dict(movie_ids, trained_model):\n",
    "    input_instances = list()\n",
    "    for s_id in movie_ids:\n",
    "        input_instances.append({'in1': [s_id]})\n",
    "    data = {'instances': input_instances}\n",
    "    movie_embeddings = trained_model.predict(data)\n",
    "    embedding_dict = {}\n",
    "    for s_id, row in zip(movie_ids, movie_embeddings['predictions']):\n",
    "        embedding_dict[s_id] = np.array(row['embeddings'])\n",
    "    return embedding_dict\n",
    "\n",
    "\n",
    "def load_movie_id_name_map(item_file):\n",
    "    movieID_name_map = {}\n",
    "    with open(item_file, 'r', encoding=\"ISO-8859-1\") as f:\n",
    "        for row in f.readlines():\n",
    "            row = row.strip()\n",
    "            split = row.split('|')\n",
    "            movie_id = split[0]\n",
    "            movie_name = split[1]\n",
    "            sparse_tags = split[-19:]\n",
    "            movieID_name_map[int(movie_id)] = movie_name \n",
    "    return movieID_name_map\n",
    "\n",
    "            \n",
    "def get_nn_of_movie(movie_id, candidate_movie_ids, embedding_dict):\n",
    "    movie_emb = embedding_dict[movie_id]\n",
    "    min_dist = float('Inf')\n",
    "    best_id = candidate_movie_ids[0]\n",
    "    for idx, m_id in enumerate(candidate_movie_ids):\n",
    "        candidate_emb = embedding_dict[m_id]\n",
    "        curr_dist = np.linalg.norm(candidate_emb - movie_emb)\n",
    "        if curr_dist < min_dist:\n",
    "            best_id = m_id\n",
    "            min_dist = curr_dist\n",
    "    return best_id, min_dist\n",
    "\n",
    "\n",
    "def get_unique_movie_ids(data_list):\n",
    "    unique_movie_ids = set()\n",
    "    for row in data_list:\n",
    "        unique_movie_ids.add(row['in1'][0])\n",
    "    return list(unique_movie_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data_list = load_csv_data(train_path, '\\t', verbose=False)\n",
    "unique_movie_ids = get_unique_movie_ids(train_data_list)\n",
    "embedding_dict = get_movie_embedding_dict(unique_movie_ids, predictor_2)\n",
    "candidate_movie_ids = unique_movie_ids.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Script below can show what is the closest movie to any movie in the data set. You can find the movie name and ID pair in the 'u.item' file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie_id_to_examine = 312 # Customize the movie ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The closest movie to Midnight in the Garden of Good and Evil (1997) in the embedding space is Men With Guns (1997)\n"
     ]
    }
   ],
   "source": [
    "candidate_movie_ids.remove(movie_id_to_examine)\n",
    "best_id, min_dist = get_nn_of_movie(movie_id_to_examine, candidate_movie_ids, embedding_dict)\n",
    "movieID_name_map = load_movie_id_name_map('./data/ml-100k/u.item')\n",
    "print('The closest movie to {} in the embedding space is {}'.format(movieID_name_map[movie_id_to_examine],\n",
    "                                                                  movieID_name_map[best_id]))\n",
    "candidate_movie_ids.append(movie_id_to_examine)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10. Close the SageMaker Instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "## clean up\n",
    "sess.delete_endpoint(predictor.endpoint)\n",
    "sess.delete_endpoint(predictor_2.endpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}