Recommendation System on Amazon SageMaker - Beginner (Factorization Machine).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommendation System on Amazon SageMaker - Beginner (Factorization Machine)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we are going to build a simple movie recommendation model with Factorization machine using Amazon Sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import sagemaker.amazon.common as smac\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.predictor import json_deserializer\n",
    "\n",
    "import boto3, csv, io, json\n",
    "import numpy as np\n",
    "from scipy.sparse import lil_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset Description including number of users/moview are in official [Movielens website](https://grouplens.org/datasets/movielens/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sagemaker session : <sagemaker.session.Session object at 0x000002F746292D48>\n",
      "S3 bucket : sagemaker-us-west-2-570447867175\n",
      "Prefix : sagemaker/movielens\n",
      "Region selected : us-west-2\n",
      "IAM role : arn:aws:iam::570447867175:role/SageMakerNotebookRole\n"
     ]
    }
   ],
   "source": [
    "sess   = sagemaker.Session()\n",
    "bucket = sess.default_bucket()                     \n",
    "prefix = 'sagemaker/movielens'\n",
    "region = boto3.Session().region_name\n",
    "role = 'arn:aws:iam::570447867175:role/SageMakerNotebookRole' # pass your IAM role name\n",
    "\n",
    "print('Sagemaker session :', sess)\n",
    "print('S3 bucket :', bucket)\n",
    "print('Prefix :', prefix)\n",
    "print('Region selected :', region)\n",
    "print('IAM role :', role)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "nbUsers=943\n",
    "nbMovies=1682\n",
    "nbFeatures=nbUsers+nbMovies\n",
    "\n",
    "nbRatingsTrain=90570\n",
    "nbRatingsTest=9430"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For each user, build a list of rated movies.\n",
    "# We'd need this to add random negative samples.\n",
    "moviesByUser = {}\n",
    "for userId in range(nbUsers):\n",
    "    moviesByUser[str(userId)]=[]\n",
    "\n",
    "with open('./data/ml-100k/ua.base','r') as f:\n",
    "    samples=csv.reader(f,delimiter='\\t')\n",
    "    for userId,movieId,rating,timestamp in samples:\n",
    "        moviesByUser[str(int(userId)-1)].append(int(movieId)-1) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loadDataset(filename, lines, columns):\n",
    "    # Features are one-hot encoded in a sparse matrix\n",
    "    X = lil_matrix((lines, columns)).astype('float32')\n",
    "    # Labels are stored in a vector\n",
    "    Y = []\n",
    "    line=0\n",
    "    with open(filename,'r') as f:\n",
    "        samples=csv.reader(f,delimiter='\\t')\n",
    "        for userId,movieId,rating,timestamp in samples:\n",
    "            X[line,int(userId)-1] = 1\n",
    "            X[line,int(nbUsers)+int(movieId)-1] = 1\n",
    "            if int(rating) >= 4:\n",
    "                Y.append(1)\n",
    "            else:\n",
    "                Y.append(0)\n",
    "            line=line+1\n",
    "            \n",
    "    Y=np.array(Y).astype('float32')\n",
    "    return X,Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, Y_train = loadDataset('./data/ml-100k/ua.base', nbRatingsTrain, nbFeatures)\n",
    "X_test, Y_test = loadDataset('./data/ml-100k/ua.test',nbRatingsTest,nbFeatures)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(90570, 2625)\n",
      "(90570,)\n",
      "Training labels: 49906 zeros, 40664 ones\n",
      "(9430, 2625)\n",
      "(9430,)\n",
      "Test labels: 5469 zeros, 3961 ones\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(Y_train.shape)\n",
    "\n",
    "assert X_train.shape == (nbRatingsTrain, nbFeatures)\n",
    "assert Y_train.shape == (nbRatingsTrain, )\n",
    "zero_labels = np.count_nonzero(Y_train)\n",
    "\n",
    "print(\"Training labels: %d zeros, %d ones\" % (zero_labels, nbRatingsTrain-zero_labels))\n",
    "\n",
    "print(X_test.shape)\n",
    "print(Y_test.shape)\n",
    "\n",
    "assert X_test.shape  == (nbRatingsTest, nbFeatures)\n",
    "assert Y_test.shape  == (nbRatingsTest, )\n",
    "zero_labels = np.count_nonzero(Y_test)\n",
    "print(\"Test labels: %d zeros, %d ones\" % (zero_labels, nbRatingsTest-zero_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Convert to protobuf and upload data to S3 bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_key      = 'train.protobuf'\n",
    "train_prefix   = '{}/{}'.format(prefix, 'train3')\n",
    "\n",
    "test_key       = 'test.protobuf'\n",
    "test_prefix    = '{}/{}'.format(prefix, 'test3')\n",
    "\n",
    "output_prefix  = 's3://{}/{}/output'.format(bucket, prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-west-2-570447867175/sagemaker/movielens/train3/train.protobuf\n",
      "s3://sagemaker-us-west-2-570447867175/sagemaker/movielens/test3/test.protobuf\n",
      "Output: s3://sagemaker-us-west-2-570447867175/sagemaker/movielens/output\n"
     ]
    }
   ],
   "source": [
    "def writeDatasetToProtobuf(X, Y, bucket, prefix, key):\n",
    "    buf = io.BytesIO()\n",
    "    smac.write_spmatrix_to_sparse_tensor(buf, X, Y)\n",
    "    buf.seek(0)\n",
    "    obj = '{}/{}'.format(prefix, key)\n",
    "    boto3.resource('s3').Bucket(bucket).Object(obj).upload_fileobj(buf)\n",
    "    return 's3://{}/{}'.format(bucket,obj)\n",
    "    \n",
    "train_data = writeDatasetToProtobuf(X_train, Y_train, bucket, train_prefix, train_key)    \n",
    "test_data  = writeDatasetToProtobuf(X_test, Y_test, bucket, test_prefix, test_key)    \n",
    "  \n",
    "print(train_data)\n",
    "print(test_data)\n",
    "print('Output: {}'.format(output_prefix))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Start Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'get_image_uri' method will be deprecated in favor of 'ImageURIProvider' class in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "174872318107.dkr.ecr.us-west-2.amazonaws.com/factorization-machines:latest\n"
     ]
    }
   ],
   "source": [
    "# Specify Docker Contatiner\n",
    "container = sagemaker.amazon.amazon_estimator.get_image_uri(\n",
    "    boto3.Session().region_name, \"factorization-machines\", \"latest\")\n",
    "print(container)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.\n",
      "WARNING:sagemaker:'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n",
      "WARNING:sagemaker:'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-06-18 23:58:24 Starting - Starting the training job...\n",
      "2020-06-18 23:58:27 Starting - Launching requested ML instances......\n",
      "2020-06-18 23:59:41 Starting - Preparing the instances for training......\n",
      "2020-06-19 00:00:56 Downloading - Downloading input data\n",
      "2020-06-19 00:00:56 Training - Downloading the training image..\u001b[34mDocker entrypoint called with argument(s): train\u001b[0m\n",
      "\u001b[34mRunning default environment configuration script\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python2.7/site-packages/pandas/util/nosetester.py:13: DeprecationWarning: Importing from numpy.testing.nosetester is deprecated, import from numpy.testing instead.\n",
      "  from numpy.testing import nosetester\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:14 INFO 140432659142464] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/resources/default-conf.json: {u'factors_lr': u'0.0001', u'linear_init_sigma': u'0.01', u'epochs': 1, u'_wd': u'1.0', u'_num_kv_servers': u'auto', u'use_bias': u'true', u'factors_init_sigma': u'0.001', u'_log_level': u'info', u'bias_init_method': u'normal', u'linear_init_method': u'normal', u'linear_lr': u'0.001', u'factors_init_method': u'normal', u'_tuning_objective_metric': u'', u'bias_wd': u'0.01', u'use_linear': u'true', u'bias_lr': u'0.1', u'mini_batch_size': u'1000', u'_use_full_symbolic': u'true', u'batch_metrics_publish_interval': u'500', u'bias_init_sigma': u'0.01', u'_num_gpus': u'auto', u'_data_format': u'record', u'factors_wd': u'0.00001', u'linear_wd': u'0.001', u'_kvstore': u'auto', u'_learning_rate': u'1.0', u'_optimizer': u'adam'}\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:14 INFO 140432659142464] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'epochs': u'10', u'feature_dim': u'2625', u'predictor_type': u'binary_classifier', u'num_factors': u'64'}\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:14 INFO 140432659142464] Final configuration: {u'factors_lr': u'0.0001', u'linear_init_sigma': u'0.01', u'epochs': u'10', u'feature_dim': u'2625', u'num_factors': u'64', u'_wd': u'1.0', u'_num_kv_servers': u'auto', u'use_bias': u'true', u'factors_init_sigma': u'0.001', u'_log_level': u'info', u'bias_init_method': u'normal', u'linear_init_method': u'normal', u'linear_lr': u'0.001', u'factors_init_method': u'normal', u'_tuning_objective_metric': u'', u'bias_wd': u'0.01', u'use_linear': u'true', u'bias_lr': u'0.1', u'mini_batch_size': u'1000', u'_use_full_symbolic': u'true', u'batch_metrics_publish_interval': u'500', u'predictor_type': u'binary_classifier', u'bias_init_sigma': u'0.01', u'_num_gpus': u'auto', u'_data_format': u'record', u'factors_wd': u'0.00001', u'linear_wd': u'0.001', u'_kvstore': u'auto', u'_learning_rate': u'1.0', u'_optimizer': u'adam'}\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:14 WARNING 140432659142464] Loggers have already been setup.\u001b[0m\n",
      "\u001b[34mProcess 1 is a worker.\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:14 INFO 140432659142464] Using default worker.\u001b[0m\n",
      "\u001b[34m[2020-06-19 00:01:14.632] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m[2020-06-19 00:01:14.637] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 0, \"duration\": 9, \"num_examples\": 1, \"num_bytes\": 64000}\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:14 INFO 140432659142464] nvidia-smi took: 0.0251770019531 secs to identify 0 gpus\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:14 INFO 140432659142464] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:14 INFO 140432659142464] [Sparse network] Building a sparse network.\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:14 INFO 140432659142464] Create Store: local\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"initialize.time\": {\"count\": 1, \"max\": 41.26715660095215, \"sum\": 41.26715660095215, \"min\": 41.26715660095215}}, \"EndTime\": 1592524874.677572, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1592524874.627642}\n",
      "\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}, \"Total Records Seen\": {\"count\": 1, \"max\": 1000, \"sum\": 1000.0, \"min\": 1000}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 1000, \"sum\": 1000.0, \"min\": 1000}, \"Reset Count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1592524874.677781, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"init_train_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1592524874.677725}\n",
      "\u001b[0m\n",
      "\u001b[34m[00:01:14] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.1.x.202835.0/AL2012/generic-flavor/src/src/kvstore/./kvstore_local.h:280: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use row_sparse_pull with row_ids.\u001b[0m\n",
      "\u001b[34m[00:01:14] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.1.x.202835.0/AL2012/generic-flavor/src/src/kvstore/./kvstore_local.h:280: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use row_sparse_pull with row_ids.\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:14 INFO 140432659142464] #quality_metric: host=algo-1, epoch=0, batch=0 train binary_classification_accuracy <score>=0.589\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:14 INFO 140432659142464] #quality_metric: host=algo-1, epoch=0, batch=0 train binary_classification_cross_entropy <loss>=0.691272155762\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:14 INFO 140432659142464] #quality_metric: host=algo-1, epoch=0, batch=0 train binary_f_1.000 <score>=0.722859069454\u001b[0m\n",
      "\u001b[34m[2020-06-19 00:01:15.370] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 2, \"duration\": 589, \"num_examples\": 91, \"num_bytes\": 5796480}\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:15 INFO 140432659142464] #quality_metric: host=algo-1, epoch=0, train binary_classification_accuracy <score>=0.560692307692\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:15 INFO 140432659142464] #quality_metric: host=algo-1, epoch=0, train binary_classification_cross_entropy <loss>=0.689987980099\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:15 INFO 140432659142464] #quality_metric: host=algo-1, epoch=0, train binary_f_1.000 <score>=0.700144763391\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"epochs\": {\"count\": 1, \"max\": 10, \"sum\": 10.0, \"min\": 10}, \"update.time\": {\"count\": 1, \"max\": 692.9299831390381, \"sum\": 692.9299831390381, \"min\": 692.9299831390381}}, \"EndTime\": 1592524875.370928, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1592524874.677651}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:15 INFO 140432659142464] #progress_metric: host=algo-1, completed 10 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Total Batches Seen\": {\"count\": 1, \"max\": 92, \"sum\": 92.0, \"min\": 92}, \"Total Records Seen\": {\"count\": 1, \"max\": 91570, \"sum\": 91570.0, \"min\": 91570}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Reset Count\": {\"count\": 1, \"max\": 2, \"sum\": 2.0, \"min\": 2}}, \"EndTime\": 1592524875.371282, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 0}, \"StartTime\": 1592524874.677961}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:15 INFO 140432659142464] #throughput_metric: host=algo-1, train throughput=130610.844 records/second\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:15 INFO 140432659142464] #quality_metric: host=algo-1, epoch=1, batch=0 train binary_classification_accuracy <score>=0.587\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:15 INFO 140432659142464] #quality_metric: host=algo-1, epoch=1, batch=0 train binary_classification_cross_entropy <loss>=0.671663574219\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:15 INFO 140432659142464] #quality_metric: host=algo-1, epoch=1, batch=0 train binary_f_1.000 <score>=0.739760554505\u001b[0m\n",
      "\u001b[34m[2020-06-19 00:01:15.959] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 4, \"duration\": 585, \"num_examples\": 91, \"num_bytes\": 5796480}\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:15 INFO 140432659142464] #quality_metric: host=algo-1, epoch=1, train binary_classification_accuracy <score>=0.566318681319\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:15 INFO 140432659142464] #quality_metric: host=algo-1, epoch=1, train binary_classification_cross_entropy <loss>=0.683364412077\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:15 INFO 140432659142464] #quality_metric: host=algo-1, epoch=1, train binary_f_1.000 <score>=0.704795529857\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 588.6240005493164, \"sum\": 588.6240005493164, \"min\": 588.6240005493164}}, \"EndTime\": 1592524875.960537, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1592524875.371023}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:15 INFO 140432659142464] #progress_metric: host=algo-1, completed 20 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Total Batches Seen\": {\"count\": 1, \"max\": 183, \"sum\": 183.0, \"min\": 183}, \"Total Records Seen\": {\"count\": 1, \"max\": 182140, \"sum\": 182140.0, \"min\": 182140}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Reset Count\": {\"count\": 1, \"max\": 3, \"sum\": 3.0, \"min\": 3}}, \"EndTime\": 1592524875.961, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 1}, \"StartTime\": 1592524875.371874}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:15 INFO 140432659142464] #throughput_metric: host=algo-1, train throughput=153693.141905 records/second\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:15 INFO 140432659142464] #quality_metric: host=algo-1, epoch=2, batch=0 train binary_classification_accuracy <score>=0.587\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:15 INFO 140432659142464] #quality_metric: host=algo-1, epoch=2, batch=0 train binary_classification_cross_entropy <loss>=0.665132446289\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:15 INFO 140432659142464] #quality_metric: host=algo-1, epoch=2, batch=0 train binary_f_1.000 <score>=0.739760554505\u001b[0m\n",
      "\u001b[34m[2020-06-19 00:01:16.587] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 6, \"duration\": 623, \"num_examples\": 91, \"num_bytes\": 5796480}\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:16 INFO 140432659142464] #quality_metric: host=algo-1, epoch=2, train binary_classification_accuracy <score>=0.576098901099\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:16 INFO 140432659142464] #quality_metric: host=algo-1, epoch=2, train binary_classification_cross_entropy <loss>=0.678237371759\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:16 INFO 140432659142464] #quality_metric: host=algo-1, epoch=2, train binary_f_1.000 <score>=0.707536941705\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 626.168966293335, \"sum\": 626.168966293335, \"min\": 626.168966293335}}, \"EndTime\": 1592524876.587865, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1592524875.960676}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:16 INFO 140432659142464] #progress_metric: host=algo-1, completed 30 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Total Batches Seen\": {\"count\": 1, \"max\": 274, \"sum\": 274.0, \"min\": 274}, \"Total Records Seen\": {\"count\": 1, \"max\": 272710, \"sum\": 272710.0, \"min\": 272710}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Reset Count\": {\"count\": 1, \"max\": 4, \"sum\": 4.0, \"min\": 4}}, \"EndTime\": 1592524876.588088, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 2}, \"StartTime\": 1592524875.96166}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:16 INFO 140432659142464] #throughput_metric: host=algo-1, train throughput=144552.306589 records/second\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:16 INFO 140432659142464] #quality_metric: host=algo-1, epoch=3, batch=0 train binary_classification_accuracy <score>=0.587\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:16 INFO 140432659142464] #quality_metric: host=algo-1, epoch=3, batch=0 train binary_classification_cross_entropy <loss>=0.659431640625\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:16 INFO 140432659142464] #quality_metric: host=algo-1, epoch=3, batch=0 train binary_f_1.000 <score>=0.739760554505\u001b[0m\n",
      "\u001b[34m[2020-06-19 00:01:17.229] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 8, \"duration\": 637, \"num_examples\": 91, \"num_bytes\": 5796480}\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:17 INFO 140432659142464] #quality_metric: host=algo-1, epoch=3, train binary_classification_accuracy <score>=0.586\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:17 INFO 140432659142464] #quality_metric: host=algo-1, epoch=3, train binary_classification_cross_entropy <loss>=0.673689472576\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:17 INFO 140432659142464] #quality_metric: host=algo-1, epoch=3, train binary_f_1.000 <score>=0.709914377233\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 641.2708759307861, \"sum\": 641.2708759307861, \"min\": 641.2708759307861}}, \"EndTime\": 1592524877.229933, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1592524876.587948}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:17 INFO 140432659142464] #progress_metric: host=algo-1, completed 40 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Total Batches Seen\": {\"count\": 1, \"max\": 365, \"sum\": 365.0, \"min\": 365}, \"Total Records Seen\": {\"count\": 1, \"max\": 363280, \"sum\": 363280.0, \"min\": 363280}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Reset Count\": {\"count\": 1, \"max\": 5, \"sum\": 5.0, \"min\": 5}}, \"EndTime\": 1592524877.230168, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 3}, \"StartTime\": 1592524876.588625}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:17 INFO 140432659142464] #throughput_metric: host=algo-1, train throughput=141147.058993 records/second\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:17 INFO 140432659142464] #quality_metric: host=algo-1, epoch=4, batch=0 train binary_classification_accuracy <score>=0.587\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:17 INFO 140432659142464] #quality_metric: host=algo-1, epoch=4, batch=0 train binary_classification_cross_entropy <loss>=0.654440124512\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:17 INFO 140432659142464] #quality_metric: host=algo-1, epoch=4, batch=0 train binary_f_1.000 <score>=0.739760554505\u001b[0m\n",
      "\u001b[34m[2020-06-19 00:01:17.960] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 10, \"duration\": 727, \"num_examples\": 91, \"num_bytes\": 5796480}\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:17 INFO 140432659142464] #quality_metric: host=algo-1, epoch=4, train binary_classification_accuracy <score>=0.594538461538\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:17 INFO 140432659142464] #quality_metric: host=algo-1, epoch=4, train binary_classification_cross_entropy <loss>=0.669661624447\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:17 INFO 140432659142464] #quality_metric: host=algo-1, epoch=4, train binary_f_1.000 <score>=0.712054877906\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 730.0980091094971, \"sum\": 730.0980091094971, \"min\": 730.0980091094971}}, \"EndTime\": 1592524877.96084, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1592524877.230022}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:17 INFO 140432659142464] #progress_metric: host=algo-1, completed 50 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Total Batches Seen\": {\"count\": 1, \"max\": 456, \"sum\": 456.0, \"min\": 456}, \"Total Records Seen\": {\"count\": 1, \"max\": 453850, \"sum\": 453850.0, \"min\": 453850}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Reset Count\": {\"count\": 1, \"max\": 6, \"sum\": 6.0, \"min\": 6}}, \"EndTime\": 1592524877.961118, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 4}, \"StartTime\": 1592524877.230701}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:17 INFO 140432659142464] #throughput_metric: host=algo-1, train throughput=123972.893859 records/second\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:17 INFO 140432659142464] #quality_metric: host=algo-1, epoch=5, batch=0 train binary_classification_accuracy <score>=0.587\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:17 INFO 140432659142464] #quality_metric: host=algo-1, epoch=5, batch=0 train binary_classification_cross_entropy <loss>=0.650061401367\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:17 INFO 140432659142464] #quality_metric: host=algo-1, epoch=5, batch=0 train binary_f_1.000 <score>=0.739102969046\u001b[0m\n",
      "\u001b[34m[2020-06-19 00:01:18.547] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 12, \"duration\": 583, \"num_examples\": 91, \"num_bytes\": 5796480}\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:18 INFO 140432659142464] #quality_metric: host=algo-1, epoch=5, train binary_classification_accuracy <score>=0.601516483516\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:18 INFO 140432659142464] #quality_metric: host=algo-1, epoch=5, train binary_classification_cross_entropy <loss>=0.666080986274\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:18 INFO 140432659142464] #quality_metric: host=algo-1, epoch=5, train binary_f_1.000 <score>=0.713719545892\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 586.5540504455566, \"sum\": 586.5540504455566, \"min\": 586.5540504455566}}, \"EndTime\": 1592524878.548363, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1592524877.960943}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:18 INFO 140432659142464] #progress_metric: host=algo-1, completed 60 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Total Batches Seen\": {\"count\": 1, \"max\": 547, \"sum\": 547.0, \"min\": 547}, \"Total Records Seen\": {\"count\": 1, \"max\": 544420, \"sum\": 544420.0, \"min\": 544420}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Reset Count\": {\"count\": 1, \"max\": 7, \"sum\": 7.0, \"min\": 7}}, \"EndTime\": 1592524878.548576, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 5}, \"StartTime\": 1592524877.961773}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:18 INFO 140432659142464] #throughput_metric: host=algo-1, train throughput=154312.47305 records/second\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:18 INFO 140432659142464] #quality_metric: host=algo-1, epoch=6, batch=0 train binary_classification_accuracy <score>=0.603\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:18 INFO 140432659142464] #quality_metric: host=algo-1, epoch=6, batch=0 train binary_classification_cross_entropy <loss>=0.646207275391\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:18 INFO 140432659142464] #quality_metric: host=algo-1, epoch=6, batch=0 train binary_f_1.000 <score>=0.746325878594\u001b[0m\n",
      "\u001b[34m[2020-06-19 00:01:19.083] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 14, \"duration\": 531, \"num_examples\": 91, \"num_bytes\": 5796480}\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:19 INFO 140432659142464] #quality_metric: host=algo-1, epoch=6, train binary_classification_accuracy <score>=0.60889010989\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:19 INFO 140432659142464] #quality_metric: host=algo-1, epoch=6, train binary_classification_cross_entropy <loss>=0.662880099454\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:19 INFO 140432659142464] #quality_metric: host=algo-1, epoch=6, train binary_f_1.000 <score>=0.716051155629\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 534.8930358886719, \"sum\": 534.8930358886719, \"min\": 534.8930358886719}}, \"EndTime\": 1592524879.084021, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1592524878.548441}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:19 INFO 140432659142464] #progress_metric: host=algo-1, completed 70 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Total Batches Seen\": {\"count\": 1, \"max\": 638, \"sum\": 638.0, \"min\": 638}, \"Total Records Seen\": {\"count\": 1, \"max\": 634990, \"sum\": 634990.0, \"min\": 634990}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Reset Count\": {\"count\": 1, \"max\": 8, \"sum\": 8.0, \"min\": 8}}, \"EndTime\": 1592524879.084245, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 6}, \"StartTime\": 1592524878.54909}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:19 INFO 140432659142464] #throughput_metric: host=algo-1, train throughput=169200.566238 records/second\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:19 INFO 140432659142464] #quality_metric: host=algo-1, epoch=7, batch=0 train binary_classification_accuracy <score>=0.616\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:19 INFO 140432659142464] #quality_metric: host=algo-1, epoch=7, batch=0 train binary_classification_cross_entropy <loss>=0.642798950195\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:19 INFO 140432659142464] #quality_metric: host=algo-1, epoch=7, batch=0 train binary_f_1.000 <score>=0.751937984496\u001b[0m\n",
      "\n",
      "2020-06-19 00:01:26 Uploading - Uploading generated training model\u001b[34m[2020-06-19 00:01:19.605] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 16, \"duration\": 518, \"num_examples\": 91, \"num_bytes\": 5796480}\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:19 INFO 140432659142464] #quality_metric: host=algo-1, epoch=7, train binary_classification_accuracy <score>=0.615461538462\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:19 INFO 140432659142464] #quality_metric: host=algo-1, epoch=7, train binary_classification_cross_entropy <loss>=0.659999435928\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:19 INFO 140432659142464] #quality_metric: host=algo-1, epoch=7, train binary_f_1.000 <score>=0.718032601952\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 520.7939147949219, \"sum\": 520.7939147949219, \"min\": 520.7939147949219}}, \"EndTime\": 1592524879.605645, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1592524879.084102}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:19 INFO 140432659142464] #progress_metric: host=algo-1, completed 80 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Total Batches Seen\": {\"count\": 1, \"max\": 729, \"sum\": 729.0, \"min\": 729}, \"Total Records Seen\": {\"count\": 1, \"max\": 725560, \"sum\": 725560.0, \"min\": 725560}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Reset Count\": {\"count\": 1, \"max\": 9, \"sum\": 9.0, \"min\": 9}}, \"EndTime\": 1592524879.60587, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 7}, \"StartTime\": 1592524879.084817}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:19 INFO 140432659142464] #throughput_metric: host=algo-1, train throughput=173778.761392 records/second\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:19 INFO 140432659142464] #quality_metric: host=algo-1, epoch=8, batch=0 train binary_classification_accuracy <score>=0.632\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:19 INFO 140432659142464] #quality_metric: host=algo-1, epoch=8, batch=0 train binary_classification_cross_entropy <loss>=0.63976751709\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:19 INFO 140432659142464] #quality_metric: host=algo-1, epoch=8, batch=0 train binary_f_1.000 <score>=0.758846657929\u001b[0m\n",
      "\u001b[34m[2020-06-19 00:01:20.169] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 18, \"duration\": 562, \"num_examples\": 91, \"num_bytes\": 5796480}\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #quality_metric: host=algo-1, epoch=8, train binary_classification_accuracy <score>=0.623164835165\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #quality_metric: host=algo-1, epoch=8, train binary_classification_cross_entropy <loss>=0.657387503488\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #quality_metric: host=algo-1, epoch=8, train binary_f_1.000 <score>=0.720826481267\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 564.1369819641113, \"sum\": 564.1369819641113, \"min\": 564.1369819641113}}, \"EndTime\": 1592524880.170637, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1592524879.605731}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #progress_metric: host=algo-1, completed 90 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Total Batches Seen\": {\"count\": 1, \"max\": 820, \"sum\": 820.0, \"min\": 820}, \"Total Records Seen\": {\"count\": 1, \"max\": 816130, \"sum\": 816130.0, \"min\": 816130}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Reset Count\": {\"count\": 1, \"max\": 10, \"sum\": 10.0, \"min\": 10}}, \"EndTime\": 1592524880.170906, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 8}, \"StartTime\": 1592524879.60646}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #throughput_metric: host=algo-1, train throughput=160418.993994 records/second\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #quality_metric: host=algo-1, epoch=9, batch=0 train binary_classification_accuracy <score>=0.639\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #quality_metric: host=algo-1, epoch=9, batch=0 train binary_classification_cross_entropy <loss>=0.637053527832\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #quality_metric: host=algo-1, epoch=9, batch=0 train binary_f_1.000 <score>=0.761085373925\u001b[0m\n",
      "\u001b[34m[2020-06-19 00:01:20.732] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 20, \"duration\": 558, \"num_examples\": 91, \"num_bytes\": 5796480}\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #quality_metric: host=algo-1, epoch=9, train binary_classification_accuracy <score>=0.630043956044\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #quality_metric: host=algo-1, epoch=9, train binary_classification_cross_entropy <loss>=0.655000303834\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #quality_metric: host=algo-1, epoch=9, train binary_f_1.000 <score>=0.723496172673\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #quality_metric: host=algo-1, train binary_classification_accuracy <score>=0.630043956044\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #quality_metric: host=algo-1, train binary_classification_cross_entropy <loss>=0.655000303834\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #quality_metric: host=algo-1, train binary_f_1.000 <score>=0.723496172673\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 561.8939399719238, \"sum\": 561.8939399719238, \"min\": 561.8939399719238}}, \"EndTime\": 1592524880.733499, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1592524880.170728}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #progress_metric: host=algo-1, completed 100 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 91, \"sum\": 91.0, \"min\": 91}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Total Batches Seen\": {\"count\": 1, \"max\": 911, \"sum\": 911.0, \"min\": 911}, \"Total Records Seen\": {\"count\": 1, \"max\": 906700, \"sum\": 906700.0, \"min\": 906700}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 90570, \"sum\": 90570.0, \"min\": 90570}, \"Reset Count\": {\"count\": 1, \"max\": 11, \"sum\": 11.0, \"min\": 11}}, \"EndTime\": 1592524880.73385, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 9}, \"StartTime\": 1592524880.171567}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #throughput_metric: host=algo-1, train throughput=161032.785892 records/second\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 WARNING 140432659142464] wait_for_all_workers will not sync workers since the kv store is not running distributed\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] Pulling entire model from kvstore to finalize\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"finalize.time\": {\"count\": 1, \"max\": 2.8998851776123047, \"sum\": 2.8998851776123047, \"min\": 2.8998851776123047}}, \"EndTime\": 1592524880.737194, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1592524880.733614}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] Saved checkpoint to \"/tmp/tmpOhV2cV/state-0001.params\"\u001b[0m\n",
      "\u001b[34m[2020-06-19 00:01:20.747] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/test\", \"epoch\": 0, \"duration\": 6114, \"num_examples\": 1, \"num_bytes\": 64000}\u001b[0m\n",
      "\u001b[34m[2020-06-19 00:01:20.788] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/test\", \"epoch\": 1, \"duration\": 41, \"num_examples\": 10, \"num_bytes\": 603520}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 10, \"sum\": 10.0, \"min\": 10}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 10, \"sum\": 10.0, \"min\": 10}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 9430, \"sum\": 9430.0, \"min\": 9430}, \"Total Batches Seen\": {\"count\": 1, \"max\": 10, \"sum\": 10.0, \"min\": 10}, \"Total Records Seen\": {\"count\": 1, \"max\": 9430, \"sum\": 9430.0, \"min\": 9430}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 9430, \"sum\": 9430.0, \"min\": 9430}, \"Reset Count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1592524880.788985, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"test_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1592524880.747321}\n",
      "\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #test_score (algo-1) : ('binary_classification_accuracy', 0.632661717921527)\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #test_score (algo-1) : ('binary_classification_cross_entropy', 0.650316421654419)\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #test_score (algo-1) : ('binary_f_1.000', 0.7510063254744106)\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #quality_metric: host=algo-1, test binary_classification_accuracy <score>=0.632661717922\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #quality_metric: host=algo-1, test binary_classification_cross_entropy <loss>=0.650316421654\u001b[0m\n",
      "\u001b[34m[06/19/2020 00:01:20 INFO 140432659142464] #quality_metric: host=algo-1, test binary_f_1.000 <score>=0.751006325474\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"totaltime\": {\"count\": 1, \"max\": 6226.672887802124, \"sum\": 6226.672887802124, \"min\": 6226.672887802124}, \"setuptime\": {\"count\": 1, \"max\": 57.28888511657715, \"sum\": 57.28888511657715, \"min\": 57.28888511657715}}, \"EndTime\": 1592524880.790028, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1592524880.737243}\n",
      "\u001b[0m\n",
      "\n",
      "2020-06-19 00:01:33 Completed - Training job completed\n",
      "Training seconds: 48\n",
      "Billable seconds: 48\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.estimator import Estimator\n",
    "# https://sagemaker.readthedocs.io/en/stable/estimators.html\n",
    "\n",
    "## ==== ##\n",
    "# Make sure to use a non-GPU instance, because Training with a sparse data set is only possible on a non-GPU instance.\n",
    "## ==== ##\n",
    "\n",
    "fm = Estimator(container,                             # The contatiner that contatins algorithm\n",
    "               role = role,                           # your IAM role\n",
    "               train_instance_count=1,                # Instance requirements\n",
    "               train_instance_type='ml.c4.xlarge' ,   # Instance type (GPU instances like 'ml.p2.xlarge' would not work in this case)\n",
    "               output_path=output_prefix,\n",
    "               sagemaker_session=sess)\n",
    "\n",
    "fm.set_hyperparameters(feature_dim=nbFeatures,\n",
    "                      predictor_type='binary_classifier',\n",
    "                      num_factors=64,\n",
    "                      epochs=10)\n",
    "\n",
    "fm.fit({'train': train_data, 'test': test_data})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We should be able to check the logs of how Sagemaker takes cares of training like above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Model Deploy "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can easily deploy the model with **.deploy** method. This creates a RESTful HTTP endpoint that can be intergrated to any of the applications that we are trying to use. You can also check whether the model is created in your Sagemaker console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import strftime, gmtime\n",
    "timestamp = strftime('%d-%H-%M-%S', gmtime())\n",
    "\n",
    "fm_predictor = fm.deploy(\n",
    "    endpoint_name = 'movielens-{}'.format(timestamp),\n",
    "    initial_instance_count=1,\n",
    "    instance_type='ml.p2.xlarge')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Predicting with the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = fm_predictor.predict(X_test[1000:1010].toarray())\n",
    "print(result)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}