Unsupervised Anomaly Detection in Amazon SageMaker (IP address detection).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unsupervised Anomaly Detection in Amazon SageMaker (IP address detection)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amazon SageMaker IP insights ingests historical data as (entity, IPv4 Address) pairs and learns the IP usage patterns of each entity. When queried with an (entity, IPv4 Address) event, an Amazon SageMaker IP Insights model returns a score that infers how anomalous the pattern of the event is. For example, when a user attempts to log in from an IP address, if the IP Insights score is high enough, a web login server might decide to trigger a multi-factor authentication system. In more advanced solutions, you can feed the IP Insights score into another machine learning model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we use the Amazon SageMaker IP Insights algorithm to train a model on synthetic data. We then use this model to perform inference on the data and show how to discover suspicious IP addresses. After running this notebook, you should be able to:\n",
    "\n",
    "- obtain, transform, and store data for use in Amazon SageMaker,\n",
    "- create an AWS SageMaker training job to produce an IP Insights model,\n",
    "- use the model to perform inference with an Amazon SageMaker endpoint.\n",
    "If you would like to know more, please check out the SageMaker IP Inisghts Documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Training and Validation**\n",
    "\n",
    "The Amazon SageMaker IP Insights algorithm supports training and validation data channels. It uses the optional validation channel to compute an area-under-curve (AUC) score on a predefined negative sampling strategy. The AUC metric validates how well the model discriminates between positive and negative samples. Training and validation data content types need to be in text/csv format. The first column of the CSV data is an opaque string that provides a unique identifier for the entity. The second column is an IPv4 address in decimal-dot notation. IP Insights currently supports only File mode. For more information and some examples, see IP Insights Training Data Formats.\n",
    "\n",
    "**Inference**\n",
    "\n",
    "For inference, IP Insights supports text/csv, application/json, and application/jsonlines data content types. For more information about the common data formats for inference provided by Amazon SageMaker, see Common Data Formats for Inference. IP Insights inference returns output formatted as either application/json or application/jsonlines. Each record in the output data contains the corresponding dot_product (or compatibility score) for each input data point. For more information and some examples, see IP Insights Inference Data Formats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np                                # For matrix operations and numerical processing\n",
    "import pandas as pd                               # For munging tabular data\n",
    "import matplotlib.pyplot as plt                   # For charts and visualizations\n",
    "import matplotlib.ticker as ticker\n",
    "%matplotlib inline\n",
    "\n",
    "from IPython.display import Image                 # For displaying images in the notebook\n",
    "from IPython.display import display               # For displaying outputs in the notebook\n",
    "from time import gmtime, strftime                 # For labeling SageMaker models, endpoints, etc.\n",
    "import time\n",
    "import tqdm\n",
    "import sys                                        # For writing outputs to notebook\n",
    "import math                                       # For ceiling function\n",
    "import json                                       # For parsing hosting outputs\n",
    "import os                                         # For manipulating filepath names\n",
    "import boto3\n",
    "import botocore\n",
    "from botocore.exceptions import ClientError\n",
    "\n",
    "import csv\n",
    "import io\n",
    "import re\n",
    "import s3fs\n",
    "import mxnet as mx\n",
    "import seaborn as sns\n",
    "import pickle\n",
    "import gzip\n",
    "import urllib\n",
    "import csv\n",
    "from datetime import datetime\n",
    "import pytz\n",
    "\n",
    "import cv2\n",
    "\n",
    "import sagemaker                                 \n",
    "from sagemaker.predictor import csv_serializer \n",
    "from sagemaker.predictor import json_deserializer\n",
    "import sagemaker.amazon.common as smac\n",
    "from sagemaker import get_execution_role"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Preparation (Specifying Sagemaker roles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sagemaker session : <sagemaker.session.Session object at 0x000001E5B61FF448>\n",
      "S3 bucket : sagemaker-us-west-2-570447867175\n",
      "Prefix : ip-insights\n",
      "Region selected : us-west-2\n",
      "IAM role : arn:aws:iam::570447867175:role/SageMakerNotebookRole\n"
     ]
    }
   ],
   "source": [
    "sess   = sagemaker.Session()\n",
    "bucket = sess.default_bucket()                     \n",
    "prefix = 'ip-insights'\n",
    "region = boto3.Session().region_name\n",
    "role = 'arn:aws:iam::570447867175:role/SageMakerNotebookRole' # pass your IAM role name\n",
    "\n",
    "print('Sagemaker session :', sess)\n",
    "print('S3 bucket :', bucket)\n",
    "print('Prefix :', prefix)\n",
    "print('Region selected :', region)\n",
    "print('IAM role :', role)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'get_image_uri' method will be deprecated in favor of 'ImageURIProvider' class in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "174872318107.dkr.ecr.us-west-2.amazonaws.com/ipinsights:latest\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "\n",
    "image = get_image_uri(region, 'ipinsights', 'latest')\n",
    "print(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training input/output will be stored in: s3://sagemaker-us-west-2-570447867175/ip-insights\n"
     ]
    }
   ],
   "source": [
    "# check if the bucket exists\n",
    "try:\n",
    "    boto3.Session().client('s3').head_bucket(Bucket=bucket)\n",
    "except botocore.exceptions.ParamValidationError as e:\n",
    "    print('Hey! You either forgot to specify your S3 bucket'\n",
    "          ' or you gave your bucket an invalid name!')\n",
    "except botocore.exceptions.ClientError as e:\n",
    "    if e.response['Error']['Code'] == '403':\n",
    "        print(\"Hey! You don't have permission to access the bucket, {}.\".format(bucket))\n",
    "    elif e.response['Error']['Code'] == '404':\n",
    "        print(\"Hey! Your bucket, {}, doesn't exist!\".format(bucket))\n",
    "    else:\n",
    "        raise\n",
    "else:\n",
    "    print('Training input/output will be stored in: s3://{}/{}'.format(bucket, prefix))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### We will generate sample web-traffic logs. We mock up a toy website example and simulate users logging into the website from mobile devices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SageMaker Team has provided a Helper function to generate random IP addresses. Please refer to \"Generate_data.py\" python file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded ASN List: 827696 ASNs.\n",
      "Starting User Activity Simulation\n",
      "Finished simulating web activity for 10000 users.\n"
     ]
    }
   ],
   "source": [
    "from generate_data import generate_dataset\n",
    "\n",
    "# We simulate traffic for 10,000 users. This should yield about 3 million log lines (~700 MB). \n",
    "NUM_USERS = 10000\n",
    "log_file = 'ipinsights_web_traffic.log'\n",
    "generate_dataset(NUM_USERS, log_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Prepare the dataset\n",
    "Now that we have our logs, we need to transform them into a format that IP Insights can use. As we mentioned above, we need to:\n",
    "\n",
    "1. Choose the resource which we want to analyze users' history for\n",
    "2. Extract our users' usage history of IP addresses\n",
    "3. In addition, we want to separate our dataset into a training and test set. This will allow us to check for overfitting by evaluating our model on 'unseen' login events.\n",
    "\n",
    "For the rest of the notebook, we assume that the Apache Access Logs are in the Common Log Format as defined by the Apache documentation. We start with reading the logs into a Pandas DataFrame for easy data exploration and pre-processing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Data Pre-Processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ip_address</th>\n",
       "      <th>rcf_id</th>\n",
       "      <th>user</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>time_zone</th>\n",
       "      <th>request</th>\n",
       "      <th>status</th>\n",
       "      <th>size</th>\n",
       "      <th>referer</th>\n",
       "      <th>user_agent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>69.55.235.87</td>\n",
       "      <td>NaN</td>\n",
       "      <td>user_63</td>\n",
       "      <td>[12/Nov/2018:12:05:07</td>\n",
       "      <td>+0000]</td>\n",
       "      <td>GET /login_success HTTP/1.1</td>\n",
       "      <td>200</td>\n",
       "      <td>476</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>69.57.9.174</td>\n",
       "      <td>NaN</td>\n",
       "      <td>user_63</td>\n",
       "      <td>[12/Nov/2018:13:20:30</td>\n",
       "      <td>+0000]</td>\n",
       "      <td>GET /login_success HTTP/1.1</td>\n",
       "      <td>200</td>\n",
       "      <td>476</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>69.57.11.177</td>\n",
       "      <td>NaN</td>\n",
       "      <td>user_63</td>\n",
       "      <td>[08/Nov/2018:13:43:03</td>\n",
       "      <td>+0000]</td>\n",
       "      <td>GET /login_success HTTP/1.1</td>\n",
       "      <td>200</td>\n",
       "      <td>476</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>69.56.56.126</td>\n",
       "      <td>NaN</td>\n",
       "      <td>user_63</td>\n",
       "      <td>[05/Nov/2018:07:16:20</td>\n",
       "      <td>+0000]</td>\n",
       "      <td>GET /login_success HTTP/1.1</td>\n",
       "      <td>200</td>\n",
       "      <td>476</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>69.57.109.29</td>\n",
       "      <td>NaN</td>\n",
       "      <td>user_63</td>\n",
       "      <td>[04/Nov/2018:14:57:55</td>\n",
       "      <td>+0000]</td>\n",
       "      <td>GET /login_success HTTP/1.1</td>\n",
       "      <td>200</td>\n",
       "      <td>476</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     ip_address  rcf_id     user              timestamp time_zone  \\\n",
       "0  69.55.235.87     NaN  user_63  [12/Nov/2018:12:05:07    +0000]   \n",
       "1   69.57.9.174     NaN  user_63  [12/Nov/2018:13:20:30    +0000]   \n",
       "2  69.57.11.177     NaN  user_63  [08/Nov/2018:13:43:03    +0000]   \n",
       "3  69.56.56.126     NaN  user_63  [05/Nov/2018:07:16:20    +0000]   \n",
       "4  69.57.109.29     NaN  user_63  [04/Nov/2018:14:57:55    +0000]   \n",
       "\n",
       "                       request  status  size  referer  \\\n",
       "0  GET /login_success HTTP/1.1     200   476      NaN   \n",
       "1  GET /login_success HTTP/1.1     200   476      NaN   \n",
       "2  GET /login_success HTTP/1.1     200   476      NaN   \n",
       "3  GET /login_success HTTP/1.1     200   476      NaN   \n",
       "4  GET /login_success HTTP/1.1     200   476      NaN   \n",
       "\n",
       "                                          user_agent  \n",
       "0  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6...  \n",
       "1  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6...  \n",
       "2  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6...  \n",
       "3  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6...  \n",
       "4  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6...  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\n",
    "    log_file,\n",
    "    sep=\" \",\n",
    "    na_values='-',\n",
    "    header=None,\n",
    "    names=['ip_address','rcf_id','user','timestamp','time_zone','request', 'status', 'size', 'referer', 'user_agent']\n",
    ")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3167344, 10)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert time stamps to DateTime objects\n",
    "df['timestamp'] = pd.to_datetime(df['timestamp'], format='[%d/%b/%Y:%H:%M:%S')\n",
    "\n",
    "# Check if they are all in the same timezone\n",
    "num_time_zones = len(df['time_zone'].unique())\n",
    "num_time_zones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### If there is more than one time_zone in your dataset, then we parse the timezone offset and update the corresponding datetime object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def apply_timezone(row):\n",
    "    tz = row[1]\n",
    "    tz_offset = int(tz[:3]) * 60   # Hour offset\n",
    "    tz_offset += int(tz[3:5])      # Minutes offset\n",
    "    return row[0].replace(tzinfo=pytz.FixedOffset(tz_offset))\n",
    "\n",
    "if num_time_zones > 1:\n",
    "    df['timestamp'] = df[['timestamp','time_zone']].apply(apply_timezone, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Select Resource\n",
    "Our goal is to train an IP Insights algorithm to analyze the history of user logins such that we can predict how suspicious a login event is.\n",
    "\n",
    "In our simulated web server, the server logs a GET request to the /login_success page everytime a user successfully logs in. We filter our Apache logs for GET requests for /login_success. We also filter for requests that have a status_code == 200, to ensure that the page request was well formed.\n",
    "\n",
    "Note: every web server handles logins differently. For your dataset, determine which resource you will need to be analyzing to correctly frame this problem. Depending on your usecase, you may need to do more data exploration and preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[(df['request'].str.startswith('GET /login_success')) & (df['status'] == 200)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Extract Users and IP address\n",
    "This Algorithm takes in a CSV file with two columns: an entity (username) ID string and the IPv4 address in decimal-dot notation. Fortunately, the Apache Web Server Access Logs output IP addresses and authentcated usernames in their own columns.\n",
    "\n",
    "Note: Each website handles user authentication differently. If the Access Log does not output an authenticated user, you could explore the website's query strings or work with your website developers on another solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[['user', 'ip_address', 'timestamp']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Create training and test dataset\n",
    "For any timeseries ML problem, we want to use out-of-time testing. Instead of randomly sampling our dataset, we split our dataset into two contiguous time windows. The first window is the training set, and the second is the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count                 3167344\n",
       "unique                 841691\n",
       "top       2018-11-13 05:50:21\n",
       "freq                       16\n",
       "first     2018-11-04 00:00:01\n",
       "last      2018-11-14 00:00:00\n",
       "Name: timestamp, dtype: object"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['timestamp'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have login events for 10 days. Let's take the first week (7 days) of data as training and then use the last 3 days for the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "time_partition = datetime(2018, 11, 11, tzinfo=pytz.FixedOffset(0)) if num_time_zones > 1 else datetime(2018, 11, 11)\n",
    "\n",
    "train_df = df[df['timestamp'] <= time_partition]\n",
    "test_df = df[df['timestamp'] > time_partition]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our training dataset, we shuffle it.\n",
    "\n",
    "Shuffling improves the model's performance since SageMaker IP Insights uses stochastic gradient descent. This ensures that login events for the same user are less likely to occur in the same mini batch. This allows the model to improve its performance in between predictions of the same user, which will improve training convergence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>ip_address</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2689795</th>\n",
       "      <td>user_8651</td>\n",
       "      <td>113.21.207.56</td>\n",
       "      <td>2018-11-08 08:53:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2302139</th>\n",
       "      <td>user_7396</td>\n",
       "      <td>50.115.220.217</td>\n",
       "      <td>2018-11-07 05:17:14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1573917</th>\n",
       "      <td>user_5200</td>\n",
       "      <td>66.237.49.103</td>\n",
       "      <td>2018-11-04 09:33:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1376593</th>\n",
       "      <td>user_4519</td>\n",
       "      <td>187.58.178.207</td>\n",
       "      <td>2018-11-08 18:42:04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1378182</th>\n",
       "      <td>user_4523</td>\n",
       "      <td>104.117.182.49</td>\n",
       "      <td>2018-11-05 00:48:52</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              user      ip_address           timestamp\n",
       "2689795  user_8651   113.21.207.56 2018-11-08 08:53:39\n",
       "2302139  user_7396  50.115.220.217 2018-11-07 05:17:14\n",
       "1573917  user_5200   66.237.49.103 2018-11-04 09:33:53\n",
       "1376593  user_4519  187.58.178.207 2018-11-08 18:42:04\n",
       "1378182  user_4523  104.117.182.49 2018-11-05 00:48:52"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Shuffle train data \n",
    "train_df = train_df.sample(frac=1)\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Upload data to S3 Bucket\n",
    "We will be doing local inference, therefore we don't need to upload our test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = train_df.to_csv(index=False, header=False, columns=['user', 'ip_address'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading data to: s3://sagemaker-us-west-2-570447867175/ip-insights\\train\\train.csv\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n"
     ]
    }
   ],
   "source": [
    "# Upload data to S3 key\n",
    "train_data_file = 'train.csv'\n",
    "key = os.path.join(prefix, 'train', train_data_file)\n",
    "s3_train_data = 's3://{}/{}'.format(bucket, key)\n",
    "\n",
    "print('Uploading data to: {}'.format(s3_train_data))\n",
    "boto3.resource('s3').Bucket(bucket).Object(key).put(Body=train_data)\n",
    "\n",
    "# Configure SageMaker IP Insights Input Channels\n",
    "input_data = {\n",
    "    'train': sagemaker.session.s3_input(s3_train_data, distribution='FullyReplicated', content_type='text/csv')\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Create a Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using GPU machines is recommeded as they will train faster. However, when the size of your dataset increases, it can become more economical to use multiple CPU machines running with distributed training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Training Job Configuration**\n",
    "- train_instance_type: the instance type to train on. We recommend p3.2xlarge for single GPU, p3.8xlarge for multi-GPU, and m5.2xlarge if using distributed training with CPU; \\\n",
    "\n",
    "- train_instance_count: the number of worker nodes in the training cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Model Hyperparameters**\n",
    "- num_entity_vectors: the total number of embeddings to train. We use an internal hashing mechanism to map the entity ID strings to an embedding index; therefore, using an embedding size larger than the total number of possible values helps reduce the number of hash collisions. We recommend this value to be 2x the total number of unique entites (i.e. user names) in your dataset; \\\n",
    "\n",
    "- vector_dim: the size of the entity and IP embedding vectors. The larger the value, the more information can be encoded using these representations but using too large vector representations may cause the model to overfit, especially for small training data sets; \\\n",
    "\n",
    "- num_ip_encoder_layers: the number of layers in the IP encoder network. The larger the number of layers, the higher the model capacity to capture patterns among IP addresses. However, large number of layers increases the chance of overfitting. num_ip_encoder_layers=1 is a good value to start experimenting with; \\\n",
    "\n",
    "- random_negative_sampling_rate: the number of randomly generated negative samples to produce per 1 positive sample; random_negative_sampling_rate=1 is a good value to start experimenting with; \\\n",
    "\n",
    "- Random negative samples are produced by drawing each octet from a uniform distributed of [0, 255]; \\\n",
    "\n",
    "- shuffled_negative_sampling_rate: the number of shuffled negative samples to produce per 1 positive sample; shuffled_negative_sampling_rate=1 is a good value to start experimenting with; \\\n",
    "\n",
    "\n",
    "**Training Hyperparameters**\n",
    "- epochs: the number of epochs to train. Increase this value if you continue to see the accuracy and cross entropy improving over the last few epochs; \\\n",
    "\n",
    "- mini_batch_size: how many examples in each mini_batch. A smaller number improves convergence with stochastic gradient descent. But a larger number is necessary if using shuffled_negative_sampling to avoid sampling a wrong account for a negative sample; \\\n",
    "\n",
    "- learning_rate: the learning rate for the Adam optimizer (try ranges in [0.001, 0.1]). Too large learning rate may cause the model to diverge since the training would be likely to overshoot minima. On the other hand, too small learning rate slows down the convergence; \\\n",
    "\n",
    "- weight_decay: L2 regularization coefficient. Regularization is required to prevent the model from overfitting the training data. Too large of a value will prevent the model from learning anything;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.\n"
     ]
    }
   ],
   "source": [
    "ip_insights = sagemaker.estimator.Estimator(\n",
    "    image, \n",
    "    role, \n",
    "    train_instance_count=1, \n",
    "    train_instance_type='ml.p2.xlarge',\n",
    "    output_path='s3://{}/{}/output'.format(bucket, prefix),\n",
    "    sagemaker_session=sess)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "ip_insights.set_hyperparameters(\n",
    "    num_entity_vectors='20000',\n",
    "    random_negative_sampling_rate='5',\n",
    "    vector_dim='128', \n",
    "    mini_batch_size='1000',\n",
    "    epochs='5',\n",
    "    learning_rate='0.01',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Start Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-07-12 21:39:28 Starting - Starting the training job...\n",
      "2020-07-12 21:39:29 Starting - Launching requested ML instances.........\n",
      "2020-07-12 21:41:06 Starting - Preparing the instances for training...\n",
      "2020-07-12 21:42:03 Downloading - Downloading input data\n",
      "2020-07-12 21:42:03 Training - Downloading the training image........\u001b[34mDocker entrypoint called with argument(s): train\u001b[0m\n",
      "\u001b[34mRunning default environment configuration script\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:26 INFO 140290456758080] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/resources/default-input.json: {u'random_negative_sampling_rate': u'1', u'shuffled_negative_sampling_rate': u'1', u'batch_metrics_publish_interval': u'1000', u'num_entity_vectors': u'100000', u'_tuning_objective_metric': u'', u'vector_dim': u'128', u'learning_rate': u'0.001', u'epochs': u'10', u'_num_gpus': u'auto', u'weight_decay': u'0.00001', u'_kvstore': u'auto_gpu', u'_num_kv_servers': u'auto', u'mini_batch_size': u'5000', u'_log_level': u'info', u'num_ip_encoder_layers': u'1'}\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:26 INFO 140290456758080] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'random_negative_sampling_rate': u'5', u'num_entity_vectors': u'20000', u'vector_dim': u'128', u'learning_rate': u'0.01', u'epochs': u'5', u'mini_batch_size': u'1000'}\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:26 INFO 140290456758080] Final configuration: {u'random_negative_sampling_rate': u'5', u'shuffled_negative_sampling_rate': u'1', u'batch_metrics_publish_interval': u'1000', u'num_entity_vectors': u'20000', u'_tuning_objective_metric': u'', u'vector_dim': u'128', u'learning_rate': u'0.01', u'epochs': u'5', u'_num_gpus': u'auto', u'weight_decay': u'0.00001', u'_kvstore': u'auto_gpu', u'_num_kv_servers': u'auto', u'mini_batch_size': u'1000', u'_log_level': u'info', u'num_ip_encoder_layers': u'1'}\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:26 WARNING 140290456758080] Loggers have already been setup.\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:26 INFO 140290456758080] nvidia-smi took: 0.0503859519958 secs to identify 1 gpus\u001b[0m\n",
      "\u001b[34mProcess 1 is a worker.\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:26 INFO 140290456758080] Using default worker.\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:26 INFO 140290456758080] Loaded iterator creator application/x-ndarray for content type ('application/x-ndarray', '1.0')\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:26 INFO 140290456758080] Loaded iterator creator text/csv for content type ('text/csv', '1.0')\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:26 INFO 140290456758080] Number of GPUs being used: 1\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"initialize.time\": {\"count\": 1, \"max\": 19.721031188964844, \"sum\": 19.721031188964844, \"min\": 19.721031188964844}}, \"EndTime\": 1594590206.479447, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594590206.459541}\n",
      "\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Records Seen\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Reset Count\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}}, \"EndTime\": 1594590206.479692, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"init_train_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594590206.479627}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:26 INFO 140290456758080] Create Store: device\u001b[0m\n",
      "\n",
      "2020-07-12 21:43:22 Training - Training image download completed. Training in progress.\u001b[34m[21:43:30] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.3.x_Cuda_10.1.x.888.0/AL2012/generic-flavor/src/src/kvstore/./kvstore_local.h:306: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use kv.row_sparse_pull() or module.prepare() with row_ids.\u001b[0m\n",
      "\u001b[34m[21:43:30] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.3.x_Cuda_10.1.x.888.0/AL2012/generic-flavor/src/src/operator/././../common/utils.h:450: Optimizer with lazy_update = True detected. Be aware that lazy update with row_sparse gradient is different from standard update, and may lead to different empirical results. See https://mxnet.incubator.apache.org/api/python/optimization/optimization.html for more details.\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:30 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=0 train binary_classification_accuracy <score>=0.534\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:30 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=0 train binary_classification_cross_entropy <loss>=0.693078674316\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:40 INFO 140290456758080] Epoch[0] Batch [1000]#011Speed: 99663.37 samples/sec#011binary_classification_accuracy=0.927379#011binary_classification_cross_entropy=0.197896\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:40 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=1000 train binary_classification_accuracy <score>=0.927378621379\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:40 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=1000 train binary_classification_cross_entropy <loss>=0.197895570814\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:50 INFO 140290456758080] Epoch[0] Batch [2000]#011Speed: 104019.79 samples/sec#011binary_classification_accuracy=0.949504#011binary_classification_cross_entropy=0.148697\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:50 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=2000 train binary_classification_accuracy <score>=0.949504247876\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:43:50 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=2000 train binary_classification_cross_entropy <loss>=0.148697323944\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:00 INFO 140290456758080] Epoch[0] Batch [3000]#011Speed: 103558.21 samples/sec#011binary_classification_accuracy=0.959047#011binary_classification_cross_entropy=0.126482\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:00 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=3000 train binary_classification_accuracy <score>=0.959046651116\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:00 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=3000 train binary_classification_cross_entropy <loss>=0.12648197314\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:09 INFO 140290456758080] Epoch[0] Batch [4000]#011Speed: 103496.33 samples/sec#011binary_classification_accuracy=0.964641#011binary_classification_cross_entropy=0.113359\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:09 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=4000 train binary_classification_accuracy <score>=0.964641339665\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:09 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=4000 train binary_classification_cross_entropy <loss>=0.113358909865\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:19 INFO 140290456758080] Epoch[0] Batch [5000]#011Speed: 104563.27 samples/sec#011binary_classification_accuracy=0.968190#011binary_classification_cross_entropy=0.104990\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:19 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=5000 train binary_classification_accuracy <score>=0.968189562088\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:19 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=5000 train binary_classification_cross_entropy <loss>=0.104989791161\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:28 INFO 140290456758080] Epoch[0] Batch [6000]#011Speed: 104192.81 samples/sec#011binary_classification_accuracy=0.970735#011binary_classification_cross_entropy=0.098729\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:28 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=6000 train binary_classification_accuracy <score>=0.970735044159\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:28 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=6000 train binary_classification_cross_entropy <loss>=0.0987285504712\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:38 INFO 140290456758080] Epoch[0] Batch [7000]#011Speed: 103452.68 samples/sec#011binary_classification_accuracy=0.972637#011binary_classification_cross_entropy=0.094153\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:38 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=7000 train binary_classification_accuracy <score>=0.972637337523\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:38 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=7000 train binary_classification_cross_entropy <loss>=0.094152812351\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:48 INFO 140290456758080] Epoch[0] Batch [8000]#011Speed: 103934.83 samples/sec#011binary_classification_accuracy=0.974105#011binary_classification_cross_entropy=0.090500\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:48 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=8000 train binary_classification_accuracy <score>=0.974105236845\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:48 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=8000 train binary_classification_cross_entropy <loss>=0.0905004167985\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:57 INFO 140290456758080] Epoch[0] Batch [9000]#011Speed: 104537.00 samples/sec#011binary_classification_accuracy=0.975318#011binary_classification_cross_entropy=0.087522\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:57 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=9000 train binary_classification_accuracy <score>=0.975318297967\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:44:57 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=9000 train binary_classification_cross_entropy <loss>=0.0875218849542\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:07 INFO 140290456758080] Epoch[0] Batch [10000]#011Speed: 103858.81 samples/sec#011binary_classification_accuracy=0.976313#011binary_classification_cross_entropy=0.085081\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:07 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=10000 train binary_classification_accuracy <score>=0.976313068693\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:07 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=10000 train binary_classification_cross_entropy <loss>=0.0850807577888\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:16 INFO 140290456758080] Epoch[0] Batch [11000]#011Speed: 104347.68 samples/sec#011binary_classification_accuracy=0.977137#011binary_classification_cross_entropy=0.083096\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:16 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=11000 train binary_classification_accuracy <score>=0.977137169348\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:16 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=11000 train binary_classification_cross_entropy <loss>=0.0830964903958\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:26 INFO 140290456758080] Epoch[0] Batch [12000]#011Speed: 104549.31 samples/sec#011binary_classification_accuracy=0.977841#011binary_classification_cross_entropy=0.081387\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:26 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=12000 train binary_classification_accuracy <score>=0.977840679943\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:26 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=12000 train binary_classification_cross_entropy <loss>=0.0813870704994\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:36 INFO 140290456758080] Epoch[0] Batch [13000]#011Speed: 104071.54 samples/sec#011binary_classification_accuracy=0.978435#011binary_classification_cross_entropy=0.079953\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:36 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=13000 train binary_classification_accuracy <score>=0.978434581955\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:36 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=13000 train binary_classification_cross_entropy <loss>=0.0799534454302\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:45 INFO 140290456758080] Epoch[0] Batch [14000]#011Speed: 104109.96 samples/sec#011binary_classification_accuracy=0.978964#011binary_classification_cross_entropy=0.078656\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:45 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=14000 train binary_classification_accuracy <score>=0.978963931148\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:45 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=14000 train binary_classification_cross_entropy <loss>=0.0786558744478\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:55 INFO 140290456758080] Epoch[0] Batch [15000]#011Speed: 104562.80 samples/sec#011binary_classification_accuracy=0.979435#011binary_classification_cross_entropy=0.077502\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:55 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=15000 train binary_classification_accuracy <score>=0.979435437637\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:45:55 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, batch=15000 train binary_classification_cross_entropy <loss>=0.0775020869475\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:00 INFO 140290456758080] Epoch[0] Train-binary_classification_accuracy=0.979671\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:00 INFO 140290456758080] Epoch[0] Train-binary_classification_cross_entropy=0.076901\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:00 INFO 140290456758080] Epoch[0] Time cost=149.674\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:00 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, train binary_classification_accuracy <score>=0.979671132294\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:00 INFO 140290456758080] #quality_metric: host=algo-1, epoch=0, train binary_classification_cross_entropy <loss>=0.0769012898197\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"epochs\": {\"count\": 1, \"max\": 5, \"sum\": 5.0, \"min\": 5}, \"update.time\": {\"count\": 1, \"max\": 153872.97892570496, \"sum\": 153872.97892570496, \"min\": 153872.97892570496}}, \"EndTime\": 1594590360.352956, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594590206.479542}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:00 INFO 140290456758080] #progress_metric: host=algo-1, completed 20 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 15526, \"sum\": 15526.0, \"min\": 15526}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 15526, \"sum\": 15526.0, \"min\": 15526}, \"Total Records Seen\": {\"count\": 1, \"max\": 15520099, \"sum\": 15520099.0, \"min\": 15520099}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 15520099, \"sum\": 15520099.0, \"min\": 15520099}, \"Reset Count\": {\"count\": 1, \"max\": 2, \"sum\": 2.0, \"min\": 2}}, \"EndTime\": 1594590360.353249, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\", \"epoch\": 0}, \"StartTime\": 1594590206.479932}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:00 INFO 140290456758080] #throughput_metric: host=algo-1, train throughput=100862.745243 records/second\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:00 WARNING 140290456758080] Already bound, ignoring bind()\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python2.7/site-packages/mxnet/module/base_module.py:502: UserWarning: Parameters already initialized and force_init=False. init_params call ignored.\n",
      "  allow_missing=allow_missing, force_init=force_init)\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:00 WARNING 140290456758080] optimizer already initialized, ignoring...\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:00 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=0 train binary_classification_accuracy <score>=0.987\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:00 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=0 train binary_classification_cross_entropy <loss>=0.0568755378723\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:09 INFO 140290456758080] Epoch[1] Batch [1000]#011Speed: 104042.85 samples/sec#011binary_classification_accuracy=0.986021#011binary_classification_cross_entropy=0.060122\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:09 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=1000 train binary_classification_accuracy <score>=0.986020979021\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:09 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=1000 train binary_classification_cross_entropy <loss>=0.0601221248711\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:19 INFO 140290456758080] Epoch[1] Batch [2000]#011Speed: 105078.97 samples/sec#011binary_classification_accuracy=0.986187#011binary_classification_cross_entropy=0.059393\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:19 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=2000 train binary_classification_accuracy <score>=0.986187406297\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:19 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=2000 train binary_classification_cross_entropy <loss>=0.0593930235362\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:29 INFO 140290456758080] Epoch[1] Batch [3000]#011Speed: 104171.27 samples/sec#011binary_classification_accuracy=0.986259#011binary_classification_cross_entropy=0.058987\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:29 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=3000 train binary_classification_accuracy <score>=0.986258580473\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:29 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=3000 train binary_classification_cross_entropy <loss>=0.0589874663011\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:38 INFO 140290456758080] Epoch[1] Batch [4000]#011Speed: 103664.93 samples/sec#011binary_classification_accuracy=0.986301#011binary_classification_cross_entropy=0.058816\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:38 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=4000 train binary_classification_accuracy <score>=0.986301424644\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:38 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=4000 train binary_classification_cross_entropy <loss>=0.0588157371859\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:48 INFO 140290456758080] Epoch[1] Batch [5000]#011Speed: 104011.09 samples/sec#011binary_classification_accuracy=0.986285#011binary_classification_cross_entropy=0.058863\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:48 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=5000 train binary_classification_accuracy <score>=0.986285342931\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:48 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=5000 train binary_classification_cross_entropy <loss>=0.0588633508345\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:57 INFO 140290456758080] Epoch[1] Batch [6000]#011Speed: 103912.50 samples/sec#011binary_classification_accuracy=0.986351#011binary_classification_cross_entropy=0.058582\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:57 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=6000 train binary_classification_accuracy <score>=0.986351274788\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:46:57 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=6000 train binary_classification_cross_entropy <loss>=0.0585820627672\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:07 INFO 140290456758080] Epoch[1] Batch [7000]#011Speed: 104337.86 samples/sec#011binary_classification_accuracy=0.986376#011binary_classification_cross_entropy=0.058502\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:07 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=7000 train binary_classification_accuracy <score>=0.986375946293\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:07 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=7000 train binary_classification_cross_entropy <loss>=0.0585016148625\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:17 INFO 140290456758080] Epoch[1] Batch [8000]#011Speed: 104582.20 samples/sec#011binary_classification_accuracy=0.986390#011binary_classification_cross_entropy=0.058380\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:17 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=8000 train binary_classification_accuracy <score>=0.986389576303\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:17 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=8000 train binary_classification_cross_entropy <loss>=0.0583798328102\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:26 INFO 140290456758080] Epoch[1] Batch [9000]#011Speed: 105040.63 samples/sec#011binary_classification_accuracy=0.986430#011binary_classification_cross_entropy=0.058220\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:26 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=9000 train binary_classification_accuracy <score>=0.986430174425\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:26 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=9000 train binary_classification_cross_entropy <loss>=0.058220140134\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:36 INFO 140290456758080] Epoch[1] Batch [10000]#011Speed: 104525.50 samples/sec#011binary_classification_accuracy=0.986458#011binary_classification_cross_entropy=0.058118\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:36 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=10000 train binary_classification_accuracy <score>=0.986458054195\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:36 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=10000 train binary_classification_cross_entropy <loss>=0.0581183660837\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:45 INFO 140290456758080] Epoch[1] Batch [11000]#011Speed: 104689.02 samples/sec#011binary_classification_accuracy=0.986455#011binary_classification_cross_entropy=0.058143\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:45 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=11000 train binary_classification_accuracy <score>=0.986454867739\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:45 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=11000 train binary_classification_cross_entropy <loss>=0.0581429156544\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:55 INFO 140290456758080] Epoch[1] Batch [12000]#011Speed: 104885.91 samples/sec#011binary_classification_accuracy=0.986476#011binary_classification_cross_entropy=0.058065\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:55 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=12000 train binary_classification_accuracy <score>=0.986476376969\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:47:55 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=12000 train binary_classification_cross_entropy <loss>=0.058065161943\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:04 INFO 140290456758080] Epoch[1] Batch [13000]#011Speed: 104441.68 samples/sec#011binary_classification_accuracy=0.986479#011binary_classification_cross_entropy=0.058081\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:04 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=13000 train binary_classification_accuracy <score>=0.986479270825\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:04 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=13000 train binary_classification_cross_entropy <loss>=0.0580814413074\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:14 INFO 140290456758080] Epoch[1] Batch [14000]#011Speed: 104465.62 samples/sec#011binary_classification_accuracy=0.986512#011binary_classification_cross_entropy=0.057999\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:14 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=14000 train binary_classification_accuracy <score>=0.986511963431\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:14 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=14000 train binary_classification_cross_entropy <loss>=0.0579992012906\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:23 INFO 140290456758080] Epoch[1] Batch [15000]#011Speed: 105081.62 samples/sec#011binary_classification_accuracy=0.986530#011binary_classification_cross_entropy=0.057979\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:23 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=15000 train binary_classification_accuracy <score>=0.986530164656\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:23 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, batch=15000 train binary_classification_cross_entropy <loss>=0.0579789409648\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:28 INFO 140290456758080] Epoch[1] Train-binary_classification_accuracy=0.986546\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:28 INFO 140290456758080] Epoch[1] Train-binary_classification_cross_entropy=0.057911\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:28 INFO 140290456758080] Epoch[1] Time cost=148.606\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:28 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, train binary_classification_accuracy <score>=0.986546245008\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:28 INFO 140290456758080] #quality_metric: host=algo-1, epoch=1, train binary_classification_cross_entropy <loss>=0.0579105245688\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 148611.01508140564, \"sum\": 148611.01508140564, \"min\": 148611.01508140564}}, \"EndTime\": 1594590508.964606, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594590360.353043}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:28 INFO 140290456758080] #progress_metric: host=algo-1, completed 40 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 15526, \"sum\": 15526.0, \"min\": 15526}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 31052, \"sum\": 31052.0, \"min\": 31052}, \"Total Records Seen\": {\"count\": 1, \"max\": 31040198, \"sum\": 31040198.0, \"min\": 31040198}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 15520099, \"sum\": 15520099.0, \"min\": 15520099}, \"Reset Count\": {\"count\": 1, \"max\": 4, \"sum\": 4.0, \"min\": 4}}, \"EndTime\": 1594590508.964906, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\", \"epoch\": 1}, \"StartTime\": 1594590360.353555}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:28 INFO 140290456758080] #throughput_metric: host=algo-1, train throughput=104434.039233 records/second\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:28 WARNING 140290456758080] Already bound, ignoring bind()\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:28 WARNING 140290456758080] optimizer already initialized, ignoring...\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:28 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=0 train binary_classification_accuracy <score>=0.981\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:28 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=0 train binary_classification_cross_entropy <loss>=0.0935693969727\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:38 INFO 140290456758080] Epoch[2] Batch [1000]#011Speed: 104158.45 samples/sec#011binary_classification_accuracy=0.986603#011binary_classification_cross_entropy=0.057363\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:38 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=1000 train binary_classification_accuracy <score>=0.986603396603\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:38 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=1000 train binary_classification_cross_entropy <loss>=0.0573629361133\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:48 INFO 140290456758080] Epoch[2] Batch [2000]#011Speed: 104156.77 samples/sec#011binary_classification_accuracy=0.986782#011binary_classification_cross_entropy=0.056670\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:48 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=2000 train binary_classification_accuracy <score>=0.986782108946\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:48 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=2000 train binary_classification_cross_entropy <loss>=0.056669622224\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:57 INFO 140290456758080] Epoch[2] Batch [3000]#011Speed: 103655.41 samples/sec#011binary_classification_accuracy=0.986813#011binary_classification_cross_entropy=0.056446\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:57 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=3000 train binary_classification_accuracy <score>=0.986813062313\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:48:57 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=3000 train binary_classification_cross_entropy <loss>=0.0564464969597\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:07 INFO 140290456758080] Epoch[2] Batch [4000]#011Speed: 103648.12 samples/sec#011binary_classification_accuracy=0.986893#011binary_classification_cross_entropy=0.056227\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:07 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=4000 train binary_classification_accuracy <score>=0.986892526868\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:07 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=4000 train binary_classification_cross_entropy <loss>=0.0562272271512\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:17 INFO 140290456758080] Epoch[2] Batch [5000]#011Speed: 103982.87 samples/sec#011binary_classification_accuracy=0.986830#011binary_classification_cross_entropy=0.056479\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:17 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=5000 train binary_classification_accuracy <score>=0.986829834033\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:17 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=5000 train binary_classification_cross_entropy <loss>=0.0564792482418\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:26 INFO 140290456758080] Epoch[2] Batch [6000]#011Speed: 104368.97 samples/sec#011binary_classification_accuracy=0.986862#011binary_classification_cross_entropy=0.056322\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:26 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=6000 train binary_classification_accuracy <score>=0.986861856357\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:26 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=6000 train binary_classification_cross_entropy <loss>=0.0563219734212\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:36 INFO 140290456758080] Epoch[2] Batch [7000]#011Speed: 104038.05 samples/sec#011binary_classification_accuracy=0.986849#011binary_classification_cross_entropy=0.056332\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:36 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=7000 train binary_classification_accuracy <score>=0.986849021568\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:36 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=7000 train binary_classification_cross_entropy <loss>=0.0563323624865\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:45 INFO 140290456758080] Epoch[2] Batch [8000]#011Speed: 103993.34 samples/sec#011binary_classification_accuracy=0.986849#011binary_classification_cross_entropy=0.056245\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:45 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=8000 train binary_classification_accuracy <score>=0.986849268841\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:45 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=8000 train binary_classification_cross_entropy <loss>=0.0562447829435\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:55 INFO 140290456758080] Epoch[2] Batch [9000]#011Speed: 104791.33 samples/sec#011binary_classification_accuracy=0.986855#011binary_classification_cross_entropy=0.056176\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:55 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=9000 train binary_classification_accuracy <score>=0.986854905011\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:49:55 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=9000 train binary_classification_cross_entropy <loss>=0.0561761958391\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:05 INFO 140290456758080] Epoch[2] Batch [10000]#011Speed: 103868.64 samples/sec#011binary_classification_accuracy=0.986879#011binary_classification_cross_entropy=0.056069\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:05 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=10000 train binary_classification_accuracy <score>=0.986879312069\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:05 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=10000 train binary_classification_cross_entropy <loss>=0.0560691429116\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:14 INFO 140290456758080] Epoch[2] Batch [11000]#011Speed: 103935.09 samples/sec#011binary_classification_accuracy=0.986889#011binary_classification_cross_entropy=0.056084\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:14 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=11000 train binary_classification_accuracy <score>=0.986889282792\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:14 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=11000 train binary_classification_cross_entropy <loss>=0.0560842336364\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:24 INFO 140290456758080] Epoch[2] Batch [12000]#011Speed: 104436.62 samples/sec#011binary_classification_accuracy=0.986904#011binary_classification_cross_entropy=0.056015\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:24 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=12000 train binary_classification_accuracy <score>=0.986904257979\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:24 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=12000 train binary_classification_cross_entropy <loss>=0.0560146840795\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:33 INFO 140290456758080] Epoch[2] Batch [13000]#011Speed: 104378.18 samples/sec#011binary_classification_accuracy=0.986920#011binary_classification_cross_entropy=0.055990\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:33 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=13000 train binary_classification_accuracy <score>=0.986919775402\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:33 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=13000 train binary_classification_cross_entropy <loss>=0.0559898139149\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:43 INFO 140290456758080] Epoch[2] Batch [14000]#011Speed: 104202.63 samples/sec#011binary_classification_accuracy=0.986932#011binary_classification_cross_entropy=0.055938\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:43 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=14000 train binary_classification_accuracy <score>=0.986932433398\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:43 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=14000 train binary_classification_cross_entropy <loss>=0.0559381256928\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:53 INFO 140290456758080] Epoch[2] Batch [15000]#011Speed: 104648.11 samples/sec#011binary_classification_accuracy=0.986938#011binary_classification_cross_entropy=0.055930\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:53 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=15000 train binary_classification_accuracy <score>=0.986938137458\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:53 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, batch=15000 train binary_classification_cross_entropy <loss>=0.0559301995356\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:58 INFO 140290456758080] Epoch[2] Train-binary_classification_accuracy=0.986953\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:58 INFO 140290456758080] Epoch[2] Train-binary_classification_cross_entropy=0.055876\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:58 INFO 140290456758080] Epoch[2] Time cost=149.088\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:58 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, train binary_classification_accuracy <score>=0.986952853278\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:58 INFO 140290456758080] #quality_metric: host=algo-1, epoch=2, train binary_classification_cross_entropy <loss>=0.0558762195239\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 149092.42415428162, \"sum\": 149092.42415428162, \"min\": 149092.42415428162}}, \"EndTime\": 1594590658.057669, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594590508.964695}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:58 INFO 140290456758080] #progress_metric: host=algo-1, completed 60 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 15526, \"sum\": 15526.0, \"min\": 15526}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 46578, \"sum\": 46578.0, \"min\": 46578}, \"Total Records Seen\": {\"count\": 1, \"max\": 46560297, \"sum\": 46560297.0, \"min\": 46560297}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 15520099, \"sum\": 15520099.0, \"min\": 15520099}, \"Reset Count\": {\"count\": 1, \"max\": 6, \"sum\": 6.0, \"min\": 6}}, \"EndTime\": 1594590658.057933, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\", \"epoch\": 2}, \"StartTime\": 1594590508.965207}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:58 INFO 140290456758080] #throughput_metric: host=algo-1, train throughput=104096.870523 records/second\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:58 WARNING 140290456758080] Already bound, ignoring bind()\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:58 WARNING 140290456758080] optimizer already initialized, ignoring...\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:58 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=0 train binary_classification_accuracy <score>=0.993\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:50:58 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=0 train binary_classification_cross_entropy <loss>=0.030358543396\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:07 INFO 140290456758080] Epoch[3] Batch [1000]#011Speed: 102102.74 samples/sec#011binary_classification_accuracy=0.987019#011binary_classification_cross_entropy=0.055592\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:07 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=1000 train binary_classification_accuracy <score>=0.987018981019\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:07 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=1000 train binary_classification_cross_entropy <loss>=0.0555919303189\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:17 INFO 140290456758080] Epoch[3] Batch [2000]#011Speed: 100617.43 samples/sec#011binary_classification_accuracy=0.987018#011binary_classification_cross_entropy=0.055376\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:17 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=2000 train binary_classification_accuracy <score>=0.987017991004\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:17 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=2000 train binary_classification_cross_entropy <loss>=0.0553757469696\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:27 INFO 140290456758080] Epoch[3] Batch [3000]#011Speed: 104611.40 samples/sec#011binary_classification_accuracy=0.987025#011binary_classification_cross_entropy=0.055160\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:27 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=3000 train binary_classification_accuracy <score>=0.987025324892\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:27 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=3000 train binary_classification_cross_entropy <loss>=0.0551604772032\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:36 INFO 140290456758080] Epoch[3] Batch [4000]#011Speed: 104648.73 samples/sec#011binary_classification_accuracy=0.987037#011binary_classification_cross_entropy=0.055206\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:36 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=4000 train binary_classification_accuracy <score>=0.987037490627\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:36 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=4000 train binary_classification_cross_entropy <loss>=0.0552062090521\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:46 INFO 140290456758080] Epoch[3] Batch [5000]#011Speed: 104158.83 samples/sec#011binary_classification_accuracy=0.987038#011binary_classification_cross_entropy=0.055262\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:46 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=5000 train binary_classification_accuracy <score>=0.987037792442\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:46 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=5000 train binary_classification_cross_entropy <loss>=0.0552620117384\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:56 INFO 140290456758080] Epoch[3] Batch [6000]#011Speed: 104566.60 samples/sec#011binary_classification_accuracy=0.987053#011binary_classification_cross_entropy=0.055129\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:56 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=6000 train binary_classification_accuracy <score>=0.987052991168\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:51:56 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=6000 train binary_classification_cross_entropy <loss>=0.0551291139756\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:05 INFO 140290456758080] Epoch[3] Batch [7000]#011Speed: 104023.76 samples/sec#011binary_classification_accuracy=0.987026#011binary_classification_cross_entropy=0.055151\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:05 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=7000 train binary_classification_accuracy <score>=0.987026139123\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:05 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=7000 train binary_classification_cross_entropy <loss>=0.0551510630345\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:15 INFO 140290456758080] Epoch[3] Batch [8000]#011Speed: 104438.59 samples/sec#011binary_classification_accuracy=0.987019#011binary_classification_cross_entropy=0.055104\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:15 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=8000 train binary_classification_accuracy <score>=0.987018622672\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:15 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=8000 train binary_classification_cross_entropy <loss>=0.0551041008087\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:24 INFO 140290456758080] Epoch[3] Batch [9000]#011Speed: 104887.55 samples/sec#011binary_classification_accuracy=0.987034#011binary_classification_cross_entropy=0.055057\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:24 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=9000 train binary_classification_accuracy <score>=0.98703421842\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:24 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=9000 train binary_classification_cross_entropy <loss>=0.0550566067661\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:34 INFO 140290456758080] Epoch[3] Batch [10000]#011Speed: 104301.41 samples/sec#011binary_classification_accuracy=0.987052#011binary_classification_cross_entropy=0.054990\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:34 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=10000 train binary_classification_accuracy <score>=0.987051994801\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:34 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=10000 train binary_classification_cross_entropy <loss>=0.0549895611008\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:43 INFO 140290456758080] Epoch[3] Batch [11000]#011Speed: 104881.99 samples/sec#011binary_classification_accuracy=0.987059#011binary_classification_cross_entropy=0.054991\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:43 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=11000 train binary_classification_accuracy <score>=0.987058721934\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:43 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=11000 train binary_classification_cross_entropy <loss>=0.0549914449415\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:53 INFO 140290456758080] Epoch[3] Batch [12000]#011Speed: 104194.65 samples/sec#011binary_classification_accuracy=0.987064#011binary_classification_cross_entropy=0.054960\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:53 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=12000 train binary_classification_accuracy <score>=0.987063578035\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:52:53 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=12000 train binary_classification_cross_entropy <loss>=0.0549602478338\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:03 INFO 140290456758080] Epoch[3] Batch [13000]#011Speed: 104139.83 samples/sec#011binary_classification_accuracy=0.987057#011binary_classification_cross_entropy=0.054985\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:03 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=13000 train binary_classification_accuracy <score>=0.987057380202\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:03 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=13000 train binary_classification_cross_entropy <loss>=0.0549845053307\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:12 INFO 140290456758080] Epoch[3] Batch [14000]#011Speed: 104969.82 samples/sec#011binary_classification_accuracy=0.987063#011binary_classification_cross_entropy=0.054959\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:12 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=14000 train binary_classification_accuracy <score>=0.98706256696\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:12 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=14000 train binary_classification_cross_entropy <loss>=0.0549588589134\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:22 INFO 140290456758080] Epoch[3] Batch [15000]#011Speed: 105126.79 samples/sec#011binary_classification_accuracy=0.987079#011binary_classification_cross_entropy=0.054934\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:22 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=15000 train binary_classification_accuracy <score>=0.987078661423\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:22 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, batch=15000 train binary_classification_cross_entropy <loss>=0.0549344107711\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:27 INFO 140290456758080] Epoch[3] Train-binary_classification_accuracy=0.987093\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:27 INFO 140290456758080] Epoch[3] Train-binary_classification_cross_entropy=0.054876\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:27 INFO 140290456758080] Epoch[3] Time cost=149.095\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:27 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, train binary_classification_accuracy <score>=0.987092747649\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:27 INFO 140290456758080] #quality_metric: host=algo-1, epoch=3, train binary_classification_cross_entropy <loss>=0.0548757663313\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 149098.92106056213, \"sum\": 149098.92106056213, \"min\": 149098.92106056213}}, \"EndTime\": 1594590807.157128, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594590658.057742}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:27 INFO 140290456758080] #progress_metric: host=algo-1, completed 80 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 15526, \"sum\": 15526.0, \"min\": 15526}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 62104, \"sum\": 62104.0, \"min\": 62104}, \"Total Records Seen\": {\"count\": 1, \"max\": 62080396, \"sum\": 62080396.0, \"min\": 62080396}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 15520099, \"sum\": 15520099.0, \"min\": 15520099}, \"Reset Count\": {\"count\": 1, \"max\": 8, \"sum\": 8.0, \"min\": 8}}, \"EndTime\": 1594590807.157476, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\", \"epoch\": 3}, \"StartTime\": 1594590658.058165}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:27 INFO 140290456758080] #throughput_metric: host=algo-1, train throughput=104092.255507 records/second\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:27 WARNING 140290456758080] Already bound, ignoring bind()\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:27 WARNING 140290456758080] optimizer already initialized, ignoring...\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:27 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=0 train binary_classification_accuracy <score>=0.991\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:27 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=0 train binary_classification_cross_entropy <loss>=0.0438944282532\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:36 INFO 140290456758080] Epoch[4] Batch [1000]#011Speed: 104265.34 samples/sec#011binary_classification_accuracy=0.987096#011binary_classification_cross_entropy=0.054600\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:36 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=1000 train binary_classification_accuracy <score>=0.987095904096\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:36 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=1000 train binary_classification_cross_entropy <loss>=0.0546003005421\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:46 INFO 140290456758080] Epoch[4] Batch [2000]#011Speed: 104854.19 samples/sec#011binary_classification_accuracy=0.987062#011binary_classification_cross_entropy=0.054534\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:46 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=2000 train binary_classification_accuracy <score>=0.987061969015\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:46 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=2000 train binary_classification_cross_entropy <loss>=0.054533971675\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:55 INFO 140290456758080] Epoch[4] Batch [3000]#011Speed: 104458.27 samples/sec#011binary_classification_accuracy=0.987125#011binary_classification_cross_entropy=0.054359\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:55 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=3000 train binary_classification_accuracy <score>=0.987124625125\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:53:55 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=3000 train binary_classification_cross_entropy <loss>=0.0543585080455\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:05 INFO 140290456758080] Epoch[4] Batch [4000]#011Speed: 103085.58 samples/sec#011binary_classification_accuracy=0.987159#011binary_classification_cross_entropy=0.054312\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:05 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=4000 train binary_classification_accuracy <score>=0.987158710322\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:05 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=4000 train binary_classification_cross_entropy <loss>=0.0543124273758\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:15 INFO 140290456758080] Epoch[4] Batch [5000]#011Speed: 104557.10 samples/sec#011binary_classification_accuracy=0.987123#011binary_classification_cross_entropy=0.054453\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:15 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=5000 train binary_classification_accuracy <score>=0.987123375325\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:15 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=5000 train binary_classification_cross_entropy <loss>=0.0544529523657\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:24 INFO 140290456758080] Epoch[4] Batch [6000]#011Speed: 104369.46 samples/sec#011binary_classification_accuracy=0.987138#011binary_classification_cross_entropy=0.054298\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:24 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=6000 train binary_classification_accuracy <score>=0.987138476921\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:24 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=6000 train binary_classification_cross_entropy <loss>=0.0542975448442\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:34 INFO 140290456758080] Epoch[4] Batch [7000]#011Speed: 104196.97 samples/sec#011binary_classification_accuracy=0.987115#011binary_classification_cross_entropy=0.054348\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:34 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=7000 train binary_classification_accuracy <score>=0.987114983574\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:34 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=7000 train binary_classification_cross_entropy <loss>=0.054348035164\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:43 INFO 140290456758080] Epoch[4] Batch [8000]#011Speed: 104005.66 samples/sec#011binary_classification_accuracy=0.987108#011binary_classification_cross_entropy=0.054297\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:43 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=8000 train binary_classification_accuracy <score>=0.987108361455\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:43 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=8000 train binary_classification_cross_entropy <loss>=0.0542970493633\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:53 INFO 140290456758080] Epoch[4] Batch [9000]#011Speed: 103632.39 samples/sec#011binary_classification_accuracy=0.987132#011binary_classification_cross_entropy=0.054267\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:53 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=9000 train binary_classification_accuracy <score>=0.987131763137\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:54:53 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=9000 train binary_classification_cross_entropy <loss>=0.0542671777924\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:03 INFO 140290456758080] Epoch[4] Batch [10000]#011Speed: 103850.04 samples/sec#011binary_classification_accuracy=0.987153#011binary_classification_cross_entropy=0.054199\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:03 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=10000 train binary_classification_accuracy <score>=0.987152684732\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:03 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=10000 train binary_classification_cross_entropy <loss>=0.0541990754425\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:12 INFO 140290456758080] Epoch[4] Batch [11000]#011Speed: 104017.97 samples/sec#011binary_classification_accuracy=0.987160#011binary_classification_cross_entropy=0.054221\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:12 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=11000 train binary_classification_accuracy <score>=0.987159894555\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:12 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=11000 train binary_classification_cross_entropy <loss>=0.0542205842116\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:22 INFO 140290456758080] Epoch[4] Batch [12000]#011Speed: 104984.63 samples/sec#011binary_classification_accuracy=0.987169#011binary_classification_cross_entropy=0.054183\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:22 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=12000 train binary_classification_accuracy <score>=0.98716940255\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:22 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=12000 train binary_classification_cross_entropy <loss>=0.054182553279\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:31 INFO 140290456758080] Epoch[4] Batch [13000]#011Speed: 103837.59 samples/sec#011binary_classification_accuracy=0.987160#011binary_classification_cross_entropy=0.054216\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:31 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=13000 train binary_classification_accuracy <score>=0.987159603107\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:31 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=13000 train binary_classification_cross_entropy <loss>=0.0542164731146\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:41 INFO 140290456758080] Epoch[4] Batch [14000]#011Speed: 104048.77 samples/sec#011binary_classification_accuracy=0.987174#011binary_classification_cross_entropy=0.054175\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:41 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=14000 train binary_classification_accuracy <score>=0.98717434469\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:41 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=14000 train binary_classification_cross_entropy <loss>=0.054175258355\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:51 INFO 140290456758080] Epoch[4] Batch [15000]#011Speed: 103722.74 samples/sec#011binary_classification_accuracy=0.987176#011binary_classification_cross_entropy=0.054183\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:51 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=15000 train binary_classification_accuracy <score>=0.987176388241\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:51 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, batch=15000 train binary_classification_cross_entropy <loss>=0.0541827544703\u001b[0m\n",
      "\n",
      "2020-07-12 21:56:07 Uploading - Uploading generated training model\n",
      "2020-07-12 21:56:07 Completed - Training job completed\n",
      "\u001b[34m[07/12/2020 21:55:56 INFO 140290456758080] Epoch[4] Train-binary_classification_accuracy=0.987198\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:56 INFO 140290456758080] Epoch[4] Train-binary_classification_cross_entropy=0.054112\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:56 INFO 140290456758080] Epoch[4] Time cost=149.156\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:56 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, train binary_classification_accuracy <score>=0.987197990468\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:56 INFO 140290456758080] #quality_metric: host=algo-1, epoch=4, train binary_classification_cross_entropy <loss>=0.0541115370908\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 149161.7112159729, \"sum\": 149161.7112159729, \"min\": 149161.7112159729}}, \"EndTime\": 1594590956.319563, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594590807.157232}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:56 INFO 140290456758080] #progress_metric: host=algo-1, completed 100 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 15526, \"sum\": 15526.0, \"min\": 15526}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 77630, \"sum\": 77630.0, \"min\": 77630}, \"Total Records Seen\": {\"count\": 1, \"max\": 77600495, \"sum\": 77600495.0, \"min\": 77600495}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 15520099, \"sum\": 15520099.0, \"min\": 15520099}, \"Reset Count\": {\"count\": 1, \"max\": 10, \"sum\": 10.0, \"min\": 10}}, \"EndTime\": 1594590956.319922, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\", \"epoch\": 4}, \"StartTime\": 1594590807.157806}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:56 INFO 140290456758080] #throughput_metric: host=algo-1, train throughput=104048.414208 records/second\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:56 INFO 140290456758080] Saved checkpoint to \"/tmp/tmpGkoM3d/state-0001.params\"\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:56 WARNING 140290456758080] wait_for_all_workers will not sync workers since the kv store is not running distributed\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"finalize.time\": {\"count\": 1, \"max\": 0.41604042053222656, \"sum\": 0.41604042053222656, \"min\": 0.41604042053222656}}, \"EndTime\": 1594590956.359999, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594590956.319667}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:56 INFO 140290456758080] Saved checkpoint to \"/tmp/tmpGXbDms/state-0001.params\"\u001b[0m\n",
      "\u001b[34m[07/12/2020 21:55:56 INFO 140290456758080] Test data is not provided.\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"totaltime\": {\"count\": 1, \"max\": 750171.2188720703, \"sum\": 750171.2188720703, \"min\": 750171.2188720703}, \"setuptime\": {\"count\": 1, \"max\": 74.95903968811035, \"sum\": 74.95903968811035, \"min\": 74.95903968811035}}, \"EndTime\": 1594590956.40052, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594590956.360097}\n",
      "\u001b[0m\n",
      "Training seconds: 850\n",
      "Billable seconds: 850\n",
      "Training job name: ipinsights-2020-07-12-21-39-26-660\n"
     ]
    }
   ],
   "source": [
    "# Start the training job (should take about ~1.5 minute / epoch to complete)  \n",
    "ip_insights.fit(input_data)\n",
    "print('Training job name: {}'.format(ip_insights.latest_training_job.job_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Deploy the model\n",
    "You could start integrating this endpoint with your production services to start querying incoming requests for abnormal behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker:Parameter image will be renamed to image_uri in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------!"
     ]
    }
   ],
   "source": [
    "predictor = ip_insights.deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type='ml.p2.xlarge'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Evaluating with the Test set (Inference)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can pass data in a variety of formats to our inference endpoint. In this example, we will pass **CSV-formmated data**. Other available formats are **JSON-formated** and **JSON Lines-formatted**. We make use of the SageMaker Python SDK utilities: csv_serializer and json_deserializer when configuring the inference endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.predictor import csv_serializer, json_deserializer\n",
    "\n",
    "predictor.content_type = 'text/csv'\n",
    "predictor.serializer = csv_serializer\n",
    "predictor.accept = 'application/json'\n",
    "predictor.deserializer = json_deserializer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'predictions': [{'dot_product': 3.5534026622772217},\n",
       "  {'dot_product': 4.504860877990723},\n",
       "  {'dot_product': 3.3484466075897217},\n",
       "  {'dot_product': 4.795417308807373},\n",
       "  {'dot_product': 3.3309245109558105}]}"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can take a few samples from the simulated dataset above, so we can see what the output looks like.\n",
    "inference_data = [(data[0], data[1]) for data in train_df[:5].values]\n",
    "predictor.predict(inference_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the predictor will only output the dot_product between the learned IP address and the online resource (in this case, the user ID). The dot product summarizes the compatibility between the IP address and online resource. The larger the value, the more the algorithm thinks the IP address is likely to be used by the user. This compatability score is sufficient for most applications, as we can define a threshold for what we constitute as an anomalous score.\n",
    "\n",
    "However, more advanced users may want to inspect the learned embeddings and use them in further applications. We can configure the predictor to provide the learned embeddings by specifing the verbose=True parameter to the Accept heading. You should see that each 'prediction' object contains three keys: ip_embedding, entity_embedding, and dot_product."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'predictions': [{'ip_embedding': [0.6174953579902649,\n",
       "    0.9170134663581848,\n",
       "    0.4442644417285919,\n",
       "    -0.4519962668418884,\n",
       "    0.13857780396938324,\n",
       "    -0.06898989528417587,\n",
       "    1.0846973657608032,\n",
       "    0.13310456275939941,\n",
       "    0.19869929552078247,\n",
       "    -0.14983734488487244,\n",
       "    -0.3549431264400482,\n",
       "    0.5736722350120544,\n",
       "    -0.16356271505355835,\n",
       "    -0.18705371022224426,\n",
       "    0.07818632572889328,\n",
       "    -0.5186281800270081,\n",
       "    -0.48681312799453735,\n",
       "    -0.17102152109146118,\n",
       "    -0.25456172227859497,\n",
       "    -0.13960635662078857,\n",
       "    1.0349712371826172,\n",
       "    0.6361879706382751,\n",
       "    -0.001113593578338623,\n",
       "    0.5804753303527832,\n",
       "    -0.3120240271091461,\n",
       "    -0.5173526406288147,\n",
       "    -0.8985670208930969,\n",
       "    0.3453097641468048,\n",
       "    0.18799936771392822,\n",
       "    0.13846957683563232,\n",
       "    -0.6546565890312195,\n",
       "    -0.1920926570892334,\n",
       "    -0.8622803688049316,\n",
       "    -0.131496399641037,\n",
       "    -0.1679571270942688,\n",
       "    0.25013887882232666,\n",
       "    -0.62872314453125,\n",
       "    0.5943201780319214,\n",
       "    -0.5346848964691162,\n",
       "    -0.10245485603809357,\n",
       "    -0.1675262600183487,\n",
       "    0.3872779607772827,\n",
       "    -0.10582689195871353,\n",
       "    0.47130727767944336,\n",
       "    -0.05572855472564697,\n",
       "    -0.6872612237930298,\n",
       "    -0.1573353111743927,\n",
       "    0.9473121166229248,\n",
       "    -0.13632941246032715,\n",
       "    -0.9338298439979553,\n",
       "    -0.4048488140106201,\n",
       "    0.44711896777153015,\n",
       "    -0.16531287133693695,\n",
       "    0.27077457308769226,\n",
       "    -0.13566967844963074,\n",
       "    -0.10924924165010452,\n",
       "    0.6484795808792114,\n",
       "    0.06591212749481201,\n",
       "    0.33427876234054565,\n",
       "    -0.07238969206809998,\n",
       "    -0.7802303433418274,\n",
       "    0.28666335344314575,\n",
       "    0.056572094559669495,\n",
       "    0.3646744191646576,\n",
       "    -0.16946174204349518,\n",
       "    -0.061726078391075134,\n",
       "    0.11723998934030533,\n",
       "    -0.04114262014627457,\n",
       "    -0.5319393873214722,\n",
       "    -0.3443294167518616,\n",
       "    -0.33173978328704834,\n",
       "    -0.28390854597091675,\n",
       "    -0.17545254528522491,\n",
       "    -0.45469120144844055,\n",
       "    1.128067135810852,\n",
       "    -0.3019171357154846,\n",
       "    -0.2719595432281494,\n",
       "    -0.14423935115337372,\n",
       "    0.208954855799675,\n",
       "    -0.25826770067214966,\n",
       "    0.05742394179105759,\n",
       "    0.46050429344177246,\n",
       "    0.06618400663137436,\n",
       "    -0.11301467567682266,\n",
       "    0.18451577425003052,\n",
       "    -0.10106965899467468,\n",
       "    0.2696163058280945,\n",
       "    0.3595592975616455,\n",
       "    -0.035764433443546295,\n",
       "    0.2151915431022644,\n",
       "    0.17129820585250854,\n",
       "    -0.06261090934276581,\n",
       "    -0.2090849131345749,\n",
       "    -0.09369642287492752,\n",
       "    0.18106520175933838,\n",
       "    -0.2366306483745575,\n",
       "    0.5014889240264893,\n",
       "    0.259149968624115,\n",
       "    0.41406115889549255,\n",
       "    -0.6286135315895081,\n",
       "    0.39750874042510986,\n",
       "    0.3622360825538635,\n",
       "    0.16487658023834229,\n",
       "    -0.011796265840530396,\n",
       "    0.7981910705566406,\n",
       "    0.21057771146297455,\n",
       "    0.07580660283565521,\n",
       "    0.02239036001265049,\n",
       "    0.7354390621185303,\n",
       "    0.23669670522212982,\n",
       "    0.5091843605041504,\n",
       "    0.053496912121772766,\n",
       "    -0.3249436616897583,\n",
       "    -0.1149739995598793,\n",
       "    -0.16545695066452026,\n",
       "    -0.4527266323566437,\n",
       "    0.019303902983665466,\n",
       "    -0.2050609290599823,\n",
       "    0.3440362215042114,\n",
       "    -0.10060646384954453,\n",
       "    -0.04315294325351715,\n",
       "    -0.29727229475975037,\n",
       "    -0.30402517318725586,\n",
       "    0.5877533555030823,\n",
       "    -0.30628666281700134,\n",
       "    0.37161463499069214,\n",
       "    -0.004016667604446411,\n",
       "    1.1603251695632935],\n",
       "   'entity_embedding': [-0.38399600982666016,\n",
       "    0.08650310337543488,\n",
       "    -0.5021734237670898,\n",
       "    0.11584401875734329,\n",
       "    0.32152292132377625,\n",
       "    -0.05314691737294197,\n",
       "    0.679578959941864,\n",
       "    -0.35320425033569336,\n",
       "    -0.4331527650356293,\n",
       "    0.25525587797164917,\n",
       "    0.5366237163543701,\n",
       "    0.4157690703868866,\n",
       "    0.3359581530094147,\n",
       "    -0.47206389904022217,\n",
       "    0.09727263450622559,\n",
       "    -0.2081962674856186,\n",
       "    0.777869462966919,\n",
       "    -0.6080018877983093,\n",
       "    -0.02951156161725521,\n",
       "    -0.08008486032485962,\n",
       "    0.9292556047439575,\n",
       "    -0.6061755418777466,\n",
       "    0.6967799663543701,\n",
       "    -1.4549436569213867,\n",
       "    -1.4787485599517822,\n",
       "    0.2963721752166748,\n",
       "    0.030443070456385612,\n",
       "    0.8191106915473938,\n",
       "    -0.34841567277908325,\n",
       "    -0.2664625942707062,\n",
       "    0.7765746712684631,\n",
       "    -0.8866427540779114,\n",
       "    -0.6923133134841919,\n",
       "    -0.10961802303791046,\n",
       "    -0.6125980019569397,\n",
       "    0.264273077249527,\n",
       "    0.11088693141937256,\n",
       "    0.11966927349567413,\n",
       "    -0.3155781626701355,\n",
       "    -0.5482757687568665,\n",
       "    -0.9239664673805237,\n",
       "    -0.8524518609046936,\n",
       "    0.18882767856121063,\n",
       "    0.7757418751716614,\n",
       "    -1.7791943550109863,\n",
       "    -0.6862668395042419,\n",
       "    -0.04339109733700752,\n",
       "    0.004090369213372469,\n",
       "    0.6029496788978577,\n",
       "    -0.5900170207023621,\n",
       "    -0.12469057738780975,\n",
       "    0.6716271042823792,\n",
       "    0.13075724244117737,\n",
       "    0.2227371335029602,\n",
       "    -0.16162151098251343,\n",
       "    0.5037687420845032,\n",
       "    0.21178533136844635,\n",
       "    -1.0266342163085938,\n",
       "    -0.15300054848194122,\n",
       "    -0.7304912209510803,\n",
       "    -0.5832581520080566,\n",
       "    0.17533813416957855,\n",
       "    0.2843809425830841,\n",
       "    -0.012580133974552155,\n",
       "    0.28980204463005066,\n",
       "    -0.22184467315673828,\n",
       "    0.664548933506012,\n",
       "    0.8095318078994751,\n",
       "    -0.05974116921424866,\n",
       "    -0.21636176109313965,\n",
       "    -0.821875274181366,\n",
       "    -0.35624292492866516,\n",
       "    1.3169825077056885,\n",
       "    1.1080094575881958,\n",
       "    1.2688813209533691,\n",
       "    0.12263240665197372,\n",
       "    -0.5493541359901428,\n",
       "    -0.8287839293479919,\n",
       "    0.00420870678499341,\n",
       "    0.12105274200439453,\n",
       "    -0.20028315484523773,\n",
       "    1.2314453125,\n",
       "    1.6283891201019287,\n",
       "    0.5314505696296692,\n",
       "    -0.220465749502182,\n",
       "    -1.688260793685913,\n",
       "    0.3505208194255829,\n",
       "    -1.2901597023010254,\n",
       "    0.7865430116653442,\n",
       "    0.7541357278823853,\n",
       "    -0.9518690705299377,\n",
       "    0.4233839213848114,\n",
       "    -0.581380307674408,\n",
       "    0.2731263339519501,\n",
       "    -0.01145147904753685,\n",
       "    -1.1117945909500122,\n",
       "    -1.5500426292419434,\n",
       "    -0.5536016821861267,\n",
       "    -0.02595449984073639,\n",
       "    0.10501621663570404,\n",
       "    0.6072013974189758,\n",
       "    -0.6914929151535034,\n",
       "    -0.15453602373600006,\n",
       "    -0.7641594409942627,\n",
       "    0.45174115896224976,\n",
       "    -0.19384446740150452,\n",
       "    0.2811153531074524,\n",
       "    -0.8162881731987,\n",
       "    0.48184338212013245,\n",
       "    0.3657425045967102,\n",
       "    -0.49435991048812866,\n",
       "    0.6009320020675659,\n",
       "    0.9616048336029053,\n",
       "    -0.5252438187599182,\n",
       "    -0.0035176011733710766,\n",
       "    -0.09965868294239044,\n",
       "    -0.06302858144044876,\n",
       "    0.4683464467525482,\n",
       "    -0.02907629683613777,\n",
       "    0.04845748096704483,\n",
       "    -0.8496536016464233,\n",
       "    -0.21202440559864044,\n",
       "    -1.0706197023391724,\n",
       "    -0.7951968908309937,\n",
       "    -0.2709662616252899,\n",
       "    -1.1493054628372192,\n",
       "    0.10899367928504944,\n",
       "    -0.3986785411834717],\n",
       "   'dot_product': 3.5534026622772217},\n",
       "  {'ip_embedding': [0.5915340185165405,\n",
       "    0.004521846771240234,\n",
       "    0.21960130333900452,\n",
       "    -0.26322877407073975,\n",
       "    0.07677116245031357,\n",
       "    0.07491442561149597,\n",
       "    -0.07528389990329742,\n",
       "    0.700626015663147,\n",
       "    0.2838284969329834,\n",
       "    0.18915674090385437,\n",
       "    -0.7755581140518188,\n",
       "    0.21150298416614532,\n",
       "    -0.4539075493812561,\n",
       "    -0.18353378772735596,\n",
       "    0.1391877681016922,\n",
       "    -1.1127885580062866,\n",
       "    -0.2515309453010559,\n",
       "    -0.08901526033878326,\n",
       "    -0.08724883198738098,\n",
       "    0.04777401685714722,\n",
       "    0.14916375279426575,\n",
       "    0.012538250535726547,\n",
       "    -0.05833077430725098,\n",
       "    -0.17041140794754028,\n",
       "    0.059089869260787964,\n",
       "    -0.2769811153411865,\n",
       "    0.4157073497772217,\n",
       "    0.40421611070632935,\n",
       "    -0.06226781755685806,\n",
       "    -0.07678717374801636,\n",
       "    -0.1949009895324707,\n",
       "    0.24347615242004395,\n",
       "    -0.1019623801112175,\n",
       "    -0.5236587524414062,\n",
       "    -0.3242836594581604,\n",
       "    -0.17526789009571075,\n",
       "    -0.2936416566371918,\n",
       "    0.23389098048210144,\n",
       "    -0.595189094543457,\n",
       "    0.15808552503585815,\n",
       "    -0.002255186438560486,\n",
       "    -0.024257779121398926,\n",
       "    -0.21298423409461975,\n",
       "    -0.04199507087469101,\n",
       "    0.3032272458076477,\n",
       "    -0.09962302446365356,\n",
       "    -0.10253657400608063,\n",
       "    -0.4721686542034149,\n",
       "    -0.6563403010368347,\n",
       "    -0.14945828914642334,\n",
       "    0.23109540343284607,\n",
       "    0.013445548713207245,\n",
       "    -0.032610900700092316,\n",
       "    0.3545110821723938,\n",
       "    -0.17212830483913422,\n",
       "    -0.010968399234116077,\n",
       "    0.3991256654262543,\n",
       "    -0.04349398612976074,\n",
       "    0.20250070095062256,\n",
       "    0.13473719358444214,\n",
       "    0.1209792047739029,\n",
       "    0.7930610775947571,\n",
       "    -0.18153788149356842,\n",
       "    -0.046378329396247864,\n",
       "    -0.22161318361759186,\n",
       "    0.09614117443561554,\n",
       "    -0.017637116834521294,\n",
       "    -0.21593989431858063,\n",
       "    0.08271980285644531,\n",
       "    -0.002512563019990921,\n",
       "    -0.14560763537883759,\n",
       "    -0.04307955503463745,\n",
       "    -0.3925350308418274,\n",
       "    -0.2531962990760803,\n",
       "    -0.2334282100200653,\n",
       "    -0.0038795843720436096,\n",
       "    0.25342124700546265,\n",
       "    0.28675737977027893,\n",
       "    -0.006841883063316345,\n",
       "    -0.06546952575445175,\n",
       "    0.5106590390205383,\n",
       "    -0.5154765844345093,\n",
       "    -0.5102471113204956,\n",
       "    0.10662864148616791,\n",
       "    0.3096982538700104,\n",
       "    -0.03970833122730255,\n",
       "    -0.11483244597911835,\n",
       "    0.5863050818443298,\n",
       "    -0.2940109968185425,\n",
       "    0.1451374739408493,\n",
       "    0.8999490737915039,\n",
       "    -0.08011655509471893,\n",
       "    0.18382342159748077,\n",
       "    -0.008337408304214478,\n",
       "    0.226502925157547,\n",
       "    0.36707010865211487,\n",
       "    0.18926791846752167,\n",
       "    0.8518780469894409,\n",
       "    -0.0710575059056282,\n",
       "    -0.291491836309433,\n",
       "    0.3542420566082001,\n",
       "    -0.5998957753181458,\n",
       "    0.025504115968942642,\n",
       "    0.3881474733352661,\n",
       "    -0.010357566177845001,\n",
       "    0.08466485142707825,\n",
       "    0.060680076479911804,\n",
       "    0.7774134874343872,\n",
       "    0.3643018305301666,\n",
       "    -0.25480085611343384,\n",
       "    0.588856041431427,\n",
       "    -0.522494912147522,\n",
       "    -0.3845176696777344,\n",
       "    0.23725686967372894,\n",
       "    -0.3552092909812927,\n",
       "    0.027572348713874817,\n",
       "    -0.03614604473114014,\n",
       "    -0.11158517748117447,\n",
       "    -0.09539991617202759,\n",
       "    -0.053902242332696915,\n",
       "    0.5616194009780884,\n",
       "    -0.044420599937438965,\n",
       "    0.7081713080406189,\n",
       "    0.20916762948036194,\n",
       "    0.1754322648048401,\n",
       "    0.6519564390182495,\n",
       "    0.10179457813501358,\n",
       "    0.00022745784372091293],\n",
       "   'entity_embedding': [-0.20535312592983246,\n",
       "    -1.0735243558883667,\n",
       "    -0.006111420225352049,\n",
       "    0.1227535605430603,\n",
       "    0.7480675578117371,\n",
       "    -0.7823497653007507,\n",
       "    -0.20504312217235565,\n",
       "    0.0913522019982338,\n",
       "    -0.2570834755897522,\n",
       "    0.6745193600654602,\n",
       "    -0.6617867946624756,\n",
       "    -0.4883136451244354,\n",
       "    0.7575210928916931,\n",
       "    -0.08927184343338013,\n",
       "    -0.21685652434825897,\n",
       "    -1.1297353506088257,\n",
       "    0.2109730839729309,\n",
       "    0.05124049261212349,\n",
       "    -0.12400240451097488,\n",
       "    -0.26416656374931335,\n",
       "    -0.4748052954673767,\n",
       "    -0.5238049030303955,\n",
       "    0.246865913271904,\n",
       "    -0.5446896553039551,\n",
       "    -0.8595126867294312,\n",
       "    -0.008815638720989227,\n",
       "    1.3591030836105347,\n",
       "    0.03924878314137459,\n",
       "    -0.4484747350215912,\n",
       "    -0.42742064595222473,\n",
       "    0.39472031593322754,\n",
       "    -0.5661177039146423,\n",
       "    -0.8289287686347961,\n",
       "    0.005233402363955975,\n",
       "    -1.1412084102630615,\n",
       "    0.25532224774360657,\n",
       "    0.8807368874549866,\n",
       "    -0.2577708661556244,\n",
       "    0.6996367573738098,\n",
       "    0.42731979489326477,\n",
       "    -0.9425326585769653,\n",
       "    -0.5327248573303223,\n",
       "    -0.31716224551200867,\n",
       "    -0.8196612000465393,\n",
       "    0.08524755388498306,\n",
       "    -0.45463645458221436,\n",
       "    1.3199799060821533,\n",
       "    -1.7853538990020752,\n",
       "    -0.8351377248764038,\n",
       "    -0.34809473156929016,\n",
       "    0.6578904986381531,\n",
       "    -1.3346830606460571,\n",
       "    0.3381901681423187,\n",
       "    0.327848881483078,\n",
       "    -0.5641584992408752,\n",
       "    -0.22140689194202423,\n",
       "    -0.2035459727048874,\n",
       "    0.1668693721294403,\n",
       "    -0.07942929863929749,\n",
       "    0.5415505170822144,\n",
       "    0.20203834772109985,\n",
       "    -0.15375573933124542,\n",
       "    -0.32032081484794617,\n",
       "    -1.3547375202178955,\n",
       "    0.4228591024875641,\n",
       "    0.012095781974494457,\n",
       "    -0.4752919673919678,\n",
       "    -0.18426191806793213,\n",
       "    -0.23563380539417267,\n",
       "    0.006705231964588165,\n",
       "    -0.6737793684005737,\n",
       "    0.7202315330505371,\n",
       "    0.65704745054245,\n",
       "    0.10756722092628479,\n",
       "    -0.5933728218078613,\n",
       "    -0.16352854669094086,\n",
       "    -0.7129353284835815,\n",
       "    0.4424625635147095,\n",
       "    0.20072035491466522,\n",
       "    -0.09626930207014084,\n",
       "    -0.13872839510440826,\n",
       "    -0.4336033761501312,\n",
       "    -0.6521335244178772,\n",
       "    0.7175868153572083,\n",
       "    0.15506288409233093,\n",
       "    0.8386318683624268,\n",
       "    -0.140714630484581,\n",
       "    -0.8812299966812134,\n",
       "    0.033969417214393616,\n",
       "    -0.022222943603992462,\n",
       "    -0.4297170639038086,\n",
       "    0.4158371388912201,\n",
       "    0.06171078979969025,\n",
       "    0.23335079848766327,\n",
       "    -0.2316177636384964,\n",
       "    -0.16541580855846405,\n",
       "    -0.8498706817626953,\n",
       "    -0.056376002728939056,\n",
       "    -0.6966305375099182,\n",
       "    0.23509985208511353,\n",
       "    -0.7787346243858337,\n",
       "    -1.0615594387054443,\n",
       "    -0.19594401121139526,\n",
       "    0.27485236525535583,\n",
       "    -0.06608263403177261,\n",
       "    0.21320509910583496,\n",
       "    0.7932630181312561,\n",
       "    0.7210513949394226,\n",
       "    0.20003871619701385,\n",
       "    -0.04413745552301407,\n",
       "    -0.2889808118343353,\n",
       "    -0.40936213731765747,\n",
       "    -0.37726789712905884,\n",
       "    -0.27192509174346924,\n",
       "    0.09032029658555984,\n",
       "    -0.033817458897829056,\n",
       "    -0.050263259559869766,\n",
       "    -0.4455338418483734,\n",
       "    -0.6070747971534729,\n",
       "    0.20836444199085236,\n",
       "    0.2064255326986313,\n",
       "    -0.011438309215009212,\n",
       "    1.2788407802581787,\n",
       "    -0.3209739029407501,\n",
       "    -0.49149569869041443,\n",
       "    0.10565263032913208,\n",
       "    -0.8291359543800354,\n",
       "    0.012865682132542133],\n",
       "   'dot_product': 4.504860877990723},\n",
       "  {'ip_embedding': [0.08530541509389877,\n",
       "    -0.12273023277521133,\n",
       "    -0.20119911432266235,\n",
       "    -0.43828362226486206,\n",
       "    -0.6055104732513428,\n",
       "    -0.048842497169971466,\n",
       "    -0.37005266547203064,\n",
       "    0.37630003690719604,\n",
       "    0.1869436800479889,\n",
       "    -0.07974901795387268,\n",
       "    0.2982284426689148,\n",
       "    -0.03202097490429878,\n",
       "    -0.5576905012130737,\n",
       "    0.3369227647781372,\n",
       "    -0.023600740358233452,\n",
       "    -0.35158419609069824,\n",
       "    -0.09817112982273102,\n",
       "    0.08504259586334229,\n",
       "    -0.2124435007572174,\n",
       "    -0.08971524983644485,\n",
       "    0.18405626714229584,\n",
       "    0.19303171336650848,\n",
       "    -0.48727157711982727,\n",
       "    0.13893035054206848,\n",
       "    -0.05206304043531418,\n",
       "    -0.3039669096469879,\n",
       "    0.1665632426738739,\n",
       "    0.28281891345977783,\n",
       "    0.1455555260181427,\n",
       "    -0.645340085029602,\n",
       "    -0.20019599795341492,\n",
       "    0.21685312688350677,\n",
       "    0.12234805524349213,\n",
       "    -0.4006022810935974,\n",
       "    -0.1589374840259552,\n",
       "    -0.014528496190905571,\n",
       "    -0.5574235916137695,\n",
       "    0.052317939698696136,\n",
       "    -0.5525795221328735,\n",
       "    -0.023332316428422928,\n",
       "    -0.07968578487634659,\n",
       "    0.14029288291931152,\n",
       "    -0.17321939766407013,\n",
       "    -0.06307157874107361,\n",
       "    -0.016319386661052704,\n",
       "    -0.616002082824707,\n",
       "    0.3258528709411621,\n",
       "    -0.0011304989457130432,\n",
       "    -0.005042499862611294,\n",
       "    -0.02509254962205887,\n",
       "    -0.06530283391475677,\n",
       "    0.6444287300109863,\n",
       "    0.02210184372961521,\n",
       "    -0.026721559464931488,\n",
       "    -0.19068588316440582,\n",
       "    -0.13253948092460632,\n",
       "    0.4942850172519684,\n",
       "    -0.19027771055698395,\n",
       "    -0.04431964457035065,\n",
       "    -0.23665255308151245,\n",
       "    0.24928361177444458,\n",
       "    0.26121285557746887,\n",
       "    -0.0543917641043663,\n",
       "    0.08330407738685608,\n",
       "    0.1768803894519806,\n",
       "    0.07075172662734985,\n",
       "    0.2744402289390564,\n",
       "    -0.12511223554611206,\n",
       "    -0.23922118544578552,\n",
       "    -0.2275295853614807,\n",
       "    -0.05464663729071617,\n",
       "    -0.25167739391326904,\n",
       "    -0.485217809677124,\n",
       "    0.40623605251312256,\n",
       "    -0.14291703701019287,\n",
       "    0.05887582153081894,\n",
       "    0.17696742713451385,\n",
       "    -0.06629592180252075,\n",
       "    0.06889011710882187,\n",
       "    -0.05131182819604874,\n",
       "    0.3290911912918091,\n",
       "    0.19706983864307404,\n",
       "    -0.13814838230609894,\n",
       "    0.16391317546367645,\n",
       "    -0.1761559247970581,\n",
       "    0.13024260103702545,\n",
       "    -0.15835019946098328,\n",
       "    0.41056028008461,\n",
       "    -0.0069942958652973175,\n",
       "    0.08479492366313934,\n",
       "    0.3965725302696228,\n",
       "    -0.32980895042419434,\n",
       "    -0.1162584200501442,\n",
       "    -0.013439260423183441,\n",
       "    0.23457781970500946,\n",
       "    0.6396196484565735,\n",
       "    -0.11210435628890991,\n",
       "    0.2717691659927368,\n",
       "    0.1714804768562317,\n",
       "    -0.0372975617647171,\n",
       "    0.4354858994483948,\n",
       "    0.25519585609436035,\n",
       "    -0.004068531095981598,\n",
       "    0.1392444670200348,\n",
       "    -0.0031845904886722565,\n",
       "    0.10692508518695831,\n",
       "    0.5666228532791138,\n",
       "    -0.39664363861083984,\n",
       "    -0.07255611568689346,\n",
       "    -0.08173221349716187,\n",
       "    0.47605347633361816,\n",
       "    -0.3338755965232849,\n",
       "    -0.518386721611023,\n",
       "    0.9063738584518433,\n",
       "    0.06141313910484314,\n",
       "    -0.010006871074438095,\n",
       "    0.19584377110004425,\n",
       "    -0.15265290439128876,\n",
       "    -0.024447597563266754,\n",
       "    -0.09825121611356735,\n",
       "    0.10821544378995895,\n",
       "    0.24519652128219604,\n",
       "    0.012086737900972366,\n",
       "    -0.3203766644001007,\n",
       "    0.05490494146943092,\n",
       "    0.7293794751167297,\n",
       "    0.11556793749332428,\n",
       "    0.1616475135087967],\n",
       "   'entity_embedding': [-0.4762589633464813,\n",
       "    -0.39504146575927734,\n",
       "    0.006291381549090147,\n",
       "    0.36105313897132874,\n",
       "    0.10661350190639496,\n",
       "    0.3035109043121338,\n",
       "    -0.5636323094367981,\n",
       "    -0.33150070905685425,\n",
       "    -0.7460122108459473,\n",
       "    0.17679642140865326,\n",
       "    0.6340569853782654,\n",
       "    0.2226601392030716,\n",
       "    -0.7841986417770386,\n",
       "    0.011705713346600533,\n",
       "    -0.17380240559577942,\n",
       "    1.440820336341858,\n",
       "    0.4820593297481537,\n",
       "    0.014963104389607906,\n",
       "    0.32045429944992065,\n",
       "    0.08340871334075928,\n",
       "    0.21454907953739166,\n",
       "    -0.13893082737922668,\n",
       "    0.07853489369153976,\n",
       "    -0.651494026184082,\n",
       "    -0.5479936003684998,\n",
       "    0.17209938168525696,\n",
       "    1.5851198434829712,\n",
       "    0.47869575023651123,\n",
       "    0.18787166476249695,\n",
       "    -0.7545486092567444,\n",
       "    0.3608000576496124,\n",
       "    -0.16934524476528168,\n",
       "    -0.11488806456327438,\n",
       "    -0.3211228847503662,\n",
       "    -0.20793473720550537,\n",
       "    -0.40379124879837036,\n",
       "    0.07086589187383652,\n",
       "    0.36286085844039917,\n",
       "    0.13920460641384125,\n",
       "    0.23844049870967865,\n",
       "    -1.0183656215667725,\n",
       "    -0.02938367798924446,\n",
       "    0.2917528450489044,\n",
       "    0.05537007749080658,\n",
       "    0.03457874432206154,\n",
       "    0.4657451808452606,\n",
       "    1.129392385482788,\n",
       "    -0.001114387298002839,\n",
       "    0.09193506091833115,\n",
       "    0.2464645802974701,\n",
       "    0.2857138514518738,\n",
       "    0.34961840510368347,\n",
       "    -0.1634356528520584,\n",
       "    -0.07076919078826904,\n",
       "    -0.5041459798812866,\n",
       "    -0.605230450630188,\n",
       "    0.14457011222839355,\n",
       "    0.10495354235172272,\n",
       "    0.1716480404138565,\n",
       "    -0.5774372816085815,\n",
       "    0.26894670724868774,\n",
       "    -0.7435979843139648,\n",
       "    -0.7092323303222656,\n",
       "    -0.7560086250305176,\n",
       "    -0.04488903284072876,\n",
       "    -0.21650661528110504,\n",
       "    -0.15381158888339996,\n",
       "    -0.6248340010643005,\n",
       "    -0.06621305644512177,\n",
       "    -0.29781123995780945,\n",
       "    0.6260724663734436,\n",
       "    -0.3252484202384949,\n",
       "    0.46520790457725525,\n",
       "    0.9114629030227661,\n",
       "    -0.05245909467339516,\n",
       "    -0.09673800319433212,\n",
       "    0.15318070352077484,\n",
       "    -0.4369019567966461,\n",
       "    0.09127993881702423,\n",
       "    0.00021479185670614243,\n",
       "    0.7439523339271545,\n",
       "    0.23351214826107025,\n",
       "    -0.14025455713272095,\n",
       "    0.20716170966625214,\n",
       "    -0.38141050934791565,\n",
       "    -0.040995024144649506,\n",
       "    0.14775654673576355,\n",
       "    -0.269500732421875,\n",
       "    0.47911378741264343,\n",
       "    -0.3567340075969696,\n",
       "    -0.04233397915959358,\n",
       "    0.2428576499223709,\n",
       "    -0.2943562865257263,\n",
       "    0.05410613492131233,\n",
       "    0.6318293809890747,\n",
       "    0.14794574677944183,\n",
       "    -0.19833754003047943,\n",
       "    -0.6307129859924316,\n",
       "    -0.6644502282142639,\n",
       "    -0.2874416708946228,\n",
       "    -0.05156590789556503,\n",
       "    0.3669848144054413,\n",
       "    -0.34680506587028503,\n",
       "    -0.9367542266845703,\n",
       "    -0.10406545549631119,\n",
       "    0.07834658026695251,\n",
       "    0.8255228996276855,\n",
       "    -1.208186149597168,\n",
       "    -0.09519656747579575,\n",
       "    1.14985191822052,\n",
       "    0.16875006258487701,\n",
       "    0.0889357179403305,\n",
       "    0.2986409366130829,\n",
       "    0.4812191128730774,\n",
       "    0.605793297290802,\n",
       "    -0.02958017587661743,\n",
       "    0.6888264417648315,\n",
       "    0.18796280026435852,\n",
       "    -0.25110578536987305,\n",
       "    -0.1477954238653183,\n",
       "    -0.30294185876846313,\n",
       "    0.9235000014305115,\n",
       "    -0.12760423123836517,\n",
       "    -0.8734511137008667,\n",
       "    0.4605773687362671,\n",
       "    0.1889042854309082,\n",
       "    0.01790972799062729,\n",
       "    -0.21489322185516357],\n",
       "   'dot_product': 3.3484466075897217},\n",
       "  {'ip_embedding': [-0.2670857310295105,\n",
       "    0.28230488300323486,\n",
       "    0.3939429521560669,\n",
       "    0.35875239968299866,\n",
       "    -0.6979749798774719,\n",
       "    0.45088717341423035,\n",
       "    0.15797531604766846,\n",
       "    -0.10769277811050415,\n",
       "    0.07436154037714005,\n",
       "    0.22540268301963806,\n",
       "    -0.04798996448516846,\n",
       "    0.2944270968437195,\n",
       "    -0.3500015139579773,\n",
       "    0.17052584886550903,\n",
       "    0.25906288623809814,\n",
       "    -0.2667100429534912,\n",
       "    0.3400273323059082,\n",
       "    -0.06710109114646912,\n",
       "    -0.03850578889250755,\n",
       "    0.10758303850889206,\n",
       "    -0.25343507528305054,\n",
       "    0.42995181679725647,\n",
       "    -0.59281986951828,\n",
       "    0.05258236452937126,\n",
       "    -0.11906873434782028,\n",
       "    -0.27056097984313965,\n",
       "    -0.11529385298490524,\n",
       "    -0.026636343449354172,\n",
       "    0.25539734959602356,\n",
       "    0.1502830684185028,\n",
       "    -0.38096344470977783,\n",
       "    -0.04926484823226929,\n",
       "    0.7562125325202942,\n",
       "    0.09798455238342285,\n",
       "    0.05542868375778198,\n",
       "    0.045311667025089264,\n",
       "    -0.14350785315036774,\n",
       "    0.18749567866325378,\n",
       "    -0.330668568611145,\n",
       "    0.08941809087991714,\n",
       "    0.12627965211868286,\n",
       "    -0.30654895305633545,\n",
       "    -0.09688974916934967,\n",
       "    0.27441903948783875,\n",
       "    0.8660655617713928,\n",
       "    -0.418342262506485,\n",
       "    -0.5026109218597412,\n",
       "    0.20073843002319336,\n",
       "    -0.08555054664611816,\n",
       "    -0.01266871951520443,\n",
       "    -0.7479530572891235,\n",
       "    0.2611791789531708,\n",
       "    0.029853567481040955,\n",
       "    0.2906605303287506,\n",
       "    0.026909667998552322,\n",
       "    -0.30254048109054565,\n",
       "    0.2593764364719391,\n",
       "    -0.31774187088012695,\n",
       "    -0.36903873085975647,\n",
       "    -0.23573003709316254,\n",
       "    -0.5875674486160278,\n",
       "    0.10888196527957916,\n",
       "    0.13868781924247742,\n",
       "    0.2709893584251404,\n",
       "    0.30465167760849,\n",
       "    -0.009596504271030426,\n",
       "    -0.020427679643034935,\n",
       "    0.36665427684783936,\n",
       "    0.24132880568504333,\n",
       "    0.18468740582466125,\n",
       "    0.4531286954879761,\n",
       "    -0.23123493790626526,\n",
       "    -0.9596786499023438,\n",
       "    -0.49795418977737427,\n",
       "    0.531369149684906,\n",
       "    0.18156196177005768,\n",
       "    0.778199315071106,\n",
       "    -0.12878967821598053,\n",
       "    -0.1313323676586151,\n",
       "    -0.03686459735035896,\n",
       "    -0.0017439499497413635,\n",
       "    0.09762651473283768,\n",
       "    -0.006910540163516998,\n",
       "    0.247987762093544,\n",
       "    0.24972310662269592,\n",
       "    0.367560476064682,\n",
       "    0.3704153299331665,\n",
       "    -0.14396867156028748,\n",
       "    -0.0868849828839302,\n",
       "    -0.23610836267471313,\n",
       "    0.3816092610359192,\n",
       "    -0.23816148936748505,\n",
       "    -0.29274988174438477,\n",
       "    -0.05071228742599487,\n",
       "    0.13278095424175262,\n",
       "    -0.11077246069908142,\n",
       "    -0.41096222400665283,\n",
       "    0.5512393712997437,\n",
       "    0.42768242955207825,\n",
       "    -0.1049836128950119,\n",
       "    0.15073451399803162,\n",
       "    0.601580023765564,\n",
       "    -0.24080051481723785,\n",
       "    0.3391762971878052,\n",
       "    -0.07406724244356155,\n",
       "    -0.26514580845832825,\n",
       "    -0.22770549356937408,\n",
       "    0.04457755386829376,\n",
       "    0.5589773654937744,\n",
       "    -0.3605540692806244,\n",
       "    0.30121269822120667,\n",
       "    -0.49797654151916504,\n",
       "    -0.2884732484817505,\n",
       "    0.0185014009475708,\n",
       "    -0.45661303400993347,\n",
       "    0.7319466471672058,\n",
       "    -0.20599304139614105,\n",
       "    0.03363528847694397,\n",
       "    0.09312237799167633,\n",
       "    -0.1816471517086029,\n",
       "    0.40543705224990845,\n",
       "    -0.31426796317100525,\n",
       "    0.11508038640022278,\n",
       "    -0.06642661243677139,\n",
       "    -0.04366326332092285,\n",
       "    0.26787737011909485,\n",
       "    -0.5200135707855225,\n",
       "    -0.22305363416671753],\n",
       "   'entity_embedding': [-0.9397528767585754,\n",
       "    0.3621082603931427,\n",
       "    0.13975679874420166,\n",
       "    1.1292411088943481,\n",
       "    -0.49383172392845154,\n",
       "    0.21288570761680603,\n",
       "    -0.7573866844177246,\n",
       "    -0.629814624786377,\n",
       "    -1.2602298259735107,\n",
       "    0.2225731760263443,\n",
       "    0.2189706265926361,\n",
       "    -0.499754399061203,\n",
       "    0.6289026141166687,\n",
       "    -0.47072017192840576,\n",
       "    -0.397925466299057,\n",
       "    -0.18853788077831268,\n",
       "    1.4031338691711426,\n",
       "    -1.4977505207061768,\n",
       "    0.8901710510253906,\n",
       "    -0.7274315357208252,\n",
       "    -0.4599848985671997,\n",
       "    0.6753523349761963,\n",
       "    0.496596097946167,\n",
       "    -0.39416369795799255,\n",
       "    -1.173071265220642,\n",
       "    0.6122972965240479,\n",
       "    0.12350968271493912,\n",
       "    -0.5571975111961365,\n",
       "    0.44472935795783997,\n",
       "    -1.0113508701324463,\n",
       "    -0.22318872809410095,\n",
       "    0.1949334442615509,\n",
       "    -0.17592722177505493,\n",
       "    1.295103907585144,\n",
       "    -0.8095986247062683,\n",
       "    -0.7216572165489197,\n",
       "    0.1450381577014923,\n",
       "    -1.445774793624878,\n",
       "    -0.5943312048912048,\n",
       "    -0.1768113076686859,\n",
       "    -1.502759337425232,\n",
       "    -0.8488940000534058,\n",
       "    0.1217726320028305,\n",
       "    -0.7147265076637268,\n",
       "    -0.39339378476142883,\n",
       "    1.7848745584487915,\n",
       "    -0.48158693313598633,\n",
       "    -1.0637449026107788,\n",
       "    -0.7392851710319519,\n",
       "    2.0637190341949463,\n",
       "    0.03590387850999832,\n",
       "    -0.4848681688308716,\n",
       "    0.5791452527046204,\n",
       "    0.3383854627609253,\n",
       "    0.4463290870189667,\n",
       "    -0.5749441981315613,\n",
       "    -0.30541741847991943,\n",
       "    0.26280322670936584,\n",
       "    -0.7154613137245178,\n",
       "    0.5927848815917969,\n",
       "    -0.3821919560432434,\n",
       "    -1.1730806827545166,\n",
       "    -0.037307363003492355,\n",
       "    0.3589475750923157,\n",
       "    0.2728520333766937,\n",
       "    0.16882680356502533,\n",
       "    -1.169325351715088,\n",
       "    1.2598131895065308,\n",
       "    1.074432373046875,\n",
       "    1.0240497589111328,\n",
       "    -0.28132620453834534,\n",
       "    0.25667959451675415,\n",
       "    -0.6489785313606262,\n",
       "    -0.3627815842628479,\n",
       "    -0.26185789704322815,\n",
       "    1.8286917209625244,\n",
       "    -0.2814028561115265,\n",
       "    -0.12387198209762573,\n",
       "    -0.4035710096359253,\n",
       "    1.138919472694397,\n",
       "    1.257529377937317,\n",
       "    -0.2678035497665405,\n",
       "    0.68021559715271,\n",
       "    0.2064870297908783,\n",
       "    0.12761029601097107,\n",
       "    -1.627913236618042,\n",
       "    -0.2159983068704605,\n",
       "    -1.1084978580474854,\n",
       "    0.6130173802375793,\n",
       "    -1.1975526809692383,\n",
       "    0.23666399717330933,\n",
       "    0.469355046749115,\n",
       "    0.604263424873352,\n",
       "    -0.7538949251174927,\n",
       "    -0.1618274301290512,\n",
       "    -0.6251475811004639,\n",
       "    -0.8826413154602051,\n",
       "    -0.2632010877132416,\n",
       "    0.45106494426727295,\n",
       "    -0.013226216658949852,\n",
       "    0.09477315098047256,\n",
       "    0.23186400532722473,\n",
       "    -0.4078301787376404,\n",
       "    -0.2578812539577484,\n",
       "    -1.0514367818832397,\n",
       "    -0.4667949974536896,\n",
       "    -0.7672724723815918,\n",
       "    -0.8577668070793152,\n",
       "    1.2416939735412598,\n",
       "    0.6565475463867188,\n",
       "    -0.3525366485118866,\n",
       "    -0.49997076392173767,\n",
       "    0.12436690181493759,\n",
       "    -1.282750129699707,\n",
       "    -0.042154185473918915,\n",
       "    1.2388720512390137,\n",
       "    -0.2112908810377121,\n",
       "    0.16018719971179962,\n",
       "    0.7745991349220276,\n",
       "    0.4639091491699219,\n",
       "    -0.01948261633515358,\n",
       "    0.359237402677536,\n",
       "    -0.1792110800743103,\n",
       "    -0.5038636326789856,\n",
       "    0.5788478851318359,\n",
       "    -1.0262523889541626,\n",
       "    -1.811168909072876,\n",
       "    -1.799295425415039],\n",
       "   'dot_product': 4.795417308807373},\n",
       "  {'ip_embedding': [0.18450278043746948,\n",
       "    0.27214697003364563,\n",
       "    0.12771786749362946,\n",
       "    0.09384215623140335,\n",
       "    -0.6173787713050842,\n",
       "    0.07012296468019485,\n",
       "    0.25918588042259216,\n",
       "    0.3377697467803955,\n",
       "    0.8635023236274719,\n",
       "    0.12473735213279724,\n",
       "    0.11336661875247955,\n",
       "    0.5927051305770874,\n",
       "    -0.2645132541656494,\n",
       "    0.3642193675041199,\n",
       "    0.4634595215320587,\n",
       "    0.42565688490867615,\n",
       "    0.30211684107780457,\n",
       "    0.2931230664253235,\n",
       "    -0.5294538140296936,\n",
       "    0.16877129673957825,\n",
       "    0.24542367458343506,\n",
       "    0.028741002082824707,\n",
       "    -0.8619400858879089,\n",
       "    0.2620870769023895,\n",
       "    -0.13386835157871246,\n",
       "    -0.5788689851760864,\n",
       "    -0.19454915821552277,\n",
       "    0.06588185578584671,\n",
       "    -0.051429376006126404,\n",
       "    -0.04740265756845474,\n",
       "    -0.29134470224380493,\n",
       "    -0.3016003668308258,\n",
       "    -0.3647770881652832,\n",
       "    0.3880982995033264,\n",
       "    -0.03137703984975815,\n",
       "    0.23241828382015228,\n",
       "    0.007088076323270798,\n",
       "    0.07522238045930862,\n",
       "    -0.054794974625110626,\n",
       "    0.202365905046463,\n",
       "    0.02178356423974037,\n",
       "    -0.20872309803962708,\n",
       "    -0.06359700858592987,\n",
       "    0.3736816942691803,\n",
       "    0.040656112134456635,\n",
       "    -0.03000902384519577,\n",
       "    -0.2445368766784668,\n",
       "    0.02792828157544136,\n",
       "    0.11167202889919281,\n",
       "    0.19801391661167145,\n",
       "    -0.501717209815979,\n",
       "    0.2675654888153076,\n",
       "    0.15650959312915802,\n",
       "    0.03732525557279587,\n",
       "    -0.41245797276496887,\n",
       "    -0.4023076891899109,\n",
       "    0.44278761744499207,\n",
       "    0.09881505370140076,\n",
       "    -0.1304853856563568,\n",
       "    0.30512678623199463,\n",
       "    -0.7521535158157349,\n",
       "    0.4308326244354248,\n",
       "    0.4909523129463196,\n",
       "    0.5205357670783997,\n",
       "    0.03654402866959572,\n",
       "    -0.08728787302970886,\n",
       "    -0.22136573493480682,\n",
       "    0.1831716001033783,\n",
       "    -0.06470655649900436,\n",
       "    0.14525508880615234,\n",
       "    -0.06723631173372269,\n",
       "    0.18213818967342377,\n",
       "    -0.34868332743644714,\n",
       "    0.05193069577217102,\n",
       "    0.4508476257324219,\n",
       "    -0.24521878361701965,\n",
       "    -0.051583658903837204,\n",
       "    0.4005081057548523,\n",
       "    -0.19585952162742615,\n",
       "    0.12980708479881287,\n",
       "    0.2989763021469116,\n",
       "    0.6285268068313599,\n",
       "    -0.04126095771789551,\n",
       "    0.046981628984212875,\n",
       "    0.30134326219558716,\n",
       "    0.2789098620414734,\n",
       "    -0.08478192239999771,\n",
       "    0.25605398416519165,\n",
       "    -0.06977075338363647,\n",
       "    -0.06896576285362244,\n",
       "    0.6655471920967102,\n",
       "    -0.07891671359539032,\n",
       "    -0.17399387061595917,\n",
       "    0.04786311835050583,\n",
       "    0.10821349918842316,\n",
       "    0.08929640799760818,\n",
       "    0.14343099296092987,\n",
       "    0.16792595386505127,\n",
       "    0.427090585231781,\n",
       "    -0.27369192242622375,\n",
       "    -0.20948606729507446,\n",
       "    -0.11025969684123993,\n",
       "    -0.19700771570205688,\n",
       "    0.20366308093070984,\n",
       "    0.01370113343000412,\n",
       "    -0.15624603629112244,\n",
       "    -0.1545754373073578,\n",
       "    0.47393232583999634,\n",
       "    0.2968006134033203,\n",
       "    -0.3560287058353424,\n",
       "    -0.17474745213985443,\n",
       "    0.0839439406991005,\n",
       "    -0.2787473201751709,\n",
       "    0.3769336938858032,\n",
       "    -0.44083815813064575,\n",
       "    -0.07610215246677399,\n",
       "    0.06552372872829437,\n",
       "    0.10503921657800674,\n",
       "    0.24494034051895142,\n",
       "    -0.06382235884666443,\n",
       "    -0.03497058153152466,\n",
       "    -0.11426234245300293,\n",
       "    -0.13563846051692963,\n",
       "    0.709717869758606,\n",
       "    0.06133721023797989,\n",
       "    0.8819040060043335,\n",
       "    0.24787047505378723,\n",
       "    0.13915883004665375],\n",
       "   'entity_embedding': [0.30115607380867004,\n",
       "    -0.2971831262111664,\n",
       "    -0.24845416843891144,\n",
       "    0.12573760747909546,\n",
       "    0.15562830865383148,\n",
       "    -0.416630357503891,\n",
       "    0.4098249077796936,\n",
       "    0.844752311706543,\n",
       "    -0.011385446414351463,\n",
       "    1.0812066793441772,\n",
       "    -0.09241710603237152,\n",
       "    0.25531071424484253,\n",
       "    0.39807748794555664,\n",
       "    -0.5651153326034546,\n",
       "    -0.3166109323501587,\n",
       "    1.175793170928955,\n",
       "    0.5277212858200073,\n",
       "    0.19530627131462097,\n",
       "    0.43986010551452637,\n",
       "    -0.0021041538566350937,\n",
       "    -0.9393190741539001,\n",
       "    -0.3199694752693176,\n",
       "    0.0870715081691742,\n",
       "    0.35745808482170105,\n",
       "    -1.03800630569458,\n",
       "    0.789326012134552,\n",
       "    0.09708458930253983,\n",
       "    -0.4787347614765167,\n",
       "    -0.1917830854654312,\n",
       "    0.5348588228225708,\n",
       "    0.4576016664505005,\n",
       "    -0.35748690366744995,\n",
       "    -1.0336534976959229,\n",
       "    0.8626435399055481,\n",
       "    -0.7767253518104553,\n",
       "    -0.19604088366031647,\n",
       "    0.6313158273696899,\n",
       "    0.14948749542236328,\n",
       "    0.5236766934394836,\n",
       "    -0.03826142102479935,\n",
       "    -0.0129996994510293,\n",
       "    -0.5497332811355591,\n",
       "    1.0808260440826416,\n",
       "    -0.007541701663285494,\n",
       "    -0.5180426836013794,\n",
       "    -0.33262574672698975,\n",
       "    0.34935998916625977,\n",
       "    -1.4306678771972656,\n",
       "    0.47523796558380127,\n",
       "    -0.20418407022953033,\n",
       "    1.6337106227874756,\n",
       "    0.21588115394115448,\n",
       "    0.1556379497051239,\n",
       "    0.33991318941116333,\n",
       "    -1.0832302570343018,\n",
       "    -0.013861696235835552,\n",
       "    0.3219185471534729,\n",
       "    -0.1278374195098877,\n",
       "    -0.763687789440155,\n",
       "    0.2532994747161865,\n",
       "    -0.2641879618167877,\n",
       "    -1.4250028133392334,\n",
       "    0.6828781366348267,\n",
       "    0.23675504326820374,\n",
       "    0.3620050549507141,\n",
       "    -0.17353200912475586,\n",
       "    -1.1341737508773804,\n",
       "    0.25928640365600586,\n",
       "    -0.3574197590351105,\n",
       "    0.05342014878988266,\n",
       "    -0.258378267288208,\n",
       "    0.12602046132087708,\n",
       "    0.9363989233970642,\n",
       "    0.9762530326843262,\n",
       "    0.25653088092803955,\n",
       "    -0.3945409059524536,\n",
       "    -1.456972360610962,\n",
       "    0.5519903302192688,\n",
       "    0.19786091148853302,\n",
       "    0.5613494515419006,\n",
       "    0.038428906351327896,\n",
       "    1.3012884855270386,\n",
       "    0.04167284816503525,\n",
       "    0.2040765881538391,\n",
       "    -0.12393803894519806,\n",
       "    -0.06797023117542267,\n",
       "    -0.8794755935668945,\n",
       "    -1.3243987560272217,\n",
       "    0.2123095691204071,\n",
       "    0.09954346716403961,\n",
       "    0.448373943567276,\n",
       "    -0.32453274726867676,\n",
       "    -0.3823782205581665,\n",
       "    0.7460602521896362,\n",
       "    0.30266639590263367,\n",
       "    -0.9773066639900208,\n",
       "    -0.2065196931362152,\n",
       "    -0.8906922340393066,\n",
       "    -0.2760157883167267,\n",
       "    0.5952996015548706,\n",
       "    -1.5768074989318848,\n",
       "    0.30741891264915466,\n",
       "    -0.043656788766384125,\n",
       "    0.25560545921325684,\n",
       "    0.21707813441753387,\n",
       "    -0.37956562638282776,\n",
       "    -0.3986319303512573,\n",
       "    0.3340977728366852,\n",
       "    0.49874624609947205,\n",
       "    -0.4796082675457001,\n",
       "    -0.6700000166893005,\n",
       "    1.328298568725586,\n",
       "    -0.10307170450687408,\n",
       "    0.005493307486176491,\n",
       "    0.061860594898462296,\n",
       "    -0.07284541428089142,\n",
       "    0.5238110423088074,\n",
       "    1.0355842113494873,\n",
       "    0.20853914320468903,\n",
       "    0.6657915711402893,\n",
       "    -0.8335227370262146,\n",
       "    0.7066413164138794,\n",
       "    -0.336885929107666,\n",
       "    0.5499586462974548,\n",
       "    0.6073265075683594,\n",
       "    0.24074381589889526,\n",
       "    -0.10078586637973785,\n",
       "    -1.7692006826400757],\n",
       "   'dot_product': 3.3309245109558105}]}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.accept = 'application/json; verbose=True'\n",
    "predictor.predict(inference_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Compute Anomaly Scores based on Test set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dot_product output of the model provides a good measure of how compatible an IP address and online resource are. However, the range of the dot_product is unbounded. This means to be able to consider an event as anomolous we need to define a threshold. Such that when we score an event, if the dot_product is above the threshold we can flag the behavior as anomolous.However, picking a threshold can be more of an art, and a good threshold depends on the specifics of your problem and dataset.\n",
    "\n",
    "In the following section, we show how to pick a simple threshold by comparing the score distributions between known normal and malicious traffic:\n",
    "\n",
    "We construct a test set of 'Normal' traffic;\n",
    "- Inject 'Malicious' traffic into the dataset;\n",
    "- Plot the distribution of dot_product scores for the model on 'Normal' trafic and the 'Malicious' traffic.\n",
    "- Select a threshold value which separates the normal distribution from the malicious traffic threshold. This value is based on your false-positive tolerance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### [1] Construct 'Normal' Traffic Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use previously created test dataset as the 'Normal' traffic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>ip_address</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>user_63</td>\n",
       "      <td>69.55.235.87</td>\n",
       "      <td>2018-11-12 12:05:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>user_63</td>\n",
       "      <td>69.57.9.174</td>\n",
       "      <td>2018-11-12 13:20:30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>user_63</td>\n",
       "      <td>69.56.173.241</td>\n",
       "      <td>2018-11-13 14:21:28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>user_63</td>\n",
       "      <td>69.56.231.51</td>\n",
       "      <td>2018-11-12 08:33:13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>user_63</td>\n",
       "      <td>69.57.13.111</td>\n",
       "      <td>2018-11-12 02:23:25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       user     ip_address           timestamp\n",
       "0   user_63   69.55.235.87 2018-11-12 12:05:07\n",
       "1   user_63    69.57.9.174 2018-11-12 13:20:30\n",
       "5   user_63  69.56.173.241 2018-11-13 14:21:28\n",
       "16  user_63   69.56.231.51 2018-11-12 08:33:13\n",
       "30  user_63   69.57.13.111 2018-11-12 02:23:25"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(950187, 3)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### [2]  Inject Fradulent Traffic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we had a dataset with enough real malicious activity, we would use that to determine a good threshold. Those are hard to come by. So instead, we manually create some malicious web traffic that mimics a realistic attack scenario.\n",
    "\n",
    "We take a set of user accounts from the test set and randomly generate IP addresses. The users should not have used these IP addresses during training. This simulates an attacker logging in to a user account without knowledge of their IP history."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "from generate_data import draw_ip\n",
    "\n",
    "# We only need the dot product. Let's reset the predictor output type.\n",
    "predictor.accept = 'application/json; verbose=False'\n",
    "\n",
    "\n",
    "def score_ip_insights(predictor, df):\n",
    "    \n",
    "    def get_score(result):\n",
    "        \"\"\"Return the negative to the dot product of the predictions from the model.\"\"\"\n",
    "        return [-prediction[\"dot_product\"] for prediction in result[\"predictions\"]]\n",
    "    \n",
    "    df = df[['user', 'ip_address']]\n",
    "    result = predictor.predict(df.values)\n",
    "    return get_score(result)\n",
    "\n",
    "\n",
    "def create_test_case(train_df, test_df, num_samples, attack_freq):\n",
    "    \"\"\"Creates a test case from provided train and test data frames. \n",
    "    \n",
    "    This generates test case for accounts that are both in training and testing data sets.\n",
    "\n",
    "    :param train_df: (panda.DataFrame with columns ['user', 'ip_address']) training DataFrame\n",
    "    :param test_df: (panda.DataFrame with columns ['user', 'ip_address']) testing DataFrame\n",
    "    :param num_samples: (int) number of test samples to use\n",
    "    :param attack_freq: (float) the ratio of negative_samples:positive_samples to generate for test case \n",
    "    :return: DataFrame with both good and bad traffic, with labels\n",
    "    \"\"\"\n",
    "    # Get all possible accounts. The IP Insights model can only make predictions on users it has seen in training\n",
    "    # Therefore, filter the test dataset for unseen accounts, as their results will not mean anything.\n",
    "    valid_accounts = set(train_df['user'])\n",
    "    valid_test_df = test_df[test_df['user'].isin(valid_accounts)]\n",
    "\n",
    "    good_traffic = valid_test_df.sample(num_samples, replace=False)\n",
    "    good_traffic = good_traffic[['user', 'ip_address']]\n",
    "    good_traffic['label'] = 0\n",
    "\n",
    "    # Generate malicious traffic\n",
    "    num_bad_traffic = int(num_samples * attack_freq)\n",
    "    bad_traffic_accounts = np.random.choice(list(valid_accounts), size=num_bad_traffic, replace=True) \n",
    "    bad_traffic_ips = [draw_ip() for i in range(num_bad_traffic)]\n",
    "    bad_traffic = pd.DataFrame({'user': bad_traffic_accounts, 'ip_address': bad_traffic_ips})\n",
    "    bad_traffic['label'] = 1\n",
    "    \n",
    "    # All traffic labels are: 0 for good traffic; 1 for bad traffic. \n",
    "    all_traffic = good_traffic.append(bad_traffic)\n",
    "\n",
    "    return all_traffic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>ip_address</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1398147</th>\n",
       "      <td>user_4490</td>\n",
       "      <td>194.98.41.250</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1913974</th>\n",
       "      <td>user_6008</td>\n",
       "      <td>168.126.246.173</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1235379</th>\n",
       "      <td>user_3909</td>\n",
       "      <td>192.135.179.191</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1989308</th>\n",
       "      <td>user_6453</td>\n",
       "      <td>201.201.237.24</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1312691</th>\n",
       "      <td>user_4149</td>\n",
       "      <td>66.207.146.193</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              user       ip_address  label\n",
       "1398147  user_4490    194.98.41.250      0\n",
       "1913974  user_6008  168.126.246.173      0\n",
       "1235379  user_3909  192.135.179.191      0\n",
       "1989308  user_6453   201.201.237.24      0\n",
       "1312691  user_4149   66.207.146.193      0"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NUM_SAMPLES = 100000\n",
    "test_case = create_test_case(train_df, test_df, num_samples=NUM_SAMPLES, attack_freq=1)\n",
    "test_case.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_case_scores = score_ip_insights(predictor, test_case)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### [3] Plot Distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXzcdZ348dd7ZpLJfadX0jalB6Wld8pVrgrIJSIol6xSj0VckOWn+1PUdWVd9efB6oqigAooohWkICoKcq+cbaFAD1rS0iM9kzT3MZnj/fvjO5OmZZpMkpnMkffz8UgzM9/r/U2Tec/nFlXFGGOMOZIr2QEYY4xJTZYgjDHGRGUJwhhjTFSWIIwxxkRlCcIYY0xUnmQHEE8VFRVaU1OT7DCMMSZtrF27tlFVK6Nty6gEUVNTw5o1a5IdhjHGpA0R2XG0bVbFZIwxJipLEMYYY6KyBGGMMSaqjGqDMMakL7/fT319PT09PckOJSPl5ORQXV1NVlZWzMdYgjDGpIT6+noKCwupqalBRJIdTkZRVZqamqivr2fatGkxH2dVTMaYlNDT00N5ebklhwQQEcrLy4dcOrMEYYxJGZYcEmc4P1tLEGZEDrT3sOq1+mSHYYxJAEsQZkRufugtPv/AG+xvs4ZFk/5EhC984Qt9z2+99VZuueWWUY3hzDPPTJkBv5YgzLA9t6WBp98+AMC2hs4kR2PMyHm9XlatWkVjY+Owjg8EAnGOKLksQZhh8QdD/NefN1Kenw3A9iZLECb9eTwerr32Wn74wx++Z9uOHTs466yzmD9/PmeddRY7d+4EYMWKFXz+859n+fLlfOlLX2LFihV89rOfZfny5RxzzDE899xzfPKTn+S4445jxYoVfef77Gc/S21tLXPnzuXrX//6aN3ikFg3VzMs97+8g7oDHdzxT0u48Xevs73REoSJn//80wY27mmL6znnTCri6xfNHXS/66+/nvnz5/PFL37xsNdvuOEGPv7xj3PNNddw9913c+ONN/LII48AsGXLFp588kncbjcrVqygubmZp59+mkcffZSLLrqIF154gV/84hcsXbqUdevWsXDhQr71rW9RVlZGMBjkrLPO4s0332T+/PlxveeRshKEGZY7n9/GyceUc+7c8Uwpz+NdSxAmQxQVFfHxj3+c22677bDXX3rpJT760Y8C8LGPfYx//OMffdsuu+wy3G533/OLLroIEWHevHmMHz+eefPm4XK5mDt3Ltu3bwfggQceYPHixSxatIgNGzawcePGxN/cEFkJwgxZKKTsb+vhsiXViAg15flWxWTiKpZP+ol00003sXjxYj7xiU8cdZ/+3Ubz8/MP2+b1egFwuVx9jyPPA4EA7777LrfeeiurV6+mtLSUFStWpOQIcitBmCFr7wkQUijJc9ofjqnMZ0dTF6GQJjkyY+KjrKyMyy+/nF/+8pd9r51yyimsXLkSgPvvv59TTz112Odva2sjPz+f4uJi9u/fz1//+tcRx5wIliDMkDV39QJQkufM6VJTno8vEGKvdXU1GeQLX/jCYb2ZbrvtNu655x7mz5/Pfffdx49+9KNhn3vBggUsWrSIuXPn8slPfpJly5bFI+S4syomM2SRBFEaLkHUVOQB8G5DJ1UluUmLy5iR6ujo6Hs8fvx4urq6+p7X1NTw9NNPv+eYe++996jPa2pqWL9+fdRtRx4X8eyzzw4p5kSyEoQZspZuPwDF4RLEtAqn/vVda4cwJqNYgjBD1nJECWJ8YQ45WS7r6mpMhrEEYYaspcspQZTkOiUIlyvck8kShDEZxRKEGbLmLj8iUJR7aOGRmvJ8q2IyJsNYgjBD1trVS1FOFm7XoX7g0yrz2XWwi0AwlMTIjDHxZAnCDFlzl5/SvMOXLZxWno8/qOxpsa6uxmQKSxBmyFq6/X2D5CJqwj2ZtjV2RDvEmLTgdrtZuHAhxx9/PBdddBEtLS1xOe+9997LDTfcEJdz9dd/avCamhrmzZvHggULeP/738++fftGfH5LEGbIWrp6+wbJRUTGQlhDtUlnubm5rFu3jvXr11NWVsbtt9+e7JCG5JlnnuGNN96gtraWb3/72yM+nyUIM2TNXb19XVwjKgu8FHo9bLV1IUyGOPnkk9m9ezcAr776KqeccgqLFi3ilFNOYfPmzYBTMrj00ks577zzmDlz5mEzwN5zzz3MmjWLM844gxdeeKHv9YGmDY9lmvBYnH766dTV1Y3wJ2Ajqc0wtHT5Kc49vAQhIsyaUMjmfe1JispklL/eDPveiu85J8yD878T067BYJCnnnqKT33qUwDMnj2b559/Ho/Hw5NPPslXvvIVHnroIQDWrVvH66+/jtfr5dhjj+Vzn/scHo+Hr3/966xdu5bi4mKWL1/OokWLgIGnDY9lmvBY/PnPf2bevHlD/Qm9hyUIMySBYIj2nsB7ShAAsycU8qc39qCqtvi8SUvd3d0sXLiQ7du3s2TJEs455xwAWltbueaaa3jnnXcQEfx+f98xZ511FsXFxQDMmTOHHTt20NjYyJlnnkllZSUAV1xxBVu2bAGcacNXrVoFONOG9y91RJsmHOibJnywBLF8+XLcbjfz58/nm9/85oh/HpYgzJC0hqfZOLINAmD2xCLuf2Une1t7mGRzMpmRiPGTfrxF2iBaW1v5wAc+wO23386NN97I1772NZYvX87DDz/M9u3bOfPMM/uO6T+dt9vt7lt2NNYPSf33G2ya8ME888wzVFRUxHTdWFgbhBmS5q6jJ4jjJhQC8Pa++K4EZsxoKy4u5rbbbuPWW2/F7/fT2tpKVVUVcPRJ9vo78cQTefbZZ2lqasLv9/Pggw/2bYvntOGJZgnCDElrd2Sq7/dWMc0KJ4hNe60dwqS/RYsWsWDBAlauXMkXv/hFvvzlL7Ns2TKCweCgx06cOJFbbrmFk08+mbPPPpvFixf3bYvntOGJJqqZs8hLbW2tRvoEm8R4cuN+Pv3rNTx6wzLmV5e8Z/uy7zzN4qml/PiqRUmIzqSzTZs2cdxxxyU7jIwW7WcsImtVtTba/laCMEPSt1hQ7ntLEADHTSzk7b1WxWRMJkhoghCR80Rks4jUicjNUbZfLSJvhr9eFJEFsR5rkqOvkTr/vW0QALMnFLGtsRNfYPBiuDEmtSUsQYiIG7gdOB+YA1wlInOO2O1d4AxVnQ/8F3DXEI41SdDc1YvbJRR6o3eAmz2xkGBIqTtgU26YocukKu9UM5yfbSJLECcAdaq6TVV7gZXAxf13UNUXVbU5/PRloDrWY01ytHT5KcnNOmoXvtkTigB42xqqzRDl5OTQ1NRkSSIBVJWmpiZycnKGdFwix0FUAbv6Pa8HThxg/08Bfx3qsSJyLXAtwJQpU4Ybq4lRS5c/ahfXiJryPLI9Luvqaoasurqa+vp6Ghoakh1KRsrJyaG6unrwHftJZIKI9hEz6kcDEVmOkyAiHYJjPlZV7yJcNVVbW2sfPRKspbs3ahfXCI/bxazxBbxtU26YIcrKymLatGnJDsP0k8gqpnpgcr/n1cCeI3cSkfnAL4CLVbVpKMea0dfc+d61II40e0KRjYUwJgMkMkGsBmaKyDQRyQauBB7tv4OITAFWAR9T1S1DOdYkR2u3n+KjdHGNmDGugMYOH+09/gH3M8aktoRVMalqQERuAB4H3MDdqrpBRK4Lb78D+A+gHPhpuNEzoKq1Rzs2UbGa2DlTfQ9cghhf5Mwh09DuozBn4H2NMakroZP1qepjwGNHvHZHv8efBj4d67EmuXyBIF29wQEbqQEqC5yeEg3tPo6pLBiN0IwxCWAjqU3MWvsm6hu4iqmy0ClBHGj3JTwmY0ziWIIwMRtoJtf+xhUeqmIyxqQvSxAmZi3heZiiLRbUX3FuFlluoaHDEoQx6cwShIlZrCUIl0uoKPBaCcKYNGcJwsQsUoIYrA0CnHYISxDGpDdLECZm+9p6EIGKgsETxLhCrzVSG5PmLEGYmNU3dzO+MAevxz3ovlaCMCb9WYIwMatv7qK6NDemfSsLvBzs9BEM2fRYxqQrSxAmZvXN3bEniEIvIYWmTitFGJOuLEGYmASCIfa29lBdmhfT/pU2FsKYtGcJwsRkX1sPwZAOoQThTLdhDdXGpC9LECYmu5u7AWIuQdhoamPSnyUIE5P6cIKoirEEUVFgCcKYdGcJwsQkkiAmlcS2pm1utptCr8cShDFpzBKEiUl9cxfji7wxjYGIqCz02nxMxqQxSxAmJk4X19jaHyIqC700tFmCMCZdWYIwMalviX2QXISVIIxJb5YgzKCCIWVvS8/wEoS1QRiTtixBmEHtb+shENJhVTF1+AJ09QYSFJkxJpEsQZhB1feNgRhaCWJc4aG1qY0x6ccShBlUfXMXAFUlQ69iAksQxqQrSxBmUIfGQAwxQdhgOWPSmiUIM6j65i7GFXrJyYp9DAT0K0FYTyZj0pIlCDOooUzz3V9ZfjYusRKEMenKEoQZ1O6WbqqG2IMJwO0S8r0e2nusF5Mx6cgShBlUU0dvX3vCUBXlZFmCMCZNWYIwA/IHQ3T4AhTnZg3r+AKvhw6fP85RGWNGgyUIM6C2bufNvSRveAmiMMeqmIxJV5YgzIBawwli2CWIHA8dPksQxqQjSxBmQCNOEF4PHVaCMCYtWYIwA2qJJIhhVzFl0WYJwpi0ZAnCDKhthCWIwhxrpDYmXVmCMANq6Rp5FVOPP4Q/GIpnWMaYUWAJwgxopG0QhTkeAGuHMCYNJTRBiMh5IrJZROpE5OYo22eLyEsi4hORfzti23YReUtE1onImkTGaY6utdtPfrabLPfwflUKvOEEYT2ZjEk7nkSdWETcwO3AOUA9sFpEHlXVjf12OwjcCHzoKKdZrqqNiYrRDK6ly09JXvawj4+UIGwshDHpJ5EliBOAOlXdpqq9wErg4v47qOoBVV0NWCtmimrt9lM0zOolcHoxAbT32H+xMekmkQmiCtjV73l9+LVYKfCEiKwVkWuPtpOIXCsia0RkTUNDwzBDNUfT2t1Lce7wC5pWxWRM+kpkgpAor+kQjl+mqouB84HrReT0aDup6l2qWquqtZWVlcOJ0wygtdtPSe7wq5gKcixBGJOuEpkg6oHJ/Z5XA3tiPVhV94S/HwAexqmyMqOspcs/7B5McKgNwgbLGZN+EpkgVgMzRWSaiGQDVwKPxnKgiOSLSGHkMfB+YH3CIjVH1drtH/ZEfQCFXudY6+ZqTPpJWC8mVQ2IyA3A44AbuFtVN4jIdeHtd4jIBGANUASEROQmYA5QATwsIpEYf6uqf0tUrCa6Hn8QXyA0okbqnCwXbpfYaGpj0lBMCUJEjlfVIX+CV9XHgMeOeO2Ofo/34VQ9HakNWDDU65n4GukgOQARsSm/jUlTsVYx3SEir4rIv4hISUIjMimjdYRrQUTYjK7GpKeYEoSqngpcjdPovEZEfisi5yQ0MpN0I52HKaLA66HdejEZk3ZibqRW1XeAfwe+BJwB3CYib4vIpYkKziRXXwliBN1cIbIutbVBGJNuYkoQIjJfRH4IbALeB1ykqseFH/8wgfGZJIpHGwTYqnLGpKtYSxA/AV4DFqjq9ar6GvSNVfj3RAVnkqulqxeITxWTtUEYk35i7eZ6AdCtqkEAEXEBOarapar3JSw6k1Rt3X5EDg12Gy7rxWRMeoq1BPEkkNvveV74NZPBWrr9FOVk4XJFmzUldgU51khtTDqKNUHkqGpH5En4cV5iQjKporV7ZNNsRBTlZNEbCOELBOMQlTFmtMSaIDpFZHHkiYgsAboTE5JJFSOdZiOib0ZXq2YyJq3EWrl8E/CgiEQm25sIXJGYkEyqGOlEfRH9p/wuL/CO+HzGmNERU4JQ1dUiMhs4Fmca77dV1Tq2Z7i2bj/VpbmD7zgIW1XOmPQ0lO4pS4Ga8DGLRARV/XVCojIpoSVObRAFliCMSUuxTtZ3HzAdWAdEWhoVsASRoVQ1bo3UfVN+W08mY9JKrCWIWmCOqg5lRTiTxjp7gwRDGpdG6kNVTFYraUw6ibUX03pgQiIDMaklXqOowZYdNSZdxVqCqAA2isirgC/yoqp+MCFRmaQ7NA/TyCbqg0O9mKwNwpj0EmuCuCWRQZjU0xqnqb4BcrLcZLtdliCMSTOxdnN9TkSmAjNV9UkRycNZRtRkqHjN5BrhzOhqbRDGpJNYp/v+Z+APwJ3hl6qARxIVlEm+ljitJhdhM7oak35ibaS+HliGs1Z0ZPGgcYkKyiRfY7vT1FReMPI2CLAZXY1JR7EmCJ+q9kaeiIgHZxyEyVCNHT4Kczx4PfGpSbRlR41JP7EmiOdE5CtAbngt6geBPyUuLJNsjR29VMZx3qTCHKtiMibdxJogbgYagLeAzwCPYSvJZbSGDh8VcU0QWbRbI7UxaSXWXkwh4OfhLzMGNHb4mD2hMG7nS/tG6t5O8OSAK1zlpgpbn4YpJ0F2fnJjMyZBYp2L6V2itDmo6jFxj8ikhKaO3riWIArCjdSqisjIVqgbdR0H4M7TIb8SrvodFE6EP98Er/0aZp0PV/4WXLEWxo1JH0OZiykiB7gMKIt/OCYV9AZCtHb745ogSvOyCISUDl+Awpz4dJ0dFaEQrLoWupvB1wF3LYeJC6Du71BzGmz5K7zwQzjtC8mO1Ji4i+ljj6o29fvarar/A7wvwbGZJGnqdLq4xjNBlOc75zrY2TvIninmhR/CtmfgvO/Ap590qpPq/g7nfhuu+RMc/2F4+puw8VEI2ZKqJrPEWsW0uN9TF06JIn4V1CalNLY7b+LxGgMBUBY+V2NHL1PL06TOvn4tPP0tmHsJLFkBInDts9D8Lkxa5Oxz0W2wfyM88DHIKYFjL4AL/xuybcl2k/5irWL6736PA8B24PK4R2NSQmNH/EsQFelWggj44I//AoUT4KIfOckBILcEchcd2s9bAJ96At4Jf73xW6fhesk1yYnbmDiKtRfT8kQHYlJHQzhBxHMcRKQE0dThG2TPFPH8rdDwNnz0QcgpHnjfnCKY9xGnumnvm7D2XksQJiPEWsX0+YG2q+oP4hOOSQV9JYjC+FUxleeHE0Q6lCD2vQX/+AHMvwJmvT/240Scqqi/fclJFBPnJyxEY0ZDrH3zaoHP4kzSVwVcB8zBaYewtogM09jeS162m7zsoSxZPrCcLDf52W6aOtIgQTz1X057wnnfGfqx8y93xku89qv4x2XMKBvKgkGLVbUdQERuAR5U1U8nKjCTPE2d8R1FHVFe4O3rIZWyOptg61Nw8g2QN4ye3HllMOdD8OYDcM43bBCdSWuxliCmAP0/+vUCNYMdJCLnichmEakTkZujbJ8tIi+JiE9E/m0ox5rEaezwxbUHU0R5QXbqN1JvfARCAadNYbiWXAO+NtjwcPziMiYJYk0Q9wGvisgtIvJ14BXg1wMdICJu4HbgfJzqqKtEZM4Rux0EbgRuHcaxJkEa2+M7ijqiPD+bxlSvYlr/EFTMgvHHD/8cU06G8pnw2n3xi8uYJIh1oNy3gE8AzUAL8AlV/fYgh50A1KnqtvBU4SuBi4847wFVXQ0cOYvboMeaxGmM80R9EeX5Xg6mchVT627Y8SIc/5FD3VqHQwQWXQ27XobGuvjFZ8woG8oEMnlAm6r+CKgXkWmD7F8F7Or3vD78WixiPlZErhWRNSKypqGhIcbTm6MJBEMc7OqlMgFVTGUF2TR19KKaokuJbFgF6MiqlyLmXwnignX3j/xcxiRJrEuOfh34EvDl8EtZwG8GOyzKa7G+M8R8rKrepaq1qlpbWVkZ4+nN0Rzs6kUVKgoTU8UUCClt3Sk6q+tbf4CJC6F8+sjPVTQRZpwDb6y0KThM2oq1BHEJ8EGgE0BV9zB499Z6YHK/59XAnhivN5JjzQhEptlITC+myFiIFKxmatoKe9c5g93iZdHV0L4Htj4Tv3MaM4piTRC96tQLKICIxNJ3bzUwU0SmiUg2cCXwaIzXG8mxZgQib96RgW3xFJmwLyUHy21Y5Xw//tL4nXPW+ZBbBusGK2wbk5piHQfxgIjcCZSIyD8Dn2SQxYNUNSAiNwCPA27gblXdICLXhbffISITgDVAERASkZuAOaraFu3Y4dygGZpDo6gTWIJIxZ5MGx6BySdCcXX8zunJdkZjr/mls6ZEwbj4nduYUTBoghBndZffA7OBNuBY4D9U9e+DHauqj+EsT9r/tTv6Pd6HU30U07Em8RJaxdRXgkixKqaGLbB//fBGTg9m6afhlZ/BmnvgzC/F//zGJNCgCUJVVUQeUdUlwKBJwaS3xg4f2W4XRTnxm2YjoixcbXUw1UoQGx4GxBkBHW8VM2Dm+2H1L+DUm8AT/8RrTKLE2gbxsogsTWgkJiU0dPioKMhOyLKg2R4XhTme1GuD2LAKpp7i9DxKhBOvg84DNrLapJ1YE8RynCSxVUTeFJG3ROTNRAZmkqOxozch7Q8RFQXevnaOlHBgkzOt99xLEneN6e+DimPh5Z9Cqo4BMSaKAesRRGSKqu7EmfLCjAGN7T7GFyUuQZTlp9h8TGvuAXHDcR9M3DVE4KTr4M//B3a9ClNOTNy1jImjwUoQjwCo6g7gB6q6o/9X4sMzoykYUnYd7GJSSW7CrlGen506vZhadsHae5zxCoXjE3uteZeDJxfW/yGx1zEmjgZLEP0roo9JZCAm+bbsb6fdF2DJ1NKEXcOZ8jtFEsTz33O+n/7FxF/LWwAzz4aNj0IolPjrGRMHgyUIPcpjk4HWbD8IQO3UYayDEKPy/Gyau3oJhZL869S0FV6/H5Z8AkomD75/PMz5EHTsg/pXR+d6xozQYAligYi0iUg7MD/8uE1E2kWkbTQCNKNnzY5mxhV6mVyWwCqmgmyCIaW1+8gJfEfZc98Fdzac9oXRu+bM94PbCxv/OHrXNGYEBkwQqupW1SJVLVRVT/hx5HnRaAVpRsea7c3U1pQmpItrRFl+CszHFAzApj/DgisS3/bQX04RzDjLSRBWzWTSwFCm+zYZbE9LN7tbuhNavQSHRmgndeGgfW+AvxOmnT76155zMbTthj2vjf61jRkiSxAGcKqXAJbWJDZB9I2mTmZD9Y6XnO9TThn9a886D1xZNmjOpAVLEAaAtdsPkpft5riJg83iPjKHJuxLYhXTzpegdFriRk4PJLcEZp0Lr98H3S2jf31jhsAShAGcEsSiKSV43In9lSjLy8Yl0NCepAQRCjnLik5dlpzrA5zxJehphRdvS14MxsTAEoShwxdg0942liS4/QHA43YxviiH+pbuhF8rqsYt0H0Qpp6cnOsDTJzvLEz08s+cacCNSVGWIAyv72wmpLC0JnED5PqrKslld3OSEsSOF5zvU5PQ/tDf8q9CwAfP35rcOIwZgCUIw9t72wGYV1U8KterKs1ld7JKEDtfgoIJThtEMpVPh0X/BGvuhvZ9yY3FmKOwBGHY2tBBRUE2JXnxX2Y0mqqSXPa19hAc7dHUquH2h1OcCfSS7ZTPQcgPbz2Y7EiMicoShGFrQwfHVBaM2vWqSnMJhJQD7T2jdk0AWnY6YxCSXb0UUTETqmrhjZXJjsSYqCxBGOoOdDB9NBNEeLbYUW+H2Pas873m1NG97kAWXOksd7rvrWRHYsx7WIIY4w529tLc5Wd6Zf6oXbO6NJwgRrsdou5JKKqCytmje92BzL3UGThnpQiTgixBjHFbGzoAmD5u9EoQkfUm6kezBBEMwLbnnNXdUqH9ISK/3Bk499aDTozGpBBLEGPc1gNOgpgxilVMedkeyvKzR7cEsXsN+Fphxtmjd81YLbgSOvYfqgIzJkVYghjjtjZ04PW4+toFRsuoj4WoewrEBcecMXrXjNXM90NOCbz5+2RHYsxhLEGMcXUHnB5MLtfoVrtUlYzyWIi6J6F6KeSOzmDAIfF4Yc4H4e2/QG9XsqMxpo8liDFua0PnqDZQR1SVOiUI1VEYC9HZBHteh+lnJf5aw3X8R5wpyLf8LdmRGNPHEsQY1uMPsqu5a1S7uEZUleTS7Q/S3DUKK8ttewbQ1Gx/iKg51Rnhvf6hZEdiTB9LEGPY9qZOVGHGKPZgiqgqHcWxEO/83alamrQw8dcaLpcb5l7ixNrTmuxojAEsQYxpWw90AiStBAGwuyXBde7BALzzBMw813kTTmXzPgJBn7McqjEpwBLEGLa1oQMRmFYx+m0QkcFyCR8LUf+qM733secn9jrxULUESqbC+j8kOxJjAEsQY1rdgQ6qSnLJzR79T9bFuVnkZbsT35Pp7b+AOxtmpHADdYQILLgKtj4NBzYlOxpjLEGMZaM9SV9/IpL4sRCqsPkxqDkNvIldSjVuTvwMZBfYOhEmJViCGMN2HeyipjwvaddP+LoQje/AwW3pUb0UkVcGSz8FG1ZBY12yozFjnCWIMaqtx09bT2DUR1D3l/DBcpsfc76nU4IAOPlz4PbC//53siMxY1xCE4SInCcim0WkTkRujrJdROS28PY3RWRxv23bReQtEVknImsSGedYFKnaqS5NXgmipjyfli4/zZ29ibnA5r/ChPlQXJ2Y8ydKQSXUfsKZeqN5e7KjMWNYwhKEiLiB24HzgTnAVSIy54jdzgdmhr+uBX52xPblqrpQVWsTFedYVd+XIJJXgpg+zuk9ta2xI/4nb9sLu16B2RfG/9yj4ZTPAQqv/TrZkZgxLJEliBOAOlXdpqq9wErg4iP2uRj4tTpeBkpEZGICYzJhu5ud8QdVyUwQ4QbyyHiMuHrrAUBh3mXxP/doKJoEM86Bdb+FUDDZ0ZgxKpEJogrY1e95ffi1WPdR4AkRWSsi1yYsyjGqvrmbnCwX5fmjsw51NNWleWS7XX1rUsSNqrMAT/UJUD49vuceTYuuhva9TrdXY5IgkQki2vSgR87MNtA+y1R1MU411PUicnrUi4hcKyJrRGRNQ0PD8KMdY3a3dFNVkoskcfEct0uYVpEf/wSx7y04sNFZZyGdzTof8srh9d8kOxIzRiUyQdQDk/s9rwb2xLqPqka+HwAexqmyeg9VvUtVa1W1trKyMk6hZ7765u6kNlBHzBhXwNaGOFcxvbHSGRw395L4nne0ebJh/hXOYL/OpmRHY8agRCaI1VXmJbgAABjNSURBVMBMEZkmItnAlcCjR+zzKPDxcG+mk4BWVd0rIvkiUgggIvnA+4H1CYx1zKlv7kpq+0PE9Mp8dh7swheIUz17MOC0P8w61xlTkO4W/ROE/M6SpMaMsoQlCFUNADcAjwObgAdUdYOIXCci14V3ewzYBtQBPwf+Jfz6eOAfIvIG8CrwF1W1ifLjpNMXoLnLn9QeTBHTxxUQDCk7m+I0ad/Wp6GzwZmyIhOMn+vM0fTqnbZmtRl1nkSeXFUfw0kC/V+7o99jBa6Pctw2YEEiYxvLIoPTkjlILiLSk6nuQAczx8dhOowNqyCn2OkBlClO/Tz8/mpnEr90b1cxacVGUo9BqTBILiIyk2xcGqoDPqe+fvZFTv19pph9IYyfB89/37q8mlFlCWIMqg+PgUiFKqZ8r4dJxTnxaaje+gz42mDuh0Z+rlQiAmd8EZrqbMU5M6osQYxB9c3dZLtdVBZ4kx0K4LRDxKUEseFhp3pp2hkjP1eqmf0BGDcHnvuetUWYUWMJYgyqb+lmUkkOLlfyxkD0N72ygK0HOnCapIYp4HMm58u06qUIlwuWfxWa3oFXjpyRxpjEsAQxBqXKGIiI6ZX5dPYG2d/mG/5Jtj6dmdVL/c2+0Bk898y3bRI/MyosQYxBu5u7U6IHU0TfnEwjqWba8EjmVi9FiMCFt4K44M+fd6YUMSaBLEGMMT3+II0dvpRooI6YPs5JEJv2tg3vBL4O2PQnOO6DmVm91F9xNZz1H7D1Kdj4SLKjMRnOEsQY0zcGIoUSxLhCL8eOL+TPb+4d3gk2/hH8nbDw6vgGlqqWfhrKjoFX7kp2JCbDWYIYY3YejHRxTZ02CBHhstpq1u1qYcv+9qGfYN1voWw6TDkp/sGlIpcbFn8cdr4IDVuSHY3JYJYgxpgnNuwnJ8vFcRPjMGo5ji5ZVIXHJTy4ZtfgO/d38F3Y8Q9Y+FGnjn6sWHg1uDzw2q+SHYnJYJYgxpDu3iB/fmMPFxw/kcKcrGSHc5jyAi9nHzeeVa/txh8MxX7gut8CMvamoCgY56y1/cbvIJCgJVvNmGcJYgx5fMM+2n0BPlKbmms0X7F0Mk2dvTz99oHYDgiFnDfI6cvTb93peFh8DXQ1wea/JDsSk6EsQYwhD67dRXVpLidNK092KFGdNrOC8UVeHlgdYzXTO49D6y5nSuyxaPr7oHgyvPRTZ6CgMXFmCWKMqG/u4sWtTVy2ZHLKjKA+ksft4uKFVTy3pYFO3yDTSag6k9eVTIXjjlzqfIxwueHML0P9q7DyavB3Jzsik2EsQYwRD63dDcCHlxy5LHhqOW1mBYGQsnr7wYF33Po07F4Lp30e3AmdtT61LboaLroN6p6E+y+zJGHiyhLEGOALBFm5eifLplekVPfWaGqnlpHlFl7aOsgSm8/fCkVVmbMw0EgsuQYuuRO2/68zDYcxcWIJYgx4aO1u9rb28Jkzjkl2KIPKzXazaEopLw6UILa/4IwBWPav4EmNGWmTbsEVsGQFvPQT2P1asqMxGcISRIbzB0Pc/kwdi6aUcOqMimSHE5NTppezfk8rrV3+6Ds8913Ir3QGi5lDzvkGFIyHP95gXV9NXFiCyHAPv7ab3S3d3Pi+mUiaDCRbNqMCVXj53SiliB0vwbvPwbKbICt1pgtJCTnFcOEP4MAG+PvXbDI/M2KWIDJYIBjiJ8/UMa+qmDOPrUx2ODFbUF1CbpY7ejvEc99xSg+1nxz9wNLB7AvghGvhlTtg1T9b91czIpYgMtgTG/ez82AXn3vfjLQpPQBke1wsnVbGi1sbD9+w82XY9qzT9pCd2o3tSXX+95wZX996EO671Jnt1phhsASRwVa9tptxhV7OOm58skMZslOml7NlfwcH2nucF1SdHjpWehicCJz2Bbj0505j/sqPgr8n2VGZNGQJIkM1d/by3JYDXLxwEu4UHRg3kFOmO6O9n93c4Lyw+hdO28Pp/xey85MYWRqZfzl86GfOz+3Ba2yMhBkySxAZ6i9v7cUfVC5emNoD445m7qRiZk8o5EdPvkPPztfh8a/AjHNg6T8nO7T0suBKuPC/Ycvf4NZZ8Mj1sPeNZEdl0oQliAz1x3W7mTGugLmTipIdyrC4XcJ/fnAu7S2NdP/2Y5BX4QwGc9mv7JAt/TSs+AvM/oCzuNI9F9g6EiYm9teWgXYd7GL19mYuWVSVVo3TRzqx6CBPFP0XBd172H/O7ZCfmpMMpoWaU+GSn8H1L4MnB35/NfiGsTiTGVMsQWSgP65z5l364IJJSY5kBOqegp+/j3HuTj6t/85XXitErV//yBVXw2X3QFMdPPJZCB5lMKIxWILIOL2BECtX72JpTSmTy9K0K+j+jfD7j0HJFFyfeZZTz/4QT719gMc37Et2ZJlh2unOqOtNf4KfLIU3H4BQMNlRmRRkCSLDrFy9k/rmbq5fPiPZoQxPd4tT/eEtgKv/ACVT+MSyGuZMLOI//riBth77xBsXJ98AH30AsgucAXV3nu7MCGulNNOPJYgM0tUb4Lan6jihpowzZqXPyOk+vZ3Om1XLTrj811A0EXDWifjOh+fR2OHj+3/bnOQgM4QIzDoXPvM8fORu6O2A33wY7vuQ9XIyfSxBZJB7X9xOY4ePL553bHo1TqvCW39wqjveeQLO/y5MOemwXeZXl7DilGn85pUd3PncVjoGW1DIxMblguM/DNevhvO+A3vfhDvPgFXXwsZH4eA2Z2lXMyZJJjX81dbW6po1a5IdRkKoKrtbuinKzaIoJ+uwbYFgiJe2NXH9/a9RW1PG3SuWJinKIQqFYPNjzspwe9fBhPnONBFTT466e6cvwLX3reGFuiaKcjxcfdJUPnbSVCaV2KR9cdPdAi/8D7z8MwiER1+XTIUTPwOLPgY56dlt2hydiKxV1dqo2yxBpLYX6hr52bNbebO+hbaeACV5Wfz4qkWcNrOSDl+AHz/9Dg+uqedgZy/FuVn8/jMnMXtCiv8Rdx10GkbX3gsNm6C0xhkhveAqZxnNQby+s5m7nt/G4xv2ISKcO3c8p82s5PhJxYwv9tLdG8QXCDGlLI+crKOfT1XZ19ZDe0+AHn+Q3c3dbNrXzruNnQTDn5rL870cN7GIaRX5tHb7aWjvYUJxLqfNrOg7d48/iNslZLkzqEDe2+X83+x9w/m/2vkSZBfC0k/ByddDwbhkR2jixBJEAjV1+CjNy47rOs+BYIgNe9r46bN1PL5hP1UluZx5bCXHTijk/pd38s6Bdj564hSe2LCfhg4fFxw/kQ8unMQZsyoHfENMuvb98I8fwtp7nE+nkxbBidfB8R8Z1rKhuw528ZuXd/DAml00R1k7wuMSZo0vpKLQS3NnLx2+ABOKcphWmU+nL8DL25rY33b4bKcugclleWS7XSiwv7WH9ijVWQVeD7U1pdQ3d7OtoYMst4v51cXMry5hYnEOFQVeJpXkUlORR2WBN72q/KLZ/Rq8+GPY8LCzSFP5DHBnOY3cRVVQXAXj5zr/p6XTnDYOkxaSliBE5DzgR4Ab+IWqfueI7RLefgHQBaxQ1ddiOTaa0U4Qf1hbz80PvcnJ08v5yVWLKc7LGvygo+jqDfD4hn08um4Pq7c30+ELkJft5vrlM/jUqdP63vg7fQG++NCb/OXNvcyvLuYbFx/Pwskl8bql+FGFpq2w6xVoesepy37n78700wuudBLDxPlxuVQopOxq7mL97jaaOn3kZ3vwuIV39nfwRrjkVZqXRb7Xw96WbrY1dpLtdnHiMeUsrSmlLD+b3Cw3lYVeZo0vPCzJqir1zd1sb+qkNC+bykIvm/e189hbe1mzo5ma8nzmTCykwxdk3a5mNuxpwxc4vM7e4xIUCIaUioJsZo4rZFJJLl29Adp6/JTle5k5roBjJxSyeEoplYUpvEpeYx28eie07XH+L31t0LYX2vdAKJxIPTlQMgXKjoGqWqiuBQ1Cyy7nmKJJTkIpnQZ5Zcm9H5OcBCEibmALcA5QD6wGrlLVjf32uQD4HE6COBH4kaqeGMux0SQyQYRCSntPAJcL8rI93PHcVr7/+GbmTipiy/52Jpfm8eOPLqK6JA9vlguvx/WeT43BkNLe42dHUxev72xm/Z42Wrp6aesJsHFPGx2+ANWlTmnhhGnlLJteTnnBe98sVJWNe9uYPaFo9CbiC4WcT/2BHhAXuDzOG4KvzakyOrjVSQgH33V6ITW8DV3h6bpdHucNY/JJcPq/Qfn00Yk5CVSVtp4ADe0+6pu72NHUxb62HtwiuAT2tfXwzoEO9rX2UOD1UJDjobHDx66DhybSqynPY2p5PmX52RR4PYRUCQQVXyBId7g6a1pFPsdUFFCSl4XX40YEOnwBOn0BgiHnbzrb46KywEt5gRevx4VLhJAqvcEQPn8Ij1vIyXKT73VTlpeNZyRVZIFep0pq92vOILyWHc50Ho2D9DrLKYHSqU4pJK8MggEI+Z0BfePmOFOs9LY7iSW/AgomOAko5Hde6+1wqsPcWc4kjtn5TqkmO9+prhRXvy83uLMPTdcSCjklnf5/p5GBgy7PmCkFDZQghl6uj90JQJ2qbgsHsRK4GOj/Jn8x8Gt1stTLIlIiIhOBmhiOjZvz/ud5egMhREBxBpv5AiFUnSqHQEhp6eoldEQu/dDCSXzvIwt4fWcz1/1mLRfe9o++bdluF0W5WXg9LucPuzdIZ+/hg5EqC72ML/JS4PXwgfkTuWRRFUtrygatrhIR5k4qjtftH27fevjdVeDvdGb/DPaGB1HF+EGiqMpJBrPOhcknwJSToWz6sKqQ0pGIUJybRXFuFjPGFcR8XFdvgE1721m74yBrtjezr62HrQ0ddPgCeFyC2+W8med43PiDIZ7YsJ/Akb+QI1Scm4UIBINKUJWQ6mG/8wJ9sXjcrr4PJ/5giEBQw8mmBo9rGoGQEgwphTkdHBuqo1dyaHJXEnB5KQ81Mi7UwKTQPqqDexjf0Ejl/g0UaTt+PCguKrWRLBLUU03coCFAnUSQUwxZeU4DfW9k+hFxElFWrrMNdY4RV7idTJwPSMFe53ye7PB5g07pORRwvlwep0rO7XUSjobPE7n+e4M7lNA0eOia4nIeB3qd87qznGQn4ryeWwqffjLuP6pE/tVWAbv6Pa/HKSUMtk9VjMcCICLXAtcCTJkyZViBHl9VjC8QIhQuTXk9h0oAqopLhLL8bIpzswip0tUbZEJRDpfXTsblEk48ppy/3Hgaz2w+gM8fotsfpL0nQGt3Lz5/iJxs5w+7KNdDUU4WE4pzWDSlhInFKdj7JqcIapY5fxTZeeFf7PAvaFaO8xycX1JxOfvnlDjVCeXTbRnQYcrL9rBkailLppZy7emD7+8Phth1sIv2nkDf726B10OB19P3xu0LBGlo76Wp04c/GOrrrZqT5SbLLQRDii8Qor3HT1NnL82dzjrWbpcLlzgTJiIgOOdTdd70I2/+ToJSst0u3C4XPYFgXwnG4xJcIiBlwBS8CuNCSiikiFTTKkIrsEE16hQqLg0w3r+HPO3E58rFj4f8QAvFgUbcGkRdbnBl0evOp1u8SNBPVqgbb6ibnFA3OdqDS5Rjx+Vx/KQi5000FHDeYIO9zpu8KwuCPuhudj4M5ZY6XyLOfoEe8Hc5a2kIOP+o84FJQ86btCvLeSMP+sN/E+FSi9vjPA4FnJJOsF9bV6Q0E62EotovMYTPFbmmK1ICcjslrcg5xeUkuQRIZIKI9jH4yN+Eo+0Ty7HOi6p3AXeBU8U0lAAjbr1swXAOO8ykklyuPnHqiM+TdCVT4JI7kh2FGUSW28UxlYOXUGZYZyMzAolMEPXA5H7Pq4E9Me6THcOxxhhjEiiRHbdXAzNFZJqIZANXAo8esc+jwMfFcRLQqqp7YzzWGGNMAiWsBKGqARG5AXgcp6vq3aq6QUSuC2+/A3gMpwdTHU43108MdGyiYjXGGPNeNlDOGGPGsIG6uWbQ3ADGGGPiyRKEMcaYqCxBGGOMicoShDHGmKgyqpFaRBqAHeGnFUBjEsNJtEy/P7B7zBSZfo/pfn9TVTXqEpQZlSD6E5E1R2uZzwSZfn9g95gpMv0eM/n+rIrJGGNMVJYgjDHGRJXJCeKuZAeQYJl+f2D3mCky/R4z9v4ytg3CGGPMyGRyCcIYY8wIWIIwxhgTVUYlCBH5voi8LSJvisjDIlLSb9uXRaRORDaLyLnJjHMkROQyEdkgIiERqT1iW6bc43nhe6gTkZuTHU+8iMjdInJARNb3e61MRP4uIu+Ev5cmM8aREJHJIvKMiGwK/47+a/j1TLrHHBF5VUTeCN/jf4Zfz5h77C+jEgTwd+B4VZ0PbAG+DCAic3DWlJgLnAf8VETcSYtyZNYDlwLP938xU+4xHPPtwPnAHOCq8L1lgntx/m/6uxl4SlVnAk+Fn6erAPAFVT0OOAm4Pvx/l0n36APep6oLgIXAeeG1bDLpHvtkVIJQ1SdUNbLS+cs4K9EBXAysVFWfqr6Ls/7ECcmIcaRUdZOqbo6yKVPu8QSgTlW3qWovsBLn3tKeqj4PHDzi5YuBX4Uf/wr40KgGFUequldVXws/bgc24awvn0n3qKraEX6aFf5SMuge+8uoBHGETwJ/DT+uAnb121Yffi2TZMo9Zsp9xGp8eBVFwt8zYhVpEakBFgGvkGH3KCJuEVkHHAD+rqoZd48RiVyTOiFE5ElgQpRNX1XVP4b3+SpOcff+yGFR9k/Z/r2x3GO0w6K8lrL3OIBMuY8xS0QKgIeAm1S1TSTaf2n6UtUgsDDcxvmwiByf7JgSJe0ShKqePdB2EbkG+ABwlh4a5FEPTO63WzWwJzERjtxg93gUaXWPA8iU+4jVfhGZqKp7RWQizqfStCUiWTjJ4X5VXRV+OaPuMUJVW0TkWZx2pYy8x4yqYhKR84AvAR9U1a5+mx4FrhQRr4hMA2YCryYjxgTKlHtcDcwUkWkiko3T8P5okmNKpEeBa8KPrwGOVkJMeeIUFX4JbFLVH/TblEn3WBnpHSkiucDZwNtk0D32l1EjqUWkDvACTeGXXlbV68LbvorTLhHAKfr+NfpZUpuIXAL8GKgEWoB1qnpueFum3OMFwP8AbuBuVf1WkkOKCxH5HXAmzvTQ+4GvA48ADwBTgJ3AZap6ZEN2WhCRU4H/Bd4CQuGXv4LTDpEp9zgfpxHajfMB+wFV/YaIlJMh99hfRiUIY4wx8ZNRVUzGGGPixxKEMcaYqCxBGGOMicoShDHGmKgsQRhjjInKEoTJCCLSEf5eIyLdIrJORDaKyB0i8p7f88j+w7jOdSLy8UH2WSEiPznKtq8M8Xonicgr4fvZJCK3DOV4Y0Yi7UZSGxODraq6UEQ8wNM4E6etGuSYmKjqHSM8xVeAbw9h/18Bl6vqG+GZbo8d4fUREXd4ughjBmQlCJOxwjP7vgjMONo+InKmiDwrIn8IryVyf3hEMCLynXAp5E0RuTX82i0i8m/hx0vD214SZy2S9f1OPUlE/hZeH+B7kfMBueHSwP0iki8ifwmvLbBeRK6IEuI4IDIJXFBVN4bPVSAi94jIW+EYPhx+/arwa+tF5Lv97rNDRL4hIq8AJ4vIP4XXNVgnInem49TwJvEsQZiMJSJ5wFk4I3sHsgi4CWf9iWOAZSJSBlwCzA2vL/LNKMfdA1ynqicDR34iXwhcAcwDrhCRyap6M9CtqgtV9WqcOXz2qOoCVT0e+FuUa/wQ2CzOAlifEZGc8OtfA1pVdV44vqdFZBLwXeB94esvFZHItNP5wHpVPRFnpoErgGWqujAc+9WD/IzMGGQJwmSi6eHpmF8A/hLDlCOvqmq9qoaAdUAN0Ab0AL8QkUuB/nN7EZ6Pp1BVXwy/9NsjzvmUqraqag+wEZga5bpvAWeLyHdF5DRVbT1yB1X9BlALPAF8lENJ5GychZUi+zUDS4FnVbUhXHq6Hzg9vEsQZxI9cJLmEmB1+Od0Fk5iNOYw1gZhMtHW8CfjWPn6PQ4CHlUNiMgJOG+eVwI34HwyjxhsDuv3nPPIHVR1i4gsAS4A/p+IPBFOCEfutxX4mYj8HGgIz/sjvHca9IFi6unX7iDAr1T1y4PcgxnjrARhTBThNQ2KVfUxnOqnwxJO+BN7uzjLTYKTRGLhD0+JTbhKqEtVfwPcCiyOEseFkTYRnBl6gziTND6Bk7Qi+5XiTIp3hohUhNsUrgKeixLDU8BHRGRc+NgyEYlWwjFjnJUgjImuEPhjuM5fgP8TZZ9PAT8XkU7gWeA9VURR3AW8KSKvAb8Gvi8iIcAPfDbK/h8DfigiXTiz9F6tqkER+SZwe7hhPAj8p6quEpEvA8+EY34s2gJTqrpRRP4deCLcBdgPXA/siCF+M4bYbK7GDJOIFETWJxaRm4GJqvqvSQ7LmLixEoQxw3dh+BO7B+fT94rkhmNMfFkJwhhjTFTWSG2MMSYqSxDGGGOisgRhjDEmKksQxhhjorIEYYwxJqr/D+2eLqJpcS32AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n, x = np.histogram(test_case_scores[:NUM_SAMPLES], bins=100, density=True)\n",
    "plt.plot(x[1:], n)\n",
    "\n",
    "n, x = np.histogram(test_case_scores[NUM_SAMPLES:], bins=100, density=True)\n",
    "plt.plot(x[1:], n)\n",
    "\n",
    "plt.legend([\"Normal\", \"Random IP\"])\n",
    "plt.xlabel(\"IP Insights Score\")\n",
    "plt.ylabel(\"Frequency\")\n",
    "\n",
    "plt.figure()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### [4] Selecting a Good Threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see in the figure above, there is a clear separation between normal traffic and random traffic. We could select a threshold depending on the application.\n",
    "\n",
    "- If we were working with low impact decisions, such as whether to ask for another factor or authentication during login, we could use a threshold = 0.0. This would result in catching more true-positives, at the cost of more false-positives.\n",
    "\n",
    "- If our decision system were more sensitive to false positives, we could choose a larger threshold, such as threshold = 10.0. That way if we were sending the flagged cases to manual investigation, we would have a higher confidence that the acitivty was suspicious."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "When threshold is set to: 0.0\n",
      "Total of 103009 flagged cases\n",
      "Total of 98256 flagged cases are true positives\n",
      "True Positive Rate: 0.953858400722267\n",
      "Recall: 0.98256\n",
      "Precision: 0.953858400722267\n"
     ]
    }
   ],
   "source": [
    "threshold = 0.0\n",
    "\n",
    "flagged_cases = test_case[np.array(test_case_scores) > threshold]\n",
    "\n",
    "num_flagged_cases = len(flagged_cases)\n",
    "num_true_positives = len(flagged_cases[flagged_cases['label'] == 1])\n",
    "num_false_positives = len(flagged_cases[flagged_cases['label'] == 0])\n",
    "num_all_positives = len(test_case.loc[test_case['label'] == 1])\n",
    "\n",
    "print(\"When threshold is set to: {}\".format(threshold))\n",
    "print(\"Total of {} flagged cases\".format(num_flagged_cases))\n",
    "print(\"Total of {} flagged cases are true positives\".format(num_true_positives))\n",
    "print(\"True Positive Rate: {}\".format(num_true_positives/float(num_flagged_cases)))\n",
    "print(\"Recall: {}\".format(num_true_positives/float(num_all_positives)))\n",
    "print(\"Precision: {}\".format(num_true_positives/float(num_flagged_cases)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (optional) 10. Compute Anomaly Scores based on Test set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SageMaker algorithms come with support for two additional services that make optimizing and using the algorithm that much easier: **Automatic Model Tuning** and **Batch Transform**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (1) Automatic Model Tuning\n",
    "The results above were based on using the default hyperparameters of the SageMaker IP Insights algorithm. If we wanted to improve the model's performance even more, we can use Amazon SageMaker Automatic Model Tuning to automate the process of finding the hyperparameters.\n",
    "\n",
    "**Validation Dataset**\n",
    "Previously, we separated our dataset into a training and test set to validate the performance of a single IP Insights model. However, when we do model tuning, we train many IP Insights models in parallel. If we were to use the same test dataset to select the best model, we bias our model selection such that we don't know if we selected the best model in general, or just the best model for that particular dateaset.\n",
    "\n",
    "Therefore, we need to separate our test set into a validation dataset and a test dataset. The validation dataset is used for model selection. Then once we pick the model with the best performance, we evaluate it the winner on a test set just as before.\n",
    "\n",
    "**Validation Metrics**\n",
    "For SageMaker Automatic Model Tuning to work, we need an objective metric which determines the performance of the model we want to optimize. Because SageMaker IP Insights is an usupervised algorithm, we do not have a clearly defined metric for performance (such as percentage of fraudulent events discovered).\n",
    "\n",
    "We allow the user to provide a validation set of sample data (same format as training data bove) through the validation channel. We then fix the negative sampling strategy to use random_negative_sampling_rate=1 and shuffled_negative_sampling_rate=0 and generate a validation dataset by assigning corresponding labels to the real and simulated data. We then calculate the model's descriminator_auc metric. We do this by taking the model's predicted labels and the 'true' simulated labels and compute the Area Under ROC Curve (AUC) on the model's performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count                  950187\n",
       "unique                 252567\n",
       "top       2018-11-13 05:50:21\n",
       "freq                       16\n",
       "first     2018-11-11 00:00:01\n",
       "last      2018-11-14 00:00:00\n",
       "Name: timestamp, dtype: object"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df['timestamp'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The test set we constructed above spans 3 days. We reserve the first day as the validation set and the subsequent two days for the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "time_partition = datetime(2018, 11, 13, tzinfo=pytz.FixedOffset(0)) if num_time_zones > 1 else datetime(2018, 11, 13)\n",
    "\n",
    "validation_df = test_df[test_df['timestamp'] < time_partition]\n",
    "test_df = test_df[test_df['timestamp'] >= time_partition]\n",
    "\n",
    "valid_data = validation_df.to_csv(index=False, header=False, columns=['user', 'ip_address'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation data has been uploaded to: s3://sagemaker-us-west-2-570447867175/ip-insights\\validation\\valid.csv\n"
     ]
    }
   ],
   "source": [
    "# Upload validation to S3\n",
    "validation_data_file = 'valid.csv'\n",
    "key = os.path.join(prefix, 'validation', validation_data_file)\n",
    "boto3.resource('s3').Bucket(bucket).Object(key).put(Body=valid_data)\n",
    "s3_valid_data = 's3://{}/{}'.format(bucket, key)\n",
    "\n",
    "print('Validation data has been uploaded to: {}'.format(s3_valid_data))\n",
    "\n",
    "# Configure SageMaker IP Insights Input Channels\n",
    "input_data = {\n",
    "    'train': s3_train_data,\n",
    "    'validation': s3_valid_data\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker:'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n",
      "WARNING:sagemaker:'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.tuner import HyperparameterTuner, IntegerParameter\n",
    "\n",
    "# Configure HyperparameterTuner\n",
    "ip_insights_tuner = HyperparameterTuner(\n",
    "    estimator=ip_insights,  # previously-configured Estimator object #\n",
    "    objective_metric_name='validation:discriminator_auc',\n",
    "    hyperparameter_ranges={'vector_dim': IntegerParameter(64, 1024)},\n",
    "    max_jobs=2,\n",
    "    max_parallel_jobs=4)\n",
    "\n",
    "# Start hyperparameter tuning job\n",
    "ip_insights_tuner.fit(input_data, include_cls_metadata=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "..............................................................................................................................................................................................................................................!\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>vector_dim</th>\n",
       "      <th>TrainingJobName</th>\n",
       "      <th>TrainingJobStatus</th>\n",
       "      <th>FinalObjectiveValue</th>\n",
       "      <th>TrainingStartTime</th>\n",
       "      <th>TrainingEndTime</th>\n",
       "      <th>TrainingElapsedTimeSeconds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>398.0</td>\n",
       "      <td>ipinsights-200712-1633-001-880df896</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.990466</td>\n",
       "      <td>2020-07-12 16:35:45-07:00</td>\n",
       "      <td>2020-07-12 16:52:47-07:00</td>\n",
       "      <td>1022.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   vector_dim                      TrainingJobName TrainingJobStatus  \\\n",
       "0       398.0  ipinsights-200712-1633-001-880df896         Completed   \n",
       "\n",
       "   FinalObjectiveValue         TrainingStartTime           TrainingEndTime  \\\n",
       "0             0.990466 2020-07-12 16:35:45-07:00 2020-07-12 16:52:47-07:00   \n",
       "\n",
       "   TrainingElapsedTimeSeconds  \n",
       "0                      1022.0  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Wait for all the jobs to finish\n",
    "ip_insights_tuner.wait()\n",
    "\n",
    "# Visualize training job results\n",
    "ip_insights_tuner.analytics().dataframe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-07-12 23:52:47 Starting - Preparing the instances for training\n",
      "2020-07-12 23:52:47 Downloading - Downloading input data\n",
      "2020-07-12 23:52:47 Training - Training image download completed. Training in progress.\n",
      "2020-07-12 23:52:47 Uploading - Uploading generated training model\n",
      "2020-07-12 23:52:47 Completed - Training job completed"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker:Parameter image will be renamed to image_uri in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mDocker entrypoint called with argument(s): train\u001b[0m\n",
      "\u001b[34mRunning default environment configuration script\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:10 INFO 140204685006656] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/resources/default-input.json: {u'random_negative_sampling_rate': u'1', u'shuffled_negative_sampling_rate': u'1', u'batch_metrics_publish_interval': u'1000', u'num_entity_vectors': u'100000', u'_tuning_objective_metric': u'', u'vector_dim': u'128', u'learning_rate': u'0.001', u'epochs': u'10', u'_num_gpus': u'auto', u'weight_decay': u'0.00001', u'_kvstore': u'auto_gpu', u'_num_kv_servers': u'auto', u'mini_batch_size': u'5000', u'_log_level': u'info', u'num_ip_encoder_layers': u'1'}\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:10 INFO 140204685006656] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'random_negative_sampling_rate': u'5', u'num_entity_vectors': u'20000', u'_tuning_objective_metric': u'validation:discriminator_auc', u'vector_dim': u'398', u'learning_rate': u'0.01', u'epochs': u'5', u'mini_batch_size': u'1000'}\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:10 INFO 140204685006656] Final configuration: {u'random_negative_sampling_rate': u'5', u'num_entity_vectors': u'20000', u'_tuning_objective_metric': u'validation:discriminator_auc', u'learning_rate': u'0.01', u'epochs': u'5', u'_num_kv_servers': u'auto', u'mini_batch_size': u'1000', u'weight_decay': u'0.00001', u'num_ip_encoder_layers': u'1', u'shuffled_negative_sampling_rate': u'1', u'batch_metrics_publish_interval': u'1000', u'vector_dim': u'398', u'_log_level': u'info', u'_num_gpus': u'auto', u'_kvstore': u'auto_gpu'}\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:10 WARNING 140204685006656] Loggers have already been setup.\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:10 INFO 140204685006656] nvidia-smi took: 0.100603103638 secs to identify 1 gpus\u001b[0m\n",
      "\u001b[34mProcess 1 is a worker.\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:10 INFO 140204685006656] Using default worker.\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:10 INFO 140204685006656] Loaded iterator creator application/x-ndarray for content type ('application/x-ndarray', '1.0')\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:10 INFO 140204685006656] Loaded iterator creator text/csv for content type ('text/csv', '1.0')\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:11 INFO 140204685006656] Number of GPUs being used: 1\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"initialize.time\": {\"count\": 1, \"max\": 17.745018005371094, \"sum\": 17.745018005371094, \"min\": 17.745018005371094}}, \"EndTime\": 1594597031.041835, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594597031.023895}\n",
      "\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Records Seen\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Reset Count\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}}, \"EndTime\": 1594597031.042057, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"init_train_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594597031.041999}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:11 INFO 140204685006656] Create Store: device\u001b[0m\n",
      "\u001b[34m[23:37:15] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.3.x_Cuda_10.1.x.888.0/AL2012/generic-flavor/src/src/kvstore/./kvstore_local.h:306: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use kv.row_sparse_pull() or module.prepare() with row_ids.\u001b[0m\n",
      "\u001b[34m[23:37:15] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.3.x_Cuda_10.1.x.888.0/AL2012/generic-flavor/src/src/operator/././../common/utils.h:450: Optimizer with lazy_update = True detected. Be aware that lazy update with row_sparse gradient is different from standard update, and may lead to different empirical results. See https://mxnet.incubator.apache.org/api/python/optimization/optimization.html for more details.\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:15 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=0 train binary_classification_accuracy <score>=0.524\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:15 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=0 train binary_classification_cross_entropy <loss>=0.692921630859\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:27 INFO 140204685006656] Epoch[0] Batch [1000]#011Speed: 87065.26 samples/sec#011binary_classification_accuracy=0.937102#011binary_classification_cross_entropy=0.178675\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:27 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=1000 train binary_classification_accuracy <score>=0.937101898102\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:27 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=1000 train binary_classification_cross_entropy <loss>=0.178675059636\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:38 INFO 140204685006656] Epoch[0] Batch [2000]#011Speed: 89587.64 samples/sec#011binary_classification_accuracy=0.954662#011binary_classification_cross_entropy=0.137988\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:38 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=2000 train binary_classification_accuracy <score>=0.954662168916\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:38 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=2000 train binary_classification_cross_entropy <loss>=0.137987602794\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:49 INFO 140204685006656] Epoch[0] Batch [3000]#011Speed: 89610.96 samples/sec#011binary_classification_accuracy=0.962076#011binary_classification_cross_entropy=0.120203\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:49 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=3000 train binary_classification_accuracy <score>=0.962075641453\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:37:49 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=3000 train binary_classification_cross_entropy <loss>=0.120203088268\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:00 INFO 140204685006656] Epoch[0] Batch [4000]#011Speed: 89504.35 samples/sec#011binary_classification_accuracy=0.966318#011binary_classification_cross_entropy=0.109892\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:00 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=4000 train binary_classification_accuracy <score>=0.966317670582\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:00 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=4000 train binary_classification_cross_entropy <loss>=0.109892193395\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:12 INFO 140204685006656] Epoch[0] Batch [5000]#011Speed: 89457.43 samples/sec#011binary_classification_accuracy=0.969020#011binary_classification_cross_entropy=0.103333\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:12 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=5000 train binary_classification_accuracy <score>=0.969020195961\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:12 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=5000 train binary_classification_cross_entropy <loss>=0.103333283692\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:23 INFO 140204685006656] Epoch[0] Batch [6000]#011Speed: 89579.22 samples/sec#011binary_classification_accuracy=0.970987#011binary_classification_cross_entropy=0.098408\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:23 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=6000 train binary_classification_accuracy <score>=0.970986668889\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:23 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=6000 train binary_classification_cross_entropy <loss>=0.0984082481319\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:34 INFO 140204685006656] Epoch[0] Batch [7000]#011Speed: 89662.30 samples/sec#011binary_classification_accuracy=0.972406#011binary_classification_cross_entropy=0.094927\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:34 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=7000 train binary_classification_accuracy <score>=0.972406227682\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:34 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=7000 train binary_classification_cross_entropy <loss>=0.0949274612994\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:45 INFO 140204685006656] Epoch[0] Batch [8000]#011Speed: 89682.23 samples/sec#011binary_classification_accuracy=0.973531#011binary_classification_cross_entropy=0.092037\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:45 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=8000 train binary_classification_accuracy <score>=0.973531183602\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:45 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=8000 train binary_classification_cross_entropy <loss>=0.092037104827\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:56 INFO 140204685006656] Epoch[0] Batch [9000]#011Speed: 89904.44 samples/sec#011binary_classification_accuracy=0.974473#011binary_classification_cross_entropy=0.089673\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:56 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=9000 train binary_classification_accuracy <score>=0.97447294745\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:38:56 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=9000 train binary_classification_cross_entropy <loss>=0.0896729038748\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:39:07 INFO 140204685006656] Epoch[0] Batch [10000]#011Speed: 89786.77 samples/sec#011binary_classification_accuracy=0.975260#011binary_classification_cross_entropy=0.087762\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:39:07 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=10000 train binary_classification_accuracy <score>=0.975259774023\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:39:07 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=10000 train binary_classification_cross_entropy <loss>=0.0877622922288\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:39:18 INFO 140204685006656] Epoch[0] Batch [11000]#011Speed: 89733.33 samples/sec#011binary_classification_accuracy=0.975891#011binary_classification_cross_entropy=0.086196\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:39:18 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=11000 train binary_classification_accuracy <score>=0.975890555404\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:39:18 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=11000 train binary_classification_cross_entropy <loss>=0.0861963293533\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:39:30 INFO 140204685006656] Epoch[0] Batch [12000]#011Speed: 89376.05 samples/sec#011binary_classification_accuracy=0.976439#011binary_classification_cross_entropy=0.084824\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:39:30 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=12000 train binary_classification_accuracy <score>=0.976439463378\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:39:30 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=12000 train binary_classification_cross_entropy <loss>=0.0848235451126\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:39:41 INFO 140204685006656] Epoch[0] Batch [13000]#011Speed: 89639.11 samples/sec#011binary_classification_accuracy=0.976905#011binary_classification_cross_entropy=0.083683\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:39:41 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=13000 train binary_classification_accuracy <score>=0.976904853473\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:39:41 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=13000 train binary_classification_cross_entropy <loss>=0.0836834286003\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:39:52 INFO 140204685006656] Epoch[0] Batch [14000]#011Speed: 89392.16 samples/sec#011binary_classification_accuracy=0.977341#011binary_classification_cross_entropy=0.082605\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:39:52 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=14000 train binary_classification_accuracy <score>=0.977340689951\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:39:52 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=14000 train binary_classification_cross_entropy <loss>=0.0826051371042\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:03 INFO 140204685006656] Epoch[0] Batch [15000]#011Speed: 89613.27 samples/sec#011binary_classification_accuracy=0.977706#011binary_classification_cross_entropy=0.081683\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:03 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=15000 train binary_classification_accuracy <score>=0.977705552963\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:03 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, batch=15000 train binary_classification_cross_entropy <loss>=0.081682705877\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:09 INFO 140204685006656] Epoch[0] Train-binary_classification_accuracy=0.977907\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:09 INFO 140204685006656] Epoch[0] Train-binary_classification_cross_entropy=0.081171\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:09 INFO 140204685006656] Epoch[0] Time cost=173.765\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:09 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, train binary_classification_accuracy <score>=0.977906737086\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:09 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, train binary_classification_cross_entropy <loss>=0.0811705356949\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:17 INFO 140204685006656] Num unique classes of label: [0. 1.]\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:18 INFO 140204685006656] #quality_metric: host=algo-1, epoch=0, validation discriminator_auc <score>=0.989875082849\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"epochs\": {\"count\": 1, \"max\": 5, \"sum\": 5.0, \"min\": 5}, \"update.time\": {\"count\": 1, \"max\": 187228.76811027527, \"sum\": 187228.76811027527, \"min\": 187228.76811027527}}, \"EndTime\": 1594597218.271038, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594597031.04192}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:18 INFO 140204685006656] #progress_metric: host=algo-1, completed 20 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 15526, \"sum\": 15526.0, \"min\": 15526}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 15526, \"sum\": 15526.0, \"min\": 15526}, \"Total Records Seen\": {\"count\": 1, \"max\": 15520099, \"sum\": 15520099.0, \"min\": 15520099}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 15520099, \"sum\": 15520099.0, \"min\": 15520099}, \"Reset Count\": {\"count\": 1, \"max\": 2, \"sum\": 2.0, \"min\": 2}}, \"EndTime\": 1594597218.271461, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\", \"epoch\": 0}, \"StartTime\": 1594597031.042226}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:18 INFO 140204685006656] #throughput_metric: host=algo-1, train throughput=82893.5096197 records/second\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:18 WARNING 140204685006656] Already bound, ignoring bind()\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python2.7/site-packages/mxnet/module/base_module.py:502: UserWarning: Parameters already initialized and force_init=False. init_params call ignored.\n",
      "  allow_missing=allow_missing, force_init=force_init)\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:18 WARNING 140204685006656] optimizer already initialized, ignoring...\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:18 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=0 train binary_classification_accuracy <score>=0.989\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:18 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=0 train binary_classification_cross_entropy <loss>=0.051865650177\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:29 INFO 140204685006656] Epoch[1] Batch [1000]#011Speed: 89180.64 samples/sec#011binary_classification_accuracy=0.982936#011binary_classification_cross_entropy=0.067308\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:29 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=1000 train binary_classification_accuracy <score>=0.982936063936\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:29 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=1000 train binary_classification_cross_entropy <loss>=0.0673077442672\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:40 INFO 140204685006656] Epoch[1] Batch [2000]#011Speed: 89531.01 samples/sec#011binary_classification_accuracy=0.983003#011binary_classification_cross_entropy=0.066781\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:40 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=2000 train binary_classification_accuracy <score>=0.983003498251\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:40 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=2000 train binary_classification_cross_entropy <loss>=0.0667813908426\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:51 INFO 140204685006656] Epoch[1] Batch [3000]#011Speed: 89225.66 samples/sec#011binary_classification_accuracy=0.983061#011binary_classification_cross_entropy=0.066386\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:51 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=3000 train binary_classification_accuracy <score>=0.983060646451\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:40:51 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=3000 train binary_classification_cross_entropy <loss>=0.0663856097828\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:03 INFO 140204685006656] Epoch[1] Batch [4000]#011Speed: 89609.43 samples/sec#011binary_classification_accuracy=0.983114#011binary_classification_cross_entropy=0.066099\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:03 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=4000 train binary_classification_accuracy <score>=0.983113971507\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:03 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=4000 train binary_classification_cross_entropy <loss>=0.0660989590373\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:14 INFO 140204685006656] Epoch[1] Batch [5000]#011Speed: 89671.61 samples/sec#011binary_classification_accuracy=0.983118#011binary_classification_cross_entropy=0.066133\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:14 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=5000 train binary_classification_accuracy <score>=0.983118376325\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:14 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=5000 train binary_classification_cross_entropy <loss>=0.0661329942046\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:25 INFO 140204685006656] Epoch[1] Batch [6000]#011Speed: 89831.61 samples/sec#011binary_classification_accuracy=0.983208#011binary_classification_cross_entropy=0.065829\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:25 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=6000 train binary_classification_accuracy <score>=0.983208465256\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:25 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=6000 train binary_classification_cross_entropy <loss>=0.0658288714506\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:36 INFO 140204685006656] Epoch[1] Batch [7000]#011Speed: 90097.51 samples/sec#011binary_classification_accuracy=0.983185#011binary_classification_cross_entropy=0.065899\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:36 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=7000 train binary_classification_accuracy <score>=0.983184973575\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:36 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=7000 train binary_classification_cross_entropy <loss>=0.0658994205517\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:47 INFO 140204685006656] Epoch[1] Batch [8000]#011Speed: 89964.45 samples/sec#011binary_classification_accuracy=0.983220#011binary_classification_cross_entropy=0.065729\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:47 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=8000 train binary_classification_accuracy <score>=0.983219722535\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:47 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=8000 train binary_classification_cross_entropy <loss>=0.0657289164471\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:58 INFO 140204685006656] Epoch[1] Batch [9000]#011Speed: 89965.55 samples/sec#011binary_classification_accuracy=0.983263#011binary_classification_cross_entropy=0.065558\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:58 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=9000 train binary_classification_accuracy <score>=0.983262637485\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:41:58 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=9000 train binary_classification_cross_entropy <loss>=0.0655581904311\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:42:09 INFO 140204685006656] Epoch[1] Batch [10000]#011Speed: 89726.85 samples/sec#011binary_classification_accuracy=0.983283#011binary_classification_cross_entropy=0.065480\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:42:09 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=10000 train binary_classification_accuracy <score>=0.983283271673\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:42:09 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=10000 train binary_classification_cross_entropy <loss>=0.0654804258433\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:42:21 INFO 140204685006656] Epoch[1] Batch [11000]#011Speed: 89311.10 samples/sec#011binary_classification_accuracy=0.983318#011binary_classification_cross_entropy=0.065426\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:42:21 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=11000 train binary_classification_accuracy <score>=0.983317880193\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:42:21 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=11000 train binary_classification_cross_entropy <loss>=0.065425674823\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:42:32 INFO 140204685006656] Epoch[1] Batch [12000]#011Speed: 89473.61 samples/sec#011binary_classification_accuracy=0.983337#011binary_classification_cross_entropy=0.065338\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:42:32 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=12000 train binary_classification_accuracy <score>=0.983336971919\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:42:32 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=12000 train binary_classification_cross_entropy <loss>=0.0653378189836\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:42:43 INFO 140204685006656] Epoch[1] Batch [13000]#011Speed: 89394.72 samples/sec#011binary_classification_accuracy=0.983373#011binary_classification_cross_entropy=0.065280\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:42:43 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=13000 train binary_classification_accuracy <score>=0.983372740558\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:42:43 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=13000 train binary_classification_cross_entropy <loss>=0.0652799198497\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:42:54 INFO 140204685006656] Epoch[1] Batch [14000]#011Speed: 89384.49 samples/sec#011binary_classification_accuracy=0.983400#011binary_classification_cross_entropy=0.065188\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:42:54 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=14000 train binary_classification_accuracy <score>=0.983399828584\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:42:54 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=14000 train binary_classification_cross_entropy <loss>=0.0651883118802\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:05 INFO 140204685006656] Epoch[1] Batch [15000]#011Speed: 89680.84 samples/sec#011binary_classification_accuracy=0.983423#011binary_classification_cross_entropy=0.065130\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:05 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=15000 train binary_classification_accuracy <score>=0.983423438437\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:05 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, batch=15000 train binary_classification_cross_entropy <loss>=0.065129519537\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:11 INFO 140204685006656] Epoch[1] Train-binary_classification_accuracy=0.983452\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:11 INFO 140204685006656] Epoch[1] Train-binary_classification_cross_entropy=0.065029\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:11 INFO 140204685006656] Epoch[1] Time cost=173.296\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:11 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, train binary_classification_accuracy <score>=0.983451500708\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:11 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, train binary_classification_cross_entropy <loss>=0.0650288106171\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:19 INFO 140204685006656] Num unique classes of label: [0. 1.]\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:20 INFO 140204685006656] #quality_metric: host=algo-1, epoch=1, validation discriminator_auc <score>=0.989926904459\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 182057.461977005, \"sum\": 182057.461977005, \"min\": 182057.461977005}}, \"EndTime\": 1594597400.329267, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594597218.271151}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:20 INFO 140204685006656] #progress_metric: host=algo-1, completed 40 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 15526, \"sum\": 15526.0, \"min\": 15526}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 31052, \"sum\": 31052.0, \"min\": 31052}, \"Total Records Seen\": {\"count\": 1, \"max\": 31040198, \"sum\": 31040198.0, \"min\": 31040198}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 15520099, \"sum\": 15520099.0, \"min\": 15520099}, \"Reset Count\": {\"count\": 1, \"max\": 4, \"sum\": 4.0, \"min\": 4}}, \"EndTime\": 1594597400.329649, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\", \"epoch\": 1}, \"StartTime\": 1594597218.271759}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:20 INFO 140204685006656] #throughput_metric: host=algo-1, train throughput=85248.0924995 records/second\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:20 WARNING 140204685006656] Already bound, ignoring bind()\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:20 WARNING 140204685006656] optimizer already initialized, ignoring...\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:20 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=0 train binary_classification_accuracy <score>=0.973\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:20 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=0 train binary_classification_cross_entropy <loss>=0.0755340881348\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:31 INFO 140204685006656] Epoch[2] Batch [1000]#011Speed: 89100.01 samples/sec#011binary_classification_accuracy=0.983826#011binary_classification_cross_entropy=0.063247\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:31 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=1000 train binary_classification_accuracy <score>=0.983826173826\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:31 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=1000 train binary_classification_cross_entropy <loss>=0.0632472056599\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:42 INFO 140204685006656] Epoch[2] Batch [2000]#011Speed: 89655.60 samples/sec#011binary_classification_accuracy=0.983751#011binary_classification_cross_entropy=0.063458\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:42 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=2000 train binary_classification_accuracy <score>=0.983750624688\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:42 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=2000 train binary_classification_cross_entropy <loss>=0.0634576681243\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:53 INFO 140204685006656] Epoch[2] Batch [3000]#011Speed: 89396.41 samples/sec#011binary_classification_accuracy=0.983778#011binary_classification_cross_entropy=0.063284\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:53 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=3000 train binary_classification_accuracy <score>=0.983778073975\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:43:53 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=3000 train binary_classification_cross_entropy <loss>=0.0632840686234\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:44:05 INFO 140204685006656] Epoch[2] Batch [4000]#011Speed: 89441.33 samples/sec#011binary_classification_accuracy=0.983863#011binary_classification_cross_entropy=0.062969\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:44:05 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=4000 train binary_classification_accuracy <score>=0.983862784304\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:44:05 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=4000 train binary_classification_cross_entropy <loss>=0.0629691904602\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:44:16 INFO 140204685006656] Epoch[2] Batch [5000]#011Speed: 89494.30 samples/sec#011binary_classification_accuracy=0.983853#011binary_classification_cross_entropy=0.063021\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:44:16 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=5000 train binary_classification_accuracy <score>=0.983852829434\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:44:16 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=5000 train binary_classification_cross_entropy <loss>=0.0630205334807\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:44:27 INFO 140204685006656] Epoch[2] Batch [6000]#011Speed: 89251.78 samples/sec#011binary_classification_accuracy=0.983886#011binary_classification_cross_entropy=0.062913\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:44:27 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=6000 train binary_classification_accuracy <score>=0.983885685719\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:44:27 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=6000 train binary_classification_cross_entropy <loss>=0.0629129410369\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:44:38 INFO 140204685006656] Epoch[2] Batch [7000]#011Speed: 89620.83 samples/sec#011binary_classification_accuracy=0.983849#011binary_classification_cross_entropy=0.063032\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:44:38 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=7000 train binary_classification_accuracy <score>=0.983848878732\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:44:38 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=7000 train binary_classification_cross_entropy <loss>=0.0630324800146\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:44:50 INFO 140204685006656] Epoch[2] Batch [8000]#011Speed: 87726.62 samples/sec#011binary_classification_accuracy=0.983880#011binary_classification_cross_entropy=0.062900\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:44:50 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=8000 train binary_classification_accuracy <score>=0.983879515061\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:44:50 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=8000 train binary_classification_cross_entropy <loss>=0.0628999100475\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:01 INFO 140204685006656] Epoch[2] Batch [9000]#011Speed: 88399.11 samples/sec#011binary_classification_accuracy=0.983895#011binary_classification_cross_entropy=0.062840\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:01 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=9000 train binary_classification_accuracy <score>=0.983895233863\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:01 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=9000 train binary_classification_cross_entropy <loss>=0.0628399790923\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:12 INFO 140204685006656] Epoch[2] Batch [10000]#011Speed: 89465.71 samples/sec#011binary_classification_accuracy=0.983893#011binary_classification_cross_entropy=0.062823\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:12 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=10000 train binary_classification_accuracy <score>=0.983893010699\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:12 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=10000 train binary_classification_cross_entropy <loss>=0.0628225001158\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:23 INFO 140204685006656] Epoch[2] Batch [11000]#011Speed: 89485.48 samples/sec#011binary_classification_accuracy=0.983886#011binary_classification_cross_entropy=0.062870\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:23 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=11000 train binary_classification_accuracy <score>=0.983886192164\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:23 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=11000 train binary_classification_cross_entropy <loss>=0.0628695536704\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:34 INFO 140204685006656] Epoch[2] Batch [12000]#011Speed: 89706.33 samples/sec#011binary_classification_accuracy=0.983908#011binary_classification_cross_entropy=0.062804\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:34 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=12000 train binary_classification_accuracy <score>=0.98390767436\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:34 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=12000 train binary_classification_cross_entropy <loss>=0.0628035931348\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:45 INFO 140204685006656] Epoch[2] Batch [13000]#011Speed: 89713.13 samples/sec#011binary_classification_accuracy=0.983908#011binary_classification_cross_entropy=0.062805\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:45 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=13000 train binary_classification_accuracy <score>=0.983907853242\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:45 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=13000 train binary_classification_cross_entropy <loss>=0.0628051318421\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:57 INFO 140204685006656] Epoch[2] Batch [14000]#011Speed: 89492.89 samples/sec#011binary_classification_accuracy=0.983918#011binary_classification_cross_entropy=0.062737\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:57 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=14000 train binary_classification_accuracy <score>=0.983917791586\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:45:57 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=14000 train binary_classification_cross_entropy <loss>=0.0627367332664\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:08 INFO 140204685006656] Epoch[2] Batch [15000]#011Speed: 89572.82 samples/sec#011binary_classification_accuracy=0.983924#011binary_classification_cross_entropy=0.062719\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:08 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=15000 train binary_classification_accuracy <score>=0.983923671755\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:08 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, batch=15000 train binary_classification_cross_entropy <loss>=0.0627191561385\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:14 INFO 140204685006656] Epoch[2] Train-binary_classification_accuracy=0.983947\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:14 INFO 140204685006656] Epoch[2] Train-binary_classification_cross_entropy=0.062612\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:14 INFO 140204685006656] Epoch[2] Time cost=173.818\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:14 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, train binary_classification_accuracy <score>=0.983946541286\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:14 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, train binary_classification_cross_entropy <loss>=0.0626116222656\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:21 INFO 140204685006656] Num unique classes of label: [0. 1.]\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:22 INFO 140204685006656] #quality_metric: host=algo-1, epoch=2, validation discriminator_auc <score>=0.989929658619\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 182495.2690601349, \"sum\": 182495.2690601349, \"min\": 182495.2690601349}}, \"EndTime\": 1594597582.825243, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594597400.329369}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:22 INFO 140204685006656] #progress_metric: host=algo-1, completed 60 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 15526, \"sum\": 15526.0, \"min\": 15526}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 46578, \"sum\": 46578.0, \"min\": 46578}, \"Total Records Seen\": {\"count\": 1, \"max\": 46560297, \"sum\": 46560297.0, \"min\": 46560297}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 15520099, \"sum\": 15520099.0, \"min\": 15520099}, \"Reset Count\": {\"count\": 1, \"max\": 6, \"sum\": 6.0, \"min\": 6}}, \"EndTime\": 1594597582.825629, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\", \"epoch\": 2}, \"StartTime\": 1594597400.329926}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:22 INFO 140204685006656] #throughput_metric: host=algo-1, train throughput=85043.5712947 records/second\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:22 WARNING 140204685006656] Already bound, ignoring bind()\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:22 WARNING 140204685006656] optimizer already initialized, ignoring...\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:22 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=0 train binary_classification_accuracy <score>=0.982\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:22 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=0 train binary_classification_cross_entropy <loss>=0.0742547607422\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:33 INFO 140204685006656] Epoch[3] Batch [1000]#011Speed: 89779.85 samples/sec#011binary_classification_accuracy=0.984024#011binary_classification_cross_entropy=0.061947\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:33 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=1000 train binary_classification_accuracy <score>=0.984023976024\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:33 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=1000 train binary_classification_cross_entropy <loss>=0.0619466742936\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:45 INFO 140204685006656] Epoch[3] Batch [2000]#011Speed: 90321.02 samples/sec#011binary_classification_accuracy=0.984154#011binary_classification_cross_entropy=0.061701\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:45 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=2000 train binary_classification_accuracy <score>=0.984154422789\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:45 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=2000 train binary_classification_cross_entropy <loss>=0.0617011348118\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:56 INFO 140204685006656] Epoch[3] Batch [3000]#011Speed: 89990.85 samples/sec#011binary_classification_accuracy=0.984151#011binary_classification_cross_entropy=0.061680\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:56 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=3000 train binary_classification_accuracy <score>=0.984151282906\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:46:56 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=3000 train binary_classification_cross_entropy <loss>=0.0616804307698\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:47:07 INFO 140204685006656] Epoch[3] Batch [4000]#011Speed: 90065.43 samples/sec#011binary_classification_accuracy=0.984250#011binary_classification_cross_entropy=0.061504\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:47:07 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=4000 train binary_classification_accuracy <score>=0.984249687578\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:47:07 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=4000 train binary_classification_cross_entropy <loss>=0.0615043577925\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:47:18 INFO 140204685006656] Epoch[3] Batch [5000]#011Speed: 89926.95 samples/sec#011binary_classification_accuracy=0.984205#011binary_classification_cross_entropy=0.061670\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:47:18 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=5000 train binary_classification_accuracy <score>=0.984204559088\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:47:18 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=5000 train binary_classification_cross_entropy <loss>=0.061669832635\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:47:29 INFO 140204685006656] Epoch[3] Batch [6000]#011Speed: 89437.75 samples/sec#011binary_classification_accuracy=0.984211#011binary_classification_cross_entropy=0.061596\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:47:29 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=6000 train binary_classification_accuracy <score>=0.984210631561\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:47:29 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=6000 train binary_classification_cross_entropy <loss>=0.0615960414733\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:47:40 INFO 140204685006656] Epoch[3] Batch [7000]#011Speed: 89812.66 samples/sec#011binary_classification_accuracy=0.984215#011binary_classification_cross_entropy=0.061626\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:47:40 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=7000 train binary_classification_accuracy <score>=0.984214683617\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:47:40 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=7000 train binary_classification_cross_entropy <loss>=0.0616256775727\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:47:51 INFO 140204685006656] Epoch[3] Batch [8000]#011Speed: 89549.01 samples/sec#011binary_classification_accuracy=0.984214#011binary_classification_cross_entropy=0.061571\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:47:51 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=8000 train binary_classification_accuracy <score>=0.984214348206\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:47:51 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=8000 train binary_classification_cross_entropy <loss>=0.0615707403552\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:03 INFO 140204685006656] Epoch[3] Batch [9000]#011Speed: 89356.31 samples/sec#011binary_classification_accuracy=0.984234#011binary_classification_cross_entropy=0.061500\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:03 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=9000 train binary_classification_accuracy <score>=0.9842341962\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:03 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=9000 train binary_classification_cross_entropy <loss>=0.0614997972945\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:14 INFO 140204685006656] Epoch[3] Batch [10000]#011Speed: 89263.19 samples/sec#011binary_classification_accuracy=0.984233#011binary_classification_cross_entropy=0.061472\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:14 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=10000 train binary_classification_accuracy <score>=0.984233276672\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:14 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=10000 train binary_classification_cross_entropy <loss>=0.0614723232951\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:25 INFO 140204685006656] Epoch[3] Batch [11000]#011Speed: 89575.63 samples/sec#011binary_classification_accuracy=0.984235#011binary_classification_cross_entropy=0.061473\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:25 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=11000 train binary_classification_accuracy <score>=0.984235251341\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:25 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=11000 train binary_classification_cross_entropy <loss>=0.0614733261172\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:36 INFO 140204685006656] Epoch[3] Batch [12000]#011Speed: 89628.86 samples/sec#011binary_classification_accuracy=0.984243#011binary_classification_cross_entropy=0.061428\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:36 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=12000 train binary_classification_accuracy <score>=0.984242813099\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:36 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=12000 train binary_classification_cross_entropy <loss>=0.0614277089453\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:47 INFO 140204685006656] Epoch[3] Batch [13000]#011Speed: 89603.90 samples/sec#011binary_classification_accuracy=0.984255#011binary_classification_cross_entropy=0.061406\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:47 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=13000 train binary_classification_accuracy <score>=0.984255441889\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:47 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=13000 train binary_classification_cross_entropy <loss>=0.0614062209825\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:58 INFO 140204685006656] Epoch[3] Batch [14000]#011Speed: 89671.01 samples/sec#011binary_classification_accuracy=0.984271#011binary_classification_cross_entropy=0.061362\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:58 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=14000 train binary_classification_accuracy <score>=0.984271052068\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:48:58 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=14000 train binary_classification_cross_entropy <loss>=0.0613620024022\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:10 INFO 140204685006656] Epoch[3] Batch [15000]#011Speed: 89515.75 samples/sec#011binary_classification_accuracy=0.984284#011binary_classification_cross_entropy=0.061298\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:10 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=15000 train binary_classification_accuracy <score>=0.984283847743\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:10 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, batch=15000 train binary_classification_cross_entropy <loss>=0.0612978278533\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:15 INFO 140204685006656] Epoch[3] Train-binary_classification_accuracy=0.984294\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:15 INFO 140204685006656] Epoch[3] Train-binary_classification_cross_entropy=0.061245\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:15 INFO 140204685006656] Epoch[3] Time cost=173.085\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:15 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, train binary_classification_accuracy <score>=0.984293894113\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:15 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, train binary_classification_cross_entropy <loss>=0.061244731665\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:23 INFO 140204685006656] Num unique classes of label: [0. 1.]\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:24 INFO 140204685006656] #quality_metric: host=algo-1, epoch=3, validation discriminator_auc <score>=0.990198590405\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 181852.20789909363, \"sum\": 181852.20789909363, \"min\": 181852.20789909363}}, \"EndTime\": 1594597764.67816, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594597582.825358}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:24 INFO 140204685006656] #progress_metric: host=algo-1, completed 80 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 15526, \"sum\": 15526.0, \"min\": 15526}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 62104, \"sum\": 62104.0, \"min\": 62104}, \"Total Records Seen\": {\"count\": 1, \"max\": 62080396, \"sum\": 62080396.0, \"min\": 62080396}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 15520099, \"sum\": 15520099.0, \"min\": 15520099}, \"Reset Count\": {\"count\": 1, \"max\": 8, \"sum\": 8.0, \"min\": 8}}, \"EndTime\": 1594597764.678554, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\", \"epoch\": 3}, \"StartTime\": 1594597582.825903}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:24 INFO 140204685006656] #throughput_metric: host=algo-1, train throughput=85344.3061858 records/second\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:24 WARNING 140204685006656] Already bound, ignoring bind()\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:24 WARNING 140204685006656] optimizer already initialized, ignoring...\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:24 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=0 train binary_classification_accuracy <score>=0.988\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:24 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=0 train binary_classification_cross_entropy <loss>=0.0261035728455\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:35 INFO 140204685006656] Epoch[4] Batch [1000]#011Speed: 89689.95 samples/sec#011binary_classification_accuracy=0.984164#011binary_classification_cross_entropy=0.060972\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:35 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=1000 train binary_classification_accuracy <score>=0.984163836164\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:35 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=1000 train binary_classification_cross_entropy <loss>=0.060972380721\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:47 INFO 140204685006656] Epoch[4] Batch [2000]#011Speed: 89562.23 samples/sec#011binary_classification_accuracy=0.984218#011binary_classification_cross_entropy=0.061315\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:47 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=2000 train binary_classification_accuracy <score>=0.984218390805\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:47 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=2000 train binary_classification_cross_entropy <loss>=0.0613147868605\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:58 INFO 140204685006656] Epoch[4] Batch [3000]#011Speed: 89523.90 samples/sec#011binary_classification_accuracy=0.984297#011binary_classification_cross_entropy=0.060976\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:58 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=3000 train binary_classification_accuracy <score>=0.984296567811\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:49:58 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=3000 train binary_classification_cross_entropy <loss>=0.060975546377\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:50:09 INFO 140204685006656] Epoch[4] Batch [4000]#011Speed: 89539.65 samples/sec#011binary_classification_accuracy=0.984384#011binary_classification_cross_entropy=0.060717\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:50:09 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=4000 train binary_classification_accuracy <score>=0.984384153962\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:50:09 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=4000 train binary_classification_cross_entropy <loss>=0.0607168460056\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:50:20 INFO 140204685006656] Epoch[4] Batch [5000]#011Speed: 89337.31 samples/sec#011binary_classification_accuracy=0.984341#011binary_classification_cross_entropy=0.060924\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:50:20 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=5000 train binary_classification_accuracy <score>=0.984341331734\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:50:20 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=5000 train binary_classification_cross_entropy <loss>=0.0609244287346\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:50:31 INFO 140204685006656] Epoch[4] Batch [6000]#011Speed: 89425.07 samples/sec#011binary_classification_accuracy=0.984350#011binary_classification_cross_entropy=0.060860\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:50:31 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=6000 train binary_classification_accuracy <score>=0.984350108315\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:50:31 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=6000 train binary_classification_cross_entropy <loss>=0.0608595595164\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:50:42 INFO 140204685006656] Epoch[4] Batch [7000]#011Speed: 89616.33 samples/sec#011binary_classification_accuracy=0.984320#011binary_classification_cross_entropy=0.060857\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:50:42 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=7000 train binary_classification_accuracy <score>=0.984320382802\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:50:42 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=7000 train binary_classification_cross_entropy <loss>=0.0608566126268\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:50:54 INFO 140204685006656] Epoch[4] Batch [8000]#011Speed: 89913.31 samples/sec#011binary_classification_accuracy=0.984292#011binary_classification_cross_entropy=0.060883\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:50:54 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=8000 train binary_classification_accuracy <score>=0.984292463442\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:50:54 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=8000 train binary_classification_cross_entropy <loss>=0.0608827378025\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:51:05 INFO 140204685006656] Epoch[4] Batch [9000]#011Speed: 89602.82 samples/sec#011binary_classification_accuracy=0.984315#011binary_classification_cross_entropy=0.060776\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:51:05 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=9000 train binary_classification_accuracy <score>=0.984315409399\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:51:05 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=9000 train binary_classification_cross_entropy <loss>=0.0607760684061\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:51:16 INFO 140204685006656] Epoch[4] Batch [10000]#011Speed: 89701.41 samples/sec#011binary_classification_accuracy=0.984337#011binary_classification_cross_entropy=0.060652\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:51:16 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=10000 train binary_classification_accuracy <score>=0.984336766323\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:51:16 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=10000 train binary_classification_cross_entropy <loss>=0.0606521139209\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:51:27 INFO 140204685006656] Epoch[4] Batch [11000]#011Speed: 89966.33 samples/sec#011binary_classification_accuracy=0.984345#011binary_classification_cross_entropy=0.060657\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:51:27 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=11000 train binary_classification_accuracy <score>=0.984344605036\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:51:27 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=11000 train binary_classification_cross_entropy <loss>=0.0606574370299\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:51:38 INFO 140204685006656] Epoch[4] Batch [12000]#011Speed: 90076.26 samples/sec#011binary_classification_accuracy=0.984357#011binary_classification_cross_entropy=0.060610\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:51:38 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=12000 train binary_classification_accuracy <score>=0.984356970252\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:51:38 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=12000 train binary_classification_cross_entropy <loss>=0.0606095053482\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:51:49 INFO 140204685006656] Epoch[4] Batch [13000]#011Speed: 89990.07 samples/sec#011binary_classification_accuracy=0.984349#011binary_classification_cross_entropy=0.060644\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:51:49 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=13000 train binary_classification_accuracy <score>=0.98434912699\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:51:49 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=13000 train binary_classification_cross_entropy <loss>=0.0606443291315\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:00 INFO 140204685006656] Epoch[4] Batch [14000]#011Speed: 89922.07 samples/sec#011binary_classification_accuracy=0.984356#011binary_classification_cross_entropy=0.060596\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:00 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=14000 train binary_classification_accuracy <score>=0.984355546032\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:00 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=14000 train binary_classification_cross_entropy <loss>=0.0605956779975\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:11 INFO 140204685006656] Epoch[4] Batch [15000]#011Speed: 89885.47 samples/sec#011binary_classification_accuracy=0.984371#011binary_classification_cross_entropy=0.060542\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:11 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=15000 train binary_classification_accuracy <score>=0.984371375242\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:11 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, batch=15000 train binary_classification_cross_entropy <loss>=0.0605415597562\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:17 INFO 140204685006656] Epoch[4] Train-binary_classification_accuracy=0.984390\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:17 INFO 140204685006656] Epoch[4] Train-binary_classification_cross_entropy=0.060464\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:17 INFO 140204685006656] Epoch[4] Time cost=173.062\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:17 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, train binary_classification_accuracy <score>=0.984390441839\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:17 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, train binary_classification_cross_entropy <loss>=0.060464225013\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:25 INFO 140204685006656] Num unique classes of label: [0. 1.]\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:26 INFO 140204685006656] #quality_metric: host=algo-1, epoch=4, validation discriminator_auc <score>=0.990466286419\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 181857.656955719, \"sum\": 181857.656955719, \"min\": 181857.656955719}}, \"EndTime\": 1594597946.536552, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594597764.678265}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:26 INFO 140204685006656] #progress_metric: host=algo-1, completed 100 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 15526, \"sum\": 15526.0, \"min\": 15526}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 77630, \"sum\": 77630.0, \"min\": 77630}, \"Total Records Seen\": {\"count\": 1, \"max\": 77600495, \"sum\": 77600495.0, \"min\": 77600495}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 15520099, \"sum\": 15520099.0, \"min\": 15520099}, \"Reset Count\": {\"count\": 1, \"max\": 10, \"sum\": 10.0, \"min\": 10}}, \"EndTime\": 1594597946.536935, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\", \"epoch\": 4}, \"StartTime\": 1594597764.678846}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:26 INFO 140204685006656] #throughput_metric: host=algo-1, train throughput=85341.7435139 records/second\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:26 INFO 140204685006656] Saved checkpoint to \"/tmp/tmpT7rZOa/state-0001.params\"\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:26 WARNING 140204685006656] wait_for_all_workers will not sync workers since the kv store is not running distributed\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"finalize.time\": {\"count\": 1, \"max\": 0.39386749267578125, \"sum\": 0.39386749267578125, \"min\": 0.39386749267578125}}, \"EndTime\": 1594597946.649995, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594597946.536646}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:26 INFO 140204685006656] Saved checkpoint to \"/tmp/tmposVddk/state-0001.params\"\u001b[0m\n",
      "\u001b[34m[07/12/2020 23:52:26 INFO 140204685006656] Test data is not provided.\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"totaltime\": {\"count\": 1, \"max\": 915994.8179721832, \"sum\": 915994.8179721832, \"min\": 915994.8179721832}, \"setuptime\": {\"count\": 1, \"max\": 119.13299560546875, \"sum\": 119.13299560546875, \"min\": 119.13299560546875}}, \"EndTime\": 1594597946.762549, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"ipinsights\"}, \"StartTime\": 1594597946.650062}\n",
      "\u001b[0m\n",
      "Training seconds: 1022\n",
      "Billable seconds: 1022\n",
      "---------------!"
     ]
    }
   ],
   "source": [
    "# Deploy best model\n",
    "tuned_predictor = ip_insights_tuner.deploy(\n",
    "    initial_instance_count=1, \n",
    "    instance_type='ml.m4.xlarge',\n",
    "    content_type='text/csv',\n",
    "    serializer=csv_serializer,\n",
    "    accept='application/json',\n",
    "    deserializer=json_deserializer\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'predictions': [{'dot_product': 4.44974422454834},\n",
       "  {'dot_product': 3.997523069381714},\n",
       "  {'dot_product': 3.1688296794891357},\n",
       "  {'dot_product': 4.983971118927002},\n",
       "  {'dot_product': 6.965197563171387}]}"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make a prediction \n",
    "tuned_predictor.predict(inference_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sess.delete_endpoint(tuned_predictor.endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Now we can determine thresholds and make predictions just like we did with the inference endpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (2) Batch Transform\n",
    "Let's say we want to score all of the login events at the end of the day and aggregate flagged cases for investigators to look at in the morning. If we store the daily login events in S3, we can use IP Insights with Amazon SageMaker Batch Transform to run inference and store the IP Insights scores back in S3 for future analysis.\n",
    "\n",
    "Below, we take the training job from before and evaluate it on the validation data we put in S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker:Parameter image will be renamed to image_uri in SageMaker Python SDK v2.\n",
      "WARNING:sagemaker:Using already existing model: ipinsights-2020-07-12-21-39-26-660\n"
     ]
    }
   ],
   "source": [
    "transformer = ip_insights.transformer(\n",
    "    instance_count=1,\n",
    "    instance_type='ml.m4.xlarge',\n",
    ")\n",
    "\n",
    "transformer.transform(\n",
    "    s3_valid_data,\n",
    "    content_type='text/csv',\n",
    "    split_type='Line'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".........................\u001b[34mDocker entrypoint called with argument(s): serve\u001b[0m\n",
      "\u001b[34mRunning default environment configuration script\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:09 INFO 140096139585344] loaded entry point class algorithm.serve.server_config:config_api\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:09 INFO 140096139585344] loading entry points\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:09 INFO 140096139585344] loaded request iterator text/csv\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:09 INFO 140096139585344] Loaded iterator creator application/x-ndarray for content type ('application/x-ndarray', '1.0')\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:09 INFO 140096139585344] Loaded iterator creator text/csv for content type ('text/csv', '1.0')\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:09 INFO 140096139585344] loaded request iterator application/json\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:09 INFO 140096139585344] loaded request iterator application/jsonlines\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:09 INFO 140096139585344] loaded response encoder application/json\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:09 INFO 140096139585344] loaded response encoder application/jsonlines\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:10 INFO 140096139585344] loaded entry point class algorithm:model\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:10 INFO 140096139585344] Loaded iterator creator application/x-ndarray for content type ('application/x-ndarray', '1.0')\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:10 INFO 140096139585344] Loaded iterator creator text/csv for content type ('text/csv', '1.0')\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:10 INFO 140096139585344] loading model...\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:10 WARNING 140096139585344] Requesting context without setting the requested num of gpus. Using 'auto'\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:10 INFO 140096139585344] nvidia-smi took: 0.0252139568329 secs to identify 0 gpus\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:10 INFO 140096139585344] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:10 INFO 140096139585344] ...model loaded.\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:10 INFO 140096139585344] Number of server workers: 4\u001b[0m\n",
      "\u001b[34m[2020-07-13 00:24:10 +0000] [1] [INFO] Starting gunicorn 19.7.1\u001b[0m\n",
      "\u001b[34m[2020-07-13 00:24:10 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)\u001b[0m\n",
      "\u001b[34m[2020-07-13 00:24:10 +0000] [1] [INFO] Using worker: sync\u001b[0m\n",
      "\u001b[34m[2020-07-13 00:24:10 +0000] [83] [INFO] Booting worker with pid: 83\u001b[0m\n",
      "\u001b[34m[2020-07-13 00:24:10 +0000] [92] [INFO] Booting worker with pid: 92\u001b[0m\n",
      "\u001b[34m[2020-07-13 00:24:10 +0000] [101] [INFO] Booting worker with pid: 101\u001b[0m\n",
      "\u001b[34m[2020-07-13 00:24:11 +0000] [110] [INFO] Booting worker with pid: 110\u001b[0m\n",
      "\u001b[32m2020-07-13T00:24:16.885:[sagemaker logs]: MaxConcurrentTransforms=4, MaxPayloadInMB=6, BatchStrategy=MULTI_RECORD\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"execution_parameters.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1594599856.876312, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"IPInsightsModel\"}, \"StartTime\": 1594599850.799442}\n",
      "\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"execution_parameters.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1594599856.876312, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"IPInsightsModel\"}, \"StartTime\": 1594599850.799442}\n",
      "\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:17 WARNING 140096139585344] Requesting context without setting the requested num of gpus. Using 'auto'\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:17 INFO 140096139585344] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:17 WARNING 140096139585344] Requesting context without setting the requested num of gpus. Using 'auto'\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:17 INFO 140096139585344] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:17 WARNING 140096139585344] Requesting context without setting the requested num of gpus. Using 'auto'\u001b[0m\n",
      "\u001b[34m[07/13/2020 00:24:17 INFO 140096139585344] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[35m[07/13/2020 00:24:17 WARNING 140096139585344] Requesting context without setting the requested num of gpus. Using 'auto'\u001b[0m\n",
      "\u001b[35m[07/13/2020 00:24:17 INFO 140096139585344] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[35m[07/13/2020 00:24:17 WARNING 140096139585344] Requesting context without setting the requested num of gpus. Using 'auto'\u001b[0m\n",
      "\u001b[35m[07/13/2020 00:24:17 INFO 140096139585344] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[35m[07/13/2020 00:24:17 WARNING 140096139585344] Requesting context without setting the requested num of gpus. Using 'auto'\u001b[0m\n",
      "\u001b[35m[07/13/2020 00:24:17 INFO 140096139585344] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1594599860.598222, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"IPInsightsModel\"}, \"StartTime\": 1594599850.799442}\n",
      "\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1594599860.598222, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"IPInsightsModel\"}, \"StartTime\": 1594599850.799442}\n",
      "\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1594599862.018344, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"IPInsightsModel\"}, \"StartTime\": 1594599856.876594}\n",
      "\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1594599862.018344, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"IPInsightsModel\"}, \"StartTime\": 1594599856.876594}\n",
      "\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1594599862.125243, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"IPInsightsModel\"}, \"StartTime\": 1594599850.799442}\n",
      "\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1594599862.125243, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"IPInsightsModel\"}, \"StartTime\": 1594599850.799442}\n",
      "\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Wait for Transform Job to finish\n",
    "transformer.wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch Transform output is at: s3://sagemaker-us-west-2-570447867175/ipinsights-2020-07-13-00-19-59-019\n"
     ]
    }
   ],
   "source": [
    "print(\"Batch Transform output is at: {}\".format(transformer.output_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11. Close the SageMaker Instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "ip_insights_tuner.delete_endpoint()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker.Session().delete_endpoint(predictor.endpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#########################################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reference\n",
    ">https://docs.aws.amazon.com/sagemaker/latest/dg/ip-insights.html"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}