16 - Monitor a Model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Monitor a Model\n",
    "\n",
    "When you've deployed a model into production as a service, you'll want to monitor it to track usage and explore the requests it processes. You can use Azure Application Insights to monitor activity for a model service endpoint."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connect to your workspace\n",
    "\n",
    "To get started, connect to your workspace.\n",
    "\n",
    "> **Note**: If you haven't already established an authenticated session with your Azure subscription, you'll be prompted to authenticate by clicking a link, entering an authentication code, and signing into Azure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Workspace\n",
    "\n",
    "# Load the workspace from the saved config file\n",
    "ws = Workspace.from_config()\n",
    "print('Ready to work with', ws.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare a model for deployment\n",
    "\n",
    "Now we need a model to deploy. Run the code below to:\n",
    "\n",
    "1. Create and register a dataset.\n",
    "2. Train a model using the dataset.\n",
    "3. Register the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Experiment\n",
    "from azureml.core import Model\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import joblib\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.metrics import roc_auc_score, roc_curve\n",
    "from azureml.core import Dataset\n",
    "\n",
    "# Upload data files to the default datastore\n",
    "default_ds = ws.get_default_datastore()\n",
    "default_ds.upload_files(files=['./data/diabetes.csv', './data/diabetes2.csv'],\n",
    "                       target_path='diabetes-data/',\n",
    "                       overwrite=True,\n",
    "                       show_progress=True)\n",
    "\n",
    "#Create a tabular dataset from the path on the datastore\n",
    "print('Creating dataset...')\n",
    "data_set = Dataset.Tabular.from_delimited_files(path=(default_ds, 'diabetes-data/*.csv'))\n",
    "\n",
    "# Register the tabular dataset\n",
    "print('Registering dataset...')\n",
    "try:\n",
    "    data_set = data_set.register(workspace=ws, \n",
    "                               name='diabetes dataset',\n",
    "                               description='diabetes data',\n",
    "                               tags = {'format':'CSV'},\n",
    "                               create_new_version=True)\n",
    "except Exception as ex:\n",
    "    print(ex)\n",
    "\n",
    "# Create an Azure ML experiment in your workspace\n",
    "experiment = Experiment(workspace=ws, name='mslearn-train-diabetes')\n",
    "run = experiment.start_logging()\n",
    "print(\"Starting experiment:\", experiment.name)\n",
    "\n",
    "# load the diabetes dataset\n",
    "print(\"Loading Data...\")\n",
    "diabetes = data_set.to_pandas_dataframe()\n",
    "\n",
    "# Separate features and labels\n",
    "X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values\n",
    "\n",
    "# Split data into training set and test set\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)\n",
    "\n",
    "# Train a decision tree model\n",
    "print('Training a decision tree model')\n",
    "model = DecisionTreeClassifier().fit(X_train, y_train)\n",
    "\n",
    "# calculate accuracy\n",
    "y_hat = model.predict(X_test)\n",
    "acc = np.average(y_hat == y_test)\n",
    "print('Accuracy:', acc)\n",
    "run.log('Accuracy', np.float(acc))\n",
    "\n",
    "# calculate AUC\n",
    "y_scores = model.predict_proba(X_test)\n",
    "auc = roc_auc_score(y_test,y_scores[:,1])\n",
    "print('AUC: ' + str(auc))\n",
    "run.log('AUC', np.float(auc))\n",
    "\n",
    "# Save the trained model\n",
    "model_file = 'diabetes_model.pkl'\n",
    "joblib.dump(value=model, filename=model_file)\n",
    "run.upload_file(name = 'outputs/' + model_file, path_or_stream = './' + model_file)\n",
    "\n",
    "# Complete the run\n",
    "run.complete()\n",
    "\n",
    "# Register the model\n",
    "print('Registering model...')\n",
    "run.register_model(model_path='outputs/diabetes_model.pkl', model_name='diabetes_model',\n",
    "                   tags={'Training context':'Inline Training'},\n",
    "                   properties={'AUC': run.get_metrics()['AUC'], 'Accuracy': run.get_metrics()['Accuracy']})\n",
    "\n",
    "# Get the registered model\n",
    "model = ws.models['diabetes_model']\n",
    "\n",
    "print('Model trained and registered.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy a model as a web service\n",
    "\n",
    "Now you're ready to deploy the registered model as a web service.\n",
    "\n",
    "First, create a folder for the deployment configuration files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "folder_name = 'diabetes_service'\n",
    "\n",
    "# Create a folder for the web service files\n",
    "experiment_folder = './' + folder_name\n",
    "os.makedirs(experiment_folder, exist_ok=True)\n",
    "\n",
    "print(folder_name, 'folder created.')\n",
    "\n",
    "# Set path for scoring script\n",
    "script_file = os.path.join(experiment_folder,\"score_diabetes.py\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you need an entry script that the service will use to score new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $script_file\n",
    "import json\n",
    "import joblib\n",
    "import numpy as np\n",
    "from azureml.core.model import Model\n",
    "\n",
    "# Called when the service is loaded\n",
    "def init():\n",
    "    global model\n",
    "    # Get the path to the deployed model file and load it\n",
    "    model_path = Model.get_model_path('diabetes_model')\n",
    "    model = joblib.load(model_path)\n",
    "\n",
    "# Called when a request is received\n",
    "def run(raw_data):\n",
    "    # Get the input data as a numpy array\n",
    "    data = json.loads(raw_data)['data']\n",
    "    np_data = np.array(data)\n",
    "    # Get a prediction from the model\n",
    "    predictions = model.predict(np_data)\n",
    "    \n",
    "    # print the data and predictions (so they'll be logged!)\n",
    "    log_text = 'Data:' + str(data) + ' - Predictions:' + str(predictions)\n",
    "    print(log_text)\n",
    "    \n",
    "    # Get the corresponding classname for each prediction (0 or 1)\n",
    "    classnames = ['not-diabetic', 'diabetic']\n",
    "    predicted_classes = []\n",
    "    for prediction in predictions:\n",
    "        predicted_classes.append(classnames[prediction])\n",
    "    # Return the predictions as JSON\n",
    "    return json.dumps(predicted_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll also need a Conda configuration file for the service environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from azureml.core.conda_dependencies import CondaDependencies \n",
    "\n",
    "# Add the dependencies for our model (AzureML defaults is already included)\n",
    "myenv = CondaDependencies()\n",
    "myenv.add_conda_package(\"scikit-learn\")\n",
    "\n",
    "# Save the environment config as a .yml file\n",
    "env_file = folder_name + \"/diabetes_env.yml\"\n",
    "with open(env_file,\"w\") as f:\n",
    "    f.write(myenv.serialize_to_string())\n",
    "print(\"Saved dependency info in\", env_file)\n",
    "\n",
    "# Print the .yml file\n",
    "with open(env_file,\"r\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can deploy the service (in this case, as an Azure Container Instance (ACI).\n",
    "\n",
    "> **Note**: This can take a few minutes - wait until the state is shown as **Healthy**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.webservice import AciWebservice, Webservice\n",
    "from azureml.core.model import Model\n",
    "from azureml.core.model import InferenceConfig\n",
    "\n",
    "# Configure the scoring environment\n",
    "inference_config = InferenceConfig(runtime= \"python\",\n",
    "                                   entry_script=script_file,\n",
    "                                   conda_file=env_file)\n",
    "\n",
    "service_name = \"diabetes-service-app-insights\"\n",
    "deployment_config = AciWebservice.deploy_configuration(cpu_cores = 1, memory_gb = 1)\n",
    "aci_service = Model.deploy(workspace=ws,\n",
    "                           name= service_name,\n",
    "                           models= [model],\n",
    "                           inference_config= inference_config,\n",
    "                           deployment_config=deployment_config)\n",
    "aci_service.wait_for_deployment(show_output = True)\n",
    "print(aci_service.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enable Application Insights\n",
    "\n",
    "Next, you need to enable Application Insights for the service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enable AppInsights\n",
    "aci_service.update(enable_app_insights=True)\n",
    "print('AppInsights enabled!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use the web service\n",
    "\n",
    "With the service deployed, now you can consume it from a client application.\n",
    "\n",
    "First, determine the URL to which these applications must submit their requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint = aci_service.scoring_uri\n",
    "print(endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you know the endpoint URI, an application can simply make an HTTP request, sending the patient data in JSON (or binary) format, and receive back the predicted class(es).\n",
    "\n",
    "> **Tip**: If an error occurs because the service endpoint isn't ready. Wait a few seconds and try again!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "\n",
    "# Create new data for inferencing\n",
    "x_new = [[2,180,74,24,21,23.9091702,1.488172308,22],\n",
    "         [0,148,58,11,179,39.19207553,0.160829008,45]]\n",
    "\n",
    "# Convert the array to a serializable list in a JSON document\n",
    "input_json = json.dumps({\"data\": x_new})\n",
    "\n",
    "# Set the content type\n",
    "headers = { 'Content-Type':'application/json' }\n",
    "\n",
    "# Get the predictions\n",
    "predictions = requests.post(endpoint, input_json, headers = headers)\n",
    "print(predictions.status_code)\n",
    "if predictions.status_code == 200:\n",
    "    predicted_classes = json.loads(predictions.json())\n",
    "    for i in range(len(x_new)):\n",
    "        print (\"Patient {}\".format(x_new[i]), predicted_classes[i] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can view the data logged for the service endpoint:\n",
    "\n",
    "1. In the [Azure portal](https://portal.azure.com), open your Machine Learning workspace.\n",
    "2. On the **Overview** page, click the link for the associated **Application Insights** resource.\n",
    "3. On the Application Insights blade, click **Logs**. \n",
    "\n",
    "    > **Note**: If this is the first time you've opened log analytics, you may need to click **Get Started** to open the query editor. If a tip explaining how to write a query is displayed, close it.\n",
    "\n",
    "4. Paste the following query into the query editor and click **Run**\n",
    "    ```\n",
    "    traces\n",
    "    |where  message == \"STDOUT\"\n",
    "      and customDimensions.[\"Service Name\"] == \"diabetes-service-app-insights\"\n",
    "    |project timestamp, customDimensions.Content\n",
    "    ```\n",
    "5. View the results. At first there may be none, because an ACI web service can take as long as five minutes to send the telemetry to Application Insights. Wait a few minutes and re-run the query until you see the logged data and predictions.\n",
    "6. When you've reviewed the logged data, close the Application Insights query page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Delete the service\n",
    "\n",
    "When you no longer need your service, you should delete it.\n",
    "\n",
    "> **Note**: If the service is in use, you may not be able to delete it immediately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    aci_service.delete()\n",
    "    print('Service deleted.')\n",
    "except Exception as ex:\n",
    "    print(ex.message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information about using Application Insights to monitor a deployed service, see the [Azure Machine Learning documentation](https://docs.microsoft.com/azure/machine-learning/how-to-enable-app-insights)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}