v1/python-sdk/tutorials/automl-with-azureml/metrics/binary-classification-metric-and-confidence-interval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/metrics/binary-classification-metric-and-confidence-interval.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automated Machine Learning\n",
    "_**New metric features in Azure AutoML**_\n",
    "\n",
    "## Contents\n",
    "1. [Introduction](#Introduction)\n",
    "1. [Setup](#Setup)\n",
    "1. [Train](#Train)\n",
    "1. [Results](#Results)\n",
    "1. [Test](#Test)\n",
    "1. [Acknowledgements](#Acknowledgements)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "In this example notebook we use the sklearn datasets, [digits](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html) and [boston](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html) to help you get familiar with binary classification metrics and confidence interval. The goal is to learn how to use these features through the examples. \n",
    "\n",
    "This notebook is using remote compute to train the model.\n",
    "\n",
    "If you are using an Azure Machine Learning Compute Instance, you are all set. Otherwise, go through the [configuration](https://github.com/Azure/MachineLearningNotebooks/blob/master/configuration.ipynb) notebook first if you haven't already to establish your connection to the AzureML Workspace. \n",
    "\n",
    "In this notebook you will learn how to:\n",
    "1. How to have binary classification metrics calculated for AutoML runs\n",
    "2. How to find binary classification metrics in UI and how to retrieve the values through code\n",
    "3. How to have confidence intervals calculated for both classification and regression AutoML runs\n",
    "4. How to find confidence intervals in UI and how to retrieve the values through code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "As part of the setup you have already created an Azure ML `Workspace` object. For Automated ML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.core.workspace import Workspace\n",
    "from azureml.core.dataset import Dataset\n",
    "from azureml.train.automl import AutoMLConfig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws = Workspace.from_config()\n",
    "\n",
    "experiment_name = \"metrics-new-feature-test\"\n",
    "\n",
    "experiment = Experiment(ws, experiment_name)\n",
    "\n",
    "output = {}\n",
    "output[\"Subscription ID\"] = ws.subscription_id\n",
    "output[\"Workspace\"] = ws.name\n",
    "output[\"Resource Group\"] = ws.resource_group\n",
    "output[\"Location\"] = ws.location\n",
    "output[\"Experiment Name\"] = experiment.name\n",
    "pd.set_option(\"display.max_colwidth\", None)\n",
    "outputDf = pd.DataFrame(data=output, index=[\"\"])\n",
    "outputDf.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create or Attach existing AmlCompute\n",
    "A compute target is required to execute the Automated ML run. In this tutorial, you create AmlCompute as your training compute resource.\n",
    "\n",
    "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
    "\n",
    "#### Creation of AmlCompute takes approximately 5 minutes. \n",
    "If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
    "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "\n",
    "# Choose a name for your CPU cluster\n",
    "cpu_cluster_name = \"cpu-cluster-1\"\n",
    "\n",
    "# Verify that cluster does not exist already\n",
    "try:\n",
    "    compute_target = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
    "    print(\"Found existing cluster, use it.\")\n",
    "except ComputeTargetException:\n",
    "    compute_config = AmlCompute.provisioning_configuration(\n",
    "        vm_size=\"STANDARD_DS12_V2\", max_nodes=6\n",
    "    )\n",
    "    compute_target = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
    "\n",
    "compute_target.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Data\n",
    "\n",
    "We load datasets from sklearn and save to local files to register them to workspace.\n",
    "\n",
    "For classification, we use [digits dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html#sklearn.datasets.load_digits)\n",
    "\n",
    "For regression, we use [boston dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html#sklearn.datasets.load_boston)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import sklearn.datasets\n",
    "\n",
    "\n",
    "def load_classification_data():\n",
    "    if os.path.exists(\"./data/digits.csv\"):\n",
    "        print(\"Find downloaded dataset. Loading\")\n",
    "    else:\n",
    "        print(\"Downloading dataset\")\n",
    "        os.makedirs(\"./data\", exist_ok=True)\n",
    "        classification_dataset = sklearn.datasets.load_digits()\n",
    "        X = classification_dataset[\"data\"]\n",
    "        y = classification_dataset[\"target\"]\n",
    "        full_data = np.concatenate([X, y.reshape(-1, 1)], axis=1).astype(\"int\")\n",
    "        columns = [\"feature_{}\".format(i) for i in range(X.shape[1])] + [\"label\"]\n",
    "        full_data = pd.DataFrame(data=full_data, columns=columns)\n",
    "        full_data.to_csv(\"./data/digits.csv\", index=False)\n",
    "        print(\"Dataset downloaded\")\n",
    "    ws = Workspace.from_config()\n",
    "    datastore = ws.get_default_datastore()\n",
    "    datastore.upload(\n",
    "        src_dir=\"./data\", target_path=\"data/new-metric-features/\", overwrite=True\n",
    "    )\n",
    "    data = Dataset.Tabular.from_delimited_files(\n",
    "        path=[(datastore, (\"data/new-metric-features/digits.csv\"))]\n",
    "    )\n",
    "    train, test = data.random_split(percentage=0.8, seed=101)\n",
    "    validation, test = test.random_split(percentage=0.5, seed=47)\n",
    "    return train, validation, test, np.arange(10), \"label\"\n",
    "\n",
    "\n",
    "(\n",
    "    digit_train,\n",
    "    digit_validation,\n",
    "    digit_test,\n",
    "    labels,\n",
    "    label_column_name,\n",
    ") = load_classification_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Binary Classification Metrics\n",
    "\n",
    "In this section we will explain how to set parameters for AutoML runs to have binary classification metrics calculated.\n",
    "\n",
    "## Binary Classification Metrics\n",
    "Binary classification metrics will be calculated for AutoML in two cases:\n",
    "1. There are exactly two classes.\n",
    "2. parameter `positive_label` in `AutoMLConfig` is specified as an existing class.\n",
    "\n",
    "When a `positive_label` is specified for multiclass classification tasks, all other classes will all be treated the negative class when calculating the binary classification metrics.\n",
    "\n",
    "When there are exactly two classes, `np.unique()` will be used to sort the classes and the class with larger index will be used as the positive class. However, we would recommend always specify a `positive_label` when you want to calculate binary classification metrics to make sure that it is calculated for the correct class. In the example below, we use class `4` as the positive class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    \"primary_metric\": \"AUC_weighted\",\n",
    "    \"enable_early_stopping\": True,\n",
    "    \"max_concurrent_iterations\": 6,\n",
    "    \"experiment_timeout_hours\": 0.25,\n",
    "    \"verbosity\": logging.INFO,\n",
    "}\n",
    "\n",
    "automl_config = AutoMLConfig(\n",
    "    task=\"classification\",\n",
    "    debug_log=\"automl_errors.log\",\n",
    "    compute_target=compute_target,\n",
    "    training_data=digit_train,\n",
    "    validation_data=digit_validation,\n",
    "    label_column_name=label_column_name,\n",
    "    positive_label=4,  # specify the positive class with this parameter\n",
    "    **automl_settings\n",
    ")\n",
    "\n",
    "classification_run = experiment.submit(automl_config, show_output=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classification_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find Binary Metrics in UI\n",
    "\n",
    "After training, you can click the link above to visit the page of this run. You can find all training runs under `Child runs` tab:\n",
    "\n",
    "![](imgs/child-runs.png)\n",
    "\n",
    "Then under `Metrics` tab, you can find some metrics names that end with `_binary`. They are the binary classification metrics with the specified positive class.\n",
    "\n",
    "![](imgs/binary-metrics.png)\n",
    "\n",
    "## Retrieve Binary Metrics with Code\n",
    "\n",
    "You can also retrieve the metrics values for any training run with codes. They returned values will be a dictionary with structure `{name: value}`. The example below retrieves the metrics of the best trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_run, fitted_model = classification_run.get_output()\n",
    "training_metrics = best_run.get_metrics()\n",
    "training_metrics[\"AUC_binary\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With data downloaded, you can also calculate the binary classification metrics with other classes as the positive class. \n",
    "\n",
    "To calculate metrics with codes, you will need to import Azure AutoML's scoring modules and specify the value of `positive_label` as desired. See example code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.automl.runtime.shared.score import constants, scoring\n",
    "\n",
    "test_df = digit_test.to_pandas_dataframe()\n",
    "y_test = test_df[label_column_name]\n",
    "test_df = test_df.drop(columns=[label_column_name])\n",
    "y_pred_proba = fitted_model.predict_proba(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for positive_label in range(10):\n",
    "    metrics = scoring.score_classification(\n",
    "        y_test,\n",
    "        y_pred_proba,\n",
    "        constants.CLASSIFICATION_SCALAR_SET,\n",
    "        labels,\n",
    "        labels,\n",
    "        positive_label=positive_label,\n",
    "    )\n",
    "    print(\n",
    "        \"AUC_binary for label {} is {:.4f}\".format(\n",
    "            positive_label, metrics[\"AUC_binary\"]\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrong Value of `positive_label` Fails the Run\n",
    "\n",
    "The value of `positive_label` passed into `AutoMLConfig` must be exactly the same as it is in the dataset. If you passed in a `positive_label` that cannot be found in the training dataset, the run will fail. See the example below, where the correct value `4` is replaced by its string version, `'4'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    \"primary_metric\": \"AUC_weighted\",\n",
    "    \"enable_early_stopping\": True,\n",
    "    \"max_concurrent_iterations\": 6,\n",
    "    \"experiment_timeout_hours\": 0.25,\n",
    "    \"verbosity\": logging.INFO,\n",
    "}\n",
    "\n",
    "automl_config = AutoMLConfig(\n",
    "    task=\"classification\",\n",
    "    debug_log=\"automl_errors.log\",\n",
    "    compute_target=compute_target,\n",
    "    training_data=digit_train,\n",
    "    validation_data=digit_validation,\n",
    "    label_column_name=label_column_name,\n",
    "    positive_label=\"4\",  # replace the correct integer value with its string version\n",
    "    **automl_settings\n",
    ")\n",
    "\n",
    "classification_run = experiment.submit(automl_config, show_output=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classification_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Confidence Interval\n",
    "\n",
    "We calculate confidence intervals for metrics by doing bootstrap and we give conservative estimates. Like binary classification metrics, you can find the confidence intervals in UI, and also retrieve them with codes. \n",
    "\n",
    "To calculate confidence intervals in AutoML runs, we need to pass two other parameters to `AutoMLConfig`:\n",
    "1. `enable_metric_confidence = True` to tell the run to calculate confidence interval\n",
    "2. `test_data` to activate a test run, as confidence intervals will only be calculated for test runs.\n",
    "\n",
    "Currently, if the task is classification, only primary metrics will have their confidence intervals logged with the run. To get confidence intervals for other metrics, you can use codes. We will provide examples below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    \"primary_metric\": \"AUC_weighted\",\n",
    "    \"enable_early_stopping\": True,\n",
    "    \"max_concurrent_iterations\": 6,\n",
    "    \"experiment_timeout_hours\": 0.25,\n",
    "    \"verbosity\": logging.INFO,\n",
    "}\n",
    "\n",
    "automl_config = AutoMLConfig(\n",
    "    task=\"classification\",\n",
    "    debug_log=\"automl_errors.log\",\n",
    "    compute_target=compute_target,\n",
    "    training_data=digit_train,\n",
    "    validation_data=digit_validation,\n",
    "    test_data=digit_test,  # if you only have a test set, you can pass validation set here, instead of at validation_data\n",
    "    label_column_name=label_column_name,\n",
    "    enable_metric_confidence=True,\n",
    "    **automl_settings\n",
    ")\n",
    "\n",
    "classification_run = experiment.submit(automl_config, show_output=False)\n",
    "classification_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find Confidence Interval in UI\n",
    "\n",
    "To locate the confidence intervals in UI, we must first find the run which gives the best model, as only the best model will be run on test set. In order to do so, click the link above for the AutoML run, and go to `Models` tab. The model listed on the top is the one with best performance:\n",
    "\n",
    "![](imgs/best-model.png)\n",
    "\n",
    "Then for this best model, go to its `Child runs` tab and click the run with tab `Test model`\n",
    "\n",
    "![](imgs/test-run.png)\n",
    "\n",
    "For this test run, under tab `Metrics`, you can find some metrics whose names end with `extras`. By switching `View as` from `Chart` to `Table`, you can find the confidence intervals for those metrics.\n",
    "\n",
    "![](imgs/confidence-intervals.png)\n",
    "\n",
    "## Find Confidence Interval with Code\n",
    "\n",
    "You can retrieve the `Run` object for test run with the following code, and get confidence interval from its metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_run, fitted_model = classification_run.get_output()\n",
    "test_run = next(best_run.get_children(type=\"automl.model_test\"))\n",
    "test_run.wait_for_completion(show_output=False, wait_post_processing=True)\n",
    "test_metrics = test_run.get_metrics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CIs = {\"metric_name\": [], \"lower_ci_95\": [], \"upper_ci_95\": [], \"value\": []}\n",
    "\n",
    "for key, ci in test_metrics.items():\n",
    "    if key.endswith(\"extras\"):\n",
    "        CIs[\"metric_name\"].append(key[:-7])  # remove \"_extras\" to get metric name\n",
    "        for ci_key, ci_value in ci.items():\n",
    "            CIs[ci_key].append(ci_value)\n",
    "\n",
    "pd.DataFrame(CIs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, you can retrieve the best model, do inference yourself, and get confidence intervals for all metrics. However, since our confidence intervals includes a large number of bootstraps, it will take some time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df = digit_test.to_pandas_dataframe()\n",
    "y_test = test_df[label_column_name]\n",
    "test_df = test_df.drop(columns=[label_column_name])\n",
    "y_pred_proba = fitted_model.predict_proba(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.automl.runtime._ml_engine.classification_ml_engine import (\n",
    "    evaluate_classifier,\n",
    ")\n",
    "\n",
    "test_metrics = evaluate_classifier(\n",
    "    y_test,\n",
    "    y_pred_proba,\n",
    "    constants.CLASSIFICATION_SCALAR_SET,\n",
    "    labels,\n",
    "    labels,\n",
    "    enable_metric_confidence=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CIs = {\"metric_name\": [], \"lower_ci_95\": [], \"upper_ci_95\": [], \"value\": []}\n",
    "\n",
    "for key, ci in test_metrics.items():\n",
    "    if key.endswith(\"extras\"):\n",
    "        CIs[\"metric_name\"].append(key[:-7])  # remove \"_extras\" to get metric name\n",
    "        for ci_key, ci_value in ci.items():\n",
    "            CIs[ci_key].append(ci_value)\n",
    "\n",
    "pd.DataFrame(CIs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Confidence Interval for Regression\n",
    "\n",
    "Confidence intervals are also supported for regression runs and all confidence intervals can be found in UI. You can find it by following the exact same steps as you do for a classification run. Here we only provide example code for a regression run, screen shots of the confidence intervals, and retrieve it with codes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_regression_data():\n",
    "    if os.path.exists(\"./data/boston.csv\"):\n",
    "        print(\"Find downloaded dataset. Loading\")\n",
    "    else:\n",
    "        print(\"Downloading dataset\")\n",
    "        os.makedirs(\"./data\", exist_ok=True)\n",
    "        regression_data = sklearn.datasets.load_boston()\n",
    "        X = regression_data[\"data\"]\n",
    "        y = regression_data[\"target\"]\n",
    "        full_data = np.concatenate([X, y.reshape(-1, 1)], axis=1)\n",
    "        columns = [\"feature_{}\".format(i) for i in range(X.shape[1])] + [\"label\"]\n",
    "        full_data = pd.DataFrame(data=full_data, columns=columns)\n",
    "        full_data.to_csv(\"./data/boston.csv\", index=False)\n",
    "        print(\"Dataset downloaded\")\n",
    "    ws = Workspace.from_config()\n",
    "    datastore = ws.get_default_datastore()\n",
    "    datastore.upload(\n",
    "        src_dir=\"./data\", target_path=\"data/new-metric-features/\", overwrite=True\n",
    "    )\n",
    "    data = Dataset.Tabular.from_delimited_files(\n",
    "        path=[(datastore, (\"data/new-metric-features/boston.csv\"))]\n",
    "    )\n",
    "    train, test = data.random_split(percentage=0.8, seed=101)\n",
    "    validation, test = test.random_split(percentage=0.5, seed=47)\n",
    "    return train, validation, test, \"label\"\n",
    "\n",
    "\n",
    "boston_train, boston_validation, boston_test, label_column_name = load_regression_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    \"primary_metric\": \"normalized_root_mean_squared_error\",\n",
    "    \"enable_early_stopping\": True,\n",
    "    \"max_concurrent_iterations\": 6,\n",
    "    \"experiment_timeout_hours\": 0.25,\n",
    "    \"verbosity\": logging.INFO,\n",
    "}\n",
    "\n",
    "automl_config = AutoMLConfig(\n",
    "    task=\"regression\",\n",
    "    debug_log=\"automl_errors.log\",\n",
    "    compute_target=compute_target,\n",
    "    training_data=boston_train,\n",
    "    validation_data=boston_validation,\n",
    "    test_data=boston_test,  # if you only have a test set, you can pass validation set here, instead of at validation_data\n",
    "    label_column_name=label_column_name,\n",
    "    enable_metric_confidence=True,\n",
    "    **automl_settings\n",
    ")\n",
    "\n",
    "regression_run = experiment.submit(automl_config, show_output=False)\n",
    "regression_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_run, fitted_model = regression_run.get_output()\n",
    "test_run = next(best_run.get_children(type=\"automl.model_test\"))\n",
    "test_run.wait_for_completion(show_output=False, wait_post_processing=True)\n",
    "test_metrics = test_run.get_metrics()\n",
    "\n",
    "CIs = {\"metric_name\": [], \"lower_ci_95\": [], \"upper_ci_95\": [], \"value\": []}\n",
    "\n",
    "for key, ci in test_metrics.items():\n",
    "    if key.endswith(\"extras\"):\n",
    "        CIs[\"metric_name\"].append(key[:-7])  # remove \"_extras\" to get metric name\n",
    "        for ci_key, ci_value in ci.items():\n",
    "            CIs[ci_key].append(ci_value)\n",
    "\n",
    "pd.DataFrame(CIs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](imgs/regression-confidence-interval.png)"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "lifengwei"
   }
  ],
  "category": "tutorial",
  "compute": [
   "AML Compute"
  ],
  "datasets": [
   "Digits",
   "Boston"
  ],
  "deployment": [
   "None"
  ],
  "exclude_from_index": false,
  "file_extension": ".py",
  "framework": [
   "None"
  ],
  "friendly_name": "New metric features in Azure AutoML",
  "index_order": 5,
  "interpreter": {
   "hash": "cc0892e042a269bcf4aec58f0c86eb5e2be478ff7be4e5f6b2680e2af1718f2e"
  },
  "kernelspec": {
   "display_name": "Python 3.7.0 64-bit ('pypi': conda)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "mimetype": "text/x-python",
  "name": "python",
  "nbconvert_exporter": "python",
  "pygments_lexer": "ipython3",
  "tags": [
   "remote_run",
   "AutomatedML"
  ],
  "task": "Classification",
  "version": "3.6.7"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}