Classification_common.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qULp_oix0Fv5",
    "tags": []
   },
   "source": [
    "## Preface and TOC\n",
    "This script is meant to classify a given polygone-based file containing footprint data of buildings of Hue City, Vietnam. \n",
    "The buildings will be classified to one of the following classes:\n",
    "\n",
    "- Shophouse `sh`\n",
    "- Single Family `sf`\n",
    "- Multi Family `mf`\n",
    "- Uninhabited Large `ul`\n",
    "- Uninhabited Small `us`\n",
    "\n",
    "**Table of content**\n",
    "\n",
    "\n",
    "1. [Head: In the first section all the variables and functions are defined and the input-data will be preprocessed](#head)\n",
    "\n",
    "    1.1. [Declaration of variables](#vars)\n",
    "    \n",
    "    1.2. [Declaration of functions](#functions)\n",
    "\n",
    "2. [Main Script](#main)\n",
    "\n",
    "    2.1 [First step: In the main-section the basic parameters are derived from the footprints](#first)\n",
    "    \n",
    "    2.1.1. [Addition of training classes: The given training classes are being added to the previously created dataset for external use in Orange's DT-classifier](#add_train)\n",
    "    \n",
    "    2.1.2. [First classification: The first three classes get classified](#class1)\n",
    "\n",
    "    2.2. [Second step: Derivation of the second step's parameters](#second)\n",
    "\n",
    "    2.2.1. [Second classification: The until here unclassified polygons are being classified](#class2)\n",
    "\n",
    "    2.2.2. [Finalization: The new parameters as well as the new classes are being merged to the dataset, created in 2.1.2. and written to HD](#final)\n",
    "\n",
    "3. [Accuracy assessment: Confusion matrices for accuracy measurements](#acc)\n",
    "\n",
    "4. [Statistics: Statistical evaluations](#statistics)\n",
    "\n",
    "    4.1. [Feature counts: Simple display of the counts of single classes](#feature_counts)\n",
    "\n",
    "    4.2. [Ward related statistics: Counting building types in wards](#wards)\n",
    "\n",
    "    4.3. [Boxplots and histograms: Generation of boxplots and histograms for each parameter, devided by classes](#boxplots)\n",
    "\n",
    "    4.4. [Residential Housings: Statistical evaluation of housings per floor area](#resi)\n",
    "\n",
    "    4.5. [Calculation of parameters' mean values and standard deviation per class](#means)\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"head\"></a>\n",
    "# Head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "xOVrdJmf3G51",
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning: 'conda-forge' already in 'channels' list, moving to the top\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting package metadata (current_repodata.json): ...working... done\n",
      "Solving environment: ...working... done\n",
      "\n",
      "# All requested packages already installed.\n",
      "\n",
      "Collecting package metadata (current_repodata.json): ...working... done\n",
      "Solving environment: ...working... done\n",
      "\n",
      "# All requested packages already installed.\n",
      "\n",
      "Collecting package metadata (current_repodata.json): ...working... done\n",
      "Solving environment: ...working... done\n",
      "\n",
      "# All requested packages already installed.\n",
      "\n",
      "Collecting package metadata (current_repodata.json): ...working... done\n",
      "Solving environment: ...working... done\n",
      "\n",
      "# All requested packages already installed.\n",
      "\n",
      "Collecting package metadata (current_repodata.json): ...working... done\n",
      "Solving environment: ...working... done\n",
      "\n",
      "# All requested packages already installed.\n",
      "\n",
      "Collecting package metadata (current_repodata.json): ...working... done\n",
      "Solving environment: ...working... done\n",
      "\n",
      "# All requested packages already installed.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# import platform\n",
    "# if platform.system() == 'Windows':\n",
    "#     !conda create -y -n Hue_classification\n",
    "#     !conda activate Hue_classification\n",
    "\n",
    "###\n",
    "# NOTE: momepy works better when installed from conda-forge\n",
    "# installation via conda in Google Colab requires condacolab installed\n",
    "# -> condacolab by Jaime Rodríguez-Guerra: https://github.com/jaimergp/condacolab\n",
    "# Not neccessary in Jupyter Notebook\n",
    "# !pip install -q condacolab\n",
    "# import condacolab\n",
    "# condacolab.install()\n",
    "###\n",
    "\n",
    "import sys\n",
    "!conda config --env --add channels conda-forge\n",
    "!conda config --env --set channel_priority strict\n",
    "!conda install -c conda-forge --yes --prefix {sys.prefix} geopandas\n",
    "!conda install -c conda-forge --yes --prefix {sys.prefix} pandas\n",
    "!conda install -c conda-forge --yes --prefix {sys.prefix} networkx\">=2.5\"\n",
    "!conda install -c conda-forge --yes --prefix {sys.prefix} momepy \n",
    "!conda install -c conda-forge --yes --prefix {sys.prefix} plotnine\n",
    "!conda install -c conda-forge --yes --prefix {sys.prefix} dask\n",
    "#%reset #clears all variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 2875,
     "status": "ok",
     "timestamp": 1623580259676,
     "user": {
      "displayName": "Bernhard Ebersbach",
      "photoUrl": "",
      "userId": "00083767612769780957"
     },
     "user_tz": -120
    },
    "id": "cKvEqs-vn2iQ",
    "outputId": "3d87fe8c-9717-43a4-8da5-ca3424d27503"
   },
   "outputs": [],
   "source": [
    "import momepy as mp\n",
    "import geopandas as gpd\n",
    "import pandas as pd\n",
    "import time \n",
    "import numpy as np\n",
    "from plotnine import *\n",
    "import sklearn.metrics as metrics\n",
    "import warnings\n",
    "import os\n",
    "from tqdm import tqdm\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f30TerRxP7VW"
   },
   "source": [
    "<a name=\"vars\"></a>\n",
    "## Declaration of variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 546,
     "status": "ok",
     "timestamp": 1623580262575,
     "user": {
      "displayName": "Bernhard Ebersbach",
      "photoUrl": "",
      "userId": "00083767612769780957"
     },
     "user_tz": -120
    },
    "id": "8sCZdKkNP6fd",
    "outputId": "93d3059b-5873-457f-baca-875026cad8a8",
    "tags": []
   },
   "outputs": [],
   "source": [
    "###### Variables ######\n",
    "\n",
    "update_mode = False  # if True, outfile_first will be loaded instead of infile\n",
    "precision = 5  # precision of calculated values\n",
    "do_tessellation = True  # calculate time demanding tessellation\n",
    "do_enclosed_tess = False  # enclosed tessellation makes only sense if the given data is of higher quality\n",
    "do_preprocessing = True  # time demandeng MP-preprocessing\n",
    "minsize = 2  # buildings smaller than this will be cut out by mp.preprocess()\n",
    "gpd.options.use_pygeos = True  # using PYGEOS speeds up certain operations\n",
    "start_time = time.time()  # starting time for time measurement\n",
    "os.chdir('C:/users/berbac/Documents/Jupyter_test')  # sets the working directory\n",
    "if not os.path.exists('./Output'): # if there is no subdirectory named 'Output' create one\n",
    "    os.mkdir('./Output')\n",
    "    os.mkdir('./Output/Daten')\n",
    "    os.mkdir('./Output/Klassifikation')\n",
    "    os.mkdir('./Output/Statistik')\n",
    "    \n",
    "# data input\n",
    "infile = './Input/Nha_buildings_2000_UTM48N.shp'\n",
    "roads_in = './Input/Hue_osm_roads_free.shp'\n",
    "water_in = './Input/Hue_osm_water_free.shp'\n",
    "boundaries_in = './Input/gadm36_VNM_3.shp'\n",
    "training_classes_in = None #\"./Input/Hue_training_classes_no_params.csv\"\n",
    "landuse_in = None  # './Input/Hue_gis_osm_landuse_a_free.shp'\n",
    "townhall_in = None  # './Input/Hue_town_hall.shp'\n",
    "max_density_spot_in = None  # './Input/Daten/Hue_max_density_spot.shp'\n",
    "k_dense_in = None  # './Input/Hue_kernal_density_1km.csv'\n",
    "reference_classes = None # './Input/Hue_reference_classes_preprocessed.csv'\n",
    "\n",
    "# data output\n",
    "infile_preprocessed = './Input/Hue_buildings_UTM48N_preprocessed.shp' \n",
    "outfile_first = './Output/Daten/Hue_buildings_UTM48N_preprocessed_first_no_classes.shp'\n",
    "data_w_training_classes_out = \"./Output/Daten/Hue_buildings_UTM48N_preprocessed_edited_w_training_classes.shp\"\n",
    "outfile_classified = './Output/Daten/Hue_buildings_UTM48N_classified_full.shp'\n",
    "tess_out = './Output/Daten/Hue_tessellation.shp'\n",
    "tess_encl_out = './Output/Daten/Hue_tessellation_enclosed.shp'\n",
    "outfile_blocks = './Output/Daten/Hue_blocks.shp'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Un3RxbnGQAbQ"
   },
   "source": [
    "<a name=\"functions\"></a>\n",
    "## Declaration of functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 28310,
     "status": "ok",
     "timestamp": 1623580293953,
     "user": {
      "displayName": "Bernhard Ebersbach",
      "photoUrl": "",
      "userId": "00083767612769780957"
     },
     "user_tz": -120
    },
    "id": "Zvl1lbX3bL-C",
    "outputId": "d9b18797-3d26-4884-b268-21f21532ba5c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Importing data...\n",
      "Preprocessing...\n",
      "Loop 1 out of 2.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cbf9e0c4cfd744f1818666603775170c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Identifying changes:   0%|          | 0/114324 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2ec425e5fb49412aba90ca7e66e82e0d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Changing geometry:   0%|          | 0/277 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loop 2 out of 2.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b722b1ecb8024b56aaf99fb2418847d1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Identifying changes:   0%|          | 0/114062 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "88af10bd355f45b79c7f1be50cfe8361",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Changing geometry:   0%|          | 0/22 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collapsed features  : 0\n",
      "Split features      : 19\n",
      "Overlapping features: 39\n",
      "58 features with erroneus geometry were dropped from the GeoDataFrame.\n",
      "1 duplicated features were dropped.\n"
     ]
    }
   ],
   "source": [
    "###### Functions ######\n",
    "\n",
    "def reproject(shp):\n",
    "    \"\"\"\n",
    "    Converts shp to metric CRS if neccesary    \n",
    "    \"\"\"\n",
    "    if shp.crs != 'EPSG:32648':\n",
    "        print(\"Reprojecting to EPSG 32648\")\n",
    "        shp = shp.to_crs(\"EPSG:32648\")\n",
    "    return shp\n",
    "\n",
    "\n",
    "def msize(gdf, radius, ignore_current=True):\n",
    "    \"\"\"\n",
    "    Calculates the mean footprint area of surrounding buildings within a given \n",
    "    range, excluding the current buildings area.\n",
    "    Adds 1 new column to the given GeoDataFrame.\n",
    "    \"\"\"\n",
    "\n",
    "    for i, row in tqdm(gdf.iterrows(), total=gdf.shape[0]):\n",
    "        idx = data.sindex.query(row.geometry.buffer(radius), predicate='contains')\n",
    "        if ignore_current:\n",
    "            idx = np.delete(idx, np.where(idx == i))\n",
    "        if len(idx) == 0:\n",
    "            gdf.loc[i, 'msize_' + str(radius)] = 0\n",
    "            continue\n",
    "        else:\n",
    "            gdf.loc[i, 'msize_' + str(radius)] = sum(gdf.iloc[idx].area) / len(idx)\n",
    "    return gdf\n",
    "\n",
    "\n",
    "def mabr(gdf):\n",
    "    \"\"\"\n",
    "    Calculates area, width and length for each opjects \n",
    "    minimum area bounding rectangle in the given gdf.\n",
    "    Adds 3 new columns to the given GeoDataFrame.\n",
    "    \"\"\"\n",
    "    for i, row in tqdm(gdf.iterrows(), total=gdf.shape[0]):\n",
    "        mabr = row.geometry.minimum_rotated_rectangle\n",
    "        gdf.loc[i, 'area_mabr'] = mabr.area\n",
    "        gdf.loc[i, 'width'] = mabr.minimum_clearance\n",
    "        gdf.loc[i, 'length'] = (mabr.length - (2 * mabr.minimum_clearance)) / 2\n",
    "\n",
    "    return gdf\n",
    "\n",
    "\n",
    "def mean_orientation(gdf, radius, ignore_current=True):\n",
    "    \"\"\"\n",
    "    Calculates building's orientations. Then, calculates mean deviation\n",
    "    of each building from surrounding buildings within the given range.\n",
    "    Adds 3 new columns to the given GeoDataFrame.\n",
    "    \"\"\"\n",
    "    if 'orient' not in gdf:\n",
    "        print(\"\\n\")\n",
    "        gdf['orient'] = mp.Orientation(gdf, verbose=True).series\n",
    "\n",
    "    for i, row in tqdm(gdf.iterrows(), total=gdf.shape[0]):\n",
    "        idx_orient = gdf.sindex.query(row.geometry.buffer(radius),\n",
    "                                      predicate='intersects')\n",
    "        if ignore_current:\n",
    "            idx_orient = np.delete(idx_orient,\n",
    "                                   np.where(idx_orient == i))\n",
    "        sum_orient = sum(gdf.iloc[idx_orient].orient)\n",
    "        try:\n",
    "            gdf.loc[i, 'orient_' + str(radius)] = sum_orient / len(idx_orient)\n",
    "        except ZeroDivisionError:\n",
    "            gdf.loc[i, 'orient_' + str(radius)] = row.orient\n",
    "    gdf['orient_dev'] = abs(gdf.orient - gdf['orient_'+ str(radius)])\n",
    "\n",
    "    return gdf\n",
    "\n",
    "\n",
    "print(\"Importing data...\")\n",
    "if update_mode:\n",
    "    data = reproject(gpd.read_file(outfile_first))\n",
    "    data.FID = data.FID.astype(np.int64)\n",
    "else:\n",
    "    data = reproject(gpd.read_file(infile))\n",
    "\n",
    "####### Preprocessings and preparations ######\n",
    "\n",
    "if 'FID' not in data:\n",
    "    data['FID'] = mp.unique_id(data)\n",
    "\n",
    "if do_preprocessing and not update_mode:\n",
    "    print(\"Preprocessing...\")\n",
    "    data = mp.preprocess(data, size=minsize)\n",
    "    defect = mp.CheckTessellationInput(data)\n",
    "    data = data.drop(defect.split.append(defect.overlap.append(defect.collapse)).index)\n",
    "    n_drop = data.shape[0] - data.FID.drop_duplicates().shape[0]\n",
    "    data.to_file(infile_preprocessed, driver='GeoJSON', encoding='utf-8')\n",
    "    print(\"{} features with erroneus geometry were dropped from the GeoDataFrame.\".format(\n",
    "        len(defect.split.append(defect.overlap.append(defect.collapse)))))\n",
    "    print(\"{} duplicated features were dropped.\".format(data.shape[0] - data.FID.drop_duplicates().shape[0]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FvW3fPXVz84o"
   },
   "source": [
    "<a name=\"main\"></a>\n",
    "# Main script\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qafjWOiBc7iY"
   },
   "source": [
    "<a name=\"first\"></a>\n",
    "## First step: generation of the base parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "cellView": "code",
    "colab": {
     "background_save": true
    },
    "id": "DOCbCTI26gZ-"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating areas...\n",
      "Calculating corners...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\berbac\\.conda\\envs\\Hue_common\\lib\\site-packages\\momepy\\shape.py:795: RuntimeWarning: invalid value encountered in arccos\n",
      "C:\\Users\\berbac\\.conda\\envs\\Hue_common\\lib\\site-packages\\momepy\\shape.py:794: RuntimeWarning: invalid value encountered in double_scalars\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating perimeters...\n",
      "Calculating rectangularity values...\n",
      "Calculating perimeter per area...\n",
      "Calculating shape index...\n",
      "Calculating length, width and area of MABRs...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████| 113982/113982 [02:33<00:00, 740.53it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating elongations...\n",
      "Calculating orientations...\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "26a47a55d5844781b260300ea3668a84",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/113982 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████| 113982/113982 [21:30<00:00, 88.35it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating shared walls...\n",
      "Evaluating proximity...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████| 113982/113982 [02:06<00:00, 901.62it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counting buildings within 50, 100, 200 m and 1 ha range...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████| 113982/113982 [06:02<00:00, 314.61it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preparing roads...\n",
      "Reprojecting to EPSG 32648\n",
      "Calculating distances to roads...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████| 113982/113982 [02:54<00:00, 653.85it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counting roads within 100 and 250 m range...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████| 113982/113982 [02:45<00:00, 687.95it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating mean area of buildings within 50 m and 100 m range\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████| 113982/113982 [21:12<00:00, 89.59it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 113982/113982 [22:10<00:00, 85.69it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating tessellation layer...\n",
      "Inward offset...\n",
      "Generating input point array...\n",
      "Generating Voronoi diagram...\n",
      "Generating GeoDataFrame...\n",
      "Dissolving Voronoi polygons...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\berbac\\.conda\\envs\\Hue_common\\lib\\site-packages\\momepy\\elements.py:380: UserWarning: Tessellation does not fully match buildings. 0 element(s) collapsed during generation - unique_id: set()\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing tessellation results to file...\n",
      "Adding tessellation related information: tess_area and CAR...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Adding \"tess_area\": 100%|█████████████████████████████████████████████████████| 113982/113982 [02:42<00:00, 701.13it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating blocks...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Adding \"bID\" to dataset: 100%|████████████████████████████████████████████████| 113982/113982 [02:13<00:00, 854.90it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating building's mean sizes and built up ratio within blocks...\n",
      "Writing output file...\n",
      "\n",
      "Process finished. It took 114 minutes.\n",
      "Output written to './Output/Daten/Hue_buildings_UTM48N_preprocessed_first_no_classes.shp.\n"
     ]
    }
   ],
   "source": [
    "###### Main script ######\n",
    "print(\"Calculating areas...\")\n",
    "data['area'] = data.geometry.area\n",
    "\n",
    "print(\"Calculating corners...\")\n",
    "data['corners'] = mp.Corners(data, verbose=False).series\n",
    "\n",
    "print(\"Calculating perimeters...\")\n",
    "data['perimeter'] = mp.Perimeter(data).series\n",
    "\n",
    "print(\"Calculating rectangularity values...\")\n",
    "data['rect'] = mp.Rectangularity(data, areas='area').series\n",
    "\n",
    "print(\"Calculating perimeter per area...\")\n",
    "data['shp_ratio'] = data.area / data.perimeter\n",
    "\n",
    "print(\"Calculating shape index...\")\n",
    "data['shp_index'] = data.perimeter / (2 * np.sqrt(np.pi * data.area))\n",
    "\n",
    "print(\"Calculating length, width and area of MABRs...\")\n",
    "data = mabr(data)\n",
    "\n",
    "print(\"Calculating elongations...\")\n",
    "data['elongation'] = data.width / data.length\n",
    "\n",
    "print(\"Calculating orientations...\")\n",
    "data = mean_orientation(data, 100)\n",
    "\n",
    "print(\"Calculating shared walls...\")\n",
    "data['shrd_walls'] = abs(np.round(mp.SharedWalls(data).series, 0))\n",
    "\n",
    "print(\"Evaluating proximity...\")\n",
    "for i, row in tqdm(data.iterrows(),\n",
    "                   total=data.shape[0]):\n",
    "    if row['shrd_walls'] == 0:\n",
    "        data.loc[i, 'proximity'] = 'detached'\n",
    "    elif row['shrd_walls'] <= int(row['length']):\n",
    "        data.loc[i, 'proximity'] = 'semi-detached'\n",
    "    else:\n",
    "        data.loc[i, 'proximity'] = 'attached'\n",
    "\n",
    "print(\"Counting buildings within 50, 100, 200 m and 1 ha range...\")\n",
    "for i, row in tqdm(data.iterrows(),\n",
    "                   total=data.shape[0]):\n",
    "    data.loc[i, 'n_in_50m'] = len(data.sindex.query(row.geometry.buffer(50),\n",
    "                                                    predicate='intersects'))\n",
    "\n",
    "    data.loc[i, 'n_in_100m'] = len(data.sindex.query(row.geometry.buffer(100),\n",
    "                                                     predicate='intersects'))\n",
    "\n",
    "    data.loc[i, 'n_in_200m'] = len(data.sindex.query(row.geometry.buffer(200),\n",
    "                                                     predicate='intersects'))\n",
    "\n",
    "    data.loc[i, 'n_in_ha'] = len(data.sindex.query(row.geometry.buffer(\n",
    "        np.sqrt(10000 / (2 * np.pi))),\n",
    "        predicate='intersects'))\n",
    "\n",
    "# Add landuse information based on OSM data\n",
    "if landuse_in is not None:\n",
    "    print(\"Adding OSM-landuse information...\")\n",
    "    landuse = reproject(gpd.read_file(landuse_in))\n",
    "    index_within = []\n",
    "    data['landuse'] = 'unknown'\n",
    "    for i in tqdm(range(len(landuse))):\n",
    "        index_within = data.geometry.within(landuse.geometry.iloc[i])\n",
    "        index_within = index_within[index_within]\n",
    "        for j in range(len(index_within)):\n",
    "            data.loc[index_within.index[j], 'landuse'] = landuse.loc[i, 'fclass']\n",
    "\n",
    "# Calculate distances to roads\n",
    "if roads_in is not None:\n",
    "    print(\"Preparing roads...\")\n",
    "    roads = reproject(gpd.read_file(roads_in))\n",
    "    roads = mp.extend_lines(roads, 0.25)\n",
    "    roads = mp.close_gaps(roads, 0.25)\n",
    "    roads = mp.remove_false_nodes(roads)\n",
    "    roads_union = roads.unary_union  # unifies all roads\n",
    "    print(\"Calculating distances to roads...\")\n",
    "    for i, row in tqdm(data.iterrows(),\n",
    "                       total=data.shape[0]):\n",
    "        data.loc[i, 'to_roads'] = row.geometry.boundary.distance(roads_union)\n",
    "\n",
    "# Roads in range\n",
    "print(\"Counting roads within 100 and 250 m range...\")\n",
    "for i, row in tqdm(data.iterrows(),\n",
    "                   total=data.shape[0]):\n",
    "    data.loc[i, 'roads_100'] = len(roads.geometry.sindex.query(\n",
    "        row.geometry.buffer(100), predicate='intersects'))\n",
    "    data.loc[i, 'roads_250'] = len(roads.geometry.sindex.query(\n",
    "        row.geometry.buffer(250), predicate='intersects'))\n",
    "\n",
    "# Caculate distances to to town hall\n",
    "if townhall_in is not None:\n",
    "    print(\"Calculating distances to town hall...\")\n",
    "    townhall = reproject(gpd.read_file(townhall_in))\n",
    "    data['to_hall'] = data.geometry.distance(townhall.loc[0, 'geometry'], align=False)\n",
    "\n",
    "if max_density_spot_in is not None:\n",
    "    print(\"Calculating distances to highest density spot...\")\n",
    "    max_density_spot = reproject(gpd.read_file(max_density_spot_in))\n",
    "    data['to_centre'] = data.geometry.distance(max_density_spot.loc[0, 'geometry'], align=False)\n",
    "\n",
    "# mean size of buildings in a given range \n",
    "# this is very time demanding\n",
    "print(\"Calculating mean area of buildings within 50 m and 100 m range\")\n",
    "data = msize(data, 50)\n",
    "data = msize(data, 100)\n",
    "\n",
    "if do_tessellation:\n",
    "    print(\"Calculating tessellation layer...\")\n",
    "    if not do_enclosed_tess:\n",
    "        limit = mp.buffered_limit(data)\n",
    "        tessellation = mp.Tessellation(data,\n",
    "                                       unique_id='FID',\n",
    "                                       limit=limit,\n",
    "                                       verbose=True\n",
    "                                       ).tessellationdsx\n",
    "        tessellation['tess_area'] = mp.Area(tessellation).series\n",
    "        print(\"Writing tessellation results to file...\")\n",
    "        tessellation.to_file(tess_out)\n",
    "\n",
    "    elif do_enclosed_tess and roads_in is not None and water_in is not None:\n",
    "        # if waters an roads are present do enclosed tessellation:\n",
    "        # enclosed tessellation needs roads and water information for the \n",
    "        # creation of boundaries. May cause problems when intersecting with buildings.\n",
    "        \n",
    "        roads = reproject(gpd.read_file(roads_in))\n",
    "        roads = mp.extend_lines(roads, 0.25)\n",
    "        roads = mp.close_gaps(roads, 0.25)\n",
    "        roads = mp.remove_false_nodes(roads)\n",
    "        water = reproject(gpd.read_file(water_in))\n",
    "        enclosures = mp.enclosures(roads, limit=gpd.GeoSeries(\n",
    "            [data.unary_union.convex_hull.buffer(10)]),\n",
    "                                   additional_barriers=[water.boundary])\n",
    "        tessellation = mp.Tessellation(data,\n",
    "                                       unique_id='FID',\n",
    "                                       enclosures=enclosures,\n",
    "                                       verbose=True\n",
    "                                       ).tessellation\n",
    "        tessellation['tess_area'] = tessellation.area\n",
    "        print(\"Writing tessellation results to file...\")\n",
    "        tessellation.to_file(tess_encl_out)\n",
    "\n",
    "    print(\"Adding tessellation related information: tess_area and CAR...\")\n",
    "    # in rare cases where only one building is situated within a bounded area\n",
    "    # this building won't be envolved in the tessellation. Thus the following \n",
    "    # statement hands the area of the untessellated cell to the building\n",
    "    \n",
    "    FID_not_tessellated = list(\n",
    "        data[data.FID.isin(tessellation[tessellation.FID.isna() == False].FID\n",
    "        ) == False].FID)\n",
    "    if len(FID_not_tessellated) > 0:\n",
    "        for i in tqdm(FID_not_tessellated,\n",
    "                      desc='Adding \"tess_area\" for single inbound buildings'):\n",
    "            tessellation.loc[\n",
    "                tessellation.sindex.query(\n",
    "                    data.loc[data[data.FID == i].index[0]].geometry,\n",
    "                    predicate='within')[0], 'FID'] = i\n",
    "    for i, row in tqdm(data.iterrows(), total=data.shape[0],\n",
    "                       desc='Adding \"tess_area\"'):\n",
    "        data.loc[i, 'tess_area'] = tessellation.loc[tessellation.FID == row.FID, \n",
    "                                                    'tess_area'].values\n",
    "    data['CAR'] = data.area / data.tess_area\n",
    "\n",
    "print(\"Generating blocks...\")\n",
    "blocks = mp.Blocks(tessellation=tessellation,\n",
    "                   edges=roads,\n",
    "                   buildings=data, \n",
    "                   id_name = 'bID',\n",
    "                   unique_id = 'FID')\n",
    "\n",
    "blocks.blocks.to_file(outfile_blocks, encoding='utf-8')\n",
    "\n",
    "for i, row in tqdm(data.iterrows(), total=data.shape[0], desc='Adding \"bID\" to dataset'):\n",
    "    data.loc[i, 'bID'] = blocks.blocks.sindex.query(data.loc[i].geometry,\n",
    "                                                    predicate='intersects')[0]\n",
    "\n",
    "print(\"Calculating building's mean sizes and built up ratio within blocks...\")\n",
    "data['b_msize'] = data.area\n",
    "data['b_builtup'] = 0\n",
    "data['b_medsize'] = data.area\n",
    "bID_set = list(set(data.bID))\n",
    "for i in range(len(bID_set)):\n",
    "    buildings_in_block = data.loc[data.bID == bID_set[i]]\n",
    "    data.loc[data.FID.isin(buildings_in_block.FID), \n",
    "             'b_msize'] = buildings_in_block.area.sum() / len(buildings_in_block)\n",
    "    data.loc[data.FID.isin(buildings_in_block.FID), \n",
    "             'b_medsize'] = buildings_in_block.area.median()\n",
    "    data.loc[data.FID.isin(buildings_in_block.FID), \n",
    "             'b_builtup'] = buildings_in_block.area.sum() / row.geometry.area\n",
    "\n",
    "# rounds all numerical values to the given precision\n",
    "data = np.round(data, precision)\n",
    "\n",
    "print('Writing output file...')\n",
    "data.to_file(outfile_first, encoding='utf-8')\n",
    "print(\"\\nProcess finished. It took {time} minutes.\\n\"\n",
    "      \"Output written to '{fname}.\".format(time=round((time.time() - start_time) / 60), fname=outfile_first))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "z026X_oBe4KQ"
   },
   "source": [
    "<a name=\"add_train\"></a>\n",
    "### Addition of taining classes\n",
    "Here, the training classes are being applied to the `data` GeoDataFrame. The output copy should only contain features with defined class in the `class_name` attribute. The `FID` attribute serves as identifier. The result is meant to be used during the training of the classifier. \n",
    "\n",
    "Inputs:\n",
    "\n",
    "`Hue_training_classes_no_params.csv` file containing the training classes\n",
    "\n",
    "`outfile_first` should be the same as `data` but is read from the hd just in case `data` doesn't exist\n",
    "\n",
    "Outputs: \n",
    "\n",
    "`Hue_buildings_UTM48N_preprocessed_edited_w_training_classes.shp`\n",
    "\n",
    "`Hue_buildings_UTM48N_preprocessed_edited_w_training_classes.shp.csv`\n",
    "\n",
    "`Hue_training_classes_w_params.csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "pGhucYx-esDh"
   },
   "outputs": [],
   "source": [
    "###### Add classes ######\n",
    "if training_classes_in:\n",
    "    classes = gpd.read_file(training_classes_in)\n",
    "    classes.FID = classes.FID.astype(np.int64)\n",
    "    data_w_training_classes = gpd.read_file(outfile_first)\n",
    "    data_w_training_classes.FID = data_w_training_classes.FID.astype(np.int64)\n",
    "    for i, row in classes.iterrows():\n",
    "        data_w_training_classes.loc[data_w_training_classes.FID == row.FID,\n",
    "                                    'class_name'] = row['class_name']\n",
    "\n",
    "    # write only features with set class_name as csv\n",
    "    data_w_training_classes[data_w_training_classes.class_name.isnull() == False].to_csv(\n",
    "        \"./Output/Klassifikation/Hue_1st_step_training_classes_w_params.csv\")\n",
    "\n",
    "    # write the whole data table as shapefile and csv\n",
    "    data_w_training_classes.to_file(data_w_training_classes_out, encoding='utf-8')\n",
    "    data_w_training_classes.loc[data_w_training_classes.class_name.isnull() == False, 'class_name']\n",
    "    data_w_training_classes.to_csv(data_w_training_classes_out+\".csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9Y_QqwfKIEVp"
   },
   "source": [
    "<a name=\"class1\"></a>\n",
    "### First classification\n",
    "\n",
    "The new dataset `data_w_classes` is created here. It contains all the information of `data` (parameters). \n",
    "In this first step the classes `us`, `sh` and `sf` are beeing added. The thresholds where calculated with help of the Orange decision tree classification.\n",
    "\n",
    "Input:\n",
    "\n",
    "`outfile_first`-defined file containing all attributes\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "Ha-pxUC7iqT8"
   },
   "outputs": [],
   "source": [
    "data_w_classes = gpd.read_file(outfile_first)\n",
    "data_w_classes.FID = data_w_classes.FID.astype(np.int64)\n",
    "\n",
    "data_w_classes.loc[data_w_classes.area <= 30, 'class_name'] = 'us'\n",
    "\n",
    "data_w_classes.loc[ (data_w_classes.class_name != 'us') &\n",
    "                    (\n",
    "                        (data_w_classes.area > 30) &\n",
    "                        (data_w_classes.area_mabr <= 141.80) &\n",
    "                        (data_w_classes.elongation <= 0.17) &\n",
    "                        (data_w_classes.b_msize <= 120.15)\n",
    "                    ),\n",
    "                    'class_name'] = 'sh'\n",
    "\n",
    "data_w_classes.loc[ (data_w_classes.class_name != 'us') &\n",
    "                    (data_w_classes.class_name != 'sh') &\n",
    "                    (\n",
    "                        (data_w_classes.area > 30) &\n",
    "                        (data_w_classes.area_mabr <= 141.80) &\n",
    "                        (data_w_classes.elongation > 0.17)\n",
    "                    ),\n",
    "                    'class_name'] = 'sf'\n",
    "\n",
    "data_w_classes.loc[data_w_classes.class_name.isnull() == True, 'class_name'] = 'mf_ul'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gkeAwUg6b_M9"
   },
   "source": [
    "<a name=\"second\"></a>\n",
    "## Second step: generation of further parameters\n",
    "After `sh`, `us` and `sf` have been classified, a subset containing only the potential mf and ul classes is generated here. The following will be done:\n",
    "\n",
    "*   Generate subset `data_mful` \n",
    "*   Calculate additional parameters based on `data_w_classes` from the first step\n",
    "*   Add training class names\n",
    "*   Save `./Output/Daten/Hue_buildings_UTM48N_preprocessed_classes_2nd_step.csv` containing new subset\n",
    "*   Save `./Output/Klassifikation/Hue_2nd_step_training_classes_w_params.csv` containing second steps training classes w/ params\n",
    "*   Save `./Output/Klassifikation/Hue_reference_classes_preprocessed_mful_w_params.csv` containing only `mf` and `ul` reference classes with params\n",
    "\n",
    "Note: These operations can be done using the index because `data_mful` inherits it from `data_w_classes`. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 281,
     "referenced_widgets": [
      "201a57ace6c14df68f078c6db7411c2e",
      "d524c1c6f0aa45a787571ddfde83f68a",
      "d2ff2ee31b1b4f19bccb592657fbafba",
      "7c509b8e62224a36af1311f3838a82bf",
      "046afab5c04643cfba91480444490605",
      "d0802212b19043c3be7b4d9235dcc495",
      "8f1f870dad824d059540b879488a4ae5",
      "1c2af11f863a4e89b769f13c24ccdcbc",
      "f58106c95326412db9edf3eefbc89140",
      "59442c7e7a4f4ef4928e1b3f7cd71843",
      "16d1d3418e3a47e4aef4ff040c1f0a64",
      "6ac3313a05884052b86ce3dd001ff160",
      "166973b412084481849fc6b37c431494",
      "b6bcc204e49e486ba2567c556069bd91",
      "360d95d8a9114dc3a2117da84a5f81a2",
      "47053e3b47d74f5ea758c34e3a07a76b",
      "246d31140fd74d7ab95d22fc2b276987",
      "6a9f009e7a9f4019a51dd6ad0adf9b9d",
      "552afccb92824ee99784496a51c89561",
      "356f0e34ef7d446992147c1450868b0c",
      "512cc211d14548b5b90adce91b7a0f5e",
      "02248b97e95e471896dec3c2c2445c2c",
      "eae35f5a58894102b5e0d4d4efbf8d66",
      "bb0c7671947e424e884441b31983fd2b",
      "e14e97bc5c15410385ffcc93c67ca4f1",
      "054ccda05e8248bc97004e0950053b3a",
      "65b11cea931f425eb0ee046ed35ccef0",
      "c7f2dc2a44e6440cb282be20c57d3017",
      "624d5c75304d4db3923b438a33774434",
      "3280f822d845445abb1f49788ce46b5a",
      "aae6bd861fb54b3ebef1e5acdde347c1",
      "2c7f4be35c9143459746b4d16a529381"
     ]
    },
    "executionInfo": {
     "elapsed": 282410,
     "status": "ok",
     "timestamp": 1623584366946,
     "user": {
      "displayName": "Bernhard Ebersbach",
      "photoUrl": "",
      "userId": "00083767612769780957"
     },
     "user_tz": -120
    },
    "id": "U5bLbv21v4pS",
    "outputId": "34a4e9f6-9125-4bfd-c67d-08667aa0dc9e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieving most common class within 50 m range...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████| 5154/5154 [00:58<00:00, 88.41it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieving nearest building's class...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████| 5154/5154 [01:02<00:00, 82.69it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieving most common class and share of mf_ul classes within blocks...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████| 1227/1227 [00:41<00:00, 29.65it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counting mf and ul classes within 50 m range...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████| 5154/5154 [00:02<00:00, 1830.04it/s]\n"
     ]
    }
   ],
   "source": [
    "data_w_classes_no_us = data_w_classes.copy()\n",
    "data_mful = gpd.GeoDataFrame(data_w_classes[data_w_classes.class_name == 'mf_ul'])\n",
    "\n",
    "# with or without centroid makes a big difference here\n",
    "print(\"Retrieving most common class within 50 m range...\")\n",
    "for i, row in tqdm(data_mful.iterrows(), total=data_mful.shape[0]):\n",
    "    found_classes = data_w_classes_no_us.iloc[data_w_classes_no_us.sindex.query(\n",
    "        row.geometry.buffer(50),\n",
    "        predicate='intersects')\n",
    "        ].class_name\n",
    "    if len(found_classes) > 0:\n",
    "        data_mful.loc[i, 'class_50'] = Counter(found_classes).most_common(1)[0][0]\n",
    "    else:\n",
    "        data_mful.loc[i, 'class_50'] = 'mf_ul'\n",
    "\n",
    "print(\"Retrieving nearest building's class...\")\n",
    "for i, row in tqdm(data_mful.iterrows(), total=data_mful.shape[0]):\n",
    "    with warnings.catch_warnings():  # suppresses overflow warning in the distance process\n",
    "       warnings.simplefilter(action='ignore', category=Warning) \n",
    "       nearest = data_w_classes_no_us.iloc[data_w_classes_no_us.sindex.query(\n",
    "            row.geometry.buffer(50),\n",
    "            predicate='intersects')].geometry.distance(row.geometry)\n",
    "       nearest = nearest.drop(index=i)  # remove current object from lcolab_classesist\n",
    "    if len(nearest) == 0:\n",
    "        continue\n",
    "    # if there are more than one dist = 0 classes \n",
    "    # and at least one of them is mf_ul, set mf_ul as nearest\n",
    "    else: \n",
    "        if len(nearest[nearest == 0]) > 1 and 'mf_ul' in data_w_classes_no_us.loc[\n",
    "                nearest[nearest == 0].index, 'class_name'].values:\n",
    "            data_mful.loc[i, 'nearest'] = 'mf_ul'\n",
    "            continue\n",
    "        nearest = data_w_classes_no_us.loc[nearest[nearest == min(nearest)\n",
    "                                                   ].index[0], 'class_name']\n",
    "        if type(nearest) is not str:\n",
    "            data_mful.loc[i, 'nearest'] = 'mf_ul'\n",
    "        else: # if nearest \n",
    "            data_mful.loc[i, 'nearest'] = nearest\n",
    "\n",
    "bID_set = list(set(data_w_classes_no_us.bID))\n",
    "print(\"Retrieving most common class and share of mf_ul classes within blocks...\")\n",
    "for i in tqdm(bID_set):\n",
    "    buildings_in_block = data_w_classes_no_us[data_w_classes_no_us.bID == i]\n",
    "    data_mful.loc[data_mful.FID.isin(buildings_in_block.FID), 'b_class'] = \\\n",
    "        Counter(buildings_in_block.class_name).most_common(1)[0][0]\n",
    "\n",
    "    b_mful_rat = np.round(\n",
    "        len(buildings_in_block[buildings_in_block.class_name == 'mf_ul']) / \\\n",
    "        len(data[data.bID == buildings_in_block.iloc[0].bID]), precision)\n",
    "    data_mful.loc[data_mful.FID.isin(buildings_in_block.FID), 'b_mful_rat'] = b_mful_rat\n",
    "\n",
    "print(\"Counting mf and ul classes within 50 m range...\")\n",
    "for i, row in tqdm(data_mful.iterrows(), total=data_mful.shape[0]):\n",
    "    mful_idx = data_mful.sindex.query(row.geometry.buffer(50), predicate='intersects')\n",
    "    mful_idx = np.delete(mful_idx, np.where(mful_idx == i))\n",
    "    data_mful.loc[i, 'mful_50m'] = len(mful_idx)\n",
    "\n",
    "# write whole dataset containing mf and ul classes without training classes\n",
    "data_mful.to_csv(\"./Output/Daten/Hue_buildings_UTM48N_preprocessed_classes_2nd_step.csv\")\n",
    "\n",
    "# write 2nd step's training classes\n",
    "if training_classes_in:\n",
    "    classes = gpd.read_file(training_classes_in)\n",
    "    classes.FID = classes.FID.astype(np.int64)\n",
    "    data_mful.FID = data_mful.FID.astype(np.int64)\n",
    "    try:\n",
    "        data_mful_training_w_params = data_mful.drop('class_name', axis=1)\n",
    "    except KeyError:\n",
    "        pass\n",
    "    data_mful_training_w_params = data_mful_training_w_params.merge(classes[['class_name', 'FID']],\n",
    "                                                                    on='FID')  # ,'class_info'\n",
    "    data_mful_training_w_params.to_csv(\"./Output/Klassifikation/Hue_2nd_step_training_classes_w_params.csv\")\n",
    "\n",
    "# write new mful-reference classes with params to file\n",
    "if reference_classes:\n",
    "    ref = gpd.read_file(reference_classes)\n",
    "    ref.FID = ref.FID.astype(np.int64)\n",
    "    try:\n",
    "        data_mful_ref = ref.merge(data_mful.drop(columns='class_name', axis=1), on='FID')\n",
    "    except KeyError:\n",
    "        data_mful_ref = ref.merge(data_mful, on='FID')\n",
    "    data_mful_ref.to_csv('./Output/Klassifikation/Hue_reference_classes_preprocessed_mful_w_params.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3kHDklAppb_H"
   },
   "source": [
    "<a name=\"class2\"></a>\n",
    "### Second classification\n",
    "Application of the DT-threhsold-based ruleset to classify `mf` and `ul` classes to the `data_mful` subset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "m6tP7wNK9j4w"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    data_mful = data_mful.drop(columns='class_name')\n",
    "except KeyError:\n",
    "    pass\n",
    "data_mful['class_name'] = None\n",
    "\n",
    "data_mful.loc[  (data_mful.class_name.isnull() == True) &\n",
    "                (data_mful.msize_50 <= 113.07) &\n",
    "                (\n",
    "                    (data_mful.nearest == 'sf') |\n",
    "                    (\n",
    "                        (\n",
    "                            (data_mful.tess_area <= 793.73) &\n",
    "                            (data_mful.b_class != 'sh')\n",
    "                        ) |\n",
    "                        (\n",
    "                            (data_mful.tess_area > 793.73) &\n",
    "                            # (data_mful.CAR <= 0.33) &\n",
    "                            (data_mful.perimeter <= 71.06)\n",
    "                        )\n",
    "                    )\n",
    "                ), 'class_name'] = 'mf'\n",
    "\n",
    "data_mful.loc[  (data_mful.msize_50 > 113.07) |\n",
    "                (data_mful.msize_50 <= 113.07) &\n",
    "                (data_mful.nearest != 'sf') & \n",
    "                (\n",
    "                    (\n",
    "                        (\n",
    "                            (data_mful.tess_area <= 793.73) &\n",
    "                            (data_mful.b_class == 'sh')\n",
    "                        )\n",
    "                    ) |\n",
    "                        (\n",
    "                            (data_mful.tess_area > 793.73) &\n",
    "                            (data_mful.perimeter > 71.06)  \n",
    "                        )\n",
    "                ), 'class_name'] = 'ul'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7JjWhYklfDFT"
   },
   "source": [
    "<a name=\"final\"></a>\n",
    "### Merging new information with main dataset and finalization\n",
    "The following block adds the `mf` and `ul` classes as well as the parameters that were calculated in the second step from `data_mful` to the fully classified dataset. The final result is written to the  `outfile_classified`-defined file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 66,
     "referenced_widgets": [
      "1712924346bb45bcafb9980ae20d59dc",
      "ca9974d5ca6f4eb2a2109bd1f63ea366",
      "3419e230eb524e37a769a08d8ebc49fd",
      "e60950da1bb74809912c0bc4a41006fd",
      "424192e71ebf46499d5845abcc514eaf",
      "ef82828170014116b6b6b8db84ed635d",
      "4ebcf19af3f54658bc1afcae579d4edb",
      "91bfd5d22fec4f26aac1caccde5d182b"
     ]
    },
    "executionInfo": {
     "elapsed": 299900,
     "status": "ok",
     "timestamp": 1623584666840,
     "user": {
      "displayName": "Bernhard Ebersbach",
      "photoUrl": "",
      "userId": "00083767612769780957"
     },
     "user_tz": -120
    },
    "id": "XpjidScvb_HR",
    "outputId": "b3996616-6f74-4ac4-ee48-807a8f696863"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████| 5154/5154 [01:47<00:00, 47.81it/s]\n"
     ]
    }
   ],
   "source": [
    "data_classified_full = data_w_classes.copy()\n",
    "\n",
    "data_classified_full['nearest'] = None\n",
    "data_classified_full['mful_50m'] = None\n",
    "data_classified_full['b_class'] = None\n",
    "data_classified_full['b_mful_rat'] = None\n",
    "data_classified_full['class_50'] = None\n",
    "for i, row in tqdm(data_mful.iterrows(), total=data_mful.shape[0]):\n",
    "    index = data_w_classes[data_w_classes.FID == row.FID].index\n",
    "    data_classified_full.loc[index, 'nearest'] = row.nearest\n",
    "    data_classified_full.loc[index, 'mful_50m'] = row.mful_50m\n",
    "    data_classified_full.loc[index, 'b_class'] = row.b_class\n",
    "    data_classified_full.loc[index, 'class_name'] = row.class_name\n",
    "    data_classified_full.loc[index, 'b_mful_rat'] = row.b_mful_rat\n",
    "    data_classified_full.loc[index, 'class_50'] = row.class_50\n",
    "data_classified_full.to_file(outfile_classified, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "m5TN-phgnSrF"
   },
   "source": [
    "<a name=\"acc\"></a>\n",
    "# Accuracy assessment\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n_mbUN03wxVG"
   },
   "source": [
    "## Confusion matrices\n",
    "For the confusion matrix the module `sklearn.metrics` is needed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 463,
     "status": "ok",
     "timestamp": 1623584667292,
     "user": {
      "displayName": "Bernhard Ebersbach",
      "photoUrl": "",
      "userId": "00083767612769780957"
     },
     "user_tz": -120
    },
    "id": "AmFrIeQdxfFi",
    "outputId": "686cab4c-3218-45bb-bc85-47225cd44a7c"
   },
   "outputs": [],
   "source": [
    "# accuracy of first classification step on training data\n",
    "if training_classes_in:\n",
    "    training_classes_first = gpd.read_file(training_classes_in)\n",
    "    training_classes_first.FID = training_classes_first.FID.astype(np.int64)\n",
    "    training_classes_first.loc[(training_classes_first.class_name == 'mf') |\n",
    "                               (training_classes_first.class_name == 'ul'),\n",
    "                               'class_name'] = 'mf_ul'\n",
    "    if not 'data_classified_full' in globals():\n",
    "        data_classified_full = gpd.read_file(outfile_classified)\n",
    "        data_classified_full.FID = data_classified_full.FID.astype(np.int64)\n",
    "\n",
    "    data_classified_first = data_classified_full.copy()\n",
    "    data_classified_first.FID = data_classified_full.FID.astype(np.int64)\n",
    "    data_classified_first.loc[(data_classified_first.class_name == 'mf') |\n",
    "                              (data_classified_first.class_name == 'ul'),\n",
    "                              'class_name'] = 'mf_ul'\n",
    "\n",
    "    pred = data_classified_first[\n",
    "        data_classified_first.FID.isin(training_classes_first.FID)]\n",
    "\n",
    "    labels = list(set(pred.class_name))\n",
    "    confusion_matrix = metrics.confusion_matrix(training_classes_first.class_name, \n",
    "                                                pred.class_name, \n",
    "                                                normalize=None,\n",
    "                                                labels=labels)\n",
    "    print(\"Accuracy assessment of the first step on training data\")\n",
    "    metrics.ConfusionMatrixDisplay(confusion_matrix, display_labels=labels).plot(cmap='PuBu')\n",
    "\n",
    "    confusion_matrix\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 349,
     "status": "ok",
     "timestamp": 1623584667632,
     "user": {
      "displayName": "Bernhard Ebersbach",
      "photoUrl": "",
      "userId": "00083767612769780957"
     },
     "user_tz": -120
    },
    "id": "d-voJ8Bwg3bQ",
    "outputId": "afb4e993-e337-4202-eec5-3443f06249ed"
   },
   "outputs": [],
   "source": [
    "# accuracy of 2nd classification step on training data\n",
    "if training_classes_in:\n",
    "    training_classes_mful = gpd.read_file(training_classes_in)\n",
    "    training_classes_mful.FID = training_classes_mful.FID.astype(np.int64)\n",
    "    training_classes_mful = training_classes_mful[(training_classes_mful.class_name == 'mf') |\n",
    "                                                  (training_classes_mful.class_name == 'ul')]\n",
    "\n",
    "    if not 'data_classified_full' in globals():\n",
    "        data_classified_full = gpd.read_file(outfile_classified)\n",
    "        data_classified_full.FID = data_classified_full.FID.astype(np.int64)\n",
    "\n",
    "    pred = data_classified_full[data_classified_full.FID.isin(training_classes_mful.FID)]\n",
    "\n",
    "    labels = list(set(pred.class_name))\n",
    "    confusion_matrix = metrics.confusion_matrix(training_classes_mful.class_name, \n",
    "                                                pred.class_name, \n",
    "                                                normalize=None,\n",
    "                                                labels=labels)\n",
    "    print(\"Accuracy assessment of the 2nd step on training data\")\n",
    "    metrics.ConfusionMatrixDisplay(confusion_matrix, display_labels=labels).plot(cmap='PuBu')\n",
    "    confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 385
    },
    "executionInfo": {
     "elapsed": 241,
     "status": "ok",
     "timestamp": 1623584667865,
     "user": {
      "displayName": "Bernhard Ebersbach",
      "photoUrl": "",
      "userId": "00083767612769780957"
     },
     "user_tz": -120
    },
    "id": "Y3bPdVL3bt6L",
    "outputId": "f33e83ca-85bf-47e9-ae0e-a96b7a66ae57"
   },
   "outputs": [],
   "source": [
    "# accuracy test on reference data\n",
    "if training_classes_in and reference_classes:\n",
    "    ref = gpd.read_file(reference_classes)\n",
    "\n",
    "    ref.FID = ref.FID.astype(np.int64)\n",
    "    if not 'data_classified_full' in globals():\n",
    "        data_classified_full = gpd.read_file(outfile_classified)\n",
    "        data_classified_full.FID = data_classified_full.FID.astype(np.int64)\n",
    "\n",
    "    pred = data_classified_full[data_classified_full.FID.isin(ref.FID)]\n",
    "\n",
    "    labels = list(set(pred.class_name))\n",
    "    confusion_matrix = metrics.confusion_matrix(ref.class_name, \n",
    "                                                pred.class_name, \n",
    "                                                normalize=None, \n",
    "                                                labels=labels)\n",
    "    print(\"Accuracy assessment for the fully classified dataset on reference data\")\n",
    "    metrics.ConfusionMatrixDisplay(confusion_matrix, display_labels=labels).plot(cmap='PuBu')\n",
    "    confusion_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gTaI2jAn1Gqn"
   },
   "source": [
    "<a name=\"statistics\"></a>\n",
    "# Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_lVG22oGueJN"
   },
   "source": [
    "<a name=\"feature_count\"></a>\n",
    "## Feature counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 19668,
     "status": "ok",
     "timestamp": 1623584687529,
     "user": {
      "displayName": "Bernhard Ebersbach",
      "photoUrl": "",
      "userId": "00083767612769780957"
     },
     "user_tz": -120
    },
    "id": "ByyLSpV5h2mk",
    "outputId": "21391237-14b6-4cf0-e18c-3dbefe055948"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Statistics for ./Output/Daten/Hue_buildings_UTM48N_classified_full.shp\n",
      "us = 31783 \n",
      "sh = 20401 \n",
      "sf = 56644 \n",
      "mf = 3198 \n",
      "ul = 1956 \n",
      "------------ \n",
      "sum = 113982\n",
      "\n",
      "Statistics for {'mf', 'ul'}\n",
      "us = 0 \n",
      "sh = 0 \n",
      "sf = 0 \n",
      "mf = 3198 \n",
      "ul = 1956 \n",
      "------------ \n",
      "sum = 113982\n"
     ]
    }
   ],
   "source": [
    "def statistics(filename):\n",
    "    if type(filename) == str:\n",
    "        gdf = gpd.read_file(filename)\n",
    "        print(\"\\nStatistics for\", str(filename))\n",
    "    else:\n",
    "        gdf = filename\n",
    "        print(\"\\nStatistics for\", set(data_mful.class_name))\n",
    "    us = gdf[gdf.class_name == 'us'].shape[0]\n",
    "    sh = gdf[gdf.class_name == 'sh'].shape[0]\n",
    "    sf = gdf[gdf.class_name == 'sf'].shape[0]\n",
    "    mf = gdf[gdf.class_name == 'mf'].shape[0]\n",
    "    ul = gdf[gdf.class_name == 'ul'].shape[0]\n",
    "    len(data[data.area <= 25])\n",
    "    print('us =', us,\n",
    "          '\\nsh =', sh,\n",
    "          '\\nsf =', sf,\n",
    "          '\\nmf =', mf,\n",
    "          '\\nul =', ul,\n",
    "          '\\n------------',\n",
    "          '\\nsum =', len(data))\n",
    "\n",
    "if training_classes_in:\n",
    "    statistics(training_classes_in)\n",
    "if reference_classes:\n",
    "    statistics(reference_classes)\n",
    "\n",
    "statistics(outfile_classified)\n",
    "\n",
    "statistics(data_mful)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e_Bq641McmMV"
   },
   "source": [
    "<a name=\"wards\"></a>\n",
    "## Ward related statistics\n",
    "The file `./Output/Statistik/Hue_wards_w_stats.csv` is created here. Under use of a GADM-provided file containing the adminstrative boundaries of Vietnam, the count of each class per ward of Hue is extracted. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "6uAljY09oqoN"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reprojecting to EPSG 32648\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████| 27/27 [00:13<00:00,  2.05it/s]\n"
     ]
    }
   ],
   "source": [
    "# Adding wards based statistics\n",
    "# Counting building types in wards\n",
    "if boundaries_in:\n",
    "    wards = reproject(gpd.read_file('./Input/gadm36_VNM_3.shp'))\n",
    "    wards_hue = gpd.GeoDataFrame(wards[wards.NAME_2 == 'Huế'])\n",
    "    if not 'data_classified_full' in globals():\n",
    "        data_classified_full = gpd.read_file(outfile_classified)\n",
    "        data_classified_full.FID = data_classified_full.FID.astype(np.int64)\n",
    "\n",
    "    for i, row in tqdm(wards_hue.iterrows(), total=wards_hue.shape[0]):\n",
    "        for class_name in ['sh', 'sf', 'mf', 'ul', 'us']:\n",
    "            wards_hue.loc[i, 'n_'+class_name] = data_classified_full[\n",
    "                data_classified_full.class_name == class_name\n",
    "                ].centroid.sindex.query(\n",
    "                    row.geometry, predicate='intersects'\n",
    "                    ).shape[0]\n",
    "\n",
    "    wards_hue.to_csv('./Output/Statistik/Hue_wards_w_stats.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"boxplots\"></a>\n",
    "## Boxplots and histograms\n",
    "Boxplots and histograms for all the features are being generated here and written to `./Output/Statistik/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 96144,
     "status": "ok",
     "timestamp": 1623581094467,
     "user": {
      "displayName": "Bernhard Ebersbach",
      "photoUrl": "",
      "userId": "00083767612769780957"
     },
     "user_tz": -120
    },
    "id": "JHQjU-rJymRo",
    "outputId": "ac3b150d-e49d-40e4-b51d-33debba16851"
   },
   "outputs": [],
   "source": [
    "if reference_classes:\n",
    "    if not os.path.exists('./Output/Statistik'):\n",
    "        os.mkdir('./Output/Statistik')\n",
    "\n",
    "    width = 14\n",
    "    height = 12\n",
    "\n",
    "    #### Statistiks for the parameters of the first step ####\n",
    "\n",
    "    ref = gpd.read_file('./Input/Hue_reference_classes_preprocessed.csv')\n",
    "    ref.FID = ref.FID.astype(np.int64)\n",
    "\n",
    "    if not 'data_classified_full' in globals():\n",
    "        data_classified_full = gpd.read_file(outfile_classified)\n",
    "        data_classified_full.FID = data_classified_full.FID.astype(np.int64)\n",
    "\n",
    "    pred = data_classified_full\n",
    "    ref_w_params = pred.copy()\n",
    "\n",
    "    for colname in list(pred.columns):\n",
    "        if colname not in [ 'geometry', 'FID', 'bID', 'class_name',\n",
    "                            'nearest', 'class_50', 'b_class', \n",
    "                            'b_mful_rat', 'mful_50']: \n",
    "            (\n",
    "                ggplot(pred)\n",
    "                + geom_boxplot(\n",
    "                    aes(x='class_name', \n",
    "                        y=colname),\n",
    "                    colour=\"#1F3552\",\n",
    "                    fill=\"#4271AE\",\n",
    "                    alpha=0.7,\n",
    "                    outlier_shape=\".\",\n",
    "                    outlier_colour=\"steelblue\",\n",
    "                    width=0.5\n",
    "                    )\n",
    "                + ggtitle(colname)\n",
    "                + theme_bw()\n",
    "            ).save(\"./Output/Statistik/boxplot_{name}.svg\".format(name=colname), units='cm', width=width, height=height)\n",
    "\n",
    "    #### Statistiks for the parameters of the first step trimmed ####\n",
    "\n",
    "    if not os.path.exists('./Output/Statistik'):\n",
    "        os.mkdir('./Output/Statistik')\n",
    "\n",
    "    ref = gpd.read_file('./Input/Hue_reference_classes_preprocessed.csv')\n",
    "    ref.FID = ref.FID.astype(np.int64)\n",
    "\n",
    "    if not 'data_classified_full' in globals():\n",
    "        data_classified_full = gpd.read_file(outfile_classified)\n",
    "        data_classified_full.FID = data_classified_full.FID.astype(np.int64)\n",
    "\n",
    "    pred = data_classified_full\n",
    "    ref_w_params = pred.copy()\n",
    "\n",
    "    # format: [param_name, max_value]\n",
    "    for i in [\n",
    "              ['area', 1000], \n",
    "              ['area_mabr', 1000],\n",
    "              ['b_msize', 500], \n",
    "              ['msize_50', 500], \n",
    "              ['perimeter', 200],\n",
    "              ['tess_area', 4000]\n",
    "             ]:\n",
    "        (\n",
    "            ggplot(pred)\n",
    "            + geom_boxplot(\n",
    "                aes(x='class_name', \n",
    "                    y=i[0]),\n",
    "                    colour=\"#1F3552\",\n",
    "                    fill=\"#4271AE\",\n",
    "                    alpha=0.7,\n",
    "                    outlier_shape=\".\",\n",
    "                    outlier_colour=\"steelblue\",\n",
    "                    width=0.5\n",
    "                    )\n",
    "            + ggtitle(i[0])\n",
    "            + theme_bw()\n",
    "            + ylim(0, i[1])\n",
    "            + xlab(\"{title}\\n\\nNote: Outliers above {limit}\\nhave been omitted in the figure\".format(title='class_name', limit=i[1]))\n",
    "        ).save(\"./Output/Statistik/cut_boxplot_{name}.svg\".format(name=i[0]), units='cm', width=width, height=height)\n",
    "\n",
    "\n",
    "    #### Statistiks for the parameters of the second step ####\n",
    "\n",
    "    ref_2_w_params = pred[(pred.class_name == 'mf') | (pred.class_name == 'ul')].copy() \n",
    "\n",
    "    for colname in ['nearest', 'class_50', 'b_class']:\n",
    "        for mful in ['mf', 'ul']:\n",
    "            (\n",
    "                ggplot(ref_2_w_params[ref_2_w_params.class_name == mful])\n",
    "                + geom_histogram(aes(x=colname), \n",
    "                                    binwidth=0.5,\n",
    "                                    colour=\"#1F3552\",\n",
    "                                    fill=\"#4271AE\",\n",
    "                                    alpha=0.7,\n",
    "                                    na_rm=True)\n",
    "                + ggtitle(mful+\": \"+colname)\n",
    "                + theme_bw()\n",
    "            ).save(\"./Output/Statistik/histogram_step2_{mful}_{name}.svg\".format(mful=mful, name=column, units='cm', width=width, height=height))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"resi\"></a>\n",
    "## Residential housings\n",
    "Statistical evaluation of housings per floor area"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0m² - 14m²: 0 (0.0%)\n",
      "15m² - 24m²: 0 (0.0%)\n",
      "25m² - 36m²: 12825 (16.0%)\n",
      "37m² - 48m²: 21408 (26.7%)\n",
      "49m² - 59m²: 14119 (17.6%)\n",
      "60m² - infm²: 27022 (33.7%)\n"
     ]
    }
   ],
   "source": [
    "## Distribution of residential housholds by area in Hue\n",
    "\n",
    "residential = data_classified_full[data_classified_full.class_name.isin(['mf', 'sh', 'sf'])]\n",
    "if not 'data_classified_full' in globals():\n",
    "    data_classified_full = gpd.read_file(outfile_classified)\n",
    "    data_classified_full.FID = data_classified_full.FID.astype(np.int64)\n",
    "\n",
    "def housings(data, start, stop):\n",
    "    \n",
    "    total = data[(data.area >= start) & (data.area <= stop)].shape[0]\n",
    "    perc = total / data.shape[0]\n",
    "    print(\"{start}m² - {stop}m²: {total} ({perc}%)\".format(\n",
    "        start=start,\n",
    "        stop=stop,\n",
    "        total=total,\n",
    "        perc=np.round(perc*100, 1)))\n",
    "\n",
    "for i in [[0, 14], [15, 24], [25, 36], [37, 48], [49, 59], [60, np.inf]]:\n",
    "    housings(residential, i[0], i[1])    \n",
    "          \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"means\"></a>\n",
    "## Means and standard deviation\n",
    "\n",
    "The parameter's mean values and standard deviations for each class area calculated here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________\n",
      "\n",
      "__________________________\n",
      "\n",
      "__________________________\n",
      "\n",
      "__________________________\n",
      "\n",
      "__________________________\n",
      "\n",
      "__________________________\n",
      "\n",
      "__________________________\n",
      "\n",
      "\n",
      "area mf\n",
      "  Mean: 186.67072111632274\n",
      "  Std.: 96.38709679474334\n",
      "\n",
      "area us\n",
      "  Mean: 20.99227907434792\n",
      "  Std.: 6.07967464418715\n",
      "\n",
      "area sh\n",
      "  Mean: 58.39712615607078\n",
      "  Std.: 23.14042294412999\n",
      "\n",
      "area ul\n",
      "  Mean: 383.63124120654396\n",
      "  Std.: 370.7850850157728\n",
      "\n",
      "area sf\n",
      "  Mean: 54.01078435986158\n",
      "  Std.: 20.69766717964151\n",
      "__________________________\n",
      "\n",
      "__________________________\n",
      "\n",
      "\n",
      "perimeter mf\n",
      "  Mean: 60.260434643527205\n",
      "  Std.: 16.240182640951346\n",
      "\n",
      "perimeter us\n",
      "  Mean: 19.026568827045903\n",
      "  Std.: 3.199277611131019\n",
      "\n",
      "perimeter sh\n",
      "  Mean: 34.16210348561345\n",
      "  Std.: 7.402555314309087\n",
      "\n",
      "perimeter ul\n",
      "  Mean: 87.45528266359916\n",
      "  Std.: 45.383944429877715\n",
      "\n",
      "perimeter sf\n",
      "  Mean: 29.55940531865688\n",
      "  Std.: 5.522055384680283\n",
      "__________________________\n",
      "\n",
      "\n",
      "rect mf\n",
      "  Mean: 0.9124016666666667\n",
      "  Std.: 0.1272314271673556\n",
      "\n",
      "rect us\n",
      "  Mean: 0.9872018783626467\n",
      "  Std.: 0.043837684173125585\n",
      "\n",
      "rect sh\n",
      "  Mean: 0.9871503784128228\n",
      "  Std.: 0.03949697731290295\n",
      "\n",
      "rect ul\n",
      "  Mean: 0.9246989877300614\n",
      "  Std.: 0.1378077327089234\n",
      "\n",
      "rect sf\n",
      "  Mean: 0.9842322018219052\n",
      "  Std.: 0.0514658313811812\n",
      "__________________________\n",
      "\n",
      "\n",
      "shp_ratio mf\n",
      "  Mean: 2.977408230143839\n",
      "  Std.: 0.7230287069382809\n",
      "\n",
      "shp_ratio us\n",
      "  Mean: 1.0796832665261302\n",
      "  Std.: 0.17927403818894214\n",
      "\n",
      "shp_ratio sh\n",
      "  Mean: 1.657016448213323\n",
      "  Std.: 0.31450808730615076\n",
      "\n",
      "shp_ratio ul\n",
      "  Mean: 3.88522072597137\n",
      "  Std.: 1.546509959140166\n",
      "\n",
      "shp_ratio sf\n",
      "  Mean: 1.7715615572699666\n",
      "  Std.: 0.3201264595115775\n",
      "__________________________\n",
      "\n",
      "\n",
      "shp_index mf\n",
      "  Mean: 1.277248771106942\n",
      "  Std.: 0.1491564826158705\n",
      "\n",
      "shp_index us\n",
      "  Mean: 1.185967895730422\n",
      "  Std.: 0.06898654157049437\n",
      "\n",
      "shp_index sh\n",
      "  Mean: 1.2806788373118965\n",
      "  Std.: 0.08992736612036005\n",
      "\n",
      "shp_index ul\n",
      "  Mean: 1.3376389775051125\n",
      "  Std.: 0.22870303005785847\n",
      "\n",
      "shp_index sf\n",
      "  Mean: 1.1525176214603488\n",
      "  Std.: 0.041448970787741256\n",
      "__________________________\n",
      "\n",
      "\n",
      "area_mabr mf\n",
      "  Mean: 209.0987311319575\n",
      "  Std.: 122.48835150274061\n",
      "\n",
      "area_mabr us\n",
      "  Mean: 21.326991525973007\n",
      "  Std.: 6.369539911111406\n",
      "\n",
      "area_mabr sh\n",
      "  Mean: 59.307010412234696\n",
      "  Std.: 23.753148594317544\n",
      "\n",
      "area_mabr ul\n",
      "  Mean: 452.68549548568507\n",
      "  Std.: 557.0769852037894\n",
      "\n",
      "area_mabr sf\n",
      "  Mean: 55.204236802309154\n",
      "  Std.: 21.958863550675538\n",
      "__________________________\n",
      "\n",
      "\n",
      "width mf\n",
      "  Mean: 10.574022861163227\n",
      "  Std.: 3.4203246094420736\n",
      "\n",
      "width us\n",
      "  Mean: 3.5573286561998554\n",
      "  Std.: 0.7891088414001397\n",
      "\n",
      "width sh\n",
      "  Mean: 4.660838067251605\n",
      "  Std.: 0.9738858787807678\n",
      "\n",
      "width ul\n",
      "  Mean: 13.20702236196319\n",
      "  Std.: 7.4819081645768\n",
      "\n",
      "width sf\n",
      "  Mean: 6.33739497793235\n",
      "  Std.: 1.3322960131938688\n",
      "__________________________\n",
      "\n",
      "\n",
      "length mf\n",
      "  Mean: 19.571743799249532\n",
      "  Std.: 6.693360664973602\n",
      "\n",
      "length us\n",
      "  Mean: 5.994045793977913\n",
      "  Std.: 1.3961172590835067\n",
      "\n",
      "length sh\n",
      "  Mean: 12.463643424341944\n",
      "  Std.: 3.213398446359017\n",
      "\n",
      "length ul\n",
      "  Mean: 30.20702100715747\n",
      "  Std.: 15.873346344070542\n",
      "\n",
      "length sf\n",
      "  Mean: 8.489569405762305\n",
      "  Std.: 1.7693087742509417\n",
      "__________________________\n",
      "\n",
      "\n",
      "elongation mf\n",
      "  Mean: 0.17839731394621636\n",
      "  Std.: 0.04701865092205385\n",
      "\n",
      "elongation us\n",
      "  Mean: 0.18875130038070667\n",
      "  Std.: 0.03624514567432187\n",
      "\n",
      "elongation sh\n",
      "  Mean: 0.1387415533552277\n",
      "  Std.: 0.02296632501630122\n",
      "\n",
      "elongation ul\n",
      "  Mean: 0.15568320040899797\n",
      "  Std.: 0.049835088689659794\n",
      "\n",
      "elongation sf\n",
      "  Mean: 0.21451622907986725\n",
      "  Std.: 0.022531401759430955\n",
      "__________________________\n",
      "\n",
      "\n",
      "orient mf\n",
      "  Mean: 22.973596575984992\n",
      "  Std.: 14.340247255444126\n",
      "\n",
      "orient us\n",
      "  Mean: 23.474438840260515\n",
      "  Std.: 13.874699022377357\n",
      "\n",
      "orient sh\n",
      "  Mean: 23.07618535758051\n",
      "  Std.: 14.124996493492603\n",
      "\n",
      "orient ul\n",
      "  Mean: 26.628529739263804\n",
      "  Std.: 14.959627229429895\n",
      "\n",
      "orient sf\n",
      "  Mean: 22.44267463932632\n",
      "  Std.: 13.659923687395464\n",
      "__________________________\n",
      "\n",
      "\n",
      "orient_100 mf\n",
      "  Mean: 22.936056278924326\n",
      "  Std.: 11.170824202403226\n",
      "\n",
      "orient_100 us\n",
      "  Mean: 23.414422602963846\n",
      "  Std.: 11.043232201934972\n",
      "\n",
      "orient_100 sh\n",
      "  Mean: 23.33934693152296\n",
      "  Std.: 11.149884492575397\n",
      "\n",
      "orient_100 ul\n",
      "  Mean: 26.95972856850716\n",
      "  Std.: 12.011760336919222\n",
      "\n",
      "orient_100 sf\n",
      "  Mean: 22.31109104053386\n",
      "  Std.: 10.528728672063762\n",
      "__________________________\n",
      "\n",
      "\n",
      "orient_dev mf\n",
      "  Mean: 5.4658567229518455\n",
      "  Std.: 5.408092585388873\n",
      "\n",
      "orient_dev us\n",
      "  Mean: 5.805280675518359\n",
      "  Std.: 5.3438917311093554\n",
      "\n",
      "orient_dev sh\n",
      "  Mean: 5.3297090784765455\n",
      "  Std.: 5.104065831896739\n",
      "\n",
      "orient_dev ul\n",
      "  Mean: 4.889872218813906\n",
      "  Std.: 6.008117690985302\n",
      "\n",
      "orient_dev sf\n",
      "  Mean: 6.191108210401809\n",
      "  Std.: 5.58210005985163\n",
      "__________________________\n",
      "\n",
      "\n",
      "shrd_walls mf\n",
      "  Mean: 7.392120075046904\n",
      "  Std.: 11.370807908451264\n",
      "\n",
      "shrd_walls us\n",
      "  Mean: 1.9613315294339742\n",
      "  Std.: 3.549912339998595\n",
      "\n",
      "shrd_walls sh\n",
      "  Mean: 4.8600558796137445\n",
      "  Std.: 8.146498989673443\n",
      "\n",
      "shrd_walls ul\n",
      "  Mean: 9.106339468302659\n",
      "  Std.: 14.73615743975546\n",
      "\n",
      "shrd_walls sf\n",
      "  Mean: 2.629440011298637\n",
      "  Std.: 4.861476988523351\n",
      "__________________________\n",
      "\n",
      "__________________________\n",
      "\n",
      "\n",
      "n_in_50m mf\n",
      "  Mean: 51.187617260787995\n",
      "  Std.: 25.099672662077527\n",
      "\n",
      "n_in_50m us\n",
      "  Mean: 49.2416071484756\n",
      "  Std.: 27.178976767184515\n",
      "\n",
      "n_in_50m sh\n",
      "  Mean: 52.24778197147199\n",
      "  Std.: 26.3852772740277\n",
      "\n",
      "n_in_50m ul\n",
      "  Mean: 30.309815950920246\n",
      "  Std.: 22.30067643583437\n",
      "\n",
      "n_in_50m sf\n",
      "  Mean: 46.75757361768237\n",
      "  Std.: 27.084258167419886\n",
      "__________________________\n",
      "\n",
      "\n",
      "n_in_100m mf\n",
      "  Mean: 150.94684177611006\n",
      "  Std.: 73.02686750834008\n",
      "\n",
      "n_in_100m us\n",
      "  Mean: 144.5849038794324\n",
      "  Std.: 79.01006756169109\n",
      "\n",
      "n_in_100m sh\n",
      "  Mean: 152.23905690897504\n",
      "  Std.: 76.0822885926222\n",
      "\n",
      "n_in_100m ul\n",
      "  Mean: 95.01124744376278\n",
      "  Std.: 65.45067825880763\n",
      "\n",
      "n_in_100m sf\n",
      "  Mean: 137.51851917237482\n",
      "  Std.: 79.04754160186764\n",
      "__________________________\n",
      "\n",
      "\n",
      "n_in_200m mf\n",
      "  Mean: 490.85209505941214\n",
      "  Std.: 223.48146943267653\n",
      "\n",
      "n_in_200m us\n",
      "  Mean: 454.2002013655099\n",
      "  Std.: 243.10128172986418\n",
      "\n",
      "n_in_200m sh\n",
      "  Mean: 484.8204989951473\n",
      "  Std.: 238.8254745768621\n",
      "\n",
      "n_in_200m ul\n",
      "  Mean: 355.7014314928425\n",
      "  Std.: 213.65869907682756\n",
      "\n",
      "n_in_200m sf\n",
      "  Mean: 435.41750229503566\n",
      "  Std.: 247.33022277589274\n",
      "__________________________\n",
      "\n",
      "\n",
      "n_in_ha mf\n",
      "  Mean: 36.85866166353971\n",
      "  Std.: 18.226581992846018\n",
      "\n",
      "n_in_ha us\n",
      "  Mean: 35.27240977881257\n",
      "  Std.: 19.429275810563652\n",
      "\n",
      "n_in_ha sh\n",
      "  Mean: 37.560952894465956\n",
      "  Std.: 19.016250580533654\n",
      "\n",
      "n_in_ha ul\n",
      "  Mean: 21.930981595092025\n",
      "  Std.: 16.220295841814362\n",
      "\n",
      "n_in_ha sf\n",
      "  Mean: 33.50529623614152\n",
      "  Std.: 19.39867934089982\n",
      "__________________________\n",
      "\n",
      "\n",
      "to_roads mf\n",
      "  Mean: 20.676819665415888\n",
      "  Std.: 30.820295175717327\n",
      "\n",
      "to_roads us\n",
      "  Mean: 29.708138016235097\n",
      "  Std.: 59.32599379411679\n",
      "\n",
      "to_roads sh\n",
      "  Mean: 24.976940539189258\n",
      "  Std.: 48.045555676538605\n",
      "\n",
      "to_roads ul\n",
      "  Mean: 31.80082259202454\n",
      "  Std.: 50.87283896013865\n",
      "\n",
      "to_roads sf\n",
      "  Mean: 32.92022472035873\n",
      "  Std.: 61.88297486921581\n",
      "__________________________\n",
      "\n",
      "\n",
      "roads_100 mf\n",
      "  Mean: 11.427454659161976\n",
      "  Std.: 9.403814600652302\n",
      "\n",
      "roads_100 us\n",
      "  Mean: 9.61598968001762\n",
      "  Std.: 9.269142597924812\n",
      "\n",
      "roads_100 sh\n",
      "  Mean: 10.502083231214156\n",
      "  Std.: 9.155487849909923\n",
      "\n",
      "roads_100 ul\n",
      "  Mean: 10.085889570552148\n",
      "  Std.: 8.89887433864322\n",
      "\n",
      "roads_100 sf\n",
      "  Mean: 8.908198573547066\n",
      "  Std.: 8.77327287487704\n",
      "__________________________\n",
      "\n",
      "\n",
      "roads_250 mf\n",
      "  Mean: 48.722951844903065\n",
      "  Std.: 36.07178122393563\n",
      "\n",
      "roads_250 us\n",
      "  Mean: 42.28549853695372\n",
      "  Std.: 37.748798100806184\n",
      "\n",
      "roads_250 sh\n",
      "  Mean: 45.63393951276898\n",
      "  Std.: 36.68033934895158\n",
      "\n",
      "roads_250 ul\n",
      "  Mean: 44.06441717791411\n",
      "  Std.: 32.02947807831267\n",
      "\n",
      "roads_250 sf\n",
      "  Mean: 38.404067509356686\n",
      "  Std.: 35.59131615419673\n",
      "__________________________\n",
      "\n",
      "\n",
      "msize_50 mf\n",
      "  Mean: 69.43469318323953\n",
      "  Std.: 18.7221104791167\n",
      "\n",
      "msize_50 us\n",
      "  Mean: 43.130453857722685\n",
      "  Std.: 14.477641052786549\n",
      "\n",
      "msize_50 sh\n",
      "  Mean: 52.39048471300427\n",
      "  Std.: 17.37338338241282\n",
      "\n",
      "msize_50 ul\n",
      "  Mean: 159.8541460633947\n",
      "  Std.: 164.98329209481165\n",
      "\n",
      "msize_50 sf\n",
      "  Mean: 49.01735007520655\n",
      "  Std.: 16.450193589986835\n",
      "__________________________\n",
      "\n",
      "\n",
      "msize_100 mf\n",
      "  Mean: 70.45449784240151\n",
      "  Std.: 26.83767227880388\n",
      "\n",
      "msize_100 us\n",
      "  Mean: 46.530396213069885\n",
      "  Std.: 15.959281357332241\n",
      "\n",
      "msize_100 sh\n",
      "  Mean: 53.83765781824422\n",
      "  Std.: 16.115939777240815\n",
      "\n",
      "msize_100 ul\n",
      "  Mean: 132.330828803681\n",
      "  Std.: 82.54451251724625\n",
      "\n",
      "msize_100 sf\n",
      "  Mean: 50.9035820505261\n",
      "  Std.: 17.71961967845132\n",
      "__________________________\n",
      "\n",
      "\n",
      "tess_area mf\n",
      "  Mean: 902.854107120075\n",
      "  Std.: 1491.8172097366473\n",
      "\n",
      "tess_area us\n",
      "  Mean: 464.8921818560237\n",
      "  Std.: 1508.710761657221\n",
      "\n",
      "tess_area sh\n",
      "  Mean: 518.3330707867262\n",
      "  Std.: 1241.4175258751643\n",
      "\n",
      "tess_area ul\n",
      "  Mean: 2131.208673164622\n",
      "  Std.: 3450.273686309346\n",
      "\n",
      "tess_area sf\n",
      "  Mean: 613.0963928299556\n",
      "  Std.: 1462.093130635741\n",
      "__________________________\n",
      "\n",
      "\n",
      "CAR mf\n",
      "  Mean: 0.3759701469668543\n",
      "  Std.: 0.20234514166080378\n",
      "\n",
      "CAR us\n",
      "  Mean: 0.2317726916905264\n",
      "  Std.: 0.1955603294681089\n",
      "\n",
      "CAR sh\n",
      "  Mean: 0.30867218763786086\n",
      "  Std.: 0.205525994956039\n",
      "\n",
      "CAR ul\n",
      "  Mean: 0.32237051124744376\n",
      "  Std.: 0.19852154567497804\n",
      "\n",
      "CAR sf\n",
      "  Mean: 0.2695766944424829\n",
      "  Std.: 0.19472383139619176\n",
      "__________________________\n",
      "\n",
      "\n",
      "bID mf\n",
      "  Mean: 687.3858661663539\n",
      "  Std.: 309.3058286472008\n",
      "\n",
      "bID us\n",
      "  Mean: 583.8996948053991\n",
      "  Std.: 354.05706665852574\n",
      "\n",
      "bID sh\n",
      "  Mean: 587.9587275133572\n",
      "  Std.: 316.6665315961786\n",
      "\n",
      "bID ul\n",
      "  Mean: 744.1226993865031\n",
      "  Std.: 329.0115087339969\n",
      "\n",
      "bID sf\n",
      "  Mean: 613.8857601864275\n",
      "  Std.: 332.8504287920866\n",
      "__________________________\n",
      "\n",
      "\n",
      "b_msize mf\n",
      "  Mean: 79.4929066572858\n",
      "  Std.: 58.93900005865109\n",
      "\n",
      "b_msize us\n",
      "  Mean: 49.76225685146147\n",
      "  Std.: 18.397594768230803\n",
      "\n",
      "b_msize sh\n",
      "  Mean: 55.480850522523404\n",
      "  Std.: 14.763765993086212\n",
      "\n",
      "b_msize ul\n",
      "  Mean: 137.36434833333334\n",
      "  Std.: 165.75923557361685\n",
      "\n",
      "b_msize sf\n",
      "  Mean: 53.47324190841042\n",
      "  Std.: 18.441449044821006\n",
      "__________________________\n",
      "\n",
      "\n",
      "b_builtup mf\n",
      "  Mean: 932.8542237210756\n",
      "  Std.: 1266.1647419662431\n",
      "\n",
      "b_builtup us\n",
      "  Mean: 878.5450916880092\n",
      "  Std.: 1249.6609089064918\n",
      "\n",
      "b_builtup sh\n",
      "  Mean: 939.8235704543894\n",
      "  Std.: 1280.3311515650119\n",
      "\n",
      "b_builtup ul\n",
      "  Mean: 857.8591120654397\n",
      "  Std.: 1093.6997359017919\n",
      "\n",
      "b_builtup sf\n",
      "  Mean: 1081.0844470681802\n",
      "  Std.: 1441.711640505738\n",
      "__________________________\n",
      "\n",
      "\n",
      "b_medsize mf\n",
      "  Mean: 54.39211258286429\n",
      "  Std.: 23.509156681729895\n",
      "\n",
      "b_medsize us\n",
      "  Mean: 39.967362588176066\n",
      "  Std.: 9.86626626679791\n",
      "\n",
      "b_medsize sh\n",
      "  Mean: 44.726866149698544\n",
      "  Std.: 9.732401172974201\n",
      "\n",
      "b_medsize ul\n",
      "  Mean: 89.15484028629857\n",
      "  Std.: 151.03785224473222\n",
      "\n",
      "b_medsize sf\n",
      "  Mean: 43.472763555363315\n",
      "  Std.: 10.358890232837563\n",
      "__________________________\n",
      "\n",
      "__________________________\n",
      "\n",
      "__________________________\n",
      "\n",
      "__________________________\n",
      "\n",
      "__________________________\n",
      "\n",
      "__________________________\n",
      "\n",
      "__________________________\n",
      "\n",
      "__________________________\n",
      "\n"
     ]
    }
   ],
   "source": [
    "## Calculation of parameters' mean values and standard deviation per class\n",
    "if not 'data_classified_full' in globals():\n",
    "    data_classified_full = gpd.read_file(outfile_classified)\n",
    "    data_classified_full.FID = data_classified_full.FID.astype(np.int64)\n",
    "\n",
    "for i in data_classified_full.columns:\n",
    "    for j in set(data_classified_full.class_name):\n",
    "        if type(data_classified_full[data_classified_full.class_name == j][i].values[1]) == np.float64:\n",
    "            print('\\n'+i, j)\n",
    "            print(\"  Mean:\", np.mean(data_classified_full[data_classified_full.class_name == j][i].values))\n",
    "            print(\"  Std.:\", np.std(data_classified_full[data_classified_full.class_name == j][i].values))\n",
    "    else:\n",
    "        pass\n",
    "    print('__________________________\\n')   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Adding \"bID\" to dataset: 100%|████████████████████████████████████████████████| 113982/113982 [02:08<00:00, 890.09it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating building's mean sizes and built up ratio within blocks...\n",
      "Writing output file...\n",
      "\n",
      "Process finished. It took 132 minutes.\n",
      "Output written to './Output/Daten/Hue_buildings_UTM48N_preprocessed_first_no_classes.shp.\n"
     ]
    }
   ],
   "source": [
    "# for i, row in tqdm(data.iterrows(), total=data.shape[0], desc='Adding \"bID\" to dataset'):\n",
    "#     data.loc[i, 'bID'] = blocks.blocks.sindex.query(data.loc[i].geometry,\n",
    "#                                                     predicate='intersects')[0]\n",
    "\n",
    "# print(\"Calculating building's mean sizes and built up ratio within blocks...\")\n",
    "# data['b_msize'] = data.area\n",
    "# data['b_builtup'] = 0\n",
    "# data['b_medsize'] = data.area\n",
    "# bID_set = list(set(data.bID))\n",
    "# for i in range(len(bID_set)):\n",
    "#     buildings_in_block = data.loc[data.bID == bID_set[i]]\n",
    "#     data.loc[data.FID.isin(buildings_in_block.FID), \n",
    "#              'b_msize'] = buildings_in_block.area.sum() / len(buildings_in_block)\n",
    "#     data.loc[data.FID.isin(buildings_in_block.FID), \n",
    "#              'b_medsize'] = buildings_in_block.area.median()\n",
    "#     data.loc[data.FID.isin(buildings_in_block.FID), \n",
    "#              'b_builtup'] = buildings_in_block.area.sum() / row.geometry.area\n",
    "\n",
    "# # rounds all numerical values to the given precision\n",
    "# data = np.round(data, precision)\n",
    "\n",
    "# print('Writing output file...')\n",
    "# data.to_file(outfile_first, encoding='utf-8')\n",
    "print(\"\\nProcess finished. It took {time} minutes.\\n\"\n",
    "      \"Output written to '{fname}.\".format(time=round((time.time() - start_time) / 60), fname=outfile_first))\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "GVN1aUjz546e",
    "mrL0P2-BcPYm"
   ],
   "name": "Hue_classification.ipynb",
   "provenance": [
    {
     "file_id": "1kR-f6rJ5XhL-VUsFGiQP7wiUJqdUTyU4",
     "timestamp": 1623590366189
    }
   ]
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  },
  "toc-autonumbering": true,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false,
  "toc-showtags": false,
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "02248b97e95e471896dec3c2c2445c2c": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "046afab5c04643cfba91480444490605": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "054ccda05e8248bc97004e0950053b3a": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "166973b412084481849fc6b37c431494": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "16d1d3418e3a47e4aef4ff040c1f0a64": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b6bcc204e49e486ba2567c556069bd91",
      "max": 5085,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_166973b412084481849fc6b37c431494",
      "value": 5085
     }
    },
    "1712924346bb45bcafb9980ae20d59dc": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_3419e230eb524e37a769a08d8ebc49fd",
       "IPY_MODEL_e60950da1bb74809912c0bc4a41006fd"
      ],
      "layout": "IPY_MODEL_ca9974d5ca6f4eb2a2109bd1f63ea366"
     }
    },
    "1c2af11f863a4e89b769f13c24ccdcbc": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "201a57ace6c14df68f078c6db7411c2e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_d2ff2ee31b1b4f19bccb592657fbafba",
       "IPY_MODEL_7c509b8e62224a36af1311f3838a82bf"
      ],
      "layout": "IPY_MODEL_d524c1c6f0aa45a787571ddfde83f68a"
     }
    },
    "246d31140fd74d7ab95d22fc2b276987": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_552afccb92824ee99784496a51c89561",
       "IPY_MODEL_356f0e34ef7d446992147c1450868b0c"
      ],
      "layout": "IPY_MODEL_6a9f009e7a9f4019a51dd6ad0adf9b9d"
     }
    },
    "2c7f4be35c9143459746b4d16a529381": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3280f822d845445abb1f49788ce46b5a": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3419e230eb524e37a769a08d8ebc49fd": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ef82828170014116b6b6b8db84ed635d",
      "max": 5085,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_424192e71ebf46499d5845abcc514eaf",
      "value": 5085
     }
    },
    "356f0e34ef7d446992147c1450868b0c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_bb0c7671947e424e884441b31983fd2b",
      "placeholder": "​",
      "style": "IPY_MODEL_eae35f5a58894102b5e0d4d4efbf8d66",
      "value": " 1102/1102 [01:01&lt;00:00, 17.94it/s]"
     }
    },
    "360d95d8a9114dc3a2117da84a5f81a2": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "424192e71ebf46499d5845abcc514eaf": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "47053e3b47d74f5ea758c34e3a07a76b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4ebcf19af3f54658bc1afcae579d4edb": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "512cc211d14548b5b90adce91b7a0f5e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "552afccb92824ee99784496a51c89561": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_02248b97e95e471896dec3c2c2445c2c",
      "max": 1102,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_512cc211d14548b5b90adce91b7a0f5e",
      "value": 1102
     }
    },
    "59442c7e7a4f4ef4928e1b3f7cd71843": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "624d5c75304d4db3923b438a33774434": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "65b11cea931f425eb0ee046ed35ccef0": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3280f822d845445abb1f49788ce46b5a",
      "max": 5085,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_624d5c75304d4db3923b438a33774434",
      "value": 5085
     }
    },
    "6a9f009e7a9f4019a51dd6ad0adf9b9d": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6ac3313a05884052b86ce3dd001ff160": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_47053e3b47d74f5ea758c34e3a07a76b",
      "placeholder": "​",
      "style": "IPY_MODEL_360d95d8a9114dc3a2117da84a5f81a2",
      "value": " 5085/5085 [03:02&lt;00:00, 27.84it/s]"
     }
    },
    "7c509b8e62224a36af1311f3838a82bf": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1c2af11f863a4e89b769f13c24ccdcbc",
      "placeholder": "​",
      "style": "IPY_MODEL_8f1f870dad824d059540b879488a4ae5",
      "value": " 5085/5085 [04:40&lt;00:00, 18.14it/s]"
     }
    },
    "8f1f870dad824d059540b879488a4ae5": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "91bfd5d22fec4f26aac1caccde5d182b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "aae6bd861fb54b3ebef1e5acdde347c1": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b6bcc204e49e486ba2567c556069bd91": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "bb0c7671947e424e884441b31983fd2b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c7f2dc2a44e6440cb282be20c57d3017": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2c7f4be35c9143459746b4d16a529381",
      "placeholder": "​",
      "style": "IPY_MODEL_aae6bd861fb54b3ebef1e5acdde347c1",
      "value": " 5085/5085 [00:07&lt;00:00, 671.67it/s]"
     }
    },
    "ca9974d5ca6f4eb2a2109bd1f63ea366": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d0802212b19043c3be7b4d9235dcc495": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d2ff2ee31b1b4f19bccb592657fbafba": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d0802212b19043c3be7b4d9235dcc495",
      "max": 5085,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_046afab5c04643cfba91480444490605",
      "value": 5085
     }
    },
    "d524c1c6f0aa45a787571ddfde83f68a": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e14e97bc5c15410385ffcc93c67ca4f1": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_65b11cea931f425eb0ee046ed35ccef0",
       "IPY_MODEL_c7f2dc2a44e6440cb282be20c57d3017"
      ],
      "layout": "IPY_MODEL_054ccda05e8248bc97004e0950053b3a"
     }
    },
    "e60950da1bb74809912c0bc4a41006fd": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_91bfd5d22fec4f26aac1caccde5d182b",
      "placeholder": "​",
      "style": "IPY_MODEL_4ebcf19af3f54658bc1afcae579d4edb",
      "value": " 5085/5085 [02:53&lt;00:00, 29.27it/s]"
     }
    },
    "eae35f5a58894102b5e0d4d4efbf8d66": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ef82828170014116b6b6b8db84ed635d": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f58106c95326412db9edf3eefbc89140": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_16d1d3418e3a47e4aef4ff040c1f0a64",
       "IPY_MODEL_6ac3313a05884052b86ce3dd001ff160"
      ],
      "layout": "IPY_MODEL_59442c7e7a4f4ef4928e1b3f7cd71843"
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}