livro_ESCD_pre_processamento.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyMO/SjxLWyGkTni6rtmgHNG",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/profkalinowski/livroescd/blob/main/livro_ESCD_pre_processamento.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aUwCbt07o7UC"
      },
      "source": [
        "# Livro Engenharia de Software para Ciência de Dados (Ed. Casa do Código)\n",
        "Marcos Kalinowski, Tatiana Escovedo, Hugo Villamizar e Hélio Lopes\n",
        "\n",
        "### Exemplo Prático de Pré-Processamento de Dados em Python"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Para exemplificar como funciona a etapa de Pré-Processamento de Dados iremos utilizar novamente o dataset Iris, que utilizamos na prática anterior, para ilustrar a limpeza de dados, em especial, o tratamento de missings. Em seguida, iremos utilizar alguns datasets artificiais para falarmos sobre as operações de Transformação de Dados, como a Normalização, Padronização, One-hot encoding e Dummy encoding. \n",
        "\n",
        "Iniciaremos esta prática importando os pacotes que utilizaremos:"
      ],
      "metadata": {
        "id": "Y8o1ROCsYMsT"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T1gTKd1qgRQg"
      },
      "source": [
        "# Importação de pacotes\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.preprocessing import MinMaxScaler # normalização\n",
        "from sklearn.preprocessing import StandardScaler # padronização\n",
        "from sklearn.preprocessing import OrdinalEncoder # ordinal encoding\n",
        "from sklearn.preprocessing import OneHotEncoder # one-hot encoding e dummy encoding"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Agora iremos importar o dataset Iris a partir de um link, e armazenaremos os dados em um dataframe, da mesma forma que fizemos na prática anterior. Exibiremos também as 5 últimas linhas do dataframe:"
      ],
      "metadata": {
        "id": "tzfnwSD-Y-ZB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# importando dados uma url para um dataframe\n",
        "\n",
        "# url a importar\n",
        "url_dados = 'https://raw.githubusercontent.com/profkalinowski/livroescd/main/iris.data'\n",
        "\n",
        "# labels dos atributos do dataset\n",
        "labels_atributos = ['comprimento_sepala', 'largura_sepala', 'comprimento_petala', 'largura_petala', 'especie']\n",
        "\n",
        "# carga do dataset através da url\n",
        "iris = pd.read_csv(url_dados, names=labels_atributos)\n",
        "\n",
        "# exibindo as últimas linhas\n",
        "iris.tail()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "Q2tm1irsKmsj",
        "outputId": "dbd51549-4e9e-440c-8645-d52ddae39070"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     comprimento_sepala  largura_sepala  comprimento_petala  largura_petala  \\\n",
              "145                 6.7             3.0                 5.2             2.3   \n",
              "146                 6.3             2.5                 5.0             1.9   \n",
              "147                 6.5             3.0                 5.2             2.0   \n",
              "148                 6.2             3.4                 5.4             2.3   \n",
              "149                 5.9             3.0                 5.1             1.8   \n",
              "\n",
              "            especie  \n",
              "145  Iris-virginica  \n",
              "146  Iris-virginica  \n",
              "147  Iris-virginica  \n",
              "148  Iris-virginica  \n",
              "149  Iris-virginica  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9dfa345a-eede-4ef9-8e07-3e5a13055050\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>comprimento_sepala</th>\n",
              "      <th>largura_sepala</th>\n",
              "      <th>comprimento_petala</th>\n",
              "      <th>largura_petala</th>\n",
              "      <th>especie</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>145</th>\n",
              "      <td>6.7</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.2</td>\n",
              "      <td>2.3</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>146</th>\n",
              "      <td>6.3</td>\n",
              "      <td>2.5</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.9</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>147</th>\n",
              "      <td>6.5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.2</td>\n",
              "      <td>2.0</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>148</th>\n",
              "      <td>6.2</td>\n",
              "      <td>3.4</td>\n",
              "      <td>5.4</td>\n",
              "      <td>2.3</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>149</th>\n",
              "      <td>5.9</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.1</td>\n",
              "      <td>1.8</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9dfa345a-eede-4ef9-8e07-3e5a13055050')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-9dfa345a-eede-4ef9-8e07-3e5a13055050 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-9dfa345a-eede-4ef9-8e07-3e5a13055050');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Como vimos na prática anterior, o dataset Iris não possui valores faltantes (missings). Agora, iremos criar uma cópia deste dataframe, adicionando uma nova linha que contém um valor missing. Exibiremos novamente as 5 últimas linhas para verificar a nova linha que inserimos:"
      ],
      "metadata": {
        "id": "Ay2URhrMZBT3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# criando um novo dataframe com iris + uma nova linha com um valor missing para o próximo exemplo\n",
        "df = iris.append({'largura_sepala': 4.0, \n",
        "                  'comprimento_petala': 5.0, \n",
        "                  'largura_petala': 0.4, \n",
        "                  'especie': 'Iris-setosa'},\n",
        "                  ignore_index=True)\n",
        "df.tail()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "PRL44Z89KyoY",
        "outputId": "b65888d9-1aa6-47aa-8bc4-ae6cf1821c79"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     comprimento_sepala  largura_sepala  comprimento_petala  largura_petala  \\\n",
              "146                 6.3             2.5                 5.0             1.9   \n",
              "147                 6.5             3.0                 5.2             2.0   \n",
              "148                 6.2             3.4                 5.4             2.3   \n",
              "149                 5.9             3.0                 5.1             1.8   \n",
              "150                 NaN             4.0                 5.0             0.4   \n",
              "\n",
              "            especie  \n",
              "146  Iris-virginica  \n",
              "147  Iris-virginica  \n",
              "148  Iris-virginica  \n",
              "149  Iris-virginica  \n",
              "150     Iris-setosa  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-68461eb1-1d44-49aa-b0f1-9a46bdfe2b5d\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>comprimento_sepala</th>\n",
              "      <th>largura_sepala</th>\n",
              "      <th>comprimento_petala</th>\n",
              "      <th>largura_petala</th>\n",
              "      <th>especie</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>146</th>\n",
              "      <td>6.3</td>\n",
              "      <td>2.5</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.9</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>147</th>\n",
              "      <td>6.5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.2</td>\n",
              "      <td>2.0</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>148</th>\n",
              "      <td>6.2</td>\n",
              "      <td>3.4</td>\n",
              "      <td>5.4</td>\n",
              "      <td>2.3</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>149</th>\n",
              "      <td>5.9</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.1</td>\n",
              "      <td>1.8</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>150</th>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.4</td>\n",
              "      <td>Iris-setosa</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-68461eb1-1d44-49aa-b0f1-9a46bdfe2b5d')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-68461eb1-1d44-49aa-b0f1-9a46bdfe2b5d button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-68461eb1-1d44-49aa-b0f1-9a46bdfe2b5d');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Em seguida, iremos executar um comando para eliminar deste dataframe as linhas que tiverem qualquer valor missing. Neste caso, será a última linha que adicionamos:"
      ],
      "metadata": {
        "id": "QLqXZS__ZDmT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# eliminando linhas que tenham ALGUM valor missing\n",
        "\n",
        "df = df.dropna(how='any')\n",
        "df.tail()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "uazM9TAgK1hs",
        "outputId": "defd1a08-c0c5-4af9-8b40-803b9d2f3173"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     comprimento_sepala  largura_sepala  comprimento_petala  largura_petala  \\\n",
              "145                 6.7             3.0                 5.2             2.3   \n",
              "146                 6.3             2.5                 5.0             1.9   \n",
              "147                 6.5             3.0                 5.2             2.0   \n",
              "148                 6.2             3.4                 5.4             2.3   \n",
              "149                 5.9             3.0                 5.1             1.8   \n",
              "\n",
              "            especie  \n",
              "145  Iris-virginica  \n",
              "146  Iris-virginica  \n",
              "147  Iris-virginica  \n",
              "148  Iris-virginica  \n",
              "149  Iris-virginica  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-531f4f4c-6041-4b39-b9b7-c4c7271cd5c2\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>comprimento_sepala</th>\n",
              "      <th>largura_sepala</th>\n",
              "      <th>comprimento_petala</th>\n",
              "      <th>largura_petala</th>\n",
              "      <th>especie</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>145</th>\n",
              "      <td>6.7</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.2</td>\n",
              "      <td>2.3</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>146</th>\n",
              "      <td>6.3</td>\n",
              "      <td>2.5</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.9</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>147</th>\n",
              "      <td>6.5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.2</td>\n",
              "      <td>2.0</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>148</th>\n",
              "      <td>6.2</td>\n",
              "      <td>3.4</td>\n",
              "      <td>5.4</td>\n",
              "      <td>2.3</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>149</th>\n",
              "      <td>5.9</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.1</td>\n",
              "      <td>1.8</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-531f4f4c-6041-4b39-b9b7-c4c7271cd5c2')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-531f4f4c-6041-4b39-b9b7-c4c7271cd5c2 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-531f4f4c-6041-4b39-b9b7-c4c7271cd5c2');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Agora iremos criar novamente um novo dataframe com a cópia do dataframe original. Em seguida, adicionamos uma nova linha com um valor missing, e uma nova linha com todos os valores missing:"
      ],
      "metadata": {
        "id": "ELTVxXqPZG5D"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# criando (novamente) um novo dataframe com iris + uma nova linha com um valor missing para o próximo exemplo\n",
        "df = iris.append({'largura_sepala': 4.0, \n",
        "                  'comprimento_petala': 5.0, \n",
        "                  'largura_petala': 0.4, \n",
        "                  'especie': 'Iris-setosa'},\n",
        "                  ignore_index=True)\n",
        "\n",
        "# adicionando uma nova linha com todos os valores missing para o próximo exemplo\n",
        "df = df.append({}, ignore_index=True)\n",
        "df.tail()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "9SSJz8t2K3Vh",
        "outputId": "bcb0164c-9f4e-4cfc-ce39-94a71ac47829"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     comprimento_sepala  largura_sepala  comprimento_petala  largura_petala  \\\n",
              "147                 6.5             3.0                 5.2             2.0   \n",
              "148                 6.2             3.4                 5.4             2.3   \n",
              "149                 5.9             3.0                 5.1             1.8   \n",
              "150                 NaN             4.0                 5.0             0.4   \n",
              "151                 NaN             NaN                 NaN             NaN   \n",
              "\n",
              "            especie  \n",
              "147  Iris-virginica  \n",
              "148  Iris-virginica  \n",
              "149  Iris-virginica  \n",
              "150     Iris-setosa  \n",
              "151             NaN  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-0df8d0e2-cb72-4e4f-befe-1827e05f1c27\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>comprimento_sepala</th>\n",
              "      <th>largura_sepala</th>\n",
              "      <th>comprimento_petala</th>\n",
              "      <th>largura_petala</th>\n",
              "      <th>especie</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>147</th>\n",
              "      <td>6.5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.2</td>\n",
              "      <td>2.0</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>148</th>\n",
              "      <td>6.2</td>\n",
              "      <td>3.4</td>\n",
              "      <td>5.4</td>\n",
              "      <td>2.3</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>149</th>\n",
              "      <td>5.9</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.1</td>\n",
              "      <td>1.8</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>150</th>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.4</td>\n",
              "      <td>Iris-setosa</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>151</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0df8d0e2-cb72-4e4f-befe-1827e05f1c27')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-0df8d0e2-cb72-4e4f-befe-1827e05f1c27 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-0df8d0e2-cb72-4e4f-befe-1827e05f1c27');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Agora iremos eliminar deste dataframe as linhas que tiverem todos os valores missing. Neste caso, será removida a última linha que adicionamos, mas não a anterior:"
      ],
      "metadata": {
        "id": "w7qR-57PZKXn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# eliminando linhas que tenham TODOS os valores missing\n",
        "df = df.dropna(how='all')\n",
        "df.tail()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "6F8ZcH3cK5JP",
        "outputId": "5e85cdb0-b252-4cc1-e564-8096547bc986"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     comprimento_sepala  largura_sepala  comprimento_petala  largura_petala  \\\n",
              "146                 6.3             2.5                 5.0             1.9   \n",
              "147                 6.5             3.0                 5.2             2.0   \n",
              "148                 6.2             3.4                 5.4             2.3   \n",
              "149                 5.9             3.0                 5.1             1.8   \n",
              "150                 NaN             4.0                 5.0             0.4   \n",
              "\n",
              "            especie  \n",
              "146  Iris-virginica  \n",
              "147  Iris-virginica  \n",
              "148  Iris-virginica  \n",
              "149  Iris-virginica  \n",
              "150     Iris-setosa  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-aef9e26a-5268-4bda-a77f-524dd28e8c68\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>comprimento_sepala</th>\n",
              "      <th>largura_sepala</th>\n",
              "      <th>comprimento_petala</th>\n",
              "      <th>largura_petala</th>\n",
              "      <th>especie</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>146</th>\n",
              "      <td>6.3</td>\n",
              "      <td>2.5</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.9</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>147</th>\n",
              "      <td>6.5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.2</td>\n",
              "      <td>2.0</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>148</th>\n",
              "      <td>6.2</td>\n",
              "      <td>3.4</td>\n",
              "      <td>5.4</td>\n",
              "      <td>2.3</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>149</th>\n",
              "      <td>5.9</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.1</td>\n",
              "      <td>1.8</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>150</th>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.4</td>\n",
              "      <td>Iris-setosa</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-aef9e26a-5268-4bda-a77f-524dd28e8c68')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-aef9e26a-5268-4bda-a77f-524dd28e8c68 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-aef9e26a-5268-4bda-a77f-524dd28e8c68');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Iremos em seguida definir valores para o preenchimento de missings, e realizar o preenchimento dos missings com estes valores definidos:"
      ],
      "metadata": {
        "id": "YiXCSB-KZMkK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# definindo valores para o preenchimento de missings\n",
        "values = {'comprimento_sepala': iris['comprimento_sepala'].median(), \n",
        "          'largura_sepala': 5.0, \n",
        "          'comprimento_petala': 4.0, \n",
        "          'largura_petala': 0.1, \n",
        "          'especie': 'Iris-setosa'}\n",
        "\n",
        "# fazendo o preenchimento\n",
        "df = df.fillna(value=values)\n",
        "df.tail()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "eBQQlyasK-fP",
        "outputId": "e78db1e0-6a5b-4718-b120-622f096cc088"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     comprimento_sepala  largura_sepala  comprimento_petala  largura_petala  \\\n",
              "146                 6.3             2.5                 5.0             1.9   \n",
              "147                 6.5             3.0                 5.2             2.0   \n",
              "148                 6.2             3.4                 5.4             2.3   \n",
              "149                 5.9             3.0                 5.1             1.8   \n",
              "150                 5.8             4.0                 5.0             0.4   \n",
              "\n",
              "            especie  \n",
              "146  Iris-virginica  \n",
              "147  Iris-virginica  \n",
              "148  Iris-virginica  \n",
              "149  Iris-virginica  \n",
              "150     Iris-setosa  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b4938904-b597-472e-98ab-c0125e0e87c4\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>comprimento_sepala</th>\n",
              "      <th>largura_sepala</th>\n",
              "      <th>comprimento_petala</th>\n",
              "      <th>largura_petala</th>\n",
              "      <th>especie</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>146</th>\n",
              "      <td>6.3</td>\n",
              "      <td>2.5</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.9</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>147</th>\n",
              "      <td>6.5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.2</td>\n",
              "      <td>2.0</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>148</th>\n",
              "      <td>6.2</td>\n",
              "      <td>3.4</td>\n",
              "      <td>5.4</td>\n",
              "      <td>2.3</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>149</th>\n",
              "      <td>5.9</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.1</td>\n",
              "      <td>1.8</td>\n",
              "      <td>Iris-virginica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>150</th>\n",
              "      <td>5.8</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.4</td>\n",
              "      <td>Iris-setosa</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b4938904-b597-472e-98ab-c0125e0e87c4')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-b4938904-b597-472e-98ab-c0125e0e87c4 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-b4938904-b597-472e-98ab-c0125e0e87c4');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BTVLPtqFqPfr"
      },
      "source": [
        "## Transformações Numéricas"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Podemos utilizar as operações de normalização e padronização usando o módulo de pré-processamento da biblioteca Scikit-learn (https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing):\n",
        "* Para normalizar os dados, usamos o MinMaxScaler;\n",
        "* Para padronizar os dados, usamos o StandardScaler.\n"
      ],
      "metadata": {
        "id": "fIxPBN61ZPKB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Observação:** Quando normalizar e quando padronizar?\n",
        "* Se a distribuição é normal, padronize. Caso contrário, normalize.\n",
        "* Os problemas de modelagem preditiva são muitas vezes complexos, não sendo clara a melhor transformação para realizar. Na dúvida, use a normalização. Se tiver tempo, explore os modelos com os dados sem transformação, com a padronização e com a normalização e veja se os resultados são significativamente diferentes e se o custo x benefício vale a pena.\n",
        "* Como a padronização resulta em valores positivos e negativos, pode ser interessante normalizar os dados após a padronização.\n",
        "* É possível definir os valores de mínimo e máximo de acordo com o conhecimento no negócio (e não simplesmente se ater aos valores observados)."
      ],
      "metadata": {
        "id": "uk7OXYcuZTOr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Para exemplificar as transformações numéricas, utilizaremos um dataset pequeno e artificial, para que seja fácil notar as transformações realizadas:"
      ],
      "metadata": {
        "id": "bgxZwV4AZcrh"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lHhKXN6dpKdW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e91f6beb-7e8c-4217-84ce-3b6646170cd2"
      },
      "source": [
        "# dados que iremos usar nos exemplos\n",
        "data = np.asarray([[100, 0.001],\n",
        "\t\t\t\t   [8,   0.05],\n",
        "\t\t\t\t   [50,  0.005],\n",
        "\t\t\t\t   [88,  0.07],\n",
        "\t\t\t\t   [4,   0.1]])\n",
        "print(data)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[1.0e+02 1.0e-03]\n",
            " [8.0e+00 5.0e-02]\n",
            " [5.0e+01 5.0e-03]\n",
            " [8.8e+01 7.0e-02]\n",
            " [4.0e+00 1.0e-01]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "A seguir, iremos normalizar os dados originais, e em seguida, padronizar os dados originais. Os comentários nos blocos de código auxiliam no seu entendimento."
      ],
      "metadata": {
        "id": "YEk3yhY4Ze0E"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sZbuLX5Sp1pX"
      },
      "source": [
        "### Normalização"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i7hGUQHIpWce",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "15f76f00-5621-4aff-8507-0b739fc1288b"
      },
      "source": [
        "# Normalização\n",
        "\n",
        "# definindo o transformador como min max scaler\n",
        "scaler = MinMaxScaler()\n",
        "\n",
        "# transformando os dados\n",
        "scaled = scaler.fit_transform(data)\n",
        "print(scaled)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[1.         0.        ]\n",
            " [0.04166667 0.49494949]\n",
            " [0.47916667 0.04040404]\n",
            " [0.875      0.6969697 ]\n",
            " [0.         1.        ]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "meF1JOIaqRmO"
      },
      "source": [
        "### Padronização\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7CelMP4Npy4r",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5ce96a0b-e783-40d9-cee0-06b583a2286a"
      },
      "source": [
        "# Padronização\n",
        "\n",
        "# definindo o transformador como standard scaler\n",
        "scaler = StandardScaler()\n",
        "\n",
        "# transformando os dados\n",
        "scaled = scaler.fit_transform(data)\n",
        "print(scaled)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[ 1.26398112 -1.16389967]\n",
            " [-1.06174414  0.12639634]\n",
            " [ 0.         -1.05856939]\n",
            " [ 0.96062565  0.65304778]\n",
            " [-1.16286263  1.44302493]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iuj3mHe4qUPO"
      },
      "source": [
        "## Transformações Categóricas"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Podemos utilizar as operações one hot encoding e dummy encoding usando o módulo de pré-processamento da biblioteca Scikit-learn (https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing):\n",
        "* Para o one hot encoding, usamos o OneHotEncoder;\n",
        "* Para o dummy encoding, também usamos o OneHotEncoder, mas usando o parâmetro drop para indicar qual categoria receberá todos os valores zero.\n",
        "\n",
        "O OneHotEncoder ordena as categorias alfabeticamente antes de aplicar a transformação. É possível especificar a lista de categorias através do parâmetro categories. Espera-se que o conjunto de treinamento contenha pelo menos um exemplo de cada categoria se as categorias não forem explicitamente definidas. Se os novos dados (conjunto de teste, por exemplo) tiverem categorias não vistas no treinamento, é possível configurar o parâmetro handle_unknown como ignore para que não ocorra um erro.\n",
        "\n",
        "Para exemplificar as transformações categóricas, utilizaremos um dataset pequeno e artificial, para que seja fácil notar as transformações realizadas:\n"
      ],
      "metadata": {
        "id": "itdVUAyNZjeW"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RKvm7cg6qa9g",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c00ff0f9-5d17-43b6-9a4b-651676a44c9e"
      },
      "source": [
        "# dados que iremos usar nos exemplos\n",
        "data = np.asarray([['castanhos'], ['verdes'], ['azuis'], ['azuis'], ['castanhos'], ['azuis']])\n",
        "print(data)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[['castanhos']\n",
            " ['verdes']\n",
            " ['azuis']\n",
            " ['azuis']\n",
            " ['castanhos']\n",
            " ['azuis']]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "A seguir, iremos aplicar o one hot encoding nos dados originais, e em seguida, aplicar o dummy encoding nos dados originais. Os comentários nos blocos de código auxiliam no seu entendimento. "
      ],
      "metadata": {
        "id": "bmSLhWDFZvLu"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TDfZ7IbPrOYN"
      },
      "source": [
        "### One Hot Encoding"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GRRAHCcxqzEN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "91e04924-3131-4498-e645-3c63793c4b3f"
      },
      "source": [
        "# definindo o transformador como one hot encoding\n",
        "encoder = OneHotEncoder(sparse=False)\n",
        "\n",
        "# transformando os dados\n",
        "onehot = encoder.fit_transform(data)\n",
        "print(onehot)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[0. 1. 0.]\n",
            " [0. 0. 1.]\n",
            " [1. 0. 0.]\n",
            " [1. 0. 0.]\n",
            " [0. 1. 0.]\n",
            " [1. 0. 0.]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-eDpdjYWrR0x"
      },
      "source": [
        "### Dummy Variable Encoding"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6vA_bmO8q85I",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "136c142f-84b5-44f9-bf01-f92782c9b38e"
      },
      "source": [
        "# definindo o transformador como one hot encoding (com Dummy variable encoder)\n",
        "encoder = OneHotEncoder(drop='first', sparse=False)\n",
        "\n",
        "# transformando os dados\n",
        "dummy = encoder.fit_transform(data)\n",
        "print(dummy)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[1. 0.]\n",
            " [0. 1.]\n",
            " [0. 0.]\n",
            " [0. 0.]\n",
            " [1. 0.]\n",
            " [0. 0.]]\n"
          ]
        }
      ]
    }
  ]
}